123

So good morning everyone. My name is Sameh Boujelbene. I'm Vice President of Research at Delor Group. We track the IT and networking industries and I personally track the switching market which as you know is very tied to the optics market.

Today I'm very excited to talk to you about the AI workloads and their impact on modern data center networks. As you know AI is not a new trend. It's not a new industry. It's been here for many, many years but we are definitely going through a new era in the age of AI. So I will first talk about what makes this new era of AI unique and different from what we've seen before. Next, I will talk about the different network design options to support these AI workloads. After that, I will talk about the AI workloads characteristics and their impact on the new AI fabric that is needed to support all these AI workloads. And finally, I will talk about the high-speed migration in AI networks.

So let's start here with a few quotes from industry pioneers. And one thing to mention here is that all of these quotes are fairly recent because a lot has changed just in the last couple of months. So a few months ago, Bill Gates wrote an open letter titled 'The Age of AI Has Begun,' and he views this new era of AI as transformative as the mobile phones and the internet. Nvidia CEO Jensen Hong also compared this new era of AI to the iPhone moment of AI.

Now I'm gonna present this chart that you may have seen this before many times but I wanted to reiterate what makes this new era of AI unique and different. It's really the number of parameters that these new large AI models have to deal with. And as you can see on this chart some have to deal with billions and even trillions of parameters. And it seems that this number of parameters is growing thousand times every three years. Some are looking at this chart and saying they are growing 10 times every year. I wanted a bigger number because thousand times seems a lot every three years.

And the complexity and size of these AI workloads dictates the number of GPUs required to run those workloads and ultimately the scale and type of the network needed to connect the GPUs. And so if you're talking about a small AI app that requires 10 or maybe hundreds of GPUs you just need a server IO and you can use a CXL, NVLink or PCIe to connect these GPUs in a scale up type of architecture. But if you are talking about moderate AI apps that require thousands of GPUs you really need a scale out type of architecture with an AI leaf. And even worse if you are talking about a large AI model that requires tens of thousands of GPUs you really need a data center scale type of architecture with an AI spine. And quite frankly most of the growth and the excitement in the industry from, you know you heard the hyperscalers talking about the AI workloads. Most of the growth is coming and most of the pressure is coming from these moderate and large AI applications which would require an AI network also known as the backend network and that backend network is different and is new and did not exist before and is quite different from the traditional frontend network that is used to connect and support general purpose servers. So the question becomes then what are the requirements for this new AI fabric?

For that let's first look at the characteristics of AI workloads. Many of you may be familiar already with these characteristics but just to reiterate I have on this chart the three step process of an AI workload. So you have compute, communicate and synchronize and it is an iterative process which creates some unique patterns for AI workloads. So as I'm listing here AI workloads consist of a large portion of elephant flows. They require a large number of short remote memory access and last but not the least the progression of all nodes can be held back by any delayed flow. So tail latency or network latency is key here and Meta showed last year that 33% of elapsed time in AI ML is spent waiting for the network. Think about the impact of that on the wasted GPU resources sitting idle waiting for the network to respond. So definitely we need to get the network out of the way.

And so these unique AI workloads characteristic drive definitely drive new requirements for this AI fabric. But also I wanted to focus and emphasize also the amount of traffic within these AI networks. So first if you look at the average size of cluster as I mentioned earlier the number of parameters that these AI workloads have to deal with is growing 1,000 times every three years. And this is why if you look at the average size of a cluster is quadrupling every two years. You may recall we used to talk about 256 GPU per cluster and then that number grew very fast to 1,000 then to 4,000 now we're talking about 32,000 and even 64,000 GPU per cluster. And what that means it means that by the time you're building your AI data center that data center is already obsolete because your requirement would have changed already. So it's very important to build the AI data center for future requirements not for current requirements. Now in terms of the network per accelerator it's really skyrocketing. Like if you're seeing the amount of bandwidth now per accelerator is 200, 400, 800 is gonna grow very soon to 1.6T and even 3.2T in the near future. So in summary not only the size of the cluster is growing in terms of GPU but the amount of bandwidth per GPU is growing which is why the AI traffic in some of these AI network is really growing 10 times every two years. And you saw the numbers that Andy shared earlier they are just mind blowing.

Now to try to meet the requirements for these AI workloads, we need an AI fabric that meets a few requirements here, so at the very least, we need high speed, low tail latency, lossless scalable fabrics. In addition to that, some potential architectural changes may be needed to meet the requirements, so we may see some denser tiers of the network to deal with the cross-sectional bandwidth. We may see some collapsed tiers of the network to deal with the latency, and we really expect cloud service providers to make distinct choices in their journey to try to achieve the best performing AI network.

Now one of the most burning questions in the industry I get often, what is the most suitable type of fabric for these AI networks? Is it gonna be Ethernet, is it gonna be InfiniBand? Before I address that question I wanted to make a quick reminder here that for an AI server we're talking about two types of connectivity here. So one to the front end for data ingest, and that's predominantly Ethernet, and the other connectivity is to the back end for GPU to GPU connectivity. That's where we see a mix of Ethernet and InfiniBand, and I do believe that in the next two years we'll see Ethernet gaining share because of all the improvement we're seeing on Ethernet as well as customers' desire to multisource. At the end of the day, the mix between Ethernet and InfiniBand will depend on the size of the clusters, the nature of the workloads, and customers' distinct choices, and this is a much bigger problem than I can address in just 10 minutes today, but I talk about all of this in my AI network report.

 Now regardless whether it's Ethernet or InfiniBand I wanted to focus on the speed migration in these back end network. And what you can see on this chart is that I'm expecting that by 2027 all of the ports will be at least 800 gig, two thirds of the port will be 1600 gig. Now if you compare this picture to what I have for the front end network, in the front end network only 10% of the ports will be 1600 gig and 800 gig will be just one third of the port. So clearly we have two different types of requirements and pictures and quite frankly if anything I do believe that the growth in AI network will crush all market expectations, quite frankly even including mine because when I think about the growth I don't think that 1.6T may be even enough here in this time frame, just unbelievable.

Okay, great timing so last slide for the summary. So clearly AI is a major inflection point in the networking industry. Back end networks are fast growing, constantly evolving, will be crushing all market expectations for sure in the next five years. Innovation at all levels is needed, not only the hardware but also software. We need all that to try to meet the requirements. The market will include and will continue to include a mix of proprietary and standard solutions but I expect standard solutions to prevail at scale. History shown as that many times. And I expect customers to have distinct choices in network topology and fabric which will create bifurcation in the market. With that I will end it here and I don't think, do we have time for questions? If not I will hand it over, okay.