Chapter 07
Scale-Out: SuperPODs
Inside the rack, every GPU has 900 GB/s of egress. One metre outside it, the same GPU has 50 or 100. That cliff is the most important number in cluster design, and nothing about the NVL72 removes it — the rack just moves it further out.
Three tiers, and the step between them
This figure is interactive and needs JavaScript. The prose around it states every number it shows.
One rack is already a very large computer, but the biggest training runs use far more than seventy-two GPUs. So racks get connected to each other — eight of them make a "SuperPOD" of 576GPUs per SuperPOD NVIDIA GPUs, and SuperPODs get connected in turn, up to 9,216Maximum GPUs in one fabric With 128 leaf switches, rail-optimised. NVIDIA GPUs in one system.
The catch is that the connection between racks is roughly ten times slower per GPU than the connection inside one. So the work has to be divided carefully: the parts that talk constantly are kept inside a rack, and only the parts that talk occasionally are allowed to cross between them.
A SuperPOD is 8NVL72 racks per SuperPOD NVIDIA NVL72 racks: 576GPUs per SuperPOD NVIDIA GPUs, roughly ~1 PB/sSuperPOD aggregate bandwidth NVIDIA aggregate and about ~240 TBSuperPOD fast memory NVIDIA of fast memory.NVIDIA Racks are joined by Quantum-X800 InfiniBand (144 × 800Quantum-X800 InfiniBand switch Q3400. NVIDIA ports on a Q3400) or Spectrum-X Ethernet, in a rail-optimised topology; the fabric scales to 9,216Maximum GPUs in one fabric With 128 leaf switches, rail-optimised. NVIDIA GPUs with 128 leaf switches. NICs are 400 Gb/sConnectX-7 SuperNIC NVIDIA ConnectX-7 on GB200 and 800 Gb/sConnectX-8 SuperNIC PCIe Gen6; shipped on GB300. NVIDIA ConnectX-8 on GB300, with BlueField-3 DPUs handling storage and isolation.
The practical consequence is a parallelism assignment. Tensor parallelism and wide expert parallelism are chatty per layer and belong inside the NVLink domain. Pipeline and data parallelism exchange far less per step and tolerate the scale-out tier. What NVL72 changed is not that this rule exists but where the line falls: the chatty dimensions now have 72 GPUs to spread across instead of 8.
Rail-optimised means each GPU's NIC connects to a dedicated leaf switch shared with the same-numbered GPU in other racks, so a ring or all-reduce along that rail crosses one switch rather than traversing the tree. It is the scale-out analogue of the port-matching trick that makes the in-rack fabric single-layer, and it exists for the same reason: collectives dominate, so build the topology the collective wants.
Multi-Node NVLink complicates the clean story slightly. The IMEX service allows GPU memory to be exported and imported across operating-system domains within an NVLink fabric, with Kubernetes modelling the boundary as a ComputeDomain resource via the GPU Operator and DRA driver.NVIDIA Docs,NVIDIA Docs The NVLink domain and the OS domain are not the same thing, and treating them as identical is a common source of confusion when scheduling on these racks.
Scale-out figures
| NVL72 racks per SuperPOD | 8 | |
|---|---|---|
| GPUs per SuperPOD | 576 | |
| SuperPOD aggregate bandwidth | ~1 PB/s | |
| SuperPOD fast memory | ~240 TB | |
| Maximum GPUs in one fabric | 9,216 | With 128 leaf switches, rail-optimised. |
| ConnectX-7 SuperNIC | 400 Gb/s | |
| ConnectX-8 SuperNIC | 800 Gb/s | PCIe Gen6; shipped on GB300. |
| Quantum-X800 InfiniBand switch | 144 × 800 Gb/s | Q3400. |
What 39 racks bought
MLPerf Training v5.0 gives a concrete data point for the whole stack working together: 2,496 Blackwell GPUs — 39 racks — completed Llama 3.1 405B in 27.3 minLlama 3.1 405B training time MLPerf Training v5.0, 4 June 2025: 27.33 minutes on 2,496 Blackwell GPUs across 39 racks running 64 active GPUs each — not fully populated 72-GPU racks, which is why 2,496 does not divide by 72. CoreWeave puts an equivalent H100 setup at around 156 racks, assuming 32 GPUs per rack. A later round reached about 10 minutes on more than 5,000 Blackwell GPUs. CoreWeave · MLCommons .CoreWeave,MLCommons The comparison worth holding onto is the rack count: an H100-equivalent run needed roughly 156 racks for the same work. A later round scaled to 5,120 GB200 GPUs and finished in about ten minutes.
Density is the point. Four times fewer racks is four times less floor, less optical cabling, fewer switch hops and a shorter critical path for every collective in the run.