Chapter 09
NVL72 vs HGX vs DGX
The differences that matter are not on the spec sheet's first page. Per-GPU NVLink bandwidth is the same in both designs. What changed is how many GPUs that bandwidth reaches, what sits between the GPU and its host, and what you are actually buying.
What actually differs
| GB200 NVL72 | HGX / DGX 8-GPU node | |
|---|---|---|
| GPUs in one NVLink domain | 72 | 8 |
| NVLink bandwidth per GPU | 1.8 TB/s | 1.8 TB/s |
| Host CPU | Grace, Arm Neoverse V2 | x86 |
| CPU ↔ GPU link | NVLink-C2C, 900 GB/s, coherent | PCIe |
| Pooled HBM in the domain | ~13.5 TB | ~1.5 TB |
| Cooling | Liquid, mandatory | Air or liquid |
| Power delivery | Rack busbar, shared | Per-server PSUs |
| Unit of purchase | Rack, or Superchip | Server |
| Unit of integration | The rack | The baseboard |
Read the first two rows together and the design becomes clear. Per-GPU NVLink bandwidth did not change. What changed is that the same bandwidth now reaches 72 peers instead of 8, and the pooled memory those peers can address grew by the same factor to ~13.5 TBPooled HBM3e per rackSources disagree: 13.4 – 13.8 TB Sources differ on rounding, on physical versus usable-after-ECC capacity, and on SKU. NVIDIA’s 192 GB per GPU gives 13.82 TB physical and 12.96 TB after ECC; Supermicro’s datasheet quotes up to 372 GB per Superchip — 186 GB per GPU — which is where the 13.4 TB figure comes from. NVIDIA · Supermicro · SemiAnalysis . NVL72 is not a faster interconnect; it is the same interconnect stretched over nine times as many endpoints.
The other structural change is on the host side. Replacing x86-over-PCIe with Grace-over-NVLink-C2C turns host memory from a staging area into a genuine second tier of the same coherent address space — which is why the rack's ~30 TBUnified memory per rack HBM3e + LPDDR5X, coherent across NVLink-C2C. NVIDIA of unified memory is quoted as a single figure rather than as HBM plus something else.
Why bandwidth, not FLOPS
The roofline model says a kernel's achievable throughput is capped by either arithmetic or by data movement, whichever binds first, and that which one binds depends on a single property of the kernel: how many FLOPs it performs per byte it moves.
This figure is interactive and needs JavaScript. The prose around it states every number it shows.
Put Blackwell's numbers in: about ~20 PFLOPSPeak FP4 per GPU Sparse. NVIDIA of sparse FP4, so roughly 10 PFLOPS dense, against 8 TB/sMemory bandwidth per GPU NVIDIA of HBM. The ridge point lands north of a thousand FLOPs per byte. Dense GEMM with large tiles clears it. Decode, at an arithmetic intensity near one, is three orders of magnitude below it and will never touch the flat part of the roof no matter how many FLOPS the chip has.
This is the whole justification for the rack. Buying more FLOPS does nothing for a bandwidth-bound phase. Buying more bandwidth — and, crucially, more places to spread the working set so that each GPU reads less — does. A 72-GPU coherent domain is a bandwidth purchase disguised as a compute purchase.
The version of this argument that matters at rack scale replaces HBM bandwidth with fabric bandwidth and cache lines with messages. Switch the widget to NVLink and the ridge point drops by roughly an order of magnitude; switch it to InfiniBand and it drops again. Any model dimension whose per-byte arithmetic falls below the ridge for a given tier is a dimension that tier cannot afford to carry.
Run that test on an MoE layer's expert all-to-all — a pure data movement with essentially no arithmetic attached — and it fails at every tier. It only becomes affordable when the bandwidth in the denominator is 900 GB/s per GPU rather than 50 or 100, which is the argument the inference chapter makes in throughput terms.
Scale-up and scale-out, stated once
Scale-up makes one machine larger: NVLink, coherent memory, load/store semantics, and a hard limit set by physics and power. Scale-out adds more machines: InfiniBand or Ethernet, message passing, and a limit set by money and floor space.
Every generation of this hardware is an argument about where the boundary between them should sit. NVL72's answer is "at the rack". The roadmap is a series of proposals to move it further out — and a matching series of power figures explaining what that costs.