The connections between GPUs change what you can run
A GPU count leaves out the paths that connect those GPUs. For a model spread across devices, the size of the fast communication domain and the network between domains can change latency, throughput and cost.
Count GPUs inside the same domain
An NVLink domain is the group of GPUs connected through the NVLink fabric. NVIDIA's HGX H200 baseboard connects 8 GPUs in that domain. GB200 NVL72 expands it to 72 GPUs through NVLink switches. Architecture source.
8 separate node domains
64 allocated GPUs, 8 per domain
Communication between these domains takes the network path.
A rack-wide domain
64 allocated GPUs within 72 connected GPUs
The illustrated worker's GPU collectives can stay within NVLink.
Scale-up links connect GPUs within a domain. Scale-out networks connect domains using adapters and switches, commonly InfiniBand or Ethernet with RDMA support. NVLink traffic within the rack uses its own switch fabric; the rack still needs networking for other traffic and communication beyond that domain. Rack architecture · Adapter protocols.
Compare the same units and direction
NVIDIA specifies 1,800 GB/s of bidirectional NVLink bandwidth per GPU for this Blackwell design. That sums transmit and receive. The example below compares one direction with a 400 Gb/s NIC port, assuming 1 dedicated port per GPU. That is an example allocation, not a claim about all cloud instances.
NVLink per GPU
900GB/s in one direction
1,800 GB/s bidirectional, split equally by direction
Example NIC allocation
50GB/s in one direction
400 gigabits converted to gigabytes per second
Nominal bandwidth ratio
18×not an inference speedup
Per-GPU aggregate NVLink rate divided by the example NIC rate
Decimal units. These are nominal link rates before overhead, not measured collective throughput or a guaranteed GPU-pair transfer rate. More NIC ports, faster links, sharing, PCIe limits and switch topology change the comparison. Rack-total bandwidth and per-GPU bandwidth are different measures.
Match the fabric to the serving strategy
Tensor parallelism
GPUs hold pieces of the same layers and exchange intermediate results. Frequent collectives make communication latency and bandwidth part of the serving path.
Expert parallelism
Mixture-of-experts workers route tokens to experts distributed across GPUs. Dispatch and combine traffic can cross domain boundaries when the experts sit on different nodes.
Pipeline parallelism
GPUs handle successive groups of layers. Traffic passes between stages, with a different communication pattern from splitting each layer across GPUs.
Independent replicas
Separate model copies serve different requests. Their GPUs do not need the same per-layer exchanges as a single model sharded across devices.
Parallelism and deployment reference
NVIDIA describes DeepSeek-R1 wide expert-parallel decode across 64 GPUs. That worker fits within the 72-GPU domain in the diagram. The source's comparison of domain sizes uses simulations; it does not establish a universal speedup or minimum GPU requirement for that model. Read the example and its conditions.
Short decode steps can expose communication delays. How much can be hidden behind compute depends on the model, batch, kernels and overlap strategy. Measure the deployed workload before assigning value to a larger domain.
Ask a cloud or neocloud for the allocated topology
Use this checklist with the actual quote. A GPU model and hourly price do not describe the full serving system.
- Allocated topology
- Exact GPU SKU, server or rack design, GPUs in one NVLink domain, and whether the quoted allocation keeps your worker inside it. A rack product name does not guarantee access to the whole rack.
- Network allocation
- NIC model, active ports and link rate per GPU or host. Ask whether links are dedicated or shared and request the path between nodes, including switch oversubscription.
- Software and isolation
- RDMA support, Ethernet/RoCE or InfiniBand configuration, collective library and driver versions, and any virtualisation limits. Have the operator verify GPU-to-GPU and GPU-to-NIC placement.
- Measured workload
- Benchmark your model, precision, context, batch size and concurrency. Record prefill and decode separately, with tail latency and useful throughput. A link-rate ratio cannot substitute for this result.
- Full operating cost
- Include the entire required allocation, network charges, minimum term and unused capacity. Compare cost per acceptable result at your latency target, then use that measured cost in the GPU ownership scenario.
For useful-life planning, track the workloads that still meet their service target on the available fabric. A functioning GPU may remain economical for an independent worker while being a poor fit for a tightly coupled model. That is a workload decision, not a physical expiry date.
This guide followed a Silicon Atlas post about NVLink domains. Specifications and deployment details above link to the manufacturer documentation; calculations and the schematic are ours.