OpenModelsHQ

The connections between GPUs change what you can run

A GPU count leaves out the paths that connect those GPUs. For a model spread across devices, the size of the fast communication domain and the network between domains can change latency, throughput and cost.

Count GPUs inside the same domain

An NVLink domain is the group of GPUs connected through the NVLink fabric. NVIDIA's HGX H200 baseboard connects 8 GPUs in that domain. GB200 NVL72 expands it to 72 GPUs through NVLink switches. Architecture source.

8 separate node domains

64 allocated GPUs, 8 per domain

8 GPUsNVLink inside
8 GPUsNVLink inside
8 GPUsNVLink inside
8 GPUsNVLink inside
8 GPUsNVLink inside
8 GPUsNVLink inside
8 GPUsNVLink inside
8 GPUsNVLink inside
NICs + scale-out network

Communication between these domains takes the network path.

A rack-wide domain

64 allocated GPUs within 72 connected GPUs

NVLink switch fabric8 GPUs outside the illustrated worker
NICs + network to other domains

The illustrated worker's GPU collectives can stay within NVLink.

Illustrative placement of a 64-GPU worker, not a cabling plan or a performance benchmark. The node comparison uses HGX H200 baseboard; the rack uses GB200 NVL72. The GPUs and systems differ.

Scale-up links connect GPUs within a domain. Scale-out networks connect domains using adapters and switches, commonly InfiniBand or Ethernet with RDMA support. NVLink traffic within the rack uses its own switch fabric; the rack still needs networking for other traffic and communication beyond that domain. Rack architecture · Adapter protocols.

Compare the same units and direction

NVIDIA specifies 1,800 GB/s of bidirectional NVLink bandwidth per GPU for this Blackwell design. That sums transmit and receive. The example below compares one direction with a 400 Gb/s NIC port, assuming 1 dedicated port per GPU. That is an example allocation, not a claim about all cloud instances.

NVLink per GPU

900GB/s in one direction

1,800 GB/s bidirectional, split equally by direction

Example NIC allocation

50GB/s in one direction

400 gigabits converted to gigabytes per second

Nominal bandwidth ratio

18×not an inference speedup

Per-GPU aggregate NVLink rate divided by the example NIC rate

Decimal units. These are nominal link rates before overhead, not measured collective throughput or a guaranteed GPU-pair transfer rate. More NIC ports, faster links, sharing, PCIe limits and switch topology change the comparison. Rack-total bandwidth and per-GPU bandwidth are different measures.

NVLink specificationNIC specification

Match the fabric to the serving strategy

Tensor parallelism

GPUs hold pieces of the same layers and exchange intermediate results. Frequent collectives make communication latency and bandwidth part of the serving path.

Expert parallelism

Mixture-of-experts workers route tokens to experts distributed across GPUs. Dispatch and combine traffic can cross domain boundaries when the experts sit on different nodes.

Pipeline parallelism

GPUs handle successive groups of layers. Traffic passes between stages, with a different communication pattern from splitting each layer across GPUs.

Independent replicas

Separate model copies serve different requests. Their GPUs do not need the same per-layer exchanges as a single model sharded across devices.

Parallelism and deployment reference

NVIDIA describes DeepSeek-R1 wide expert-parallel decode across 64 GPUs. That worker fits within the 72-GPU domain in the diagram. The source's comparison of domain sizes uses simulations; it does not establish a universal speedup or minimum GPU requirement for that model. Read the example and its conditions.

Ask a cloud or neocloud for the allocated topology

Use this checklist with the actual quote. A GPU model and hourly price do not describe the full serving system.

Allocated topology
Exact GPU SKU, server or rack design, GPUs in one NVLink domain, and whether the quoted allocation keeps your worker inside it. A rack product name does not guarantee access to the whole rack.
Network allocation
NIC model, active ports and link rate per GPU or host. Ask whether links are dedicated or shared and request the path between nodes, including switch oversubscription.
Software and isolation
RDMA support, Ethernet/RoCE or InfiniBand configuration, collective library and driver versions, and any virtualisation limits. Have the operator verify GPU-to-GPU and GPU-to-NIC placement.
Measured workload
Benchmark your model, precision, context, batch size and concurrency. Record prefill and decode separately, with tail latency and useful throughput. A link-rate ratio cannot substitute for this result.
Full operating cost
Include the entire required allocation, network charges, minimum term and unused capacity. Compare cost per acceptable result at your latency target, then use that measured cost in the GPU ownership scenario.

For useful-life planning, track the workloads that still meet their service target on the available fabric. A functioning GPU may remain economical for an independent worker while being a poor fit for a tightly coupled model. That is a workload decision, not a physical expiry date.

This guide followed a Silicon Atlas post about NVLink domains. Specifications and deployment details above link to the manufacturer documentation; calculations and the schematic are ours.