Buy memory first. A GPU that cannot hold the model and its context cannot run it at full speed. Among the devices that fit, pick the one with higher memory bandwidth, then check that your software supports it.

1. Memory first

NVIDIA’s inference guide names the two main contributors to GPU memory use: the model weights and the key-value (KV) cache. The cache grows linearly with sequence length, so long contexts need extra room. Ollama’s docs agree: a larger context length increases the memory required, and Ollama’s default context length rises with available VRAM.

Quantisation lowers the cost of the weights. The llama.cpp README lists integer quantisation from 1.5 to 8 bits for faster inference and reduced memory use. The gpt-oss-120b card, for example, says that model runs on a single 80 GB GPU, and its smaller sibling gpt-oss-20b runs within 16 GB of memory.

If the model does not fit, it cannot run at full speed. llama.cpp can split work between GPU and CPU, but that is a fallback.

2. Bandwidth second

NVIDIA’s guide says model execution is frequently memory-bandwidth bound, particularly in the weights. Between two devices that both hold your model, the one with higher bandwidth should generate tokens faster. The vendor figures are far apart: 1792 GB/sec for the GeForce RTX 5090, 1.2 TB/s for the Apple M5 Ultra, up to 614 GB/s for the M5 Max and 273 GB/s for the DGX Spark.

3. Software support third

Ollama’s hardware docs cover NVIDIA GPUs by compute capability, AMD Radeon through ROCm or Vulkan, and Apple GPUs through Metal. llama.cpp lists backends including CUDA, HIP, Metal, Vulkan and SYCL. MLX is the Apple framework for Mac. Check the hardware list of the tool you plan to use before you buy.

The four categories

Consumer GPUs, used or new

NVIDIA lists the RTX 3090 and RTX 4090 with 24 GB of GDDR6X, and the RTX 5090 with 32 GB of GDDR7. These cards pair high bandwidth with modest memory. If you buy used, check condition and warranty.

Workstation GPUs

The RTX PRO 6000 Blackwell has 96 GB of GDDR7, and NVIDIA lists Workstation, Max-Q and Server editions. AMD’s Radeon AI PRO R9700 appears in Ollama’s ROCm list. One workstation card holds models that would otherwise need several consumer cards.

Mac unified memory

Apple says the Mac Studio with M5 Ultra offers up to 512 GB of unified memory, and M5 Max versions go up to 128 GB. For the MacBook Pro, Apple lists the M5 Max at up to 128 GB and up to 614 GB/s. Capacity is the strength. Bandwidth sits below the top NVIDIA cards.

AI boxes

The DGX Spark has 128 GB of unified memory at 273 GB/s. It holds large models in a small case, with lower bandwidth than a high-end GPU.

Rent instead

Renting a cloud GPU by the hour makes sense when you want to test a model before buying, run a one-off job, or need more memory than any single device you could afford. It also suits work that runs only a few hours a week. Buying makes more sense for steady daily use, or for data that must stay local. We do not quote prices, so compare current rates with providers.

FAQ

Is more VRAM or more bandwidth better?

Memory capacity decides whether a model runs at all, so check it first. Bandwidth then affects how fast it generates.

Is a used RTX 3090 still a good choice?

NVIDIA lists it with 24 GB of GDDR6X, and Ollama lists it as supported. It is an older generation, so check the seller and the card’s condition.

Do I need an NVIDIA GPU?

No. Ollama and llama.cpp support AMD GPUs and Apple silicon too, though the exact cards supported vary by tool.

Mac or PC for local models?

A Mac gives you large unified memory in one machine. An NVIDIA card gives higher bandwidth at smaller memory sizes.