OpenModelsHQ

CUDA and alternatives 9

Pick a compute stack by hardware and runtime: which of PyTorch, vLLM, llama.cpp and MLX support CUDA, ROCm, Metal, Vulkan, SYCL and CANN, with sources.

llama.cpp supports GPU backends including CUDA, HIP, Metal, Vulkan and SYCL. vLLM has a different hardware matrix and uses plugins for some accelerators. Choose a compute stack against the runtime and model you intend to use, then check whether it supports the exact device and operating system.

If you are running PyTorch

PyTorch offers CUDA builds for NVIDIA and HIP builds for supported AMD GPUs. PyTorch for HIP deliberately reuses the torch.cuda interfaces, reducing the source changes needed for many applications. AMD publishes GPU and operating-system compatibility tables; a device outside those tables is not officially supported. A framework build must also match the selected ROCm release.

Intel GPUs use a separate device string, xpu. PyTorch documents support arriving as a prototype in 2.5 for client GPUs and the Data Center GPU Max Series, installed from a dedicated wheel index, with validated hardware spanning Arc A-Series and B-Series and Core Ultra integrated graphics. The old Intel Extension for PyTorch was to be maintained until the end of March 2026 and then marked End-of-Life unless there was a solid need, according to its notice; its repository is now archived, and Intel recommends using PyTorch directly.

On a Mac, the mps device needs Apple silicon, macOS 14.0 or later and Python 3.10 or later per Apple’s requirements, which also describe the MPS backend as being in beta.

If you are serving with vLLM

vLLM documents CUDA, ROCm and Intel XPU installation paths, with separate requirements for each. Native Windows installation is not supported; the guide points Windows users to WSL. The ROCm instructions warn that an unsupported Python version can cause an installer to select a CUDA wheel. Match the wheel, Python environment and accelerator stack before loading a model.

For Apple GPUs, vLLM points to vLLM-Metal, a community-maintained plugin using MLX. Ascend NPUs use vllm-ascend, which publishes its own compatible CANN, PyTorch and TorchNPU versions. Plugin availability does not mean every model supported by core vLLM runs on that backend.

If you are running llama.cpp

llama.cpp offers a broader set of device backends. The backend list covers CUDA, HIP, Metal, Vulkan, SYCL, CANN, MUSA, OpenCL, Hexagon, OpenVINO, zDNN, ZenDNN, WebGPU, BLAS, BLIS and RPC. Metal is enabled by default on macOS.

The release assets show which builds you can install without compiling. llama.cpp publishes CPU packages and accelerator packages for backends including CUDA, ROCm, Vulkan and SYCL. The combinations vary by platform and release. Its build guide documents other backends, including CANN. Vulkan offers a path for consumer GPUs with a suitable Vulkan driver, without requiring the vendor’s full compute toolkit.

If you are on Apple silicon

Apple exposes several software paths. Metal is the API, PyTorch reaches it through mps, and MLX is Apple’s own array framework with a unified memory model. Running language models on MLX goes through mlx-lm, covered on our MLX page. llama.cpp’s Metal backend is on by default, and vLLM arrives only through the MLX-backed plugin.

On Intel GPUs, llama.cpp also offers SYCL. Its documentation records performance changes between toolchain releases. That makes the compiler version part of a meaningful comparison with Vulkan; a result from a different build is not proof that either backend will be faster on your workload.

FAQ

What is the safest choice if I want every runtime to work?

No GPU guarantees support across every runtime. NVIDIA GPUs have CUDA paths in PyTorch, vLLM and llama.cpp, but architecture, driver and model requirements still differ. Check the requirements for the versions you plan to deploy.

Can I serve production traffic on an AMD consumer card?

vLLM lists the Radeon RX 7900 and RX 9000 series in its ROCm requirements, so it is possible, though AMD’s support footnotes narrow the operating systems. Test before committing.

Does Windows work for local inference?

For llama.cpp, yes, with prebuilt CPU, CUDA, ROCm, SYCL, Vulkan, OpenVINO and OpenCL binaries. vLLM does not support Windows natively and suggests WSL.

Where do I check whether my exact GPU is supported?

AMD’s compatibility matrix for ROCm, PyTorch’s Intel GPU page for xpu, and the runtime’s own requirements section. Vendor and runtime tables often disagree, and the stricter one governs.

Keep exploring

Sources