CUDA and alternatives 9
Pick a compute stack by hardware and runtime: which of PyTorch, vLLM, llama.cpp and MLX support CUDA, ROCm, Metal, Vulkan, SYCL and CANN, with sources.
llama.cpp supports GPU backends including CUDA, HIP, Metal, Vulkan and SYCL. vLLM has a different hardware matrix and uses plugins for some accelerators. Choose a compute stack against the runtime and model you intend to use, then check whether it supports the exact device and operating system.
If you are running PyTorch
PyTorch offers CUDA builds for NVIDIA and HIP builds for supported AMD GPUs. PyTorch for HIP deliberately reuses the torch.cuda interfaces, reducing the source changes needed for many applications. AMD publishes GPU and operating-system compatibility tables; a device outside those tables is not officially supported. A framework build must also match the selected ROCm release.
Intel GPUs use a separate device string, xpu. PyTorch documents support arriving as a prototype in 2.5 for client GPUs and the Data Center GPU Max Series, installed from a dedicated wheel index, with validated hardware spanning Arc A-Series and B-Series and Core Ultra integrated graphics. The old Intel Extension for PyTorch was to be maintained until the end of March 2026 and then marked End-of-Life unless there was a solid need, according to its notice; its repository is now archived, and Intel recommends using PyTorch directly.
On a Mac, the mps device needs Apple silicon, macOS 14.0 or later and Python 3.10 or later per Apple’s requirements, which also describe the MPS backend as being in beta.
If you are serving with vLLM
vLLM documents CUDA, ROCm and Intel XPU installation paths, with separate requirements for each. Native Windows installation is not supported; the guide points Windows users to WSL. The ROCm instructions warn that an unsupported Python version can cause an installer to select a CUDA wheel. Match the wheel, Python environment and accelerator stack before loading a model.
For Apple GPUs, vLLM points to vLLM-Metal, a community-maintained plugin using MLX. Ascend NPUs use vllm-ascend, which publishes its own compatible CANN, PyTorch and TorchNPU versions. Plugin availability does not mean every model supported by core vLLM runs on that backend.
If you are running llama.cpp
llama.cpp offers a broader set of device backends. The backend list covers CUDA, HIP, Metal, Vulkan, SYCL, CANN, MUSA, OpenCL, Hexagon, OpenVINO, zDNN, ZenDNN, WebGPU, BLAS, BLIS and RPC. Metal is enabled by default on macOS.
The release assets show which builds you can install without compiling. llama.cpp publishes CPU packages and accelerator packages for backends including CUDA, ROCm, Vulkan and SYCL. The combinations vary by platform and release. Its build guide documents other backends, including CANN. Vulkan offers a path for consumer GPUs with a suitable Vulkan driver, without requiring the vendor’s full compute toolkit.
If you are on Apple silicon
Apple exposes several software paths. Metal is the API, PyTorch reaches it through mps, and MLX is Apple’s own array framework with a unified memory model. Running language models on MLX goes through mlx-lm, covered on our MLX page. llama.cpp’s Metal backend is on by default, and vLLM arrives only through the MLX-backed plugin.
On Intel GPUs, llama.cpp also offers SYCL. Its documentation records performance changes between toolchain releases. That makes the compiler version part of a meaningful comparison with Vulkan; a result from a different build is not proof that either backend will be faster on your workload.
FAQ
What is the safest choice if I want every runtime to work?
No GPU guarantees support across every runtime. NVIDIA GPUs have CUDA paths in PyTorch, vLLM and llama.cpp, but architecture, driver and model requirements still differ. Check the requirements for the versions you plan to deploy.
Can I serve production traffic on an AMD consumer card?
vLLM lists the Radeon RX 7900 and RX 9000 series in its ROCm requirements, so it is possible, though AMD’s support footnotes narrow the operating systems. Test before committing.
Does Windows work for local inference?
For llama.cpp, yes, with prebuilt CPU, CUDA, ROCm, SYCL, Vulkan, OpenVINO and OpenCL binaries. vLLM does not support Windows natively and suggests WSL.
Where do I check whether my exact GPU is supported?
AMD’s compatibility matrix for ROCm, PyTorch’s Intel GPU page for xpu, and the runtime’s own requirements section. Vendor and runtime tables often disagree, and the stricter one governs.
- AMD ROCm and HIP for AI: what is supportedWhich AMD GPUs ROCm officially supports, the PyTorch and vLLM versions AMD validates, and what HIPIFY does to CUDA source.
- GPU interconnects: NVLink domains and cloud networkingCompare domain size and bandwidth, and check the topology behind a cloud GPU quote.
- Huawei Ascend, CANN and other non-CUDA acceleratorsWhat CANN is, how torch_npu and MindSpore reach Ascend NPUs, the versions vLLM Ascend pins, and where Tenstorrent, TPU and AWS Neuron stacks sit.
- Intel oneAPI, SYCL and PyTorch XPU supportWhat SYCL and oneAPI are, which Intel GPUs PyTorch validates for the xpu device, and what Intel's notice says about the Intel Extension for PyTorch.
- llama.cpp backends: Vulkan, CUDA, Metal, SYCL and moreThe full llama.cpp backend list, which ones ship prebuilt binaries, and why Vulkan is the practical choice on consumer AMD and Intel GPUs.
- Metal, MPS and MLX: the Apple silicon AI stackHow Metal, the PyTorch mps device and Apple's MLX framework fit together on a Mac, with the macOS and hardware requirements each one states.
- ZLUDA and CUDA translation layers: where things standWhat ZLUDA translates, which AI workloads its developer reports working, and how funding changes affected its scope.
- CUDA explained: why it became the default for AIWhat CUDA is, how cuDNN, NCCL and TensorRT deepen the dependency, and what the CUDA Toolkit licence actually permits, from NVIDIA docs and EULA.
- Triton and Mojo: portable kernels above CUDAOpenAI Triton as a kernel language PyTorch already depends on, and what Modular claims for Mojo and MAX, with the platform support each one documents.
Keep exploring
- WatchNew labs and architectures, with vendor claims checked against sources.
- LicensesWhat each model license allows, in plain English.
- China DeskWho owns the Chinese labs and what their models are.
- Run open modelsCommands to run open models with Ollama, MLX and GGUF.
- GuidesWhich hardware to pick and what it can run.
- Model typesText, coding, image, video, audio, embedding and vision models.
Sources
- llama.cpp build guide
- MLX repository
- Intel Extension for PyTorch retirement notice
- llama.cpp SYCL documentation
- vLLM Ascend plugin
- vLLM Apple GPU plugin installation
- ROCm compatibility matrix
- llama.cpp README
- llama.cpp releases
- vLLM GPU installation doc
- Getting Started on Intel GPU (PyTorch)
- PyTorch HIP (ROCm) semantics
- Accelerated PyTorch training on Mac (Apple)
- vLLM installation overview