Ascend NPUs use Huawei’s CANN software stack. PyTorch reaches the devices through TorchNPU, while vLLM uses the separate vllm-ascend plugin. These projects publish compatibility requirements that must be matched to the installed driver, firmware and accelerator.
How PyTorch reaches an Ascend NPU
The route is a plugin rather than a fork. TorchNPU describes itself as a deep learning adapter plugin built by Ascend for PyTorch, letting the framework invoke Ascend NPUs directly, and says it inherits and reuses capabilities from upstream PyTorch while adding Ascend-specific adaptation. Its licence badge reads BSD 3-clause, and it ships on PyPI as torch-npu.
The install sequence makes the dependency structure clear. CANN comes first, at version 9.1.0 in the current instructions, along with the NPU driver and firmware matching the hardware. Then a CPU build of PyTorch, then the matching torch-npu wheel, then sourcing set_env.sh from the CANN install. Version pairs are tight: the README’s example installs torch==2.12.0 with torch-npu==2.12.0, and points at a compatibility matrix covering TorchNPU, PyTorch, CANN and Python together.
Device handling differs from the AMD approach. Where PyTorch on ROCm reuses torch.cuda, Ascend gets its own device type, and tensors print as device='npu:0'. The project notes that since TorchNPU 2.5.1, import torch_npu is no longer mandatory because registration happens automatically, though it still recommends the explicit import. Its feature list includes graph mode, FSDP2 for distributed training, collective primitives such as AllGather, AllReduce and AllToAll, and integration into PyTorch through the PrivateUse1 device hook.
MindSpore is the other framework path. Its README calls it an open source deep learning training and inference framework for mobile, edge and cloud, with native support for Ascend processors and software and hardware co-optimisation, licensed under Apache 2.0.
Serving on Ascend
vLLM’s Ascend support is an out-of-tree plugin, vllm-ascend, which the project describes as community maintained and as the recommended approach for the Ascend backend within the vLLM community, built on the hardware-pluggable interface from vLLM’s hardware plugin design.
The plugin lists supported Atlas inference and training hardware, with some devices marked experimental. Its software requirements pin compatible CANN, PyTorch and TorchNPU versions and require a matching vLLM release. Those requirements are separate from TorchNPU’s standalone installation example. Mixing the newest release of every dependency is not the documented installation path.
llama.cpp also has a CANN backend, listed in its README with Ascend NPU as the target device and built with -DGGML_CANN=on. It is a source build; the project’s release assets do not include a prebuilt CANN package.
Other stacks
Tenstorrent publishes tt-metal, its own low-level programming model and libraries, in the open.
Google’s TPUs are reached from PyTorch through PyTorch/XLA, which bridges the framework to the XLA compiler, and that repository is the reference for what is supported. A note in its README dated 4/22/2026 says: “Once TorchTPU is public it will replace PyTorch/XLA.”
AWS has Neuron, the SDK for its Trainium and Inferentia accelerators, documented at its own site.
These toolchains have their own compilers and operator support. Framework and serving-runtime integration differs by accelerator. A PyTorch plugin can make an application reachable on new hardware without making every library, model architecture or optimisation portable.
FAQ
Does CUDA code run on an Ascend NPU?
No. The supported route is a framework plugin, either TorchNPU for PyTorch or MindSpore, compiled against CANN. There is no drop-in CUDA layer in these projects.
Which CANN version do I need?
It depends on what sits above it. TorchNPU’s current instructions name CANN 9.1.0, and vllm-ascend pins CANN to exactly 9.1.0 along with specific PyTorch and TorchNPU versions. Check the compatibility matrix for your framework version rather than installing the newest of each.
Can I serve open models on Ascend with vLLM?
Through the vllm-ascend plugin, yes. Its README says Transformer-like, mixture-of-experts, embedding and multimodal models run on Ascend NPUs, and points to a support matrix for specifics.
Is llama.cpp an option on Ascend?
It has a CANN backend built from source with -DGGML_CANN=on, requiring the CANN toolkit. Unlike CUDA, ROCm, SYCL and Vulkan, it is not among the prebuilt release binaries.