A GGUF file packs a model’s weights, often quantised to 4 bits or so, together with the metadata needed to run them, into a single file. llama.cpp reads it, and so do the tools built on llama.cpp, including Ollama and LM Studio.
What GGUF is
Hugging Face describes GGUF as a binary format optimised for quick loading and saving of models, designed for use with GGML and other executors. Unlike tensor-only formats such as safetensors, a GGUF file stores both the tensors and a standard set of metadata. It was developed by ggerganov, who also develops llama.cpp. Models trained in PyTorch can be converted to GGUF, and the Hub can show a file’s metadata and tensor list in a viewer.
llama.cpp
Its README describes llama.cpp as LLM inference in C/C++, aiming for minimal setup on a wide range of hardware. Apple silicon is a first-class target through Metal, and the README lists CUDA for NVIDIA GPUs, HIP for AMD GPUs, Vulkan and SYCL. It supports CPU and GPU hybrid inference, so a model larger than VRAM can be partly accelerated.
Install it with brew install llama.cpp or winget install llama.cpp, from pre-built releases, or by building from source. The two commands that matter are llama-cli and llama-server. The README’s newer quick start also shows them as llama cli and llama serve.
llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
llama-server -m models/7B/ggml-model.gguf -c 2048
The -hf flag downloads a model from Hugging Face. Its quant suffix is optional and case-insensitive. The default is Q4_K_M, falling back to the first file in the repo if that quant is missing. -m loads a local file, -c sets context size, and -ngl sets how many layers go to the GPU. llama-server listens on 127.0.0.1:8080 by default and offers OpenAI-compatible routes such as /v1/chat/completions, plus a built-in web interface.
Reading quant names
A name like Q4_K_M has three parts. Q4 is the nominal bit width. K marks the k-quant family from llama.cpp pull request 1684. The last letter, S, M or L, picks a mix of tensor types. In that pull request, Q4_K_S uses the Q4_K type for all tensors, while Q4_K_M uses Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest. The quantize README’s size table shows Q4_K_M files larger than Q4_K_S, and the pull request’s benchmark shows lower perplexity for Q4_K_M, meaning output closer to the original model.
Hugging Face’s type table gives the layout of the base types. Q4_K stores 8 blocks of 32 weights per super-block at 4.5 bits per weight. Q5_K comes to 5.5 bits per weight and Q6_K to 6.5625. Types starting with IQ use an importance matrix and go as low as 1.56 bits per weight for IQ1_S. Q8_0 is 8-bit round-to-nearest, and the table lists it with Q4_0, Q5_0 and similar as legacy methods. F16 and BF16 are 16-bit floats, the unquantised baseline. The quantize tool’s README notes that quantisation shrinks files and can speed up inference, but may lose accuracy.
Making your own GGUF
The quantize README shows two steps. Convert a Hugging Face model to a high-precision GGUF with convert_hf_to_gguf.py, then run llama-quantize with a target such as Q4_K_M. Hugging Face’s GGUF-my-repo space does the same without local setup.
LM Studio and Ollama
LM Studio, a desktop app built on llama.cpp, runs on Mac (Apple silicon), Windows and Linux. Its in-app downloader lists the quantisations of each model and highlights the one recommended for your hardware. It also has a command line tool, lms, and Hugging Face’s docs say its “Use this model” option covers both GGUF and MLX models.
Ollama is also based on llama.cpp. It can run GGUF repos directly with ollama run hf.co/{username}/{repository}, or import a local GGUF through a Modelfile. It does not quantise on import, so quantise first with a tool such as llama-quantize.
FAQ
What does Q4_K_M mean?
Q4 is 4-bit, K is the k-quant method, and M is the medium mix of tensor types. It is the default quant for llama.cpp’s -hf downloads and for Ollama’s Hugging Face pulls.
Is GGUF the same as the model the lab released?
No. A GGUF file is a conversion of the original weights, often quantised, and many are published by third parties such as bartowski. The original release is typically in a format like safetensors.
Can llama.cpp run a model bigger than my GPU memory?
Yes, in part. Its CPU and GPU hybrid inference can partially accelerate a model that is larger than VRAM.
GGUF or MLX on a Mac?
GGUF runs through llama.cpp, LM Studio or Ollama on any platform they support. MLX is Apple’s framework built for Apple silicon, covered on the MLX page.