On an Apple silicon Mac, pip install mlx-lm followed by mlx_lm.generate runs a language model pulled from Hugging Face. mlx-lm is the Python package for running and fine-tuning language models on MLX, Apple’s open-source array framework for machine learning on Apple silicon.

What MLX and mlx-lm are

MLX comes from Apple machine learning research. Its README describes a NumPy-like Python API and a unified memory model: arrays live in shared memory, and operations can run on the CPU or GPU without copying data. The mlx-lm package is built on it. Its README calls it a Python package for generating text and fine-tuning large language models on Apple silicon, with Hugging Face Hub integration, quantisation, and LoRA and full fine-tuning.

The standard pip install mlx needs Apple silicon, a native Python 3.10 or newer and macOS 14 or newer. Intel Macs are not supported by the PyPI package. MLX also publishes CUDA and CPU-only Linux packages, but the Mac is the main path and the one covered here.

Install and generate

pip install mlx-lm
mlx_lm.generate --prompt "How tall is Mt Everest?"
mlx_lm.chat

The README also gives a conda route, conda install -c conda-forge mlx-lm. The default model for generate and chat is mlx-community/Llama-3.2-3B-Instruct-4bit. Choose another with --model:

mlx_lm.generate --model mistralai/Mistral-7B-Instruct-v0.3 --prompt "hello"

That downloads the model from the Hugging Face Hub and generates text with your prompt, according to the README. mlx_lm.generate -h lists the options.

Run a local server

mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit

This starts a server on port 8080 of localhost, with an API meant to be similar to the OpenAI chat API. Test it with curl localhost:8080/v1/chat/completions. The server docs say it is not recommended for production, because it implements only basic security checks.

Finding models: mlx-community

The MLX Community organisation on Hugging Face hosts thousands of MLX-format models, usually in several quantisation levels. If a model has no MLX version, mlx_lm.convert --model <repo> -q quantises one locally, and --upload-repo publishes it.

Unified memory and the GPU limit

On Apple silicon the CPU and GPU share one memory pool, so there is no separate video memory to fill, and a Mac with more memory can load larger models. macOS does cap how much memory can be wired for the GPU. The mlx-lm README says a model that is large relative to total RAM can be slow, and that on macOS 15 or newer you can raise the system wired memory limit:

sudo sysctl iogpu.wired_limit_mb=N

N should be larger than the model size in megabytes and smaller than the machine’s memory. mlx-lm’s docs do not state the default limit, and we do not quote one. To save memory on long prompts, mlx-lm offers --max-kv-size for a rotating cache and --kv-bits on the server for a quantised cache.

Vision models with mlx-vlm

mlx-vlm is a separate package for inference and fine-tuning of vision language models on a Mac, and of omni models with audio and video. Install it with pip install -U mlx-vlm.

mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg

There is a FastAPI server, started with mlx_vlm.server --model <repo>, and an optional Gradio chat interface.

FAQ

Does MLX run on Windows or on Intel Macs?

The standard PyPI package needs Apple silicon and macOS 14 or newer. MLX also ships Linux CUDA and CPU-only packages. Windows is not listed in its install docs.

How do I serve an MLX model to other apps?

Run mlx_lm.server --model <repo> and send OpenAI-style chat requests to port 8080. Use it for local work, not production.

Why is a large model slow on my Mac?

mlx-lm warns when a model is large relative to RAM. On macOS 15 or newer, raising iogpu.wired_limit_mb with sudo sysctl can help.

What is the difference between mlx-lm and mlx-vlm?

mlx-lm runs text models. mlx-vlm runs models that take images, and sometimes audio and video.