MLX is Apple’s array framework, and mlx-lm is the package that runs language models on it. It is macOS and Apple Silicon only: there is no Windows or Linux build, and it will not use an external NVIDIA GPU. On M-series chips it is tuned for the shared CPU/GPU memory and can beat the llama.cpp backend other runners use for some models; measure your own case, since it varies by model.
1. Confirm you are on Apple Silicon
uname -m
Prints arm64 on an M-series Mac; if you see x86_64, MLX will not run and you want Ollama or LM Studio instead. macOS 15 or newer is needed for the large-model speedups.
2. Install mlx-lm
pip install mlx-lm
Installs the CLI entry points (mlx_lm.generate, mlx_lm.chat, mlx_lm.server) and the Python API. Conda users can run conda install -c conda-forge mlx-lm instead.
3. Generate from a model
mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --prompt "Explain shared memory in one line"
Downloads the model from Hugging Face on first run, then prints a completion. Run mlx_lm.generate --help for every flag.
4. Chat and serve an OpenAI-style API
mlx_lm.chat
Opens an interactive REPL that keeps context for the session. For an HTTP endpoint instead:
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit
Starts a local server on port 8080 whose API is modelled on OpenAI’s chat API, with a /v1/chat/completions endpoint; the docs call it unsuitable for production. Any OpenAI client works by pointing at http://localhost:8080/v1.
5. Get quantized models from Hugging Face
The mlx-community organisation on the Hub publishes ready-made MLX models, usually 4-bit, which the commands above pull by repo ID or you fetch with the hf CLI. To convert and quantize a model yourself:
mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q
Writes a 4-bit MLX copy to ./mlx_model. A 4-bit build needs roughly a quarter of the 16-bit memory; see how much RAM a model needs.
If it fails:
command not found: mlx_lm.generate: the pip scripts are not on PATH, or you are in the wrong environment; reinstall in the active venv.- Generation is very slow or swaps to disk: the model is larger than free memory; pick a 4-bit build, or raise the limit with
sudo sysctl iogpu.wired_limit_mb=N. - A model fails to load asking for remote code: add
--trust-remote-code, but only for repos you trust. Full CLI in the mlx-lm README.