Tens of thousands of GGUF repos on the Hub run locally with one command, no Modelfile and no manual download. Pick a quant, then point Ollama or llama.cpp straight at the repo.
1. Find a GGUF repo and pick a quant
Search the Hub for a model with “GGUF” in the name, for example bartowski/Llama-3.2-3B-Instruct-GGUF. Open Files and versions: each .gguf file is one quantization. Q4_K_M is the usual size-versus-quality balance; Q8_0 is near-lossless and larger. See how to choose a quantization for the trade-offs. The Use this model button on the page generates the exact command for Ollama or llama.cpp.
2. Run it with Ollama
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF
Downloads the default Q4_K_M quant and starts an interactive chat. Pick a different quant with a tag:
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
Both hf.co and huggingface.co work as the domain, and the tag is case-insensitive. Per the Ollama on the Hub docs, if Q4_K_M is absent Ollama picks another reasonable quant from the repo.
3. Run it with llama.cpp
llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
Downloads that quant from the Hub and opens a chat (the cache location is set by LLAMA_CACHE). To serve an OpenAI-compatible API instead:
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
Starts the server on 127.0.0.1:9931. Exact syntax is in the llama.cpp Hub docs. New to the engine? Start with the llama.cpp quickstart.
4. Handle gated repos
Community requants (bartowski, unsloth) are usually ungated and run straight away. Official repos such as Meta Llama and FLUX are gated: accept the licence and authenticate first, or the pull fails with a 401 or 403. For private or gated Ollama pulls you also add an SSH key to your Hugging Face account. See access gated models.
Verify it worked
ollama list
Shows the model you just pulled, named after the repo. Send one prompt:
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF "Reply with just: ok"
A short reply means the model downloaded, loaded and runs.
If it fails:
Error: pull model manifest: 401— the repo is gated or private; see step 4.- llama.cpp exits with a 404 or “file not found” for the tag — the quant tag does not match a file in the repo. Copy the exact quant from the Files tab or the Use this model button.
ollama: command not foundorllama-cli: command not found— the tool is not installed or not on PATH. Install Ollama, or build or install llama.cpp (the quickstart covers both).
Next: choose a GGUF quantization · run models with Ollama on Windows.