# Run any GGUF from Hugging Face with Ollama or llama.cpp

> Pick a GGUF quant in the Files tab, then run it with ollama run hf.co/user/repo or llama-cli -hf user/repo:QUANT.

- Canonical: https://guides-ai.com/guides/run-hugging-face-gguf-with-ollama-llama-cpp/
- Plate 16.05 · Topic: Hugging Face (https://guides-ai.com/topics/huggingface/)
- Published: 12 Oct 2026 · 3 min read
- Source site: guides-ai — https://guides-ai.com/

Tens of thousands of GGUF repos on the Hub run locally with one command, no Modelfile and no manual download. Pick a quant, then point Ollama or llama.cpp straight at the repo.

## 1. Find a GGUF repo and pick a quant

Search the Hub for a model with "GGUF" in the name, for example `bartowski/Llama-3.2-3B-Instruct-GGUF`. Open **Files and versions**: each `.gguf` file is one quantization. `Q4_K_M` is the usual size-versus-quality balance; `Q8_0` is near-lossless and larger. See [how to choose a quantization](/guides/choose-llm-quantization/) for the trade-offs. The **Use this model** button on the page generates the exact command for Ollama or llama.cpp.

## 2. Run it with Ollama

```bash
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF
```

Downloads the default `Q4_K_M` quant and starts an interactive chat. Pick a different quant with a tag:

```bash
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
```

Both `hf.co` and `huggingface.co` work as the domain, and the tag is case-insensitive. Per the [Ollama on the Hub docs](https://huggingface.co/docs/hub/ollama), if `Q4_K_M` is absent Ollama picks another reasonable quant from the repo.

## 3. Run it with llama.cpp

```bash
llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
```

Downloads that quant from the Hub and opens a chat (the cache location is set by `LLAMA_CACHE`). To serve an OpenAI-compatible API instead:

```bash
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
```

Starts the server on `127.0.0.1:9931`. Exact syntax is in the [llama.cpp Hub docs](https://huggingface.co/docs/hub/gguf-llamacpp). New to the engine? Start with the [llama.cpp quickstart](/guides/llama-cpp-quickstart/).

## 4. Handle gated repos

Community requants (bartowski, unsloth) are usually ungated and run straight away. Official repos such as Meta Llama and FLUX are gated: accept the licence and authenticate first, or the pull fails with a 401 or 403. For private or gated Ollama pulls you also add an SSH key to your Hugging Face account. See [access gated models](/guides/hugging-face-gated-models-access/).

## Verify it worked

```bash
ollama list
```

Shows the model you just pulled, named after the repo. Send one prompt:

```bash
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF "Reply with just: ok"
```

A short reply means the model downloaded, loaded and runs.

**If it fails:**

- `Error: pull model manifest: 401` — the repo is gated or private; see step 4.
- llama.cpp exits with a 404 or "file not found" for the tag — the quant tag does not match a file in the repo. Copy the exact quant from the Files tab or the **Use this model** button.
- `ollama: command not found` or `llama-cli: command not found` — the tool is not installed or not on PATH. Install Ollama, or build or install llama.cpp (the quickstart covers both).

---

Next: [choose a GGUF quantization](/guides/choose-llm-quantization/) · [run models with Ollama on Windows](/guides/run-local-llm-ollama-windows/).