Which Local LLM Should You Run?
Choose a local LLM by what fits your RAM and VRAM first, then by task, with starter picks for 8GB, 16GB, and 24GB machines.
On this page5 sections
Answer first: run the largest model your memory holds, then the best one for your task. Chasing a bigger name than your box fits only makes it swap to disk and crawl. Decide hardware, then task, then size, in that order — the same discipline as picking an AI model.
1. Start with your hardware, not the model
A model’s parameter count times its quantization has to fit in RAM for CPU inference or VRAM for GPU inference, with headroom for context. Do that math in RAM and VRAM requirements before you download anything, because it rules out most choices immediately.
2. Then choose by task
Within what fits, match the family to the job: general chat and drafting (llama3.2, qwen3, gemma3), code (qwen2.5-coder), vision questions about images (llava), and embeddings for search or RAG (nomic-embed-text). All of these exist in the Ollama library today.
3. Set the quantization tradeoff
Quantization shrinks a model so a larger one fits your box. A 4-bit quant is the usual sweet spot, trading a little quality for roughly half the memory of an 8-bit build. Weigh it in choose a quantization.
4. Starter picks by memory tier
Concrete starting points, assuming a 4-bit quant:
8 GB: a 3B-class model (llama3.2) for chat and drafting
16 GB: a 7-8B-class model (qwen2.5-coder for code, qwen3 for chat)
24 GB+: a 14B+ model, or run nomic-embed-text alongside a chat model
Move up a size only when a real task fails, not on spec-sheet envy.
5. Pull one and test
Pick your tier’s model and run it to confirm it actually fits.
ollama run llama3.2 "Say hello and name yourself"
If it replies fast and coherently, keep it; if it stalls or swaps, drop a size. Not sure which runner? Compare Ollama vs LM Studio, and browse exact tags in the Ollama model library.
If it fails:
- Out of memory or very slow: the model plus context exceeds your RAM or VRAM — pick a smaller size or a lower quant.
- “model not found”: that tag does not exist — run
ollama pull <name>or check the name in the library. - Answers are too weak: you undersized for the task — move one tier up if memory allows, or switch to a task-specific family.