Ollama offloads to the GPU automatically when it can. When it runs on CPU instead, generation crawls. The cause is usually a driver, a container, or a model too big for your VRAM, not a missing setting.
1. Check whether the GPU is actually used
ollama ps
Shows each loaded model and a PROCESSOR column: 100% GPU, 100% CPU, or a split like 48%/52% CPU/GPU.
Watch utilisation live while a prompt runs:
nvidia-smi -l 1
A busy GPU during generation confirms offload; a flat 0% means it fell back to CPU. On Apple Silicon nothing needs configuring, since Metal is automatic, so skip to step 4.
2. Update the NVIDIA driver and CUDA runtime
Ollama needs compute capability 5.0+ and driver 550 or newer (570+ for cards from 5.0 to 6.2). Check what you have:
nvidia-smi
Prints the driver version and card; if it fails, the host has no working driver and Ollama cannot see a GPU. On Windows the GPU build is used once the driver is current; see run Ollama on Windows.
3. Fix GPU passthrough in Docker and WSL
In Docker the container needs the NVIDIA Container Toolkit, so confirm the runtime first:
docker run --gpus all ubuntu nvidia-smi
If that prints your GPU, run Ollama with --gpus all; if it errors, install the toolkit first. Under WSL2 the GPU comes from the Windows driver, so install that, not a driver inside the Linux distro.
4. Size the model to your VRAM
If a model is larger than free VRAM, Ollama keeps some layers on the GPU and the rest on CPU, the 48%/52% split from step 1. Pick a smaller quantization or a smaller model for a full 100% GPU load; see how much RAM or VRAM a model needs and choose a quantization.
5. Set the right GPU with device variables
CUDA_VISIBLE_DEVICES=GPU-<uuid> ollama serve
Pins Ollama to one card; list UUIDs with nvidia-smi -L. For AMD the equivalent is ROCR_VISIBLE_DEVICES, and an unsupported AMD card may need HSA_OVERRIDE_GFX_VERSION.
If it fails:
ollama psstill shows100% CPUafter a Linux suspend/resume: reload the driver withsudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm.- The container worked, then dropped to CPU: add
"exec-opts": ["native.cgroupdriver=cgroupfs"]to/etc/docker/daemon.jsonand restart Docker. - An AMD GPU is ignored on Linux: add the ollama user to the
rendergroup, or setAMD_LOG_LEVEL=3and read the discovery error. Full matrix in the Ollama GPU docs.