Fix Ollama Not Using Your GPU

Ollama offloads to the GPU automatically when it can. When it runs on CPU instead, generation crawls. The cause is usually a driver, a container, or a model too big for your VRAM, not a missing setting.

1. Check whether the GPU is actually used

ollama ps

Shows each loaded model and a PROCESSOR column: 100% GPU, 100% CPU, or a split like 48%/52% CPU/GPU.

Watch utilisation live while a prompt runs:

nvidia-smi -l 1

A busy GPU during generation confirms offload; a flat 0% means it fell back to CPU. On Apple Silicon nothing needs configuring, since Metal is automatic, so skip to step 4.

2. Update the NVIDIA driver and CUDA runtime

Ollama needs compute capability 5.0+ and driver 550 or newer (570+ for cards from 5.0 to 6.2). Check what you have:

nvidia-smi

Prints the driver version and card; if it fails, the host has no working driver and Ollama cannot see a GPU. On Windows the GPU build is used once the driver is current; see run Ollama on Windows.

3. Fix GPU passthrough in Docker and WSL

In Docker the container needs the NVIDIA Container Toolkit, so confirm the runtime first:

docker run --gpus all ubuntu nvidia-smi

If that prints your GPU, run Ollama with --gpus all; if it errors, install the toolkit first. Under WSL2 the GPU comes from the Windows driver, so install that, not a driver inside the Linux distro.

4. Size the model to your VRAM

If a model is larger than free VRAM, Ollama keeps some layers on the GPU and the rest on CPU, the 48%/52% split from step 1. Pick a smaller quantization or a smaller model for a full 100% GPU load; see how much RAM or VRAM a model needs and choose a quantization.

5. Set the right GPU with device variables

CUDA_VISIBLE_DEVICES=GPU-<uuid> ollama serve

Pins Ollama to one card; list UUIDs with nvidia-smi -L. For AMD the equivalent is ROCR_VISIBLE_DEVICES, and an unsupported AMD card may need HSA_OVERRIDE_GFX_VERSION.

If it fails:

ollama ps still shows 100% CPU after a Linux suspend/resume: reload the driver with sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm.
The container worked, then dropped to CPU: add "exec-opts": ["native.cgroupdriver=cgroupfs"] to /etc/docker/daemon.json and restart Docker.
An AMD GPU is ignored on Linux: add the ollama user to the render group, or set AMD_LOG_LEVEL=3 and read the discovery error. Full matrix in the Ollama GPU docs.

Next: fix CUDA out of memory when running a model.