Fix CUDA out of memory when running a model locally

torch.OutOfMemoryError: CUDA out of memory means your GPU has no free VRAM left for the next allocation. The fix is to make the job smaller: a lower batch size or resolution, a smaller or quantized model, half precision, or CPU offload.

1. Lower the batch size, resolution, or sequence length

The quickest win is to ask for less at once. In Transformers drop the batch size; in diffusers for FLUX or Stable Diffusion cut height, width, or num_frames.

pipe(prompt, height=512, width=512)

Smaller inputs mean smaller activations.

2. Load the model in half precision

float32 weights need about 4 bytes per parameter; bfloat16 or float16 halve that.

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("your-model", dtype=torch.bfloat16)

Use bfloat16 on Ampere or newer, float16 on older cards. Most models are trained in bfloat16, so inference quality holds. See how much RAM or VRAM a model needs.

3. Quantize to 8-bit or 4-bit

If half precision still overflows, quantize the weights with bitsandbytes.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    "your-model",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
)

load_in_4bit=True cuts a 15B model from about 32 GB to about 9 GB. See how to choose a quantization.

4. Offload layers to the CPU

When the model still will not fit on one GPU, spill the rest to system RAM.

model = AutoModelForCausalLM.from_pretrained("your-model", device_map="auto")

device_map="auto" places layers on the GPU first, then the CPU; see the Transformers pipeline quickstart. For diffusers pipelines call enable_model_cpu_offload(), or enable_sequential_cpu_offload() for the largest saving at the cost of speed, plus vae.enable_tiling().

5. Clear the cache and defragment memory

Free cached blocks between runs, and let the allocator expand segments to fight fragmentation.

import torch
torch.cuda.empty_cache()

empty_cache() returns cached memory to other GPU apps but will not grow what PyTorch can use. If the error mentions reserved but unallocated memory, set the allocator config.

macOS/Linux:

export PYTORCH_ALLOC_CONF=expandable_segments:True

Windows (PowerShell):

$env:PYTORCH_ALLOC_CONF="expandable_segments:True"

Verify it worked

nvidia-smi -l 1

A load that stays under total VRAM and finishes without the error is fixed.

If it fails:

torch.OutOfMemoryError on the first batch: the model alone exceeds VRAM; quantize or offload, do not just lower the batch size.
Memory climbs each step until it crashes: you are holding tensors in a list; detach with .detach() or wrap inference in torch.no_grad().
reserved memory far above allocated: fragmentation; set PYTORCH_ALLOC_CONF=expandable_segments:True (its alias is PYTORCH_CUDA_ALLOC_CONF). Full notes in the PyTorch CUDA guide.