# Fix CUDA out of memory when running a model locally

> torch.OutOfMemoryError: CUDA out of memory means the GPU is full. Lower the batch size, use float16 or 4-bit, offload to CPU, or clear the cache.

- Canonical: https://guides-ai.com/guides/fix-cuda-out-of-memory/
- Plate 09.15 · Topic: Local LLMs (https://guides-ai.com/topics/local-llm/)
- Published: 21 Oct 2026 · 3 min read
- Source site: guides-ai — https://guides-ai.com/

`torch.OutOfMemoryError: CUDA out of memory` means your GPU has no free VRAM left for the next allocation. The fix is to make the job smaller: a lower batch size or resolution, a smaller or quantized model, half precision, or CPU offload.

## 1. Lower the batch size, resolution, or sequence length

The quickest win is to ask for less at once. In Transformers drop the batch size; in diffusers for FLUX or Stable Diffusion cut height, width, or num_frames.

```python
pipe(prompt, height=512, width=512)
```

Smaller inputs mean smaller activations.

## 2. Load the model in half precision

float32 weights need about 4 bytes per parameter; bfloat16 or float16 halve that.

```python
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("your-model", dtype=torch.bfloat16)
```

Use bfloat16 on Ampere or newer, float16 on older cards. Most models are trained in bfloat16, so inference quality holds. See [how much RAM or VRAM a model needs](/guides/local-llm-ram-vram-requirements/).

## 3. Quantize to 8-bit or 4-bit

If half precision still overflows, [quantize](/glossary/#quantization) the weights with bitsandbytes.

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    "your-model",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
)
```

`load_in_4bit=True` cuts a 15B model from about 32 GB to about 9 GB. See [how to choose a quantization](/guides/choose-llm-quantization/).

## 4. Offload layers to the CPU

When the model still will not fit on one GPU, spill the rest to system RAM.

```python
model = AutoModelForCausalLM.from_pretrained("your-model", device_map="auto")
```

`device_map="auto"` places layers on the GPU first, then the CPU; see the [Transformers pipeline quickstart](/guides/transformers-pipeline-quickstart/). For diffusers pipelines call `enable_model_cpu_offload()`, or `enable_sequential_cpu_offload()` for the largest saving at the cost of speed, plus `vae.enable_tiling()`.

## 5. Clear the cache and defragment memory

Free cached blocks between runs, and let the allocator expand segments to fight fragmentation.

```python
import torch
torch.cuda.empty_cache()
```

`empty_cache()` returns cached memory to other GPU apps but will not grow what PyTorch can use. If the error mentions reserved but unallocated memory, set the allocator config.

macOS/Linux:

```bash
export PYTORCH_ALLOC_CONF=expandable_segments:True
```

Windows (PowerShell):

```powershell
$env:PYTORCH_ALLOC_CONF="expandable_segments:True"
```

## Verify it worked

```bash
nvidia-smi -l 1
```

A load that stays under total VRAM and finishes without the error is fixed.

**If it fails:**

- `torch.OutOfMemoryError` on the first batch: the model alone exceeds VRAM; quantize or offload, do not just lower the batch size.
- Memory climbs each step until it crashes: you are holding tensors in a list; detach with `.detach()` or wrap inference in `torch.no_grad()`.
- reserved memory far above allocated: fragmentation; set `PYTORCH_ALLOC_CONF=expandable_segments:True` (its alias is `PYTORCH_CUDA_ALLOC_CONF`). Full notes in the [PyTorch CUDA guide](https://docs.pytorch.org/docs/stable/notes/cuda.html).