# QLoRA on a Consumer GPU: 4-bit Fine-Tuning

> Fine-tune a 7B model on one 16-24GB GPU: load it in 4-bit NF4 with bitsandbytes, attach LoRA via PEFT, train with TRL SFTTrainer.

- Canonical: https://guides-ai.com/guides/qlora-consumer-gpu/
- Plate 16.15 · Topic: Hugging Face (https://guides-ai.com/topics/huggingface/)
- Published: 09 Nov 2026 · 4 min read
- Source site: guides-ai — https://guides-ai.com/

QLoRA is LoRA on top of a frozen 4-bit base model. bitsandbytes stores the base weights in 4-bit (NF4), and only the small LoRA adapter matrices are trained. Reach for it when you want to tune a roughly 7B model but 16-bit weights plus training overhead don't fit on your single 16GB or 24GB card.

Not sure fine-tuning is the right tool? Read [fine-tuning vs RAG vs prompting](/guides/fine-tuning-vs-rag-vs-prompting/) first. For plain LoRA without quantization, see [LoRA fine-tune a small model](/guides/lora-fine-tune-small-model/). Get your data into shape with [prepare a dataset for fine-tuning](/guides/prepare-dataset-for-fine-tuning/).

## Why it fits in VRAM

The [Transformers bitsandbytes docs](https://huggingface.co/docs/transformers/en/quantization/bitsandbytes) say 4-bit quantization reduces memory use by 4x. The base is frozen, so gradients and optimizer state exist only for the adapters. The PEFT docs say QLoRA can tune a 65B model on a single 48GB GPU, and the Transformers docs give the example of nested quantization tuning Llama-13b on a 16GB T4 at sequence length 1024, batch size 1, and 4 accumulation steps. Your own peak depends on model, sequence length, and batch size, so measure it.

NF4 on NVIDIA needs a Pascal (GTX 10-series) or newer GPU.

```bash
pip install --upgrade transformers accelerate bitsandbytes peft trl datasets
```

```python
import torch
from datasets import load_dataset
from peft import LoraConfig, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

model_id = "Qwen/Qwen2.5-7B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)
model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    bias="none",
    task_type="CAUSAL_LM",
)

dataset = load_dataset("trl-lib/Capybara", split="train[:1000]")

args = SFTConfig(
    output_dir="qlora-out",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=1e-4,
    max_length=1024,
    gradient_checkpointing=True,
    bf16=True,
    num_train_epochs=1,
    logging_steps=10,
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    peft_config=peft_config,
)
trainer.train()
trainer.model.save_pretrained("qlora-adapter")
```

`SFTTrainer` wraps the quantized model with your `peft_config`, so you don't call `get_peft_model` yourself. The learning rate of 1e-4 follows the TRL tip that adapters usually take a higher rate than full fine-tuning. The Capybara dataset is conversational, so the trainer applies the model's chat template. The saved folder holds the adapter; share it with [Hugging Face upload](/guides/hugging-face-upload-model-or-dataset/).

## VRAM & pitfalls

- **Double quant:** `bnb_4bit_use_double_quant=True` quantizes the quantization constants too, saving about 0.4 bits per parameter at no extra performance cost, per the Transformers docs.
- **Compute dtype:** the default is float32; bf16 speeds up computation. If your card lacks bf16, use `torch.float16` here and set `fp16=True, bf16=False` in `SFTConfig`.
- **Gradient checkpointing:** it trades speed for memory. It is on by default in both `prepare_model_for_kbit_training` and `SFTConfig`. Keep it on.
- **Still OOM?** Cut `max_length`, keep batch size 1 and raise accumulation, or follow [fix CUDA out of memory](/guides/fix-cuda-out-of-memory/).
- **Merging:** the adapter is a separate small file. Don't assume you can fold it into the 4-bit weights. Reload the base in 16-bit, attach the adapter, then merge.

---

Next: [LoRA fine-tune a small model](/guides/lora-fine-tune-small-model/) | [Prepare a dataset for fine-tuning](/guides/prepare-dataset-for-fine-tuning/) | [Fix CUDA out of memory](/guides/fix-cuda-out-of-memory/)
## Questions people ask

### Can I train the 4-bit weights themselves?

No. The Transformers docs state that 8 and 4-bit training is only supported for training extra parameters, which is why QLoRA trains LoRA adapters on top of a frozen 4-bit base.

### Do I still need prepare_model_for_kbit_training?

The PEFT quantization guide says to call it to preprocess a quantized model for training, so the example below does. It also turns on gradient checkpointing by default.