QLoRA on a Consumer GPU: 4-bit Fine-Tuning

QLoRA is LoRA on top of a frozen 4-bit base model. bitsandbytes stores the base weights in 4-bit (NF4), and only the small LoRA adapter matrices are trained. Reach for it when you want to tune a roughly 7B model but 16-bit weights plus training overhead don’t fit on your single 16GB or 24GB card.

Not sure fine-tuning is the right tool? Read fine-tuning vs RAG vs prompting first. For plain LoRA without quantization, see LoRA fine-tune a small model. Get your data into shape with prepare a dataset for fine-tuning.

Why it fits in VRAM

The Transformers bitsandbytes docs say 4-bit quantization reduces memory use by 4x. The base is frozen, so gradients and optimizer state exist only for the adapters. The PEFT docs say QLoRA can tune a 65B model on a single 48GB GPU, and the Transformers docs give the example of nested quantization tuning Llama-13b on a 16GB T4 at sequence length 1024, batch size 1, and 4 accumulation steps. Your own peak depends on model, sequence length, and batch size, so measure it.

NF4 on NVIDIA needs a Pascal (GTX 10-series) or newer GPU.

pip install --upgrade transformers accelerate bitsandbytes peft trl datasets

import torch
from datasets import load_dataset
from peft import LoraConfig, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

model_id = "Qwen/Qwen2.5-7B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)
model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    bias="none",
    task_type="CAUSAL_LM",
)

dataset = load_dataset("trl-lib/Capybara", split="train[:1000]")

args = SFTConfig(
    output_dir="qlora-out",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=1e-4,
    max_length=1024,
    gradient_checkpointing=True,
    bf16=True,
    num_train_epochs=1,
    logging_steps=10,
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    peft_config=peft_config,
)
trainer.train()
trainer.model.save_pretrained("qlora-adapter")

SFTTrainer wraps the quantized model with your peft_config, so you don’t call get_peft_model yourself. The learning rate of 1e-4 follows the TRL tip that adapters usually take a higher rate than full fine-tuning. The Capybara dataset is conversational, so the trainer applies the model’s chat template. The saved folder holds the adapter; share it with Hugging Face upload.

VRAM & pitfalls

• Double quant: bnb_4bit_use_double_quant=True quantizes the quantization constants too, saving about 0.4 bits per parameter at no extra performance cost, per the Transformers docs.
• Compute dtype: the default is float32; bf16 speeds up computation. If your card lacks bf16, use torch.float16 here and set fp16=True, bf16=False in SFTConfig.
• Gradient checkpointing: it trades speed for memory. It is on by default in both prepare_model_for_kbit_training and SFTConfig. Keep it on.
• Still OOM? Cut max_length, keep batch size 1 and raise accumulation, or follow fix CUDA out of memory.
• Merging: the adapter is a separate small file. Don’t assume you can fold it into the 4-bit weights. Reload the base in 16-bit, attach the adapter, then merge.

Next: LoRA fine-tune a small model | Prepare a dataset for fine-tuning | Fix CUDA out of memory