Prepare a Dataset for Fine-Tuning

The shape of your dataset decides what a fine-tune learns. TRL’s SFTTrainer takes language-modeling or prompt-completion data, in standard (plain text) or conversational form. For chat models, use the conversational layout: each row holds a list of {"role", "content"} messages. Hand the trainer a conversational dataset and it applies the chat template for you, so your job is clean rows, not tokenization. The exact formats are in the TRL dataset formats docs.

Pick one layout

• messages: a full conversation in one column. Loss covers the whole sequence unless you set assistant_only_loss=True in SFTConfig.
• prompt + completion: both are message lists. Loss is computed on the completion only by default.
• text: a plain string, for non-chat data.

Keep one layout across the whole file.

Build, save, load, split

Write one JSON object per line. The datasets docs call this the most efficient JSON layout, and load_dataset("json", data_files=...) reads it directly. With no split names given, everything lands in train, so split it yourself.

import json
from datasets import load_dataset
from transformers import AutoTokenizer

SYSTEM = "Answer in one short sentence."
pairs = [
    ("What color is the sky?", "The sky is blue."),
    ("What color is grass?", "Grass is green."),
    ("What color is snow?", "Snow is white."),
    ("What color is a ripe banana?", "A ripe banana is yellow."),
]

with open("train.jsonl", "w", encoding="utf-8") as f:
    for q, a in pairs:
        row = {"messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": q},
            {"role": "assistant", "content": a},
        ]}
        f.write(json.dumps(row, ensure_ascii=False) + "\n")

ds = load_dataset("json", data_files="train.jsonl", split="train")
splits = ds.train_test_split(test_size=0.25, seed=42)  # DatasetDict: "train", "test"

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
print(tok.apply_chat_template(
    splits["train"][0]["messages"], tokenize=False, add_generation_prompt=False
))

Each line in train.jsonl looks like this:

{"messages": [{"role": "system", "content": "Answer in one short sentence."}, {"role": "user", "content": "What color is the sky?"}, {"role": "assistant", "content": "The sky is blue."}]}

Check the rendered text

The preview uses tokenize=False and add_generation_prompt=False. The chat templating docs say to turn the generation prompt off for training, since the extra assistant-start tokens don’t help there. You print it only to eyeball roles and turn boundaries; SFTTrainer does the same step internally. If you render to a text column yourself, drop messages and tokenize with add_special_tokens=False, because templates already include special tokens.

Then pass both splits in:

from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=SFTConfig(output_dir="out", eval_strategy="steps", eval_steps=50),
    train_dataset=splits["train"],
    eval_dataset=splits["test"],
)
trainer.train()

Quality checklist

• Dedupe. Repeated or near-identical rows get memorized and overweight one pattern.
• One output shape. If answers are one sentence, make every answer one sentence, with the same system prompt, tone and formatting.
• Enough examples. Rule of thumb, not a documented figure: a few hundred clean rows can teach a format or style. Teaching new facts is usually a job for retrieval; see fine-tuning vs RAG vs prompting.
• Hold out a test split, and keep it clean. Split before any augmentation, and keep every eval question and its near-duplicates out of train.

Once the rows are right, train with LoRA or QLoRA. To filter or stream large files first, see load, stream, filter; to share the result, upload it.

Next: LoRA fine-tune a small model, QLoRA on a consumer GPU, Upload a model or dataset