Fine-Tune a Model on Hugging Face 2026: The TRL + PEFT Workflow for SFT and DPO

Hugging Face offers several training libraries and the choice is confusing. Here is the exact TRL + PEFT workflow used for supervised fine-tuning and preference alignment.

💡 What You Will Learn

Hugging Face offers several training libraries and the choice is confusing. Here is the exact TRL + PEFT workflow used for supervised fine-tuning and preference alignment.

📜 Table of Contents

• The Modern HF Training Stack
• Workflow 1: SFT (Supervised Fine-Tuning)
• Workflow 2: DPO (Direct Preference Optimization)
• The Evaluation Trap
• Full Pipeline

The Modern HF Training Stack

Two libraries cover 90% of 2026 fine-tuning needs: - PEFT (21,522 stars) - parameter-efficient methods (LoRA, QLoRA) so you train adapters, not full weights - TRL (19,040 stars) - high-level trainers for SFT, DPO, PPO on top of transformers (163,377 stars)

Workflow 1: SFT (Supervised Fine-Tuning)

Goal: teach the model a format or domain. Dataset: instruction-response pairs (e.g. 1,000 support tickets).

from trl import SFTTrainer
from peft import LoraConfig

lora = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"])
trainer = SFTTrainer(
    model="Qwen/Qwen3-1.5B",          # or any chat model
    train_dataset=dataset,
    peft_config=lora,
    args=TrainingArguments(output_dir="./sft", num_train_epochs=2),
)
trainer.train()

QLoRA variant: add load_in_4bit=True via BitsAndBytesConfig to cut VRAM about 4x.

Workflow 2: DPO (Direct Preference Optimization)

Goal: make the model prefer good answers over bad ones. Dataset: (prompt, chosen, rejected) triples - e.g. chosen = human-approved reply, rejected = the old bot reply.

from trl import DPOTrainer
dpo = DPOTrainer(
    model=model,             # your SFT model
    ref_model=model_ref,     # frozen reference
    train_dataset=dpo_data,
    peft_config=lora,
    args=TrainingArguments(output_dir="./dpo", num_train_epochs=1),
)
dpo.train()

DPO needs only 3 columns and no reward model - that is why it replaced most RLHF in open source.

The Evaluation Trap

Loss going down does not mean the model is better. After SFT/DPO, evaluate on a held-out set with clear metrics: format compliance, refusal rate, and a small human preference test. TRL models are easy to destroy with a bad dataset, so always keep the base model and compare side by side.

Full Pipeline

Raw data, format into instruction pairs, SFT with LoRA, merge adapter, DPO with preference triples, evaluate, push to the Hub. Two training runs, one weekend, a genuinely better model.