Hugging Face ships several training libraries, which makes choosing confusing. PEFT plus TRL covers about 90% of real fine-tuning: one builds LoRA adapters, the other provides SFT and DPO trainers.

📜 Table of Contents

• Stack and data format
• SFT
• DPO
• Evaluation: loss is not quality
• FAQ

Hugging Face Fine-Tuning in Practice 2026: SFT and DPO with TRL + PEFT

Fine-tuning answers two questions: teach a format or domain (SFT), and make the model prefer better answers (DPO). PEFT builds parameter-efficient adapters; TRL ships both trainers.

Stack and data format

Python 3.10+ with a matching PyTorch build; 8GB VRAM means QLoRA, 16GB+ allows regular LoRA.

Quality beats quantity: a few hundred clean examples usually outperform thousands of mixed ones.

SFT

from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

lora = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","k_proj","v_proj","o_proj"],
                  task_type="CAUSAL_LM")
trainer = SFTTrainer(model=model, peft_config=lora, train_dataset=dataset,
    args=SFTConfig(output_dir="./sft", num_train_epochs=2, learning_rate=1e-4,
                   max_length=2048))
trainer.train()

For QLoRA, load the model in 4-bit (nf4) to cut VRAM at some speed cost.

DPO

from trl import DPOTrainer, DPOConfig
dpo = DPOTrainer(model=model, train_dataset=dpo_data, peft_config=lora,
    args=DPOConfig(output_dir="./dpo", num_train_epochs=1, learning_rate=5e-5, beta=0.1))
dpo.train()

beta controls drift from the reference model. A few hundred to a few thousand clearly contrasted pairs suffice. Merge adapters with model.merge_and_unload() before serving, or keep them separate to switch tasks cheaply.

Evaluation: loss is not quality

Measure format compliance rate, refusal rate, and blind human preference against the base model. If human preference does not improve, suspect the data - not the epoch count or r.

FAQ

SFT before DPO? Yes, standard order; the reverse converges more slowly.

How much data? Hundreds of examples for format tasks, thousands for domain knowledge.

Out of VRAM? QLoRA, shorter max_length, smaller batch with accumulation, then a smaller base model.

Does fine-tuning make models dumber? It can - catastrophic forgetting. Mix in general data and keep the base model for comparison.

Library APIs change between versions - check official docs.