Fine Tuning Best Practices 2026: Optimize Open Source LLMs for Your Specific Task
Fine tuning improves LLM performance on specific tasks, but it's easy to overfit, lose generalization, or waste compute. These best practices help you fine tune efficiently.
💡 What You Will Learn
Fine tuning improves LLM performance on specific tasks, but it's easy to overfit, lose generalization, or waste compute. These best practices help you fine tune efficiently.
📜 Table of Contents
Fine Tuning Best Practices 2026: Optimize Open Source LLMs for Your Specific Task
Fine tuning adapts a pre-trained LLM to your specific task. With LoRA and Unsloth, it's accessible on consumer GPUs.
When Should You Fine Tune?
| Scenario | Fine Tune? | Better Alternative |
|---|---|---|
| Need specific output format | Yes | --- |
| Domain-specific terminology | Yes | --- |
| Follow instructions better | No | Use better base model |
| General knowledge gaps | No | Use RAG instead |
| Rule: If RAG + prompt engineering solves it, don't fine tune. | ||
| ## Step 1: Prepare Your Dataset | ||
| Format as conversations: |
[
{
"messages": [
{"role": "system", "content": "You are a medical coding assistant."},
{"role": "user", "content": "Convert this diagnosis to ICD-10"},
{"role": "assistant", "content": "E11.40"}
]
}
]
Checklist: Minimum 100 examples, balanced classes, no duplicates, 80/10/10 split.
Step 2: Choose Your Method
| Method | VRAM | Speed | Quality | Best For |
|---|---|---|---|---|
| Full Fine Tune | 48GB+ | Slow | Best | Massive data |
| LoRA (rank=16) | 12-16GB | Fast | Great | Most use cases |
| QLoRA (4-bit) | 6-8GB | Fastest | Good | Consumer GPUs |
| Start with QLoRA on a single RTX 3090/4090. | ||||
| ## Step 3: Implement with Unsloth |
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen3-7B-Instruct-bnb-4bit",
max_seq_length=2048,
dtype=torch.bfloat16,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model, r=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=16, lora_dropout=0, bias="none",
)
from trl import SFTTrainer
from transformers import TrainingArguments
trainer = SFTTrainer(
model=model, tokenizer=tokenizer,
train_dataset=dataset, max_seq_length=2048,
args=TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
learning_rate=2e-4,
num_train_epochs=3,
fp16=True,
output_dir="outputs",
),
)
trainer.train()
Step 4: Hyperparameter Best Practices
| Parameter | Recommended | Notes |
|---|---|---|
| LoRA rank (r) | 16-32 | Higher = more expressive |
| Learning rate | 1e-4 to 5e-4 | QLoRA needs higher LR |
| Batch size | 2-8 | Use gradient accumulation |
| Epochs | 3 | Watch for overfitting |
| ## Common Mistakes | ||
| 1. Too little data (< 100 examples) โ overfitting | ||
| 2. Learning rate too high โ loss spikes | ||
| 3. Too many epochs โ catastrophic forgetting | ||
| 4. Not evaluating on held-out test set | ||
| ## FAQ | ||
| Q: Can I fine tune on 8GB VRAM? A: Yes. Use QLoRA with a 7B model in 4-bit. | ||
| Q: How long does it take? A: 500 examples on 7B with QLoRA: ~1-2 hours on RTX 4090. | ||
| Q: Mix fine tuning with RAG? A: Yes. Fine tune for format/style, RAG for facts. |
❓ FAQ
Can I fine tune on 8GB VRAM?
Yes. Use QLoRA with a 7B model in 4-bit.
How long does it take?
500 examples on 7B with QLoRA: ~1-2 hours on RTX 4090.
Mix fine tuning with RAG?
Yes. Fine tune for format/style, RAG for facts.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
