Fine Tuning Best Practices 2026: Optimize Open Source LLMs for Your Specific Task
🩺 Summary
Fine tuning improves LLM performance on specific tasks, but it's easy to overfit, lose generalization, or waste compute. These best practices help you fine tune efficiently.
📝 Details
# Fine Tuning Best Practices 2026: Optimize Open Source LLMs for Your Specific Task
Fine tuning adapts a pre-trained LLM to your specific task. With LoRA and Unsloth, it's accessible on consumer GPUs.
## When Should You Fine Tune?
| Scenario | Fine Tune? | Better Alternative |
|----------|-----------|-------------------|
| Need specific output format | Yes | --- |
| Domain-specific terminology | Yes | --- |
| Follow instructions better | No | Use better base model |
| General knowledge gaps | No | Use RAG instead |
**Rule**: If RAG + prompt engineering solves it, don't fine tune.
## Step 1: Prepare Your Dataset
Format as conversations:
```json
[
{
"messages": [
{"role": "system", "content": "You are a medical coding assistant."},
{"role": "user", "content": "Convert this diagnosis to ICD-10"},
{"role": "assistant", "content": "E11.40"}
]
}
]
```
**Checklist**: Minimum 100 examples, balanced classes, no duplicates, 80/10/10 split.
## Step 2: Choose Your Method
| Method | VRAM | Speed | Quality | Best For |
|--------|------|-------|---------|----------|
| Full Fine Tune | 48GB+ | Slow | Best | Massive data |
| LoRA (rank=16) | 12-16GB | Fast | Great | Most use cases |
| QLoRA (4-bit) | 6-8GB | Fastest | Good | Consumer GPUs |
**Start with QLoRA on a single RTX 3090/4090.**
## Step 3: Implement with Unsloth
```python
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen3-7B-Instruct-bnb-4bit",
max_seq_length=2048,
dtype=torch.bfloat16,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model, r=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=16, lora_dropout=0, bias="none",
)
from trl import SFTTrainer
from transformers import TrainingArguments
trainer = SFTTrainer(
model=model, tokenizer=tokenizer,
train_dataset=dataset, max_seq_length=2048,
args=TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
learning_rate=2e-4,
num_train_epochs=3,
fp16=True,
output_dir="outputs",
),
)
trainer.train()
```
## Step 4: Hyperparameter Best Practices
| Parameter | Recommended | Notes |
|-----------|------------|-------|
| LoRA rank (r) | 16-32 | Higher = more expressive |
| Learning rate | 1e-4 to 5e-4 | QLoRA needs higher LR |
| Batch size | 2-8 | Use gradient accumulation |
| Epochs | 3 | Watch for overfitting |
## Common Mistakes
1. Too little data (< 100 examples) → overfitting
2. Learning rate too high → loss spikes
3. Too many epochs → catastrophic forgetting
4. Not evaluating on held-out test set
## FAQ
**Q: Can I fine tune on 8GB VRAM?** A: Yes. Use QLoRA with a 7B model in 4-bit.
**Q: How long does it take?** A: 500 examples on 7B with QLoRA: ~1-2 hours on RTX 4090.
**Q: Mix fine tuning with RAG?** A: Yes. Fine tune for format/style, RAG for facts.
💬 Comments (0)