LoRA微调2026：单卡GPU训练定制模型完整指南
提示词工程到天花板了。模型需要学习公司风格、术语和数据——全量微调太贵。LoRA是中间路线。
LoRA Fine-Tuning: The One-GPU Path to a Custom Model
LoRA (Low-Rank Adaptation) is the technique that made fine-tuning practical for small teams. Instead of updating all billions of parameters, it freezes the base model and trains a small set of low-rank matrices - typically 1-5% of the parameters. That means a 7B-14B model fine-tunes on a single consumer GPU with 16-24GB VRAM, in hours instead of days.
Why LoRA Works
Research on parameter-efficient fine-tuning (PEFT) showed that the weight updates needed to adapt a model to a task live in a low-dimensional subspace. LoRA approximates the update as the product of two small matrices. The base model stays frozen and untouched; the LoRA weights are a small file you load at inference time. You can even keep multiple LoRA adapters for one base model and swap them per task - one model, ten specialized personalities.
The Practical Workflow
- Prepare data. 500-5,000 high-quality examples in the format you need (chat, Q&A, structured output). Quality beats quantity here.
- Choose a base. A strong base beats a weak fine-tune: Qwen2.5, Llama 3.x, or Mistral models are the common starting points.
- Train with a framework. LLaMA-Factory or Unsloth handle the details; the config is: rank 8-64, alpha 2x rank, learning rate ~2e-4.
- Evaluate, not admire. Compare adapter vs base on a held-out set - if your custom terms appear and general ability did not collapse, ship it.
The Failure Modes
- Catastrophic forgetting - the adapter overfits your data and the model forgets general knowledge. Mitigate with a small learning rate and mixed data.
- Data leakage - test examples that look like training examples inflate your numbers.
- Wrong rank - rank too low underfits, too high overfits; start at 16 and move up only if underfitting.
QLoRA: The Memory Trick
QLoRA adds 4-bit quantization of the base model, letting you fine-tune a 13B model on a single 24GB card. The trade-off is a small quality hit versus full LoRA; the ability to train locally usually wins.
FAQ
Is LoRA free? The technique and frameworks (LLaMA-Factory, Unsloth) are open source; you pay for GPU time.
LoRA vs full fine-tuning? LoRA: faster, cheaper, 80-95% of the quality for most tasks. Full: best quality, needs serious hardware.
Can I combine LoRA with RAG? Yes - they solve different problems: LoRA teaches style/format, RAG supplies facts.
What about copyright on my data? Your fine-tune is your model - just make sure you own the training data.
