Pretraining, continued pretraining, fine-tuning and alignment get mixed together, so projects budget for the wrong stage. Nine out of ten needs land at the end of the pipeline; knowing where you are saves a server's worth of money.

📜 Table of Contents

• Stage 1: Pretraining (rarely yours to do)
• Stage 2: Continued pretraining (domain adaptation)
• Stage 3: Instruction tuning (where 99% of projects live)
• Stage 4: Alignment
• A workflow that lands
• Cost structure
• FAQ
• Related

AI Model Training 2026: From Pretraining to Fine-Tuning, the Complete Workflow

"Train our own model" hides four different jobs: pretraining from scratch, continued pretraining, instruction tuning, and preference alignment. Costs differ by three or four orders of magnitude.

Most projects need the third stage but budget for the first.

Stage 1: Pretraining (rarely yours to do)

Training from random initialization on trillions of tokens. Frontier models cost tens of millions in GPU time plus data pipelines and distributed training engineering. You consume the output — download an open base model — and check three things: target-language ability, licence, and tooling support.

Stage 2: Continued pretraining (domain adaptation)

If you have large unlabelled domain text and the base model's vocabulary and phrasing are clearly off, keep training on your own corpus.

This step is overrated. Most "the model doesn't understand our industry" complaints are prompt or retrieval problems.

Stage 3: Instruction tuning (where 99% of projects live)

Memory is weights + gradients + optimizer states. AdamW keeps fp32 first and second moments, which is why 7B full fine-tuning needs 80GB+ while QLoRA fits on a consumer card.

Stage 4: Alignment

Fine-tuning teaches the task; alignment teaches conduct. DPO dominates open pipelines because it needs only chosen/rejected pairs, no reward model. Usually the model provider's job; do it yourself only after instruction tuning converges and you have a preference-labelling process.

A workflow that lands

1. Pick a base model by language, licence and tooling — not leaderboard rank.
2. Build a 100-500 item eval set with gold answers, frozen before training.
3. Label 500-5000 instruction pairs; validate the process with 100 first.
4. Run QLoRA for one epoch on a single GPU.
5. If eval scores don't move, fix data — don't add epochs or a bigger model.
6. Merge adapters, convert to GGUF or AWQ.
7. Log production failures as seeds for the next labelling round.

Step 5 is the one people skip and the one that pays.

Cost structure

QLoRA on 7B is a few GPU hours — electricity. Full fine-tuning is multi-GPU hours to days at cloud rates that vary widely by provider. Continued pretraining is usually dominated by data-cleaning labour. Serving costs live in power and VRAM.

FAQ

Can prompts and RAG replace fine-tuning? Often yes, and you should try first. Prompts fix format and role, RAG fixes knowledge freshness, fine-tuning fixes consistent style and per-task inference cost.

7B or 70B? Extraction, classification and format conversion work at 7B after tuning. Multi-step reasoning and long-document understanding are base-model capabilities that tuning rarely closes.

Does tuning break existing skills? Catastrophic forgetting is real. Mix 10-20% general instruction data, keep epochs at 1-3, don't over-raise the learning rate.

When is continued pretraining justified? When the model mangles domain vocabulary. When it merely lacks your latest policy, that's retrieval.

Can an LLM grade the eval set? For triage yes, for final judgement no — graders carry style and length biases.

Note: VRAM, time and price figures are order-of-magnitude; verify on your hardware and provider pricing pages.

Related

• Unsloth QLoRA教程2026：8GB显存就能跑的4-bit微调
• LLM微调工具横评2026：LLaMA-Factory、Unsloth、PEFT、TRL怎么选
• LLM微调数据标注2026：Label Studio、LLM辅助标注与质量控制