GPT-2 is small and free, the ideal model for learning fine-tuning - but most tutorials assume you already know transformers internals. Here is a path that runs from zero.

📜 Table of Contents

• Why GPT-2 for practice
• Environment and data
• Full code
• Hyperparameters
• How long it takes
• Overfitting versus working
• Verify with generation
• Next step: a 7B model
• FAQ
• Related

Fine-Tune GPT-2 in 2026: A Complete Beginner Tutorial With Real Code

The expensive part of learning fine-tuning is not compute, it is not understanding the pipeline. GPT-2 is around 124M parameters, trains in minutes on a laptop GPU, is fully supported by transformers, and exposes every step of the chain. Once it runs end to end, swapping in a 7B model with QLoRA is the 2026 mainstream path.

Why GPT-2 for practice

• Low VRAM needs; consumer GPUs and even CPUs work.
• Public weights and tokenizer, no access requests.
• Fast iterations - change a hyperparameter and see results in minutes.
• It surfaces problems common to all models: padding, special tokens, overfitting, save/reload.

Its weakness is real: GPT-2's Chinese ability is weak. Treat it as a teaching model, not a product backbone.

Environment and data

pip install torch transformers datasets accelerate

A plain text file is enough - one paragraph or one document per line. Causal language modeling needs no labels: the text is both input and target.

Aim for 500-2000 paragraphs. Too little and no pattern is learned; beyond that the gain flattens. Keep the content stylistically consistent with the output you want.

Full code

from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForCausalLM,
                          DataCollatorForLanguageModeling,
                          Trainer, TrainingArguments)

MODEL = "openai-community/gpt2"
tok = AutoTokenizer.from_pretrained(MODEL)
if tok.pad_token is None:          # GPT-2 has no pad token
    tok.pad_token = tok.eos_token

raw = load_dataset("text", data_files={"train": "mydata.txt"})["train"]
raw = raw.train_test_split(test_size=0.05, seed=42)

def tokenize(batch):
    return tok(batch["text"], truncation=True, max_length=256)

ds = raw.map(tokenize, batched=True, remove_columns=["text"])
model = AutoModelForCausalLM.from_pretrained(MODEL)

args = TrainingArguments(
    output_dir="./gpt2-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=5e-5,
    warmup_steps=100,
    logging_steps=20,
    save_strategy="epoch",
    eval_strategy="epoch",   # older versions: evaluation_strategy
    fp16=False,
)

collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False)

trainer = Trainer(model=model, args=args,
                  train_dataset=ds["train"], eval_dataset=ds["test"],
                  data_collator=collator)
trainer.train()
trainer.save_model("./gpt2-finetuned")
tok.save_pretrained("./gpt2-finetuned")

Common traps: mlm=False for causal LM (wrong value gives nonsense loss); pad_token must be set or batching fails; remove_columns=["text"] avoids collator type errors; max_length drives VRAM use.

Hyperparameters

Do not start with a learning-rate sweep. Get a loss curve that decreases first.

How long it takes

Shorter max_length usually beats new hardware for wall-clock time.

Overfitting versus working

Training loss falls while validation loss falls then rises - that is overfitting. Add data, cut epochs, use dropout.

Output side is more telling: repeating one sentence, or verbatim regurgitation of training text, means memorization, not learning. Garbled output usually means learning rate too high or too few steps.

Verify with generation

from transformers import pipeline
gen = pipeline("text-generation", model="./gpt2-finetuned",
               tokenizer="./gpt2-finetuned", device_map="auto")
print(gen("Today's product update", max_new_tokens=80, do_sample=True,
          temperature=0.8, top_p=0.95, repetition_penalty=1.1)[0]["generated_text"])

do_sample=False gives deterministic output for A/B comparison; repetition_penalty above 1 suppresses loops. Keep prompt and sampling fixed when comparing models.

Next step: a 7B model

Same pipeline, three changes: load a 7B model (Llama, Qwen) in 4-bit; train with QLoRA so only low-rank adapters are updated; save adapters (tens of MB) and merge at load time.

By then you already understand padding, truncation, learning rate and overfitting - none of which depend on model size.

FAQ

How much data? About 500 samples shows style adaptation. Factual knowledge belongs in retrieval, not fine-tuning.

Chinese data on GPT-2? Possible but inefficient - Chinese characters split into many tokens. Use a Qwen-family model for Chinese.

Commercial use? Depends on the base model license. Read the model card rather than assuming.

Loss dropped but answers got no better? Loss measures next-token prediction, not your quality bar. Build your own test prompts.

CPU training too slow? Shorten sequences, cut samples, run 1-2 epochs to validate the pipeline, then move to a GPU.

Note: transformers parameter names change between versions (eval_strategy vs evaluation_strategy). Check the current official docs before running.

Related

• Unsloth QLoRA tutorial 2026: 4-bit fine-tuning in 8GB
• Open-source coding models in 2026