How LLMs Actually Work 2026: NanoGPT (62k Stars) - Build GPT From Scratch and Truly Understand It
You use ChatGPT daily but have no idea what happens inside. NanoGPT - Karpathy's minimal GPT implementation - trains a real transformer on a laptop. Walking through it is the fastest way to truly understand LLMs.
💡 What You Will Learn
You use ChatGPT daily but have no idea what happens inside. NanoGPT - Karpathy's minimal GPT implementation - trains a real transformer on a laptop. Walking through it is the fastest way to truly unde
📜 Table of Contents
The short answer
NanoGPT (61,797 stars, MIT, Andrej Karpathy) is the smallest complete GPT implementation: ~300 lines of PyTorch that train a character-level transformer on your laptop in minutes. Reading it - or better, his 2-hour video walkthrough - is the fastest way to understand how LLMs actually work.
What NanoGPT teaches you
- Tokenization: text becomes integer IDs (via a tiny BPE tokenizer or character-level).
- Embeddings: each token ID gets a learned vector.
- Transformer blocks: self-attention (how tokens relate), feed-forward layers, layer norm, residual connections.
- Training loop: sampling batches, computing cross-entropy loss, backprop, gradient updates.
- Generation: sampling token by token with temperature.
Run it yourself
git clone https://github.com/karpathy/nanoGPT
cd nanoGPT
python data/shakespeare_char/prepare.py # tiny dataset
python train.py config/train_shakespeare_char.py # ~3 min on CPU
python sample.py # generate Shakespeare-like text
That's it - a real transformer trained from scratch, on a laptop, no GPU needed (though GPU is faster).
Why this matters in 2026
Every serious AI engineer or power user benefits from mental models of what's under the hood:
- Understand context windows: why they're limited, how attention scales quadratically.
- Understand hallucinations: it's a next-token predictor - confident text is its only mode.
- Understand fine-tuning: the same architecture, just continued training on your data.
- Evaluate honestly: when someone claims a "new architecture", you can ask real questions.
Learning path
- Watch Karpathy's "Let's build GPT" video (free, ~2 hours).
- Run NanoGPT locally with the Shakespeare dataset.
- Modify: change model size, add a tiny dataset of your own, compare outputs.
- Then read the official GPT-2/3 papers - the code will make them click.
FAQ
Do I need a GPU? No - the default config trains on CPU in ~3 minutes; a GPU makes it faster and lets you scale up.
Is NanoGPT useful for real projects? It's for learning; production LLMs use much larger models and infrastructure. But the concepts transfer 100%.
Is this enough to understand modern LLMs? It's the foundation; add attention papers and fine-tuning guides (see our LoRA guide) for the full picture.
Related
❓ FAQ
Do I need a GPU?
No - the default config trains on CPU in ~3 minutes; a GPU makes it faster and lets you scale up.
Is NanoGPT useful for real projects?
It's for learning; production LLMs use much larger models and infrastructure. But the concepts transfer 100%.
Is this enough to understand modern LLMs?
It's the foundation; add attention papers and fine-tuning guides (see our LoRA guide) for the full picture.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
