How LLMs Actually Work 2026: NanoGPT (62k Stars) - Build GPT From Scratch and Truly Understand It

You use ChatGPT daily but have no idea what happens inside. NanoGPT - Karpathy's minimal GPT implementation - trains a real transformer on a laptop. Walking through it is the fastest way to truly understand LLMs.

## The short answer **NanoGPT** (61,797 stars, MIT, Andrej Karpathy) is the smallest complete GPT implementation: ~300 lines of PyTorch that train a character-level transformer on your laptop in minutes. Reading it - or better, his 2-hour video walkthrough - is the fastest way to understand how LLMs actually work. ## What NanoGPT teaches you - **Tokenization**: text becomes integer IDs (via a tiny BPE tokenizer or character-level). - **Embeddings**: each token ID gets a learned vector. - **Transformer blocks**: self-attention (how tokens relate), feed-forward layers, layer norm, residual connections. - **Training loop**: sampling batches, computing cross-entropy loss, backprop, gradient updates. - **Generation**: sampling token by token with temperature. ## Run it yourself ```bash git clone https://github.com/karpathy/nanoGPT cd nanoGPT python data/shakespeare_char/prepare.py # tiny dataset python train.py config/train_shakespeare_char.py # ~3 min on CPU python sample.py # generate Shakespeare-like text ``` That's it - a real transformer trained from scratch, on a laptop, no GPU needed (though GPU is faster). ## Why this matters in 2026 Every serious AI engineer or power user benefits from mental models of what's under the hood: - **Understand context windows**: why they're limited, how attention scales quadratically. - **Understand hallucinations**: it's a next-token predictor - confident text is its only mode. - **Understand fine-tuning**: the same architecture, just continued training on your data. - **Evaluate honestly**: when someone claims a "new architecture", you can ask real questions. ## Learning path 1. Watch Karpathy's "Let's build GPT" video (free, ~2 hours). 2. Run NanoGPT locally with the Shakespeare dataset. 3. Modify: change model size, add a tiny dataset of your own, compare outputs. 4. Then read the official GPT-2/3 papers - the code will make them click. ## FAQ **Do I need a GPU?** No - the default config trains on CPU in ~3 minutes; a GPU makes it faster and lets you scale up. **Is NanoGPT useful for real projects?** It's for learning; production LLMs use much larger models and infrastructure. But the concepts transfer 100%. **Is this enough to understand modern LLMs?** It's the foundation; add attention papers and fine-tuning guides (see our LoRA guide) for the full picture. ## Related - [LLM Inference Engines: vLLM vs SGLang](/post/llm-inference-vllm-sglang-guide-20260802) - [Fine-Tuning With LoRA 2026](/post/fine-tuning-lora-guide-20260802)