LLM Tokenizer Explained 2026: Why Your Prompt Costs More Than You Think

Token counts never match your intuition - a Chinese character can cost 1-3 tokens while English words cost 1-2. Understanding the tokenizer explains your API bill, context limits, and weird model behavior.

💡 What You Will Learn

Token counts never match your intuition - a Chinese character can cost 1-3 tokens while English words cost 1-2. Understanding the tokenizer explains your API bill, context limits, and weird model beha

📜 Table of Contents

• What a Tokenizer Does
• The Surprising Math
• The Cost Multiplier You Are Ignoring
• How Models Count Context
• Tools to Check Your Own Text
• One Rule That Fixes Most Waste

What a Tokenizer Does

Before an LLM reads text, a tokenizer splits it into tokens - chunks of 1 to about 6 characters. The model predicts tokens, not characters or words. This single fact explains most of your API costs.

The Surprising Math

• English: about 0.75 tokens per word (GPT-4o). hello world = 2 tokens.
• Chinese: about 1.5-2 tokens per character. A 100-character Chinese paragraph can cost as much as a 300-character English one.
• Code: JSON and code tokenize poorly - braces, quotes and keywords each burn tokens. A minified JSON payload can cost 2-3x its character count.
• Emoji and exotic Unicode: 3+ tokens each.

The Cost Multiplier You Are Ignoring

System prompts are re-sent on every request. A 2,000-token system prompt on 10,000 requests/day = 20M tokens/day just for instructions. This is why prompt caching and shorter system prompts save real money.

How Models Count Context

The context window (e.g. 128k tokens) is measured in tokens, not characters. Practical consequence: a 128k model fits roughly 90k English words, or only about 64k Chinese characters. Localization teams constantly mis-plan around this.

Tools to Check Your Own Text

• OpenAI tokenizer (platform.openai.com/tokenizer) - the classic playground
• tiktoken - the open source BPE tokenizer library behind GPT models
• Hugging Face tokenizers - the library (used by transformers, 163,377 stars) for all open models; every model card shows tokenizer stats

One Rule That Fixes Most Waste

Never send what the model does not need. Trim conversation history aggressively (keep the last N turns plus a summary), compress JSON, and move static instructions into cached system prompts. Tokenizer-aware prompting is the cheapest optimization in the LLM stack.