Models read tokens, not words. The same paragraph can cost nearly twice as many tokens on a different model, and your bill and context limit scale with it. Here is how tokenizers split text, how to count tokens yourself, and which habits quietly burn money.

📜 Table of Contents

• What a Tokenizer Does
• Anchors You Can Check Yourself
• Counting Tokens: Three Routes
• Context Windows in Words
• Four Habits That Cut Cost Immediately
• FAQ
• Related

LLM Tokenizer Explained 2026: Why Your Prompt Costs More Than You Think

Models read tokens, not words. The same paragraph can cost nearly twice as many tokens on a different model, and your bill and context limit scale with it.

What a Tokenizer Does

A tokenizer maps text to integer IDs. The dominant algorithm is BPE (byte pair encoding): start from raw bytes, repeatedly merge the most frequent adjacent pairs, and end up with a vocabulary of tens to hundreds of thousands of entries. The model never sees characters, only IDs.

Three consequences matter for cost:

• Frequent words already merged in the vocabulary cost one token. Rare words fall back to subwords or single bytes.
• Chinese is 3 UTF-8 bytes per character. Chinese-first models usually map common characters to one token; many Western models split rare characters, names and mixed-script strings into 2-4 tokens.
• Long digit strings get shredded, which is a large part of why models are bad at arithmetic.

Vocabularies differ per model. OpenAI publishes tiktoken vocabularies (e.g. o200k_base); Claude, Gemini and most Chinese providers do not. Token counts are not portable across models - always measure with the model you will actually call.

Anchors You Can Check Yourself

Treat these as estimates; measure once with the real tokenizer and the guessing stops.

Counting Tokens: Three Routes

# OpenAI
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("count this prompt")))

# Open models
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
print(len(tok("count this prompt")["input_ids"]))

No-code route: OpenAI's tokenizer page shows the exact split, which makes it obvious which segment is being shredded.

Context Windows in Words

Capacity is not usable quality: information in the middle of a long context is often ignored. See the context-window guide.

Four Habits That Cut Cost Immediately

1. Trim history - keep the last few turns verbatim, summarise the rest. Agents resend history every turn.
2. Compress JSON - drop indentation, shorten keys, send only the fields downstream uses.
3. Cache the static prefix - system prompt, tool definitions and corpus background unchanged at the front. Cached-input discounts differ per provider; check official pricing.
4. Cap output - set max_tokens and ask for conclusions only. Output usually costs several times input.

One multiplication people miss: a 2,000-token system prompt at 10,000 requests per day is 20 million input tokens per day, for text that never changes.

FAQ

Same text, different token count? Something dynamic is in the prompt: timestamps, random IDs, retrieval order, tool order. Move variable parts to the end.

Is Chinese more expensive? Depends on the vocabulary. Measure the same text with each model's tokenizer instead of converting heuristics.

How do I count tokens for a local model? Use that model's AutoTokenizer or llama.cpp's tokenize interface.

Why is output priced higher? Generation is sequential, one token at a time. Multiples vary by provider; see official pricing pages.

Note: vocabularies and prices change with releases; verify against official documentation.

Related

• LLM evaluation leaderboards on Hugging Face
• Unified LLM API gateways compared