12 Sentences, 4 Languages: Japanese Costs 1.5x English Tokens, Chinese 1.05x

📘 Tutorials 2026-11-23 7 min read

Context budgets are set by word count, or by the folklore that Chinese costs two to three times the tokens. The first number is wrong for anything but English, and the second is wrong in the direction people expect.

📜 Table of Contents

Twelve sentences, four languages, four tokenizers. Chinese came back at 1.03 to 1.10 times the English token count. Japanese at 1.19 to 1.63, German at 1.21 to 1.41. Measured per character on the same Wikipedia article, the order flips hard: Chinese costs 3.1 to 4.1 times the English tokens for every 1,000 characters. A 32k window that holds 141,241 English characters holds 45,899 Chinese ones on the same tokenizer.

Both numbers are correct. They answer different questions, and most budgets people set use neither.

The claim this test was built to check

The number everyone quotes is that one token is about 0.75 English words, and that Chinese or Japanese cost two to three times more tokens for the same content. The first half is close for English. The second half is where things get muddled, because there are two different "same content" comparisons and they do not agree.

Same sentence, same meaning: how many tokens? That is the number that matters when you price a request, because the request carries meaning, not bytes. Same volume of text: how many tokens per 1,000 characters? That is the number that matters when you decide how much of a document fits in a window, because a window holds tokens, not ideas.

How the numbers were produced

Four tokenizer files, downloaded from Hugging Face on 23 November 2026: Qwen3-32B, Llama-3.1-8B-Instruct, gpt-oss-120b, and Gemma-3-27B. Llama came from the NousResearch mirror and Gemma from the unsloth mirror, because both first-party repos need an auth token. Each file was loaded with tokenizers 0.23.3 and encoded with encode(text).ids, no special tokens added, no model inference anywhere in this test.

The sample is 12 sentences I wrote in English and translated into Chinese, Japanese and German. Six are plain everyday sentences. Six are packed with the things that usually break tokenizers: an invoice number with a year, a four-digit HTTP status, a batch size, a street address, a storage figure in TB, and an email address. The translations are mine, checked sentence by sentence for the same content; that is the softest part of this test and it is stated here rather than buried.

Plain sentences: Chinese was cheaper than English on three of four tokenizers

tokenizerEnglishChineseJapaneseGermanChinese / EnglishJapanese / English
Qwen3-32B504283830.841.66
Llama-3.1-8B565790901.021.61
gpt-oss-120b504789660.941.78
Gemma-3-27B565068750.891.21

Chinese is not expensive here. It is slightly cheaper than English on Qwen3, gpt-oss and Gemma, and level on Llama. The reason is vocabulary: a Chinese character is one to three UTF-8 bytes, common characters sit in the vocabulary whole, and Qwen3 ships 151k entries while Gemma ships 262k. English pays for the space before every word and for subword splits on anything past the top few thousand words.

Japanese is the opposite case. Kanji, hiragana, katakana and Latin loanwords in one sentence, and none of the four vocabularies covers the mix well. Best case is Gemma at 1.21; worst is gpt-oss at 1.78. That is a 78% surcharge on identical meaning.

Add invoice numbers and email addresses and the ordering changes

tokenizerENZHJADEZH / ENJA / ENDE / EN
Qwen3-32B1011161491261.151.481.25
Llama-3.1-8B921061381181.151.501.28
gpt-oss-120b86981321011.141.531.17
Gemma-3-27B1091201291251.101.181.15

Six sentences of dates, codes and addresses and the Chinese advantage is gone: 1.10 to 1.15 times English instead of 0.84 to 1.02. Full-width punctuation, the spacing rules around Latin runs, and the way a date like 2026 年 12 月 5 日 spreads across characters all cost. Mixed content is the normal case in production, so the honest summary is the combined one:

languageQwen3-32BLlama-3.1-8Bgpt-oss-120bGemma-3-27B
English (baseline)151148136165
Chinese158 (1.05x)163 (1.10x)145 (1.07x)170 (1.03x)
Japanese232 (1.54x)228 (1.54x)221 (1.63x)197 (1.19x)
German209 (1.38x)208 (1.41x)167 (1.23x)200 (1.21x)

Twelve sentences is a small sample and these ratios carry roughly a tenth either way. The direction is stable across all four tokenizers, which is the point: Japanese and German are the expensive languages for the same meaning, and Chinese is nearly free.

What 32,768 tokens holds, in characters

Different question, different answer. Take one Wikipedia article, the Coffee page, in the four languages: 57,003 characters in English, 12,526 in Chinese, 27,040 in Japanese, 85,471 in German. The articles are not translations of each other, so read this as a per-character rate, not a translation test.

languagetokens per 1,000 chars, Qwen3Llama-3.1gpt-ossGemma-3chars that fit in 32k, Qwen3
English232.0220.1216.6227.7141,241
Chinese713.9905.8875.3706.545,899
Japanese677.8748.7809.8578.048,346
German308.4300.9246.1259.0106,253

Qwen3 holds 141,241 English characters in 32,768 tokens. The same window holds 45,899 Chinese characters, under a third. Not because Chinese sentences are expensive but because a Chinese character carries far more content per glyph and the vocabulary stores it in fewer, denser units. If you are deciding whether a 12,000-character Chinese report fits, the answer on Qwen3 is yes, with room for the prompt and the answer. If you are deciding whether a 120,000-character English log file fits, the answer is no, on any tokenizer here.

Word counting fails in both directions. A 32k window is about 25,200 English words at 1.3 tokens per word, and English is the only language where that conversion is safe.

On English the tokenizer choice is a 2.9% rounding error; on Japanese it is 32%

One fixed 1,337-character English passage, four tokenizers: 278, 283, 285, 286 tokens. A 2.9% spread, which is the real size of the tokenizer wars on English text. Per word that works out to 1.27 to 1.31, which is why the 0.75 words-per-token rule survives.

Japanese on the same six plain sentences: 68 tokens on Gemma, 89 on gpt-oss, 90 on Llama. Choosing a tokenizer changes the Japanese bill by a third while changing the English bill by 3%. If your product is Japanese and you are picking a model on price, the tokenizer is part of the price.

Run the same check on your own corpus

Ten lines, no inference, takes a second per thousand sentences. Download tokenizer.json from the model repo you actually plan to serve, then:

from tokenizers import Tokenizer

tk = Tokenizer.from_file("tokenizer.json")
samples = {"en": english_rows, "zh": chinese_rows, "ja": japanese_rows}

for lang, rows in samples.items():
    tokens = sum(len(tk.encode(r).ids) for r in rows)
    chars = sum(len(r) for r in rows)
    print(lang, tokens, round(tokens / chars * 1000, 1), "tokens per 1000 chars")

Use the output for two settings and nothing else. First, a per-language character budget for retrieval and for cutting long inputs: never plan a cut by word or character count across languages, plan it by measured tokens. Second, a per-language price sheet: if you bill customers per token and your customers write in Japanese, your margin is already 1.5 times thinner than the English price list suggests.

The context side of this arithmetic, meaning what a token of KV cache costs once it is in the model, is in the per-token KV cache table. What the retrieval bill looks like per 1,000 questions is in the RAG cost measurement, and what a tool list costs before the user types anything is in the MCP token count. The mechanism behind byte-level BPE, if you want the why rather than the numbers, is in the tokenizer explainer.

What this test does not show

Twelve sentences, translated by me, encoded without special tokens. No generation, so nothing here says anything about output length or quality per language. Chat templates add per-message overhead that this test does not count. The Gemma and Llama tokenizer files were read from mirrors, and gpt-oss uses the o200k_harmony vocabulary the open weights ship with. One tokenizer version, one day, 23 November 2026. The ratios are stable enough across four vocabularies to act on; the exact numbers are not constants.

Related Articles
2026-07-22
Model Quantization 2026: GGUF AWQ GPTQ
2026-08-07
AI Subtitle Translator: Open-Source Pipeline for Any Video
2026-07-19
AnythingLLM Tutorial: Manage All AI Models in One Interface
2026-08-01
Run Ollama on Android in 2026: 4 Apps Compared
2026-07-26
AI for Business Analysts 2026
2026-08-01
llms.txt Generator 2026: The AI Site Map That Gets Your Content Cited

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment