AI Test Maker in Chinese: 3 of 120 Wrong Options Are Unverifiable

🔧 AI Tools 2026-11-15 9 min read

Chinese generation rarely invents options that are absent from the passage, 3 of 120 against 17 of 120 in English, but the failure moves: options are short quantifier swaps, the key lands in position A in 21 of 40 items once refutability is required, and the judge mislabels half of the few options it flags.

📜 Table of Contents

Chinese test maker: 3 of 120 wrong options cannot be ruled out
Eight Chinese passages, 40 items, 240 wrong options audited one by one: 3 cannot be ruled out, against 17 of 120 in the English batch. Run on 2026-11-15.

Last week's audit of generated quizzes ran on English passages, and it ended with a caveat: Chinese material was not tested, and the rate of wrong options absent from the passage could be different there. This is that test. Eight Chinese passages, same protocol, same models, same judge. The rate is not different, it is five times lower. Three wrong options out of 120 are not in the passage, against seventeen in the English batch.

That reads like good news and is only half of one. The other half is what happens to the items when the wrong options stop being absent: the key slides to the first position in 21 of 40 items, and the wrong options turn into true statements lifted from other sentences of the passage.

How it was run

Eight passages of 167 to 201 Chinese characters, the same eight subjects as the English batch so that the comparison has something to stand on: tidal locking, the Namib fog-basking beetle, tunnel boring machines, caffeine as a plant defence, the Fresnel lens, the honeybee waggle dance, freeze-dried instant coffee, the Dead Sea. They are not translations; I wrote them fresh in Chinese, which is a confound and is noted again at the end.

Each passage went once to deepseek-flash at temperature 1.0 with an instruction asking for five four-option items, one unambiguous key, the key as a 0-based index, JSON out. All 40 items came back from all eight passages in the first run and again in the second: no empty responses this time, because Chinese output is short enough to fit inside a 9,000-token cap even with the reasoning pass. Each of the 240 wrong options then went to deepseek-v4-pro at temperature 0 with the passage, the stem and the option, asked for one of three labels and a verbatim quote. Separately, each of the 80 items went to the same judge with the key hidden, asked to answer it from the passage; that is the key check, and it agreed with the stated key on 80 of 80.

Labels the judge gave, and what the items actually are

On the default instruction, the judge called 6 of 120 wrong options absent. I read all nine flagged options against their passages. Three of the six labels are wrong, and wrong in the same direction.

relation between the passage and the wrong optionthe judge saidafter reading each one
a sentence in the passage rules it out113116
the passage never mentions it, and never contradicts it63
the passage supports it, but it does not answer the question11

Take the beetle item. The stem asks how much annual rain the habitat gets, options are 10, 20, 50 and 100 millimetres, the key is 20, and the passage says annual rain is often under 20 millimetres. The judge called 50 contradicted with a quote, and called 100 absent without one. Both are above 20; if 50 is refutable, 100 is refutable by the same sentence. That is a judge inconsistency, not a property of the item.

The other two are the same pattern. One distractor says the jacking-group stroke difference stays at zero, and the passage says the stroke difference is how the axis is corrected, so the passage rules it out; the judge called it absent. One says freeze-drying needs no extraction, and the passage's first step is extraction, again refutable by a sentence; the judge called it absent. What the judge is actually sensitive to is wording: an option that reuses passage vocabulary while flipping a quantity gets called contradicted, and one that paraphrases a negation gets called absent.

So the number to carry is three, not six: 3 of 120 wrong options in the Chinese batch cannot be ruled out from the passage at all, and they are all short noun phrases: dune height, ten millimetres, matching the screw speed to the segment assembly speed. That last one is the giveaway. It swaps one speed-matching pair from the passage, discharge-versus-excavation, for another pair, so a student who has read the passage carefully can rule it out, and a student who has not cannot.

Against the English batch

Same protocol, same models, same judge, run on 2026-11-14. The English numbers include my own reading of the ten flagged items.

eight English passages (11-14)eight Chinese passages (today)
items4040
wrong options the passage never mentions173
share of 120 wrong options14%2.5%
items carrying at least one10 of 403 of 40
mean option length, characters37.08.7
the key is the longest option24 of 40, 60%16 of 40, 40%

Seventeen to three is a large gap on small samples, and the honest statement is that the direction is clear and the size is not. The option length is a better clue to the mechanism than the counts. Chinese wrong options average 8.7 characters; English ones average 37.0. The Chinese generator writes items in a fill-in-the-blank shape: a one-line stem with a quantity or a term to retrieve, then three rival terms. A rival term has to come from somewhere in the passage to be plausible, so it usually can be checked against a sentence. The English generator writes whole clauses of outside-world detail, solar panels, desalination plants, and those cannot be checked because they were never there. In the English batch the absent options were longer than the average option, 40.5 against 34.6 characters, which is the fingerprint of the clause-shaped distractor.

What this means for a teacher is not that Chinese generation is safer. It means the failure moves. Chinese items fail by being answerable without the passage: a stem like "how much water does it drink" with options that differ only in the quantifier, 百分之十几, 百分之几, 百分之几十, is a test of attention, not of reading. That item came out of the batch; the key is fine and the item is still close to worthless.

The same one line, in Chinese

Yesterday's fix was a constraint in the request rather than a filter on the output: every wrong option must be refutable by citing a specific sentence, and options the passage never mentions are out of bounds. The Chinese version of that line, same eight passages, same model, same temperature, same item count.

default instructionone line added
items4040
wrong options the passage never mentions30
key in position A12 of 4021 of 40
key in position D3 of 408 of 40
mean option length, characters8.715.5
the key is the longest option40%42%
stems that also appeared in the other run1 of 40

Absent options go to zero, so the instruction transfers. Two costs are larger in Chinese than they were in English. Option length nearly doubles, 8.7 to 15.5 characters, which is expected: a refutable distractor needs the specific detail that makes it refutable. And the key position collapses toward the front, 12 of 40 in position A becoming 21 of 40, while position D goes from 3 to 8. English moved the other way on the same instruction, so this is a Chinese-side artefact of the model, not a property of the constraint. If the bank will be handed out as printed sheets, shuffle the options in code after generation; do not ask the model to balance them.

The zero deserves one more caveat. Refutability is not relevance. Two items in the stricter batch were labelled supported, and both are shapes the constraint pushes the model toward. One carries a wrong option the passage states outright and that does not answer the stem: asked what holds the face and keeps chamber pressure, one option is that a laser measures axis deviation. As a statement it is true in the passage, as an answer it is out, and the judge was asked about the relation between passage and option, not about whether the option answers the question, so it counted as contradicted. The other is a quantifier variant: asked how much water the beetle drinks, the options are 百分之十几, 百分之几, 百分之几十 and 完全不喝水, and the one labelled supported is the passage sentence with a character removed, neither true nor foreign. Forbidding absent options pushes the generator into exactly this: instead of inventing content, it lifts other true sentences out of the same passage, or blurs the original. A teacher can live with that trade, but it needs a second pass that asks whether the option answers the stem and whether it differs from another option by a quantifier only, which this audit did not do.

The key was not the problem

Each of the 80 items was answered independently, key hidden, passage in front, and asked for the index plus the sentence relied on. All 80 matched the stated key, in both instruction sets. That is consistent with the English batch, 40 of 40 both times, and it sets the boundary of this problem: what breaks in generated quizzes is not the answer key, it is everything around it, the wrong options in English and the item shape in Chinese.

Config to copy

The Chinese generation instruction, with the line that did the work:

就下面这段材料出 {n} 道单项选择题,每题四个选项,只有一个是正确的,可以在材料中找到依据。
每个错误选项都必须能被材料里的某一句话直接否掉;材料压根没提的内容不许写成错误选项。
答案用0到3的下标表示。
返回 JSON:{"items":[{"stem":"题干","options":["选项一","选项二","选项三","选项四"],"answer":0}]}

材料:
{passage}

Two settings that matter more than they look. max_tokens at 9,000 with the default reasoning pass, because a truncated response parses as fewer items rather than as an error. And temperature 1.0 for generation, so the two runs are not the same questions with a token changed: only one stem of 40 appeared in both instruction sets.

The audit prompt is four lines: the passage, the stem, the option, and three labels, contradicted, absent, supported, with the shortest supporting sentence quoted verbatim, empty quote for absent. Requiring the quote is what exposed the judge's inconsistency above; without it there is nothing to compare two labels against. Script is standard library only, 10 workers, 240 options in about four minutes.

What this does not cover

The Chinese passages are rewrites, not translations, so language and text both changed; a cleaner design would machine-translate the same eight passages and run both versions. One judge, one generation model, both DeepSeek, and the judge mislabels about half of the few options it flags, which is why the manual count is reported next to it. Forty items per instruction set is small; 17 against 3 is a direction, not a coefficient. Chinese numerals and quantifiers make the "absent" judgement harder than it is in English, so part of the gap could be measurement. And nothing here touches the question a teacher actually cares about, which is whether a student who has not read the passage can beat one who has. That was measured on English material only, at 71 of 80, and it is the next thing to run in Chinese.

Practical version. If the material is Chinese and the items are for print, use the refutability line, then shuffle options in code, then read ten items yourself looking for quantifier-only distractors. If the material is English, audit the wrong options rather than the keys, because the keys held in every batch measured so far. The free-tier limits and export rules of nine quiz tools are here: what each free quiz maker actually gives you. The English batch this one is compared against is here: 17 of 120 wrong options were never in the passage. And where 96 English keys landed across four positions, 5 of them in D, is here: answer position bias. Two numbers to take away: 3 of 120 Chinese wrong options cannot be ruled out from the passage, against 17 of 120 in English, and the refutability line takes both to zero while pushing 21 of 40 keys into position A.

The batch that came next stopped auditing options and counted coverage: ten items per passage rest on 5.1 sentences, the same number of sentences the passages contain, and an instruction to test different sentences changed nothing: how many items one passage can actually carry.

Related Articles
2026-08-17
Best AI Avatar Generator in Istanbul 2026: 7 Tools for Turkish E-commerce and Ads
2026-10-14
Best AI Podcast Editing Tool in Cork 2026: 5 Picks for Two-Host Shows Recorded Apart
2026-08-12
Databricks Mosaic AI Model Serving 2026: Setup, Pricing and When It Beats Self-Hosting
2026-08-14
AI Diagram Generator From Text 2026: 6 Free Tools That Draw From Prompts
2026-08-10
AI Legal Tools 2026: 8 Free and Open Source Options for Research and Review
2026-08-15
Best AI Video Editor in Singapore 2026: 6 Tools for Content Creators and SMEs

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment