AI Quiz Answer Key Recheck: 188 Blind Answers and 376 Rewrites, Nothing Moved

🔧 AI Tools 2026-11-20 7 min read

Re-answering looks like the obvious check on a generated key. Measured, it agrees 188 times out of 188 on items whose correct option is usually a lifted sentence, and catches nothing.

📜 Table of Contents

Every check a teacher can run on an AI test maker comes back to one question: is the marked answer right. I tried to break the key on 188 items and could not. Two models answered every item blind and each matched the key 188 times. Then the stems were reworded 376 times and answered again: 376 answers, none moved.

That result is less reassuring than it sounds. The key survives because it can barely fail. Twenty-seven percent of the keys are copied word for word out of the passage, and 106 of 188 have 80% or more of their content words inside a single sentence, so a reworded question still points at the same option. The trouble is one layer out: in 74 of 188 items (39%) a wrong option can be defended by quoting a sentence from the passage, and in 37 of those the defence comes out of the same sentence as the key.

How it was run

Twenty-eight passages, 22 English of 118 to 133 words and six Chinese, the same set used for the cue audit on 18 November. Eight items per English passage, five per Chinese one, four options each, key checkable against the passage, deepseek-flash at temperature 0.8: 28 calls in 42.4 seconds, 206 items asked, 188 usable after dropping any item that came back without four options and a valid index. 7,639 prompt tokens in, 27,016 out.

Three passes over those 188. Blind: passage, stem, four options, no key, answered by deepseek-flash at temperature 0 and by deepseek-v4-pro, 376 calls in 62.3 seconds, 121,936 prompt tokens in. Reworded: the model rewrote each stem without seeing the key, twice, options kept verbatim; mean stem length 54.6 characters against 47.3 before, and mean content-word overlap with the original 0.44, so about half the words changed. Both rewrites were answered blind, 376 calls in 209.7 seconds. Defence: with the key shown and excluded, name one wrong option a student could argue for and paste the sentence that supports it, 188 calls in 376.2 seconds. Every quote that came back, all 74 of them, was checked against the passage and is in it.

Nothing moved

checkwhat was runanswers that matched the key
blind, second model188 items, deepseek-v4-pro, no key shown188
blind, same model at temperature 0188 items, deepseek-flash, no key shown188
stem reworded, answered again188 items x 2 rewrites = 376 answers376 of 376 unchanged

Two model families, 376 blind answers, no disagreement with the key. The rewording pass is the harsher of the two, because a model rewriting a stem with no access to the key has no reason to preserve the cue that made the item work, and 376 of 376 answers still landed on the marked option, 256 in English and 120 in Chinese.

A key built out of a copied sentence cannot fail either test

A key that is a verbatim span of the passage, and 106 of 188 keys sit 80% inside one sentence, is answerable by string matching. Reword the question and the matching still works; hand it to a different model and the matching still works. It is the same mechanism that left 71 of 80 items answerable with the passage hidden in the closed-book test, and the same one the option wording leaked in the cue audit. A tool whose items are built this way will pass any re-answering check without being any good.

The items that would cost a teacher an argument

countitems
a wrong option can be defended by quoting a sentence from the passage74 of 188 (39%)
of those, the defended option itself has 80% or more of its content words inside one sentence32 of 74
of those, the defence comes out of the same sentence as the key37 of 74
of those, rated strong by the model doing the defending (the rest were weak)0 of 74
of those, also flagged by the option count below46 of 74
flagged by the count: two or more options 80% inside one sentence60 of 188 (32%)
items where the model defends an option and the count clears it28
items where the count flags and the model clears it26
union of the two lists88 of 188 (47%)

Three of the 74, verbatim from the run:

questionmarked keywrong option a student can defendthe sentence they would quote
When is an internal sizing agent added during papermaking?To the pulp before the sheet formsOnly at the size press as a filmSurface sizing applies starch or a polymer film at the size press.
At about what temperature is crystallisation fastest?13 to 14 degrees50 to 51 degreesHeating a set jar to about 50 degrees dissolves the crystals again…
What is the residual alignment error called?OverlayDriftWafer heating during exposure, lens drift and stage vibration all move overlay during a run…

All 74 were rated weak by the model doing the defending, none strong: each is resolvable by reading the stem closely. That is the shape of the risk. Not a wrong key, but a distractor built from a real sentence, which costs a student who skims a mark and costs the teacher the conversation about it. Half the list, 37 items, draws the defence from the key's own sentence, where a swapped clause does the damage; the other 37 transplant a sentence from somewhere else in the passage.

Two counts, 54 items apart

Counting options by machine gives 60 of 188 items (32%) with two or more options whose content words sit 80% inside one sentence, 55 with exactly one and 73 with none. That count and the model's defence list agree on 46 items and disagree on 54: 28 items the model defends and the count clears, 26 the count flags and the model clears. The union is 88 items, 47% of the paper, which is too much to hand a teacher for a ten-item quiz. The intersection is 46, a quarter of the paper, and that is the list worth reading.

What I read by hand

Twelve items drawn at random, six English and six Chinese, seed 1120, marked on screen without the model answers in front of me: twelve keys correct, none with two defensible options. The ids: papersizing g8 item 7, battery g8 item 1, sourdough g8 items 7 and 2, ballast g8 item 5, firstcrack g8 item 3, shuilunji junior item 4, lidian junior item 4, daocha junior items 0 and 1, tankong junior item 1, yanjian junior item 3. Twelve is not a sample that clears the set; it is enough to say the automated passes were not missing obvious errors.

The check to copy over

Export the items as JSON, one object per item carrying id, options, answer and the passage the item was built from, and run this.

import json, re, sys

STOP = set("a an the of in on at to for from by with and or but if then that this these is are was were be been "
           "it its as which who what when where why how not no more most such into over under between".split())

def words(s):
    return [w for w in re.findall(r"[a-z][a-z'-]+", s.lower()) if w not in STOP and len(w) > 2]

def flat(s):
    return re.sub(r"[^a-z0-9 ]+", " ", s.lower())

def support(option, passage):
    cw = words(option)
    if not cw:
        return 0.0
    best = 0.0
    for s in re.split(r"(?<=[.!?])\s+", passage):
        f = flat(s)
        best = max(best, sum(1 for w in cw if w in f) / len(cw))
    return best

for it in json.load(open(sys.argv[1])):
    n = sum(1 for o in it["options"] if support(o, it["passage"]) >= 0.8)
    if n >= 2:
        print(it["id"], n)

On this set it prints 60 ids. Above a fifth of the paper, the wording is what to review, not the answer. Do not spend the time on a re-answer pass: on these 188 items it found nothing, and on items like these it cannot find anything.

What this does not measure

Two models from one vendor, temperatures 0 and 0.8, 28 expository passages: no poems, no lab reports, no maths. Nobody sat these items, so nothing here is an observed classroom error rate. The 80% threshold is a choice and a lower one raises the flagged count along with the false positives. The rewrites were written by the same model family that wrote the items, so they may preserve the very cue that makes the key obvious; a human rewriting the stem might break more than zero. And the defence pass was shown the key, which is right for a marking question and wrong for an answerability question.

What is safe to act on: on 188 items, re-answering, either with a second model or after rewording, caught no wrong key, and an item whose correct option is a lifted sentence cannot fail that test by construction. Review time goes further on the 46 items where the model and the count both point at the same two options. Two numbers to carry: 188 of 188 blind answers matching the key, and 74 of 188 items with a wrong option a student can quote the passage for. The free tools these prompts go into are in nine free test makers; what the grade field actually does is in the grade dial audit; how much of a passage ten items cover is in the coverage count; and the checks worth running before a paper goes out are in the six-check gate.

Re-answering cannot see this one either. Two arms, 60 items each, blind answer agreed 60 of 60 in both, including the 18 items whose calculated answer mixes millimetres with days: the table run.

Related Articles
2026-11-13
AI Test Maker: 12 of 16 Items Labelled Hard Answered Without the Passage
2026-08-06
AI Avatar Generator Open Source: SadTalker, LivePortrait and More
2026-08-18
Best AI Headshot Generator for Actors 2026: 6 Tools for Casting and Spotlight
2026-08-12
Free LLM API Platform 2026: Run Your Own OpenAI-Compatible Gateway With Zero Cost
2026-08-02
LLM Evaluation Framework 2026: How to Test Prompts Like OpenAI Does (Promptfoo Guide)
2026-07-26
Claude Cowork: Anthropic Collaboration Platform

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment