AI Quiz Answer Key Recheck: 188 Blind Answers and 376 Rewrites, Nothing Moved
Re-answering looks like the obvious check on a generated key. Measured, it agrees 188 times out of 188 on items whose correct option is usually a lifted sentence, and catches nothing.
📜 Table of Contents
Every check a teacher can run on an AI test maker comes back to one question: is the marked answer right. I tried to break the key on 188 items and could not. Two models answered every item blind and each matched the key 188 times. Then the stems were reworded 376 times and answered again: 376 answers, none moved.
That result is less reassuring than it sounds. The key survives because it can barely fail. Twenty-seven percent of the keys are copied word for word out of the passage, and 106 of 188 have 80% or more of their content words inside a single sentence, so a reworded question still points at the same option. The trouble is one layer out: in 74 of 188 items (39%) a wrong option can be defended by quoting a sentence from the passage, and in 37 of those the defence comes out of the same sentence as the key.
How it was run
Twenty-eight passages, 22 English of 118 to 133 words and six Chinese, the same set used for the cue audit on 18 November. Eight items per English passage, five per Chinese one, four options each, key checkable against the passage, deepseek-flash at temperature 0.8: 28 calls in 42.4 seconds, 206 items asked, 188 usable after dropping any item that came back without four options and a valid index. 7,639 prompt tokens in, 27,016 out.
Three passes over those 188. Blind: passage, stem, four options, no key, answered by deepseek-flash at temperature 0 and by deepseek-v4-pro, 376 calls in 62.3 seconds, 121,936 prompt tokens in. Reworded: the model rewrote each stem without seeing the key, twice, options kept verbatim; mean stem length 54.6 characters against 47.3 before, and mean content-word overlap with the original 0.44, so about half the words changed. Both rewrites were answered blind, 376 calls in 209.7 seconds. Defence: with the key shown and excluded, name one wrong option a student could argue for and paste the sentence that supports it, 188 calls in 376.2 seconds. Every quote that came back, all 74 of them, was checked against the passage and is in it.
Nothing moved
| check | what was run | answers that matched the key |
|---|---|---|
| blind, second model | 188 items, deepseek-v4-pro, no key shown | 188 |
| blind, same model at temperature 0 | 188 items, deepseek-flash, no key shown | 188 |
| stem reworded, answered again | 188 items x 2 rewrites = 376 answers | 376 of 376 unchanged |
Two model families, 376 blind answers, no disagreement with the key. The rewording pass is the harsher of the two, because a model rewriting a stem with no access to the key has no reason to preserve the cue that made the item work, and 376 of 376 answers still landed on the marked option, 256 in English and 120 in Chinese.
A key built out of a copied sentence cannot fail either test
A key that is a verbatim span of the passage, and 106 of 188 keys sit 80% inside one sentence, is answerable by string matching. Reword the question and the matching still works; hand it to a different model and the matching still works. It is the same mechanism that left 71 of 80 items answerable with the passage hidden in the closed-book test, and the same one the option wording leaked in the cue audit. A tool whose items are built this way will pass any re-answering check without being any good.
The items that would cost a teacher an argument
| count | items |
|---|---|
| a wrong option can be defended by quoting a sentence from the passage | 74 of 188 (39%) |
| of those, the defended option itself has 80% or more of its content words inside one sentence | 32 of 74 |
| of those, the defence comes out of the same sentence as the key | 37 of 74 |
| of those, rated strong by the model doing the defending (the rest were weak) | 0 of 74 |
| of those, also flagged by the option count below | 46 of 74 |
| flagged by the count: two or more options 80% inside one sentence | 60 of 188 (32%) |
| items where the model defends an option and the count clears it | 28 |
| items where the count flags and the model clears it | 26 |
| union of the two lists | 88 of 188 (47%) |
Three of the 74, verbatim from the run:
| question | marked key | wrong option a student can defend | the sentence they would quote |
|---|---|---|---|
| When is an internal sizing agent added during papermaking? | To the pulp before the sheet forms | Only at the size press as a film | Surface sizing applies starch or a polymer film at the size press. |
| At about what temperature is crystallisation fastest? | 13 to 14 degrees | 50 to 51 degrees | Heating a set jar to about 50 degrees dissolves the crystals again… |
| What is the residual alignment error called? | Overlay | Drift | Wafer heating during exposure, lens drift and stage vibration all move overlay during a run… |
All 74 were rated weak by the model doing the defending, none strong: each is resolvable by reading the stem closely. That is the shape of the risk. Not a wrong key, but a distractor built from a real sentence, which costs a student who skims a mark and costs the teacher the conversation about it. Half the list, 37 items, draws the defence from the key's own sentence, where a swapped clause does the damage; the other 37 transplant a sentence from somewhere else in the passage.
Two counts, 54 items apart
Counting options by machine gives 60 of 188 items (32%) with two or more options whose content words sit 80% inside one sentence, 55 with exactly one and 73 with none. That count and the model's defence list agree on 46 items and disagree on 54: 28 items the model defends and the count clears, 26 the count flags and the model clears. The union is 88 items, 47% of the paper, which is too much to hand a teacher for a ten-item quiz. The intersection is 46, a quarter of the paper, and that is the list worth reading.
What I read by hand
Twelve items drawn at random, six English and six Chinese, seed 1120, marked on screen without the model answers in front of me: twelve keys correct, none with two defensible options. The ids: papersizing g8 item 7, battery g8 item 1, sourdough g8 items 7 and 2, ballast g8 item 5, firstcrack g8 item 3, shuilunji junior item 4, lidian junior item 4, daocha junior items 0 and 1, tankong junior item 1, yanjian junior item 3. Twelve is not a sample that clears the set; it is enough to say the automated passes were not missing obvious errors.
The check to copy over
Export the items as JSON, one object per item carrying id, options, answer and the passage the item was built from, and run this.
import json, re, sys
STOP = set("a an the of in on at to for from by with and or but if then that this these is are was were be been "
"it its as which who what when where why how not no more most such into over under between".split())
def words(s):
return [w for w in re.findall(r"[a-z][a-z'-]+", s.lower()) if w not in STOP and len(w) > 2]
def flat(s):
return re.sub(r"[^a-z0-9 ]+", " ", s.lower())
def support(option, passage):
cw = words(option)
if not cw:
return 0.0
best = 0.0
for s in re.split(r"(?<=[.!?])\s+", passage):
f = flat(s)
best = max(best, sum(1 for w in cw if w in f) / len(cw))
return best
for it in json.load(open(sys.argv[1])):
n = sum(1 for o in it["options"] if support(o, it["passage"]) >= 0.8)
if n >= 2:
print(it["id"], n)
On this set it prints 60 ids. Above a fifth of the paper, the wording is what to review, not the answer. Do not spend the time on a re-answer pass: on these 188 items it found nothing, and on items like these it cannot find anything.
What this does not measure
Two models from one vendor, temperatures 0 and 0.8, 28 expository passages: no poems, no lab reports, no maths. Nobody sat these items, so nothing here is an observed classroom error rate. The 80% threshold is a choice and a lower one raises the flagged count along with the false positives. The rewrites were written by the same model family that wrote the items, so they may preserve the very cue that makes the key obvious; a human rewriting the stem might break more than zero. And the defence pass was shown the key, which is right for a marking question and wrong for an answerability question.
What is safe to act on: on 188 items, re-answering, either with a second model or after rewording, caught no wrong key, and an item whose correct option is a lifted sentence cannot fail that test by construction. Review time goes further on the 46 items where the model and the count both point at the same two options. Two numbers to carry: 188 of 188 blind answers matching the key, and 74 of 188 items with a wrong option a student can quote the passage for. The free tools these prompts go into are in nine free test makers; what the grade field actually does is in the grade dial audit; how much of a passage ten items cover is in the coverage count; and the checks worth running before a paper goes out are in the six-check gate.
Re-answering cannot see this one either. Two arms, 60 items each, blind answer agreed 60 of 60 in both, including the 18 items whose calculated answer mixes millimetres with days: the table run.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.