Broken Quiz Explanations: 78 of 80 Caught by a Judge
A generated explanation can quote the passage perfectly and still not explain the answer: a quote swapped for a neighbouring sentence passes any string check and is rejected only by reading. Cheap deterministic filters catch one failure mode out of four.
📜 Table of Contents

Every AI test maker ships an explanation with each item, and teachers read it as a second safety net: if the key were wrong, the reasoning would look off. That net only holds if somebody checks the explanation, and nobody does. So I wrote eighty broken explanations on purpose, mixed them with one hundred and forty real ones, and gave the whole pile to a judge model.
The judge caught 78 of the 80. The check most people would build first, is the quoted sentence copied verbatim from the passage, caught 44 of 80, and passed every one of the 20 built by quoting a real but irrelevant sentence. A string match answers a narrower question than the one that matters.
How it was run
Twelve passages of 118 to 133 words, written for this test: sourdough microbiology, Antarctic ice cores, damping in cable-stayed bridges, paper sizing chemistry, honey crystallisation, railway ballast tamping, espresso puck resistance, satellite drag in the thermosphere, leading-edge erosion on turbine blades, first crack in coffee roasting, lithography overlay, lightning protection. Four to six sentences each.
Each passage went to deepseek-flash at temperature 1.0 twice. The plain prompt asked for five items with four options, one key, and a one or two sentence explanation of why the key is right. The second prompt added one line: the explanation must start with the decisive sentence copied verbatim from the passage inside double quotation marks, then one sentence of its own, and if no sentence settles the item then do not write that item.
Twenty-four calls, 120 items, none dropped and no malformed JSON. The escape hatch in the second prompt, return fewer items rather than one you cannot quote for, went unused: every item came back with a quotable sentence.
What the quote requirement bought
| plain prompt | quote required | |
|---|---|---|
| items returned | 60 | 60 |
| explanations that quote the passage | 0 | 60 |
| quotes found verbatim in the passage | 0 | 60 |
| mean explanation length, characters | 120.1 | 250.8 |
| judge: explanation settles the key | 60 | 60 |
| judge: asserts a fact outside the passage | 0 | 0 |
| explanation leans to a wrong option | 1 | 0 |
Under the plain prompt, 0 of 60 explanations quote anything. They are paraphrases of the key, and they are short, 120 characters on average. Under the quote requirement every explanation opens with the passage's own words, every quote is found verbatim in the passage, and the length doubles. That is the whole cost: the explanation gets 130 characters longer.
What it does not do is make the explanation right. All 60 quotes were genuine, and a genuine quote can still be the wrong sentence. The judge passed both batches at 60 of 60, which on its own would suggest the explanations were fine either way. That is the reason for the next part: a judge that passes everything is worthless as a gate, so I gave it things that were definitely broken.
Eighty broken explanations
From the twenty-party batch I took 20 items and made four corrupted versions of each:
C1, wrong direction. Rewrite the explanation so it argues, confidently and in the same style, that a neighbouring option is correct. This is the failure a teacher fears most, an explanation that teaches the wrong answer.
C2, invented fact. Keep the explanation and add one specific, confident detail the passage does not contain, a number, a name or a mechanism.
C3, generic filler. Replace the explanation with a sentence that says the item tests comprehension and the correct option matches the passage, which is true of every item and therefore explains nothing.
C4, quote drift. Keep the structure and the second sentence, but replace the quoted sentence with a different sentence from the same passage, picking the one whose words overlap the key least. The quote stays verbatim, the claim about the key stays, and the two no longer connect.
All 80 went to deepseek-v4-pro at temperature 0 with the passage, the item, the key, and one question set: reading only the explanation, does it show why the keyed option is correct according to the passage, and does it assert a fact the passage does not contain.
| batch | items | judge says it settles the key | judge flags outside fact | passes the verbatim-quote check |
|---|---|---|---|---|
| real explanations (control) | 20 | 20 | 0 | 20 |
| C1 argues for a wrong option | 20 | 0 | 20 | 6 |
| C2 one invented fact added | 20 | 20 | 20 | 10 |
| C3 generic filler | 20 | 0 | 0 | 0 |
| C4 quote swapped for another sentence | 20 | 2 | 8 | 20 |
Read down the third column. Real explanations pass 20 of 20 and the judge never invents an outside-fact complaint against them, so the gate is not a rubber stamp. C1 dies completely, 20 of 20 rejected, and worth noting, the judge also flagged all 20 as asserting outside facts: an explanation for a wrong option has to import something the passage does not say, and the judge noticed. C2 is the interesting row. It passes the first question, because the explanation does argue for the correct key, and it is caught by the second, 20 of 20 flagged as asserting a fact the passage lacks. C3 dies at 0 of 20, and that cell is not much of a test: the filler was written in Chinese against English items, so a judge could catch it on language alone. C4 is the row that matters.
The string check is the wrong question
Column five is the verbatim check, the one-liner that asks whether the quoted sentence exists in the passage. Against C4 it passes 20 of 20, because by construction the quote is a real sentence. Against C2 it passes 10 of 20, because the invented fact was appended after a real quote. Across the 80 corrupted explanations it catches 44: 14 of the wrong-direction batch, 10 of the invented-fact batch, all 20 of the filler batch, and none of the 20 drift items. It is a filter for the explanations that do not quote at all, and it is blind to the ones that quote the wrong sentence.
I tried a second cheap check, counting content words in the explanation that never appear in the passage, and flagging anything with three or more. It flagged 18 of the 20 real explanations. A paraphrase cannot avoid words the passage does not use, so the check measures paraphrase, not error. Nine in ten false alarms is not a gate.
The pattern is worth stating plainly. Deterministic checks work on the surface of the explanation, whether the words are present, and the defects live in the relation between the explanation and the item. Only a reader can judge that relation, and the reader can be a model as long as you ask it the narrow question and give it the key.
Two examples
From C4, caught. Item: what do the yeasts in a sourdough starter produce? Key: carbon dioxide. Explanation: “A sourdough starter is a culture of wild yeasts and lactic acid bacteria living in a flour and water paste.” This sentence directly identifies carbon dioxide as the yeast product. Every word of the quote is in the passage. The quoted sentence says nothing about carbon dioxide. A student who checks the quote against the passage finds it, trusts it, and learns nothing about why the answer is right. The judge returned settles: no.
From C2, caught by the second question. The real explanation ended with the definition of the starter; the corrupted version added, with the bacteria commonly including Lactobacillus sanfranciscensis. The species is real and it is not in the passage. The judge passed the first question and flagged the second. That split is the useful part: an explanation can be directionally right and still teach a fact the material never gave.
From C1, caught. Option 1 is correct: the passage describes the starter as a yeast mixture that is kept in the refrigerator, noting that “a starter held in the fridge for a week still ferments, just slowly.” The quote is genuine. The inference is backwards: the passage says refrigeration slows fermentation, not that refrigeration defines a starter.
The two prompts to copy
Generation, with the quote requirement:
Write {n} multiple-choice items from the passage below. Each item needs four options, exactly one correct, and the correct answer must be checkable against the passage.
Add an "explanation" field that starts with the decisive sentence copied verbatim from the passage inside double quotation marks, followed by one sentence of your own telling a student why that sentence settles the item.
If you cannot find a sentence that settles an item, do not write that item; return fewer items rather than one you cannot quote for.
Return JSON only: {"items":[{"stem":"...","options":["...","...","...","..."],"answer":0,"explanation":"..."}]}
Passage:
{passage}
Judge, one call per item:
Below is a passage, one multiple-choice item, its keyed answer, and the explanation that will be shown to students.
Answer two questions and nothing else.
1. Reading only the explanation, does it show why the keyed option is correct according to the passage?
2. Does the explanation assert a fact that the passage does not contain?
Return JSON only: {"settles":"yes","outside":"no"}
Passage:
{passage}
Item: {stem}
Options:
0. {o0}
1. {o1}
2. {o2}
3. {o3}
Keyed answer: {key}
Explanation: {explanation}
Reject on either answer. Keep the verbatim check ahead of it as a free filter, since it costs nothing and removes the batch with no quote at all. Drop the word-list check. Do not skip the key from the judge's prompt to make it harder: with the key removed, the judge starts grading whether the option is plausible rather than whether the explanation supports it, which is a different question.
One operational detail that cost three runs. The judge here is a reasoning model, and its thinking is billed against the same completion budget. With max_tokens at 1500 the easy rows returned in about 160 reasoning tokens and the hard ones spent the entire 1500 thinking and returned an empty string, which looks exactly like an API outage. Raising the cap to 6000 fixed it. Twenty judge calls at four-way concurrency take 15.2 seconds, so the whole gate on a thirty-item quiz is a handful of seconds and thirty calls.
What this does not measure
Twelve English passages of 118 to 133 words, five items each, one generator at temperature 1.0, one judge at temperature 0, both DeepSeek, so a shared-vendor preference cannot be ruled out. The corruptions were machine-made and blunt: a subtly wrong explanation, one that reverses a mechanism while sounding right, is the case teachers actually meet and I did not test it. The C4 swap used the sentence with the least word overlap with the key, which is the easiest drift to spot; a drift that half-supports the key would score worse. C3 is confounded by language. Twenty per cell is small, and every rate here should be read as this batch, not a benchmark.
What is safe to act on: a plain prompt gives you explanations with no citation at all, a quote requirement gives you citations for free and doubles the length, and a two-question judge rejected 78 of 80 broken explanations while passing 140 of 140 real ones. The gates that did not work are the cheap ones, and they are the ones most likely to be built first.
The item-level checks that run before any of this, the ones that read an export the way an exam reads, are in six checks against a 15,214-item human bank. Why ten items from one passage test five sentences, and what that does to a score, is in the coverage count. What the free test makers actually give you and where their limits sit is in nine free test maker tools. Take two numbers with you: of 80 deliberately broken explanations, 78 were rejected, and the verbatim-quote check alone rejected 44, none of them from the quote-drift batch.
The gate above reads the text under each item. The options leak earlier than that: 128 items, 34 with an absolute-word option, one of them keyed, and a blind student scoring 50.1% on two rules: the cue in the option itself.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.