AI Test Maker: 17 of 120 Wrong Options Are Never in the Passage
The keys are fine and the distractors are not: 17 of 120 wrong options are simply absent from the passage, so a student can only eliminate them by noticing an absence rather than by finding the sentence that rules them out. Seven of the ten affected items had stems with no anchor to the passage.
📜 Table of Contents
Ask a quiz generator for a batch of questions and the answer keys usually hold up. What fails more quietly is the other three options. Some of them are not wrong in the sense of contradicting the passage. They are simply not there. A student who works only from the text has nothing to check them against, and the item turns into a test of whether you noticed an absence.
I audited 120 wrong options from 40 generated items, one call each, asking a second model to say whether the passage rules the option out, never mentions it, or actually supports it. Seventeen of the 120 were never mentioned. Ten of the 40 items carried at least one of them. Then I added a single line to the generation instruction and ran the same eight passages again: zero.
How the audit works
Eight passages of 117 to 134 words were written for this test: tidal locking, the Namib fog-basking beetle, tunnel boring machines, caffeine as a plant defence, the Fresnel lens, the honeybee waggle dance, freeze-dried instant coffee, and the Dead Sea. Each was sent once to deepseek-flash at temperature 1.0, asking for five four-option items with exactly one correct option, returned as JSON with the key as a 0-based index. All 40 items that came back had four distinct options and a valid key, so nothing was dropped.
Each wrong option then went to deepseek-v4-pro at temperature 0 with the passage, the stem, the option, and one question: is the passage incompatible with this option, silent about it, or supportive of it. The judge had to quote the sentence it relied on. Six of the 120 calls came back with empty content; three of those answered on a re-run, and three I labelled myself by reading the passage. The three manual labels are the ones noted below.
What the wrong options actually are
| relation between the passage and the wrong option | options | share |
|---|---|---|
| a sentence in the passage rules it out | 102 | 85% |
| the passage never mentions it, and never contradicts it | 17 | 14% |
| the passage supports it, but it does not answer the question | 1 | 1% |
The 85 percent is the good part, and it is better than the usual complaint suggests. These were not two-answer items. Both answerers, deepseek-flash and deepseek-v4-pro, each given the passage and asked to pick the best option and quote the sentence that settles it, matched the key on 40 of 40 items, independently. Whatever else is wrong with machine-written quizzes, the key was not the problem in this batch, and no item had a second defensible answer.
The 14 percent is the part that reaches students. Take one item: "What human applications have copied the beetle's surface?" The key is fog-harvesting nets and trials of self-filling water bottles, which the passage states. Two of the three distractors are solar panels and wind turbines, and desalination plants and irrigation pipes. The passage says nothing about either. A reader cannot eliminate them by finding the sentence that rules them out, because there is no such sentence; the only move available is "I did not see that in the text", which is a test-taking habit rather than a reading skill. Seven of the ten affected items had stems with no anchor to the passage at all, which makes the absence worse: nothing in the question tells the student that outside knowledge is out of scope.
One item went the other way. Asked why freeze-drying costs more per kilo, one distractor was "the granules are lighter in bulk". The passage does say the granules are lighter in bulk, so the option is true, and still wrong, because it does not answer why the process costs more. That is a fair item, and it is also the kind of item a generator produces by accident: the option is true because it was copied from the passage, not because anyone checked that it fails the question.
One line in the instruction
The fix is not a filter on the output. It is a constraint in the request: every wrong option must be refutable by citing a specific sentence of the passage, and options the passage simply never mentions are out of bounds. I added that line, kept the model, the temperature, the passages and the item count identical, and audited the new batch the same way.
| default instruction | one line added | |
|---|---|---|
| items | 40 | 40 |
| wrong options the passage never mentions | 17 | 0 |
| items carrying at least one such option | 10 of 40 | 0 of 40 |
| stems that say "According to the passage" | 10 | 14 |
| the key is the longest option | 24 of 40, 60% | 19 of 40, 48% |
| mean option length, characters | 37.0 | 52.7 |
| stems that also appeared in the other run | 1 of 40 | |
Seventeen to zero, on the same eight passages. Two caveats keep this honest. Only one of the 40 stems appeared in both runs, so this is not the same items cleaned up; the stricter instruction produces different questions, and the comparison is between two instruction sets, not between a batch and its repair. And the option text got longer, from 37 characters on average to 52.7, because a refutable distractor needs the specific detail that makes it refutable.
The longer options did not make the length cue worse. In the default batch the key was the longest option in 24 of 40 items, 60 percent; under the stricter instruction it was 19 of 40, 48 percent, because the distractors grew along with the keys. That is the opposite of the usual trade-off, and it is worth measuring rather than assuming, since the length cue was the single largest leak in an earlier batch of 80 items at 28.6 percent.
The failure that cost the most time
On the first attempt at the stricter batch, three of the eight passages returned no items at all. The calls succeeded, the responses were empty. The cause was the token budget: at the default reasoning setting the visible answer never arrived inside a 2,600-token cap, and an empty string parses as nothing rather than as an error, so the loss is silent. Dropping the reasoning setting to low and raising the cap to 9,000 tokens gave eight of eight passages and 40 items. The same thing happened on five of the 120 audit calls in the first batch.
Two habits follow. Log the item count per passage as the response arrives, so a silent zero is visible instead of being absorbed into a total. And never salvage a truncated array: a batch that returns four items when you asked for five is usually a truncated response, and the fourth item is fine while the fifth is missing, which is exactly the kind of gap that gets noticed after the quiz is handed out.
Config to copy
The generation instruction, with the line that mattered in the middle:
Write {n} multiple-choice questions about the passage below.
- Four options each, exactly one correct.
- Answerable from the passage alone.
- Every wrong option must be refutable by citing a specific sentence
of the passage. Do not write a wrong option that the passage simply
never mentions.
- Vary the position of the correct option.
Return JSON: {"items":[{"stem":"...","options":["..."],"answer":0}]}
The audit prompt is shorter than the generation prompt. Passage, stem, option, three labels, and a required quote: contradicted if a sentence is incompatible with the option, absent if the passage is silent, supported if the passage states it. Requiring the quote is what makes the label checkable; without it the judge will happily call an unmentioned option contradicted. The script is distractor_audit.py, standard library only, 12 workers, no dependencies: about two minutes for 120 options.
What this does not cover
Both roles were played by DeepSeek models, and the audit label is a model's judgement, not a teacher's. I read all ten flagged items myself and agreed with the label on all ten, which is a check on those ten and no more. The material is English expository prose; Chinese passages were not tested, and the absent-option rate could differ there. Forty items is a small sample, and the difference between 25 percent and zero rests on two runs of the same eight passages, so the effect size is not settled, only the direction. The stricter instruction also pushes stems toward the formulaic "According to the passage", from 10 to 14 of 40, which is a real cost in variety for a bank that will be reused.
The prescription is short. Audit the wrong options, not just the keys, because the keys were fine in both batches and the wrong options were not. If you would rather not audit, put the refutability requirement in the instruction and spot-check ten items; on this evidence that is where the difference lives.
If you are picking a tool rather than building one, the free limits and export rules of nine of them are here: what each free quiz maker actually gives you. The same eight passages also went through a difficulty-label check, where 12 of 16 items labelled hard were answered with the passage removed: the difficulty label test. The length cue measured on an earlier batch of 80 items is here: 28.6% pickable by option length. Two numbers to take away: 17 of 120 wrong options were never mentioned in the passage, and 0 of 120 were after one line was added to the instruction.
The same protocol run on Chinese passages moves the failure rather than removing it: 3 of 120 wrong options cannot be ruled out from the passage, the refutability line still takes that to zero, and the cost lands on key position instead of option quality: the Chinese batch.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.