AI Test Maker: 12 of 16 Items Labelled Hard Answered Without the Passage
The difficulty dropdown promises hard items and returns longer wording instead. Twelve of sixteen items labelled hard were answered correctly with the passage removed, against fifteen of sixteen labelled easy: a 19-point spread, not a difficulty.
📜 Table of Contents
Most quiz generators have a difficulty dropdown: easy, medium, hard. Pick hard and the items come back longer and more convoluted, which looks like the right answer. This week I measured whether the label means anything. It points in the right direction, and that is about all it does.
Two batches, two questions
The material was twelve short explanatory passages I wrote for this test, 117 to 151 words each: heat pump efficiency, the Aral Sea, Krakatoa, runway numbering, noise-cancelling headphones, the shipping container, stalactites, bridge expansion joints, tree-ring dating, espresso crema, the Newfoundland cod fishery, and fibre optics. They were used in two batches. One checked the answer keys. The other checked the difficulty labels.
Items were written by deepseek-flash at temperature 1.0, six per passage, four options, answerable from the passage alone, one correct option, key position varied. Answering and key checking were done by deepseek-v4-pro at temperature 0, with the passage in front of it.
The labels rank the items, a little
Forty-eight items, sixteen in each band, answered with the passage removed.
| label | items | answered right, passage hidden | said it could not tell | call failed |
|---|---|---|---|---|
| easy | 16 | 15 | 1 | 0 |
| medium | 16 | 14 | 2 | 0 |
| hard | 16 | 12 | 2 | 2 |
Forty-one right, five refusals, two failed calls, and not one wrong option picked. The judge either knew the item or said it could not tell; it never guessed badly. That is the shape of the result worth keeping in mind.
The labels do order the items: 94%, 88%, 75%. The problem is the distance. Twelve of the sixteen items labelled hard were answered correctly with the passage taken away, three quarters of the band. An item that genuinely requires the passage should be answerable by nobody in this setup.
What the label actually tracks
Line up the surface features of the three bands and the origin of the label becomes obvious.
| label | stem, words | key, characters | distractors, characters | key is the longest option |
|---|---|---|---|---|
| easy | 11.8 | 31 | 25 | 8 of 16 |
| medium | 12.4 | 57 | 44 | 11 of 16 |
| hard | 14.9 | 84 | 59 | 12 of 16 |
The harder the label, the longer the stem and the longer the correct option. Half of the easy band opens with "According to the passage" and asks for a sentence that is on the page; seven of the sixteen hard items open with "Which statement best captures" and three with "Why", asking for a summary or an inference. The wording changed. How much the item depends on the text barely did.
A long correct option is a cue in itself. In 31 of the 48 items, 65%, the key is the longest option; restricting to the 45 items with a single longest option, picking that option by length alone scores 30, 67%. Seven days earlier I measured the same thing on a different batch of 80 items and got 28.6%. The generation settings were not identical, so the two figures should not be compared as if they were one series, but they point the same way: the length of an option carries its own answer.
The answer keys held up
A second batch of 56 items went through a different check. The passage and the question went to the stronger model together, and it had to pick an option and copy, word for word, the sentence from the passage that settled it.
Fifty-six of 56 picks matched the key. Fifty-six of 56 quoted sentences were found verbatim in the passage, none of them invented. The 21 figures that appear in a stem or in a key all appear in the passage they came from, with nothing made up. The 56 items rest on 46 distinct supporting sentences, so the coverage is not as narrow as six items per passage might suggest.
Getting this far took some housekeeping. Four of the twelve passages never produced parseable JSON in the label run, and two of twelve in the key run; six items at once is a long output and truncation is the usual cause, so the fix is a larger token budget and a full retry rather than salvaging half an array. Three more items were dropped because two of their four options were identical. Those losses are why the counts are 48 and 56 rather than the 72 that were asked for.
What to do with it
Treat difficulty as something to calibrate, not something to request. If you want the label, measure it first: run the same items with the passage hidden, compute the hit rate per band, and see the gap before you put marks on it. On this evidence a "hard" band that sits at 75% is a label, not a difficulty.
Three things that moved the numbers more than the label did. Ask for the shape of a hard item instead of the word: the answer spans two paragraphs, or two figures have to be combined, or the stem uses vocabulary the passage does not. Give the generator room to finish; six items plus four options overruns a small token budget and a truncated array cannot be repaired. And flatten option length, keeping keys and distractors within a few words of each other, because a 84-character key against 59-character distractors is a gift to anyone who reads only the options.
The script is difficulty_calibrate.py, standard library only. It reads a CSV with the header stem,a,b,c,d,answer,difficulty, sends each item to a model with no passage, and prints the hit rate per band. --model sets the judge, --workers the concurrency, --report prints the per-item table.
What this does not show
Both roles were played by DeepSeek models, and another family would move the numbers. The passages are English expository text; Chinese material was not tested. The judge is a model, not a student, so whether a class would also answer those twelve hard items without reading is an open question, and probably a different answer. Forty-eight and 56 items are small samples, and a 19-point gap between the outer bands could narrow on another set of topics. What survives is narrower: the difficulty label is not a quantity, and in this batch the keys were not the part that went wrong.
If you are choosing between tools, the free limits and export rules of nine of them are here: what each free quiz maker actually gives you. The same batch answered with the text removed, and what a filter keeps, is the previous test: the closed-book run. The length cue measured on an earlier batch is here: 28.6% pickable by option length. Two numbers to take away: 12 of 16 items labelled hard were answered without the passage, and 56 of 56 answer keys in the other batch agreed with a stronger model.
The label turned out to be a label. The same eight passages also went through a distractor audit, and the wrong options were the weaker half: 17 of 120 were never mentioned in the passage, and one line in the instruction took that to zero: the distractor audit.
The labels were the first hint that these knobs do less than they promise. The grade setting behaves the same way: 2.11 grades delivered for 8 asked, with 82% of grade-12 keys still sitting in one sentence: the grade dial, measured.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.