AI Test Maker: Hide the Passage, 71 of 80 Still Answered

A quiz written from a passage can be answered without the passage. On 80 generated items, 71 came back correct with the text removed and only 3 were refused; the nine that survived were all detail or number items.

📜 Table of Contents

• How the test was run
• The numbers
• Which items leak
• What to do with it
• The filter script
• What this does not establish

A quiz written from a passage has one job. Someone who never read the passage should not be able to pass it. That is worth measuring on AI-generated items, and the number that comes back is not comfortable.

Eighty items, ten passages, one scorer. With the passage in front of it, the scorer answered all eighty correctly. With the passage removed, it still answered seventy-one.

How the test was run

Ten passages of 111 to 134 words, written for this test, one per topic: GPS positioning, the stages of sleep, coral bleaching, a sourdough starter, tea oxidation, the patent history of the zipper, Inca quipu, Roman concrete, the North Sea flood of 1953, and the adoption of the UPC barcode. Each passage was sent once to deepseek-chat at temperature 1.0, asking for eight four-option questions that can be answered from the passage and have exactly one correct answer, returned as JSON with the key as an index from 0 to 3.

That produced 80 items, every one of them with four distinct options and a valid key, so nothing was quietly dropped from the counts. Each item was then sent back on its own: the stem and the four options, no passage, temperature 0, one call per item so that no item could use another as context. The scorer was allowed to answer N for "cannot tell". A control run put the passage back in front of the same model at temperature 0.

The numbers

The control is what makes the other rows readable. Eighty of eighty with the passage attached means the items are legible, correctly keyed, and unambiguous enough for the model that wrote them. The nine that failed the closed-book pass are not broken items; they are items whose answer the model did not have in memory.

Running the identical closed-book pass a second time returned the same 71 and the same nine items. At temperature 0 the check is reproducible, so an item does not drift in and out of the filter between runs.

Which items leak

Split the 80 by the shape of the correct option and the leak stops looking uniform.

The nine that survived, in full. A figure-eight knot on a quipu meant one, not ten thousand. A single high knot could mean ten thousand. The 1953 surge ran more than five metres above mean sea level in places. Dutch defences failed at more than 150 locations. Tea oxidation runs one to three hours. The grocery industry adopted the UPC in 1974, a year after the standard was settled. The 1851 clasp locker was Elias Howe's patent, not Sundback's. Adenosine falls during sleep rather than rising. And brightness against body in tea comes from temperature and humidity.

Look at what is missing from that list: the questions about mechanism almost all passed straight through. How a receiver turns a signal delay into a range. Why a fourth satellite is needed for the clock error. What pozzolana does inside Roman concrete. Why the Delta Works were built. Anyone who knows the subject answers those with the passage closed, which makes the passage decoration.

The items that survived ask for a fact that exists only in the text: a knot value, a count of breaches, a date one year off from the obvious one. This is also why choosing an obscure topic is no protection. Quipu and the 1953 flood are not common knowledge, and each still lost only two items of eight. A generator reaches for the most public facts in any passage, and those are exactly the facts a solver already holds.

There is a second cue in the same 80 items: 41 of them, 51%, put the key on the longest option. That is the same weakness measured another way six days earlier, when 28.6% of generated items were pickable by option length alone. Length is a cue a student can learn. World knowledge is not a cue at all; it is the answer.

What to do with it

The filter costs one call per item and it is the cheapest check in the pipeline: drop everything the closed-book pass answered correctly and keep what it missed or refused. Nine of 80 survived, 11%. A 60-item quiz that genuinely requires the passage therefore costs about 530 generated items, which is 66 passages at eight items each. That is the honest price of a reading check. Not paying it buys a quiz that measures what the class already knew.

Two notes on using it. Filter before editing, not after, because there is no point polishing 80 items down to nine. And ask for detail in the generation prompt, numbers and names and quantities as stated in the passage, since that is the shape that survives.

The filter script

closedbook_filter.py, stdlib only, Python 3.8 or newer. It reads a CSV with the header stem,a,b,c,d,answer, shows the model each item without its passage, and writes out the ones it missed. --keep miss also drops refusals, --drop-errors discards items whose check call failed instead of keeping them, and --report prints the per-item table.

Run on the sixteen items from two of the passages in this test, with the per-item report:

checked 16 items in 4.0s (deepseek-chat, 4 workers)
answered without the passage: 12 (75.0%)   refused: 2   wrong: 2   errors: 0
kept 4 of 16 (25.0%) -> kept.csv
  cut  guessed B key B | What material were quipu cords made from according to the passage?
  KEEP guessed N key B | According to the passage, what did a figure-eight knot represent?
  KEEP guessed A key C | What did a single knot high on a cord potentially signify?
  cut  guessed B key B | Who were the accountants described by Spanish chroniclers as using quipu?
  KEEP guessed B key C | By how much did water levels rise above mean sea level in places?
  KEEP guessed N key B | At how many locations did Dutch defences fail?
  cut  guessed C key C | When was the last storm surge barrier of the Delta Works completed?

Four of sixteen, and the four are the ones asking for a knot value, a surge height, a count of breaches, a date. The other twelve came back from memory, including the one asking which material the cords were made from.

What this does not establish

One model family wrote the items and scored them. A different family may leak differently, so 89% is not a universal constant; the direction is what carries, that detail items survive a closed-book pass and concept items do not. The passages are English and expository. Whether a well-read student would also answer 71 of 80 is untested, and a student is not a model, so classroom leakage is probably lower. N is a self-report, which makes the three refusals a floor. And none of this says a surviving item is fair, correctly keyed, or at the right difficulty, only that its answer was not in the model's memory.

If the question is which generator to use in the first place, the free tiers and export limits across nine of them are compared here: what the nine free AI test makers actually give you. For the other failure that shows up when a passage is reused, generating the same quiz twice returned 35 of 64 items: the duplicate-rate check. Two numbers to carry: 71 of 80 items answered with the passage hidden, and 9 of 80 left once the filter has run.