Regenerate an AI Quiz and 35 of 64 Items Come Back the Same

Generating the same quiz twice returns 35 of 64 items a second time at Jaccard 0.5, 15 of them identical, and the repeats keep the same answer letter 20 times out of 35. Deduplicate across exports before building a bank; a passage pair yields about 11.6 distinct items.

📜 Table of Contents

• What was run
• How much repeats
• Where the repeats come from
• What a bank of 60 costs
• The check
• What this does not cover

A quiz is generated for the morning section, then generated again before the afternoon section from the same passage. The second set opens in a new tab and looks different. More than half of it is the first set with the sentences turned around.

The number below comes from 128 items: eight passages, eight questions each, two independent runs, 11 November 2026. It answers one question. Ask the same generator for the same quiz twice, and how much of it comes back?

What was run

Eight passages of 110 to 130 words, all written for this test: coral bleaching, the patent history of the zipper, the Indian monsoon, a sourdough starter, GPS positioning, Inca quipu, sleep stages, and tea oxidation. Each passage was sent twice, in two separate calls, asking for eight four-option questions with exactly one correct answer, returned as JSON with the key as an index from 0 to 3. Temperature 1.0, model deepseek-chat, and the responses came back stamped deepseek-flash.

An item counted only when four distinct option texts came back and the key was a valid index. All 128 passed that filter, so the counts below have nothing quietly dropped. The two calls share no context: the second one is not a cache hit, it is a fresh generation from the same passage and the same prompt.

How much repeats

Thirty-five of the 64 items in the second run match an item from the first run at a token Jaccard of 0.5 or higher. Fifteen of them are the same question word for word. Raise the bar to 0.7 and 22 remain, so the repeats are mostly close paraphrases rather than distant cousins.

The threshold is a choice, so here is what sits on it. Between 0.5 and 0.6 there were four pairs, and all four ask the same thing: "What drives the Indian monsoon" against "What is the Indian monsoon described as", "How often does a counter-top starter need feeding" against "How often should a starter on the counter be fed". Below 0.5 the pairs start asking about different facts. A lower threshold would inflate the number, and it would be measuring vocabulary rather than repetition.

Where the repeats come from

The zipper passage repeated six of its eight items, leaving ten distinct questions out of sixteen. That passage is dates and proper names: 1851, Howe, 1913, Sundback, the Navy in 1918, the 1923 advertisement. A passage holds a finite list of testable facts, and a generator asked twice walks the same list in a slightly different order. Coral bleaching and tea oxidation repeated three each, leaving thirteen.

Among the 35 repeats, 29 kept the exact text of the correct answer and 20 kept the same letter. No case had a first-run distractor turn into the second-run key. So a repeated item is safe to reuse; it just costs you the item you thought you were buying. The shared letter is the part that travels: the position audit run the day before found 5 of 96 generated keys in D, and a repeat carries its letter along with everything else.

What a bank of 60 costs

Across the eight passages, one run pair produced 11.6 distinct items on average, with a range of 10 to 13. Sixty questions therefore take five to six passages run twice, about 90 generated items, rather than eight passages run once. Running twice is worth the calls only if the repeats are removed; without that step you pay for 64 items and keep 29.

The check

Stdlib only, Python 3.8 or newer. Both exports are CSV with the header stem,a,b,c,d,answer and a lower-case letter in answer. Without flags it only reports: each pair above the threshold, then the count. With --merge it writes a combined bank that keeps the first run in full and adds only the non-repeating items from the second. --th changes the threshold; 0.5 suits short stems, 0.45 is reasonable for long ones.

Run on the two exports from this test, output trimmed to one pair and the totals:

A=64 items  B=64 items  threshold=0.50
DUP 1.00
  A: In what year did Gideon Sundback redesign the teeth?
  B: In what year did Gideon Sundback redesign the teeth?
B items repeating A: 35 of 64 (55%)
merged bank: 93 items

What this does not cover

One model family, one day of generation, English passages, all eight of them expository. Narrative or argumentative text may offer fewer testable facts and repeat more; longer passages offer more facts and repeat less. The comparison reads stems, so the same fact asked from another angle, "In what year" against "how many years after Howe's patent", counts as two items. Repetition at the level of facts is therefore at least as high as the number above, never lower. Whether the repeated items are correct, ambiguous, or the right difficulty is a separate audit: 28.6% of items guessable by option length, and the answer-leak check on 15,186 exam items.

If the question is which generator to use in the first place, the free tiers and export limits across nine of them are compared here: what the nine free AI test makers actually give you. Two numbers to carry into the next export: 35 of 64 items came back twice, and one passage pair yields 11.6 usable questions.

Duplicates are one way a generated bank goes wrong. Another is that the item never needed the passage: with the text hidden, 71 of 80 items were still answered correctly, and the nine that survived were all detail questions: the closed-book test.