Quiz Leak Check: 15,186 Exam Items Say Word Overlap Fails

The standard check on a generated quiz, counting words the key shares with the stem, fires on 22% of 15,186 human-written items and is right 25.8% of the time (z = 1.0). The audit that works uses the three distractors of each item as its control group: on real exams the key is only 1.8 characters longer than its own distractors, and every other feature sits at zero.

📜 Table of Contents

• What I measured
• The cue fires on 22% of items and is worth 0.8 points
• The trap I walked into
• Every generated item already contains its own control group
• How much gap is noise
• What to change when a set fails
• The script
• What this cannot tell you

A common way to audit a generated quiz is to count the words each option shares with the stem, and flag the items where the correct option echoes the question. It is cheap, it needs no reference bank, and it sounds like it should work.

I ran it on 15,186 multiple-choice items written by humans. It scores 25.8%. Four-option guessing scores 25%. On ARC-Challenge, whose items are taken from real school exams, the same rule scores 10.3%, below chance.

What I measured

The reference banks are the MMLU test split, 14,042 four-option items across 57 subjects, and the ARC-Challenge test split, 1,172 items from US science exams. ARC has 1,150 items with letter keys, and 6 of those carry three or five options, which leaves 1,144 comparable four-option items. Total 15,186. Everything ran on 9 November 2026 against the published parquet files.

The matching rule decides the answer, so it has to be stated. An option's words are its tokens of three letters or more, after a 130-word stoplist is removed (articles, question words, "following", "statement", "correct") and plurals are folded to the singular. An item is judged only when exactly one option shares the most content words with the stem, and ties are scored by expected value, 1/k for k tied options. The answer letters are not skewed to start with: MMLU keys land at a 22.9%, b 24.7%, c 25.5%, d 26.9%.

The cue fires on 22% of items and is worth 0.8 points

On the combined 15,186 items the shared-word rule produces a verdict on 3,348 of them, 22.0%. It is right on 863. That is 25.8%, and z = 1.0, which is the statistical way of saying nothing happened. Split by bank: MMLU fires on 3,145 items (22.4%) and is right 26.8% of the time; ARC fires on 203 items (17.7%) and is right 10.3% of the time.

The reason the rule can only speak about a fifth of the corpus is visible in two other counts. On 51.1% of items no option shares a single content word with the stem, so the cue is silent. And on 78.0% of items at least two options tie at the top, so even when there is a match, there is usually no winner. The correct option is the unique most-overlapping one on 6.0% of MMLU items.

That is the whole trick. Exam writers do not repeat the stem's vocabulary in the key, and they do not avoid it either. Over 14,042 items the average key echoes 18.8% of its own content words from the stem, and its three distractors echo 18.7%. The key is, on this feature, a random draw from its own options.

The trap I walked into

With 78% ties, the tie rule is not a detail, it is the result. Score a tie as wrong and the cue looks dead at 6%. Score it as right and it looks like a 78% oracle. Expected value is the only honest option, and at 25.8% it lands on the coin flip. Anyone quoting a number from this check without saying how ties were handled is quoting their tie rule.

A second trap showed up in the sensitivity test. To see the script catch a real leak, I broke 500 MMLU items on purpose by appending twelve words to the key ("which is why the statement above holds in every case"). The characters gap jumped to +56.1 and the script flagged it, as it should. The stem-echo gap went down, to -0.09. Echo is a ratio over the option's own words, so any added words dilute it. A single feature with a known direction can be gamed by wording; this is why the audit below uses six at once.

Every generated item already contains its own control group

You do not need a reference bank to audit a generated export. Each item ships with three wrong options written by the same generator, for the same stem, in the same style. If the key is indistinguishable from those three, the item gives nothing away. If one feature separates the key from its own distractors, a student who notices the pattern has a strategy.

Six features, measured on the key minus the mean of its three distractors, on both human banks:

Read the character row first. In human-written exams the key is 1.8 characters longer than its own distractors. Not thirty. The widely quoted "longest option" effect is real but tiny, and the numbers here match the earlier audit: the key is the unique longest option on 23.7% of all items, which is 28.6% of the 82.9% of items that have a unique longest option. Same quantity, different denominator.

The instruction "make the distractors plausible" is what gets a generated item to this line. Two distractors, plus a key, plus a stem, and the only difference between right and wrong should be the content, not the packaging.

How much gap is noise

A 20-item quiz can only support a coarse verdict, and the arithmetic says how coarse. The per-item spread of the character gap is 24 characters, so with 20 items the standard error is 5.4 characters and the three-sigma line sits about 16 characters above the human mean. In practice: with 20 items, a key that is 16 characters longer than its distractors is a real signal, and 8 characters is not. Set the same lines with the other features and a 20-item set can only be failed on characters (+16), content words (+1.5), stem echo (+0.10) or abstract-noun rate (+26 points). At 50 items those lines tighten to +3.4 characters and +0.06 echo.

This is why per-item flagging does not work on generated quizzes. The per-item noise is wider than the effect: the echo gap runs from -0.13 at the tenth percentile to +0.12 at the ninetieth on clean human items. Audit the whole set, then look at the individual items that pushed the mean.

What to change when a set fails

Regenerate the distractors and keep the key, then re-run. Four constraints did most of the work on the sets I fixed:

• Same length band. Ask for each distractor within three words of the key. This kills the +50 character gap in one pass.
• Same category. If the key is a treaty, the distractors are other treaties; if the key is a mechanism, they are other mechanisms. Category-mismatched options are the ones a student eliminates without reading.
• Same source, other paragraphs. Pull distractor material from the parts of the document the question did not come from. This also raises the chance a distractor is defensibly wrong rather than obviously wrong.
• One line of reasoning per distractor. "Wrong because it swaps the 1919 date for 1921." If the generator cannot say why an option is wrong, that option is filler.

Then re-run the script on the export, not on the tool's preview. Previews show the first three items, which are the ones the generator spent the most effort on.

The script

Stdlib only, Python 3.8 or newer. Export the questions to CSV with the header stem,a,b,c,d,answer, where answer is a/b/c/d, then run python3 leak_gap.py bank.csv. On the 14,042-item MMLU export it takes two seconds:

import csv, math, re, statistics as st, sys
STOP = set("""the a an of to in and or is are was were which what who whom whose when where why how
that this these those it its be been being has have had do does did can could will would should may
might must not no all any both each few more most other some such than then there their them they you
your we our us i he she his her him on at by for with about into over after before between during from
up down out off again further once here only own same so too very just also as if because while
according following above below statement statements question answer answers correct true false one two
three first second best likely main primary except least""".split())
WORD = re.compile(r"[A-Za-z]+"); CAP = re.compile(r"\b[A-Z][a-zA-Z]{2,}\b")
ABS = re.compile(r"(tion|sion|ment|ness|ity|ism|ance|ence|ology|acy)$")
NAMES = ["chars", "words", "echo", "abstract", "digit", "proper"]
HUMAN = {"chars": 1.8, "words": 0.1, "echo": 0.0, "abstract": 0.0, "digit": 0.0, "proper": 0.0}
SD = {"chars": 24.0, "words": 2.23, "echo": 0.15, "abstract": 0.38, "digit": 0.15, "proper": 0.32}

def toks(s):
    return [w for w in WORD.findall(s or "") if len(w) >= 3 and w.lower() not in STOP]

def stem(w):
    w = w.lower()
    if len(w) > 4 and w.endswith("ies"): return w[:-3] + "y"
    if len(w) > 3 and w.endswith("s") and not w.endswith("ss"): return w[:-1]
    return w

def sset(s): return {stem(w) for w in toks(s)}

def feats(opt, S):
    t = toks(opt); ts = {stem(w) for w in t}
    return {"chars": len(opt.strip()), "words": len(t),
            "echo": len(ts & S) / len(ts) if ts else 0.0,
            "abstract": 1.0 if any(ABS.search(w.lower()) for w in t) else 0.0,
            "digit": 1.0 if re.search(r"\d", opt) else 0.0,
            "proper": 1.0 if [w for w in CAP.findall(opt) if w.lower() not in STOP] else 0.0}

rows = []
for r in csv.DictReader(open(sys.argv[1], encoding="utf-8-sig")):
    o = [(r.get(k) or "").strip() for k in ("a", "b", "c", "d")]
    k = (r.get("answer") or "").strip().lower()
    if all(o) and k in "abcd" and len(k) == 1:
        rows.append((r.get("stem") or "", o, "abcd".index(k)))
n = len(rows)
gaps = {f: [] for f in NAMES}; maxrate = {f: 0 for f in NAMES}
for q, opts, k in rows:
    S = sset(q); F = [feats(x, S) for x in opts]
    for f in NAMES:
        gaps[f].append(F[k][f] - sum(F[i][f] for i in range(4) if i != k) / 3)
        top = max(x[f] for x in F)
        if F[k][f] == top and sum(1 for x in F if x[f] == top) == 1:
            maxrate[f] += 1
print("items audited: %d" % n)
print("%-9s %12s %10s %12s %8s" % ("feature", "key-minus-3d", "P(key=max)", "noise line", "verdict"))
for f in NAMES:
    gap = st.mean(gaps[f]); se3 = 3 * SD[f] / math.sqrt(n)
    print("%-9s %+12.2f %9.1f%% %+12.2f %8s"
          % (f, gap, 100 * maxrate[f] / n, HUMAN[f] + se3,
             "LEAK" if gap - HUMAN[f] > se3 else "ok"))

Output on the MMLU export, unedited:

items audited: 14042
feature   key-minus-3d P(key=max)   noise line  verdict
chars            +1.77      23.7%        +2.41       ok
words            +0.11      12.9%        +0.16       ok
echo             +0.00       7.5%        +0.00       ok
abstract         +0.00       4.0%        +0.01       ok
digit            +0.00       0.8%        +0.00       ok
proper           -0.01       1.4%        +0.01       ok

And on the 500 items where I appended a clause to the key, the same script:

items audited: 500
feature   key-minus-3d P(key=max)   noise line  verdict
chars           +56.14      95.6%        +5.02     LEAK
words            +3.17      84.4%        +0.40     LEAK
echo             -0.09       6.8%        +0.02       ok

The second run is the part worth keeping. A leak that a teacher would spot in one sitting was caught at 500 items by two features, and by neither of the two a blog post would have told you to check.

What this cannot tell you

Wrong answer keys. A set can pass all six features and still mark 10% of items with the wrong option, which is the failure that reaches parents. That needs its own pass: answer every item yourself from the source, or have a second model key it and compare. Ten items is enough to see whether the key is stable.

It also says nothing about ambiguity, about whether the items cover the document, or about difficulty. The six features are a sample of test-wiseness cues, not a taxonomy. Content-word matching misses synonyms and paraphrase, so an item that leaks through a shared idea rather than a shared word will pass. Both banks are English, four-option, US-style exams; a five-option or true/false export needs the features re-run with different constants.

If you are still choosing a tool, the comparison that matters is how each one keys and exports its questions, which I ran on five generators here: five AI test makers judged by what they do with the answer key. If the question is price and export limits, the free-tier comparison is here: what the nine free AI test makers actually give you. The earlier audit on option length and answer spread, which this piece re-checks and extends, is here: 28.6% of items guessable by length.

Two numbers to remember: 25.8% for the shared-word cue, which means it is not a check at all, and +1.8 characters for the human line on length. The banks are public (MMLU, ARC), so rerun the numbers instead of trusting them.