AI Test Maker QA: Six Checks

🔧 AI Tools 2026-11-17 7 min read

Generated sets look fine item by item and fail as a group: keys pile onto one letter, the right answer is written longer than the wrong ones, and two items test one sentence.

📜 Table of Contents

An AI test maker returns twenty usable-looking items in eight seconds. Nobody has time to read all twenty twice, so the set goes to class unchallenged, and the failure shows up as a bad grade instead of a bad item. The fix is a gate that runs in one second and answers a narrow question: does this export look like an exam, or does it look like a language model writing four options?

I measured what a real exam looks like on nine simple counters, using 15,214 human-written items: the MMLU test split, 14,042 four-option items across 57 subjects, and the ARC-Challenge test split, 1,172 science items taken from US school exams, of which 1,165 have four options. Everything below ran on 17 November 2026 against the published parquet files. Then I ran the same counters as a script on a 300-item slice, and again on a deliberately broken copy of that slice.

What a human bank looks like on nine counters

checkMMLU 14,042ARC-Challenge 1,172flag the set if
answer letters spreadA 22.9 / B 24.7 / C 25.5 / D 26.922.6 / 26.5 / 26.5 / 24.4one letter over 40 per cent, spread over 15 points
longest option is the key3,332 of 11,682, 28.5%256 of 947, 27.0%over 40%
key inside the two longest53.8%57.0%over 80%
mean key length vs distractors43.2 vs 41.5 chars, +1.830.2 vs 28.8, +1.4gap over 15 chars
stems using not or except2,219, 15.8%79, 6.7%over 30%
duplicate stems after cleanup184, 1.3%2, 0.2%over 5%
stems sharing an 8-word opening2,002, 14.3%6, 0.5%over 30%
a distractor repeats most of the key words2,443, 20.2%143, 12.8%over 40%
options per item4 for every item4 on 1,165, 3 on 4, 5 on 3mixed counts in one export

Four of those numbers are worth reading carefully, because they explain why the checks work at all.

Answer letters are almost flat. MMLU puts 22.9 per cent of keys on A and 26.9 per cent on D, a 4.0 point spread across 14,042 items. ARC spans 3.9 points. Exam writers rotate the key on purpose. A generated set that puts 60 per cent of keys on B is not harder, it is easier to guess: a student who notices scores 60 per cent without reading anything.

The key is slightly longer than its distractors, and not much. MMLU keys average 43.2 characters against 41.5 for the other three options. The gap is 1.8 characters. That is why picking the longest option scores only 28.5 per cent, barely above the 25 per cent a coin flip gives, which matches the earlier length audit on this site. A model that writes a full clause for the right answer and two labels for the wrong ones produces a gap in the tens of characters and a hit rate near 90 per cent. The counter sees it in one line of output.

Negation is rarer than teachers think. 15.8 per cent of MMLU stems carry a not, except or false; ARC is at 6.7 per cent. These are the items most likely to be argued about in class, because a double negative survives a spellcheck. If a generated set runs above 30 per cent, it is not being rigorous, it is reusing one construction.

Real banks repeat their openings more than their text. Only 184 MMLU items are exact duplicates after cleanup, 1.3 per cent. But 2,002 items, 14.3 per cent, share their first eight words with another item: which-of-the-following phrasing is a template, not a duplicate. So the duplicate counter has to be strict about what it calls a duplicate, or it flags every history paper ever written.

One more number that matters for anyone exporting to a scanner: three ARC items have five options and four have three. Seven items out of 1,172, and those seven are exactly the kind of thing a bubble sheet cannot score. Option-count consistency is a boring check that catches real breakage, because AI exports mix three, four and five option items in one file more often than they should.

The same checks as a script you can run on your own export

The script below is the short version: eight counters, standard library only, takes a JSON export and prints percentages. Feed it the file your test maker gives you, or the file you convert from its CSV.

#!/usr/bin/env python3
# quizcheck.py  ->  python3 quizcheck.py items.json
# items.json: [{"stem": "...", "options": ["a","b","c","d"], "key": 2}, ...]
# key 可以是 0 起的下标,也可以是字母 A-D
import json, re, sys, collections

STOP = set("the a an of to in for on with and or is are was were be been by as at from it its which that this "
           "following statements statement question answer correct true false best most likely according".split())
NEG = re.compile(r"(not|except|incorrect|false|least)", re.I)

def words(s):
    out = []
    for w in re.findall(r"[A-Za-z][A-Za-z\-]+", s.lower()):
        if len(w) > 2 and w not in STOP:
            out.append(w[:-1] if len(w) > 4 and w.endswith("s") and not w.endswith("ss") else w)
    return out

def key_index(item, n):
    k = item.get("key")
    if isinstance(k, int):
        return k if 0 <= k < n else None
    k = str(k).strip().upper()
    return "ABCDE".index(k) if len(k) == 1 and k in "ABCDE"[:n] else None

items = [i for i in json.load(open(sys.argv[1])) if len(i["options"]) == 4]
keyed = [(i, k) for i, k in [(i, key_index(i, 4)) for i in items] if k is not None]

pos = collections.Counter(k for _, k in keyed)
print("answer letters  ", {c: round(100.0*pos.get(j,0)/len(keyed), 1) for j, c in enumerate("ABCD")})

uniq = hits = two = 0
for i, k in keyed:
    L = [len(o.strip()) for o in i["options"]]
    if L.count(max(L)) == 1:
        uniq += 1
        hits += L.index(max(L)) == k
    if k in sorted(range(4), key=lambda j: -L[j])[:2]:
        two += 1
print("longest is key", round(100.0*hits/uniq, 1), " in two longest", round(100.0*two/len(keyed), 1))

norm = [re.sub(r"\s+", " ", re.sub(r"[^a-z0-9 ]", "", i["stem"].lower())).strip() for i in items]
dup = sum(c-1 for c in collections.Counter(norm).values() if c > 1)
print("duplicate stems", round(100.0*dup/len(items), 1), " negation",
      round(100.0*sum(1 for i in items if NEG.search(i["stem"]))/len(items), 1))

amb = 0
for i, k in keyed:
    kc = set(words(i["options"][k]))
    worst = 0.0
    for j in range(4):
        if j == k or not kc:
            continue
        dc = set(words(i["options"][j]))
        if dc:
            worst = max(worst, len(kc & dc)/len(kc | dc))
    amb += worst >= 0.6
print("key-option overlap", round(100.0*amb/len(keyed), 1), " items", len(items))

Run against 300 MMLU items on 17 November 2026 it printed letters 19.0 / 23.3 / 31.7 / 26.0, longest-is-key 25.6 per cent, key in the two longest 55.3 per cent, duplicates 1.0 per cent, negation 6.3 per cent, key-option overlap 16.7 per cent. Every number lands inside the human range and no flag fires, which is what a clean bank does.

Then I doctored that same slice the way a model does it: put the right answer first in every item and added a short justifying clause to it. The script printed A 100.0 per cent and longest-is-key 99.3 per cent, and raised both flags. Two of the six checks caught a set that would have passed a human skim, because a teacher reading item by item does not count how often the answer sits on A.

How to use the output without over-reading it

These counters test the shape of an export, not the pedagogy. A set can pass all eight and still ask about trivia, and a set can fail one and still be fine: 20.2 per cent of MMLU items have a distractor that repeats most of the key words, and MMLU is a benchmark people publish papers about. Treat a flag as a reason to open ten items and read them, not as a verdict.

The order that saves the most time: run the letter spread and the length check first, because they catch the failure that actually hurts, a set that can be beaten without knowing the material. Then duplicates, because a repeat inside one worksheet is visible to students and embarrassing. Then negation, because it predicts arguments. The overlap check comes last and decides the least.

Two limits worth stating. The baselines come from two banks, so a vocational or language exam may sit outside these ranges; measure your own old exams first and use those numbers instead, the script does not care where the items came from. And none of this replaces the closed-book test, where you hide the passage and answer the items yourself. That costs ten minutes per set and catches what the script cannot see: an item whose answer is common knowledge.

Related audits on this site: 15,186 items and the word-overlap leak rule, where AI test makers put the key, what comes back when you regenerate, and how many sentences ten items really cover.

The counters above read the export. The next batch read the explanation that ships under each item: eighty corrupted explanations, four kinds of breakage, one judge asked two narrow questions, 78 rejected and 140 of 140 real explanations passed: whether the explanation explains anything.

Related Articles
2026-08-23
Best AI Video Generator in 2026: 7 Tools for Product Demo Videos
2026-07-27
Best AI Note-Taking App 2026: Top 8 Compared for Productivity
2026-08-16
Best AI Scheduling Assistant in Chicago 2026: 6 Tools for Consultants and Corporates
2026-08-18
Best AI Interior Design Tool for Airbnb Hosts 2026: 6 Tools That Make Your Listing Stand Out
2026-08-23
Best AI Headshot Generator in Malaga 2026: 6 Tools for Digital Nomads and Remote Workers
2026-11-17
Qwen4-Max released today: 60% cheaper per task, but what I'm more excited about is the 27B that can be deployed locally

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment