AI Test Maker Audit: 28.6% of Items Guessable by Length
A generated test can look right and still be beatable without the reading. I ran three checks a teacher can repeat on any export against 14,042 MMLU items and 1,172 ARC-Challenge items; on well-built items the longest-option trick is worth only 3.6 points over chance, which is the line a generated set should be compared to.
💡 What You Will Learn
A generated test can look right and still be beatable without the reading. I ran three checks a teacher can repeat on any export against 14,042 MMLU items and 1,172 ARC-Challenge items; on well-built
📜 Table of Contents
Twenty multiple-choice questions take about eight seconds to generate. Checking whether a student who skipped the reading can still pass takes an hour, so the check does not happen, and the test goes to class untested.
The three checks below are the cheap version of that hour. I ran them on two human-authored reference banks to get a baseline, because a number on its own says nothing until you know what normal looks like. The banks are the MMLU test split (14,042 four-option items across 57 subjects) and the ARC-Challenge test split (1,172 grade-school science items drawn from real exams). Everything was run on 6 November 2026 against the published parquet files, and the script at the end reproduces these exact numbers on your own export.
Check one: does the longest option carry the answer?
In MMLU, 11,645 of 14,042 items (82.9%) have exactly one longest option. Pick it every time and you score 3,328 of 11,645, or 28.6%. Chance is 25%. The gap is 3.6 points, and at that sample size it is not noise (z = 8.9). It is also not a strategy — 3.6 points will not pass a test.
ARC-Challenge behaves the same way: 954 of 1,172 items have a unique longest option, and picking it scores 258, or 27.0%, z = 1.5. On 1,172 items the length signal is indistinguishable from chance.
Take the two longest options and guess between them, and the numbers move to 53.8% (MMLU) and 57.1% (ARC). Four to seven points over a coin flip — that is the honest size of the length signal in real exams.
So a healthy four-option bank sits at 27-29% on this check. A generated set that scores 55% or 70% is not a harder test; it is a test where the model wrote a full explanatory clause for the right answer and two-word labels for the wrong ones. A student finds that pattern in about one sitting.
Check two: are the answer letters spread out?
MMLU keys land at a 22.9%, b 24.7%, c 25.5%, d 26.9%. ARC-Challenge: a 22.6%, b 26.5%, c 26.5%, d 24.4%.
Here is the trap I fell into. My first version failed a bank when chi-square cleared 7.81 at df = 3. MMLU came back at 45.9 and the reference corpus failed its own check. That 45.9 comes from a 1.6-point deviation, because at 14,042 items the test detects almost nothing. Statistical significance and pedagogical significance are different sizes here.
The usable rule is the spread, not the p-value: highest letter share minus lowest letter share, which is 3.9 points on both banks (3.945 on MMLU, 3.949 on ARC-Challenge). Above 5 points, one letter is doing the work, and a student who always picks the same letter gets a free head start. If your export puts 40% of its keys on c, the chi-square will say so, but so will four numbers you can count by hand.
I kept the chi-square in the script output for the extreme case and let the spread drive the decision. That distinction is the whole reason the audit was worth running on a corpus before running it on generated items.
Check three: duplicates and ties
MMLU has 161 items (1.1%) where two options normalise to the same string. Real exam banks are not spotless, so 1.1% is the floor of acceptable, not a scandal. ARC has 3 (0.3%). A generated bank above 3% is emitting options it never read back, which is also the cheapest defect to fix.
Ties matter for the opposite reason. 17.1% of MMLU items and 18.6% of ARC items have two options of the same character length, so a length-only guesser cannot even make a decision. A generated bank at 0% ties has been length-padded, deliberately or by accident, until every option looks the same size.
One more count worth keeping: 380 MMLU items (2.7%) include an "all of the above" or "none of the above" option. Those are legitimate in real exams and dangerous in generated ones, because a model will happily pair them with a key that contradicts the other three options.
The spread inside one corpus: audit the deck, not the average
The pooled 28.6% hides a wide range. Across the 53 MMLU subjects with at least 60 uniquely-longest items, the median is 28.9% and the 90th percentile is 36.4%. At the top: international_law 53.4% (n=121) and jurisprudence 43.6% (n=108), where the correct statement carries its qualifications and the distractors stay crisp. At the bottom: public_relations 15.2% (n=110) and human_aging 16.8% (n=223), both below chance, meaning those banks are actively anti-guessable.
The consequence is practical. One threshold applied to a pooled export will miss the single subject where length leaks, and that subject will be the one your students complain about. Run the check per subject or per deck, then pool.
The script
Stdlib only, Python 3.8+, nothing to install. Export the generated items to CSV with the header stem,a,b,c,d,answer, where answer is one of a/b/c/d, then run python3 item_audit.py bank.csv.
Run against MMLU, it prints this:
items audited: 14042 answer position a/b/c/d: 3222/3462/3582/3776 chi2=45.9 picked longest option: 3328/11645 = 28.58% (chance 25%, z=8.92) tied-longest 17.1% | top-2-longest 53.8% | duplicate options 1.1% letter spread: 3.9 points
That output is the corpus failing my own chi-square tripwire while passing the spread rule, which is how check two got rewritten.
What to change when a set fails
- Shuffle the keys per item with a fixed seed. One pass, reproducible, fixes check two outright.
- Strip the trailing clause on the correct answer. "ATP synthase, which phosphorylates ADP using the proton gradient" becomes "ATP synthase". The clause is the mechanical cause of length bias; padding the distractors only hides the same signal under noise.
- Delete duplicate options instead of rewording them. A model that emitted the same option twice did not track its own output, and the item is suspect beyond the duplicate.
- Never accept all-of-the-above from a generator. It is a shortcut for the model to avoid writing a fourth plausible wrong answer.
What these three checks cannot tell you
They measure guessability, not correctness. A bank can pass all three and still key 10% of its items wrong, which is the failure that actually hurts a class — a correct answer marked incorrect, then a parent email. That is a different pass: answer every item from the source text yourself, or key it a second time with another model and read the disagreements. Ten items is enough to see whether the key is stable.
If you are still choosing a tool, the comparison that matters most is what each one does with the answer key and the export, which is the test I ran on five test makers last month: five AI test makers judged by what they do with the answer key. If price and export formats are the open question instead, the free-tier breakdown is here: the free quiz and test makers compared on price and export.
Two numbers to carry: 28.6% and 3.9 points. If the set your tool produced scores far above them, the students will find out before you do. The corpus files used here are public — MMLU and ARC — so the baseline can be re-run, not taken on trust.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.