AI Test Maker Cue Audit: 34 Items Gave Away an Option

🔧 AI Tools 2026-11-18 11 min read

A generated test can be beaten without the passage: the wrong option is written as an exaggerated version of the right one, so any option containing only, never or all can be deleted on sight. One prompt sentence fixes that cue and leaves the length cue standing.

📜 Table of Contents

An item leaks through its wording. The wrong option built as a caricature of the right one, “it stops fermenting completely” against “it still ferments, just slowly”, reads as wrong to anyone who has taken a few tests, and that reading is available without the passage. So I generated four hundred items with one model, measured that leak with a regex, and ran the same two lines of arithmetic over 14,042 human-authored items to see what normal looks like.

Absolute-word cue in generated items: 1 of 34 absolute-word options held the key; blind guess after dropping them and taking the longest scores 50.1%
Of 128 generated items, 34 carried an absolute-word option and one of those 34 held the key; the same cue keys the answer 37.1% of the time in MMLU. Dropping absolute options and taking the longest of the rest scores 50.1% blind on the generated set, 28.1% on MMLU.

The plain prompt left an absolute-word option in 34 of 128 English items, and exactly one of those 34 carried the key. In MMLU the same cue appears in 14.0% of items and the key carries it 37.1% of the time. A student who drops any option containing always, never, all, none, only, entirely, completely, impossible or solely and guesses among the rest scores 29.2% on the generated set and 25.0% on MMLU. Add a second rule, take the longest of the survivors, and the same student reaches 50.1% on the generated set against 28.1% on MMLU.

How it was run

Sixteen English passages of 118 to 133 words: the twelve from the explanation test plus four on tides, glacier flow, lithium-ion cells and soil structure. Eight items per passage, four options each, key checkable against the passage. The model was deepseek-flash at temperature 0.8, fifty-six calls in 101 seconds, 400 items, none short of the count asked for, none malformed. A Chinese half used a separate set of twelve passages, on 点茶, 伸缩缝, 道岔, 酱油, 光纤色散, 盐碱化, 探空, 析锂, 越剧, 空化, 热岛 and 古籍脱酸, six items per passage.

The second prompt adds one sentence of house rules: no negative stems, no absolute qualifiers in any option, no all-or-none-of-the-above, options of the same kind and similar length. Everything else is identical, including the temperature.

The human side is the MMLU test split (14,042 four-option items) and the ARC-Challenge test split (1,144 four-option items out of 1,172), read on the ECS box from the same parquet files used for the earlier length audit. The regex is one line, the arithmetic is the same function for both banks, and the length measure counts characters after stripping punctuation, which is what reproduced 28.6% on MMLU and matches the number published on 6 November.

What the plain prompt leaves in

MMLU test, 14,042plain prompt, 128house rules, 128
items with an absolute-word option1,967 (14.0%)34 (26.6%)0
the keyed option carries the absolute word730 (37.1%)1 (2.9%)—
exactly one absolute option, and it is keyed420 of 1,390 (30.2%)0 of 22—
two or more absolute options577 (4.1%)12 (9.4%)0
“all / none of the above” option380 (2.7%)00

The rate is the smaller half of the story. The bigger half is where the absolute word sits. In MMLU, 730 of the 1,967 cue items put it on the key, so the cue points at the answer more often than chance; when exactly one option carries it, that option is the key 30.2% of the time, roughly the 25% a blind guess would give. Human item writers use absolute words in both directions.

The generator used them as a signpost for the wrong answer. Of 34 cue items, one keyed an absolute option. Of the 22 items where exactly one option carried the word, none did. That is not a coincidence of one passage: the same pattern repeats in Chinese, 11 cue items, one keyed, none of the 7 single-cue items.

ARC-Challenge, the grade-school bank, carries the cue less often, 53 of 1,144 items, 4.6%, and puts it on the key 26 times, 49.1%. Real exam writers use “always” and “all” as content, in items about set membership or universal claims. The generated items use them as decoration on a distractor.

The Chinese half, same pattern

plain prompt, 72house rules, 72
items with an absolute-word option11 (15.3%)0
the keyed option carries the absolute word1 (9.1%)—
exactly one absolute option, and it is keyed0 of 7—
drop absolute options, guess among the rest26.5%25.0%
the same rule, only on items that carry the cue34.9% (11)—
drop absolute options, then take the longest left55.6%31.4%
the unique longest option is the key69.8% (43)47.1% (17)

Twelve passages, six items each, and the cue appears at a similar rate, 11 of 72 items (15.3%), one of them keyed, none of the seven single-cue items. What differs is the length cue: 43 of 72 Chinese items came back with a unique longest option and 30 of those carried the key, 69.8%, so the combined blind score reaches 55.6% against 50.1% in English. The house rules removed all eleven absolute options and pushed the combined score down to 31.4%, most of that from the length clause, which is the only place in this test where the length part of the sentence did visible work.

Two rules, half the paper

a student who never reads the passageMMLU, 14,042plain, 128house rules, 128
guess at random25.0%25.0%25.0%
drop absolute-word options, guess among the rest25.0%29.2%25.0%
the same rule, only on items that carry the cue26.1% (1,871)41.9% (33)—
drop absolute options, then take the longest of what is left28.1%50.1%41.1%
the unique longest option is the key28.6% (11,645)48.3% (118)44.9% (107)

Read the third column. Dropping absolute options is worth 4.2 points over chance on the generated set and nothing at all on MMLU, because in MMLU the cue is as likely to sit on the key. The second rule is where the damage is: longest-of-the-survivors reaches 50.1%, twice chance, on 128 items.

Two rules, no reading, half marks. The length half is the older finding, published here on 6 November against the same two human banks, and the generated number is far above the human line, 48.3% against 28.6%. The absolute-word half is new, and it is the one a prompt can switch off in a single sentence.

One sentence of house rules

The second prompt adds this and changes nothing else:

House rules for the item set: no negative stems (no “which is NOT”, no “except”, no “least”);
no absolute qualifiers in any option (always, never, all, none, only, entirely, completely, impossible, solely);
no “all of the above” or “none of the above”; all four options the same kind of thing, similar in length.

Across 200 items after that line, English and Chinese, the count of absolute-word options is zero. Not reduced, zero. Every call still returned the requested number of items, all with four options and no duplicates. Mean stem length moved from 11.0 to 10.4 words in English and from 19.2 to 20.0 characters in Chinese, so the rule did not buy compliance by hollowing the stems out. What did change inside the item is the spread of option length: the within-item standard deviation fell from 1.19 to 0.52 words in English, and Chinese options got shorter as well as more even, 7.8 characters on average down to 6.3.

The Chinese run shows what the same line does to the length cue: 69.8% of unique-longest items keyed on the plain prompt, 47.1% after, and the number of items with a unique longest option fell from 43 of 72 to 17 of 72, which is the “similar length” clause doing its work. English moved much less, 48.3% to 44.9%, because English distractors differ in wording and a length instruction does not reach that.

What the house rules did not fix

The combined strategy on the corrected English set is still 41.1% against 28.1% on MMLU, and the length cue alone is still 44.9% against 28.6%. Turning off the absolute-word cue moves a student from 50.1% to 41.1%; it does not give you a fair item set. The remaining gap is option-length variance, and it needs either a hard cap on option characters or an editing pass, not another clause.

The negative-stem worry measured zero

The house rules also ban negative stems, and that clause cost nothing, because the plain prompt produced none: 0 of 400 items across both languages contained a stem built on not, except, least, false or incorrect. MMLU carries 2,219 such stems, 15.8% of the bank, so the worry is real in human exams, where “which of the following is NOT” is a standard form. This generator did not reach for it once in 400 items.

That is a result about one model at one temperature, and it should not be copied into a checklist as “generators do not write negative stems”. What it does say is that the clause is free here, so keep it and stop budgeting review time for a failure mode that did not appear.

Two items, verbatim

From the plain English set, passage on sourdough. Stem: “How does a starter held in the fridge for a week compare with one fed that morning?” Options include the key, “It still ferments slowly, so the dough will be more acidic”, and “It stops fermenting completely”. The passage says the opposite of the second option in plain words. A student who has learned that completely is a warning sign deletes it in under a second.

From the Chinese set, passage on 桥梁伸缩缝. Stem asks for the main reason the expansion joint is the most damage-prone part of a deck; the distractor is 「桥梁长度一定过长」, an absolute claim that does not appear anywhere in the passage, and the key is 「车辆冲击、雨水和杂物集中在缝隙里」. Same construction, different language, and it is the distractor that carries the signpost.

The check to copy over

Export the items to CSV with the header stem,a,b,c,d,answer, answer as a lower-case letter, and run this. It prints four lines: how many items carry the cue, how often the cue sits on the key, and what the two student rules score.

import csv, re, sys

ABS = re.compile(r"\b(always|never|all|none|only|entirely|completely|impossible|solely|sole)\b", re.I)

def norm(s):
    return len(re.sub(r"[^a-z0-9 ]+", " ", str(s).lower()).strip())

def load(path):
    rows = []
    with open(path, newline="", encoding="utf-8-sig") as f:
        for r in csv.DictReader(f):
            opts = [r.get(k, "").strip() for k in "abcd"]
            ans = str(r.get("answer", "")).strip().lower()
            if not all(opts) or ans not in "abcd" or len(ans) != 1:
                continue
            rows.append((opts, "abcd".index(ans)))
    return rows

def main(path):
    rows = load(path)
    n = len(rows)
    cue = sum(1 for o, a in rows if any(ABS.search(x) for x in o))
    keyed = sum(1 for o, a in rows if ABS.search(o[a]))
    drop = comb = 0.0
    for o, a in rows:
        keep = [i for i, x in enumerate(o) if not ABS.search(x)]
        if not keep:
            continue
        drop += (1.0 / len(keep)) if a in keep else 0.0
        top = [i for i in keep if norm(o[i]) == max(norm(o[j]) for j in keep)]
        comb += (1.0 / len(top)) if a in top else 0.0
    print("items %d" % n)
    print("items with an absolute-word option %d (%.1f%%)" % (cue, 100.0 * cue / n))
    print("keyed option carries an absolute word %d (%.1f%% of the cue items)" % (keyed, 100.0 * keyed / max(cue, 1)))
    print("drop absolute options, guess        %.1f%%" % (100.0 * drop / n))
    print("drop absolute, then longest         %.1f%%" % (100.0 * comb / n))
    print("chance                              25.0%")

if __name__ == "__main__":
    main(sys.argv[1])

On the 128 English items from the plain prompt it prints 34, 2.9%, 29.2% and 50.1%; on the same 128 after the house rules it prints 0, 0.0%, 25.0% and 41.1%. The thresholds to use: any rate of absolute-word options above the 14.0% MMLU line means the set carries a cue the human bank does not, and any combined score within ten points of the one the plain prompt produces means the set is not ready. For Chinese items the word list changes and the arithmetic does not: 全部, 所有, 一定, 唯一, 从不, 完全, 必然, 绝对, 只能, 任何.

What this does not measure

One model, one temperature, 128 English items per variant and 72 Chinese items per variant. The regex is mechanical: only also fires inside “not only”, all inside “all real numbers”, so the 14.0% MMLU rate includes some benign hits and the generated 26.6% probably includes a few too. MMLU is knowledge-heavy, where absolute words are content; passage-based items are a different genre, so the 14.0% line is a reference, not a target. Nobody sat the tests; the scores are computed expectations for a student using the rule, not observed answers. And the house rules were tested as a bundle, so the 69.8% to 47.1% Chinese drop cannot be attributed to the length clause alone.

What is safe to act on: the plain prompt puts an absolute word on a distractor 34 times and on the key once, the human bank splits the same cue 37.1% toward the key, and one sentence of house rules removes the cue entirely without losing a single item. The check that reads the explanation under each item is in the explanation gate; where the length cue sits in human banks was measured on 6 November in the 28.6% audit; how many sentences ten items actually test is in the coverage count; and the nine free tools these prompts are meant to be pasted into are in nine free test makers. Two numbers to carry: 34 cue items, one keyed.

The cue above lives in the option wording. The instruction line leaks too: 252 items at grades 4, 8 and 12, eight grades of request worth 2.11 grades of stem, and 82% of grade-12 keys still inside a single sentence: the grade dial audit.

The cue above is one way an option gives itself away. The other is structural: 74 of 188 items carry a wrong option whose defence is a sentence from the passage, and the count of options sitting inside one sentence reaches 60: the key recheck, measured.

Related Articles
2026-08-24
Best AI Photo Editor in Cancun 2026: 6 Tools for Hotels, Tours and Tourism Marketing
2026-08-18
Best AI Headshot Generator for Actors 2026: 6 Tools for Casting and Spotlight
2026-08-15
AI Podcast Show Notes Generator 2026: 5 Free Tools From Episode to Notes
2026-08-28
Best AI Medical Scribe in Tampa 2026: 6 Tools That Cut Charting Time
2026-07-27
AI Video Generation Pricing 2026: Best Value Per Dollar
2026-08-22
Best AI Productivity App in Chiang Mai 2026: 6 Tools for Digital Nomads

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment