Where AI Test Makers Put the Answer

A generator left to place its own answer keys used D 5 times in 96 items and B 37 times, so answering B every time scores 38.5% without reading. Asking for an even spread fixed the totals (chi-square 0.67) and produced ABCDABCD in three of six batches. Shuffle in code, check the printed sequence, and verify the key text travelled with the option.

📜 Table of Contents

• What was run
• D is close to unused, and the two runs agree
• Asking for an even spread works, and swaps one pattern for a worse one
• Fix it in code, and check the fix
• The script
• What this does not cover

Twenty multiple-choice questions take a few seconds to generate. Checking where the correct options landed takes one line of arithmetic, and nobody runs it, so the test goes to class with its key pattern intact. Students are better at spotting that pattern than teachers are at hiding it.

The number below comes from 144 items: six short passages, one model, three runs on 10 November 2026. It answers one question. Left to decide for itself, does a generator spread the four letters out?

What was run

Six passages of about 120 words each, all written for this test: the water cycle, Roman road building, the light reactions of photosynthesis, Gutenberg's press, tides, and Fleming's contaminated plate. Each passage got one API call asking for eight four-option questions with exactly one correct answer, returned as JSON with the options labelled A to D. Two runs used that plain instruction at the default temperature. A third run added one sentence: spread the correct answers evenly, exactly two of each letter across eight questions, and never the same letter twice in a row. Same passages, same model, same parser.

The model was deepseek-chat on the vendor's OpenAI-compatible endpoint, and the responses came back stamped deepseek-flash. An item counted only when four distinct option texts came back and the key was a single letter from A to D. All 144 items passed that filter, so nothing is quietly missing from the counts below.

D is close to unused, and the two runs agree

Of 96 items written under the plain prompt, five put the answer in D. An even spread would be 24. Answering B every time scores 37 of 96, or 38.5%, against 25% for guessing. On a 20-question quiz that is 7.7 correct without reading a word, against 5.0. The two runs came from separate API calls and land within one item of each other on every letter, which makes this a property of the generator on this task rather than one unlucky batch.

Batch by batch it is worse than the total suggests. Eight of the twelve batches of eight never used D at all. The most lopsided batch put five of its eight keys in B. A student who meets two quizzes from the same tool sees the same shape twice, which is how a pattern gets noticed in the first place.

Asking for an even spread works, and swaps one pattern for a worse one

The third run obeyed the instruction. Twelve A, fourteen B, twelve C, ten D: chi-square 0.67, p = 0.88, nothing left to detect at the set level. Then the batches themselves: three of the six came back as ABCDABCD, letter for letter, all eight items. Measured by how often each key follows the ABCD cycle from wherever it starts, the plain runs sit at 31% and 33%, which is the coin-flip line; the quota run sits at 73%, with three of its six batches perfect cycles.

That trade is the part worth keeping. A quota prompt replaces a mild bias, where a guesser gets 13 extra points out of B, with a deterministic cycle that a student who spots it rides to 8 out of 8. The instruction was followed literally, including the no-repeat clause, because a four-letter cycle never repeats a letter back to back. Compliance on the sentence, failure on the intent.

Fix it in code, and check the fix

Shuffling after generation is the reliable fix. Two details cost a run each to learn.

Detail one: a per-item random shuffle does not balance a set. On a 32-item export drawn with the plain prompt the keys were 5/13/11/3. Shuffling each item's four options independently, with a fixed seed, gave 5/13/7/7: chi-square 4.5, p = 0.21, and a run of four identical letters. Random is unbiased in expectation and still visibly lumpy in the file you hand out, because a 32-item set is small enough for chance to leave marks.

Detail two: the balance has to be dealt, and the no-repeat rule has to be checked on the printed sequence. Deal the letters from a deck of equal counts, then move each key into its assigned slot. My first version enforced the no-repeat rule on the deck and then dealt through a shuffled index; the printed sequence still contained a run of three. The check belongs on the final order, not on the deck it came from.

Output on that 32-item export, unedited:

before  n=32 a=5 b=13 c=11 d=3 chi2=8.50 p=0.037 run=3
after   n=32 a=8 b=8 c=8 d=8 chi2=0.00 p=1.000 run=2
key text changed on 0 items

The last line is the one to keep. Reordering four options is easy, and carrying the key along with them is where a shuffle quietly wrecks an export. The script re-reads every row and compares the text of the marked option before and after, because a perfect distribution over moved keys is worse than no shuffle at all. On a single eight-item export the same script turns 1/3/2/2 into 2/2/2/2.

The script

Stdlib only, Python 3.8 or newer. Export to CSV with the header stem,a,b,c,d,answer, answer as a lower-case letter, then run python3 fix_keys.py bank.csv. It writes bank.fixed.csv beside the input and prints the distribution before and after. The seed is fixed at 20261110, so the same export always produces the same file; change the seed when you reuse an export for another class.

import csv, hashlib, math, random, sys, collections
L = "abcd"

def chi2_p(x):  # p-value for 3 degrees of freedom
    return math.erfc(math.sqrt(x / 2)) + math.sqrt(2 * x / math.pi) * math.exp(-x / 2)

def longest_run(s):
    best = cur = 1
    for a, b in zip(s, s[1:]):
        cur = cur + 1 if a == b else 1
        best = max(best, cur)
    return best if s else 0

def report(rows, tag):
    seq = "".join(r[5] for r in rows)
    c = collections.Counter(seq); n = len(seq); e = n / 4.0
    cnt = [c[k] for k in L]
    chi = sum((o - e) ** 2 / e for o in cnt)
    print("%-7s n=%d a=%d b=%d c=%d d=%d chi2=%.2f p=%.3f run=%d"
          % (tag, n, *cnt, chi, chi2_p(chi), longest_run(seq)))

def main(path, seed=20261110):
    rows = []
    for r in csv.DictReader(open(path, encoding="utf-8-sig", newline="")):
        opts = [(r.get(k) or "").strip() for k in L]
        key = (r.get("answer") or "").strip().lower()
        if all(opts) and key in L:
            rows.append([r["stem"].strip()] + opts + [key])
    report(rows, "before")

    rng = random.Random(seed)
    deck = [L[i % 4] for i in range(len(rows))]
    for _ in range(500):
        rng.shuffle(deck)
        if longest_run("".join(deck)) <= 2:
            break

    out = []
    for row, target in zip(rows, deck):
        rest = [o for i, o in enumerate(row[1:5]) if i != L.index(row[5])]
        random.Random(int(hashlib.sha256(row[0].encode()).hexdigest()[:8], 16)).shuffle(rest)
        new = rest[:]; new.insert(L.index(target), row[1 + L.index(row[5])])
        out.append([row[0]] + new + [target])
    report(out, "after")
    print("key text changed on %d items" % sum(
        row[1 + L.index(row[5])] != new[1 + L.index(new[5])] for row, new in zip(rows, out)))

    with open(path.replace(".csv", ".fixed.csv"), "w", encoding="utf-8", newline="") as f:
        w = csv.writer(f); w.writerow(["stem", "a", "b", "c", "d", "answer"]); w.writerows(out)

if __name__ == "__main__":
    main(sys.argv[1])

What this does not cover

One model family, one day of generation, English passages. A hosted test maker may already shuffle server-side, in which case the export arrives balanced and the check above takes ten seconds to confirm. Nothing here says anything about whether the keys are right, whether the items are ambiguous, or whether they are the right difficulty; each of those needs its own pass. Forty-eight items is a coarse batch: an unbiased generator produces three or fewer D's in 48 items about once in 1,300 batches, so one batch is enough to raise the question and not enough to settle it, which is why the two runs were compared before anything was concluded.

If the open question is whether a key can be reached from the shape of the options rather than their order, that audit is here: 28.6% of items guessable by option length. If you want the check that turns each item's three distractors into a control group, it is here: the answer-leak audit on 15,186 exam items. And if you are still choosing a generator, the free tiers and export limits across nine of them are compared here: what the nine free AI test makers actually give you.

Two numbers to carry into the next export: 5 of 96 keys in D, and ABCDABCD in half the batches once you ask for an even spread. Check the sequence, then hand out the quiz.

The next question is what a second run buys. Regenerating from the same passage returned 35 of 64 items from the first run, 15 of them word for word, so a bank of 60 takes five to six passages run twice: the duplicate-rate check.