AI Test Maker Grade Dial: Asked for Grade 12
The grade field feels like a reading-level control. Measured, it moves the stems a quarter of the way and leaves the one-sentence answerability above 80%.
📜 Table of Contents
The grade field is the first control a teacher touches. Type a number, get items for that grade. I took ten passages and asked for six items each at grade 4, at grade 8 and at grade 12, which is 180 items over thirty calls, then ran a readability formula over the stems. Eight grades of request moved the stems 2.11 grades, and a request for grade 12 came back reading at 7.97, below a science bank written for grades three to nine.
The second half of the result is worse for the dial. At grade 12, 82% of items still had a key whose content words sit inside a single passage sentence, so a student who matches strings can answer them without reading for meaning. Asking for a higher grade cut that from 93% to 82% and left it there.
How it was run
Ten English passages of 118 to 133 words, ten of the sixteen used for the cue audit on 18 November, on sourdough, ice cores, cable stays, paper sizing, honey, ballast, espresso, satellite drag, blade erosion and first crack. Six items per passage per requested grade, four options each, key checkable against the passage. The model was deepseek-flash at temperature 0.8, forty-eight calls in 62.7 seconds, 252 items in total, none short of the count asked for.
The instruction was one sentence inside an otherwise identical prompt: “for students in grade 4”, “grade 8”, “grade 12”. Nothing else changed. The Chinese arm used six separate passages, four items each at 小学四年级, 初中二年级 and 高中三年级.
Readability is Flesch-Kincaid grade level, 0.39 words-per-sentence plus 11.8 syllables-per-word minus 15.59, with a vowel-group syllable counter that drops a silent final e. A stem counts as one sentence, because almost every item is a single question. That convention is the one weak point of the measure and I come back to it below.
The human lines are the same function over the MMLU test split (13,977 stems scored out of 14,042) and the ARC-Challenge test split (1,171 of 1,172), both read on the box from the parquet files used for the November audits. MMLU averages 10.51, ARC 8.39.
One grade delivered for every four asked
| asked for | items | mean stem FK grade | median | words per stem | within two grades of the ask |
|---|---|---|---|---|---|
| grade 4 | 60 | 5.86 | 5.86 | 10.1 | 28 (47%) |
| grade 8 | 60 | 6.75 | 6.73 | 11.3 | 29 (48%) |
| grade 12 | 60 | 7.97 | 8.09 | 12.4 | 16 (27%) |
Read the first and last rows. A request for grade 12 produced stems at 7.97, roughly the reading level of a request for grade 8 plus one. The slope is 2.11 grades of output for 8 grades of input, 26%. Mean stem length does move in step, 10.1 words to 12.4, and that is most of what the formula is reacting to.
The third column also kills a hope. At grade 4, only 47% of stems landed within two grades of the request; 57% of them came back above grade 5. At grade 12, 97% of stems came back at or below grade 13, and 55 of 60 came back below grade 12. The dial errs in both directions, but the larger error is at the top: nothing came back harder than asked.
Why the passage sets the ceiling
The ten source passages average FK 12.11, from 10.15 on sourdough to 14.73 on satellite drag. They are the hardest text in the pipeline and every stem is built from them. Between 30% and 37% of the words in a stem appear in the passage itself, and the technical nouns cannot be swapped out without breaking the item: a question about the electrolytic capacitor in the cable-stay passage has to say capacitor.
What the model can change, it changes at the margins. It adds a subordinate clause, which pushes words per stem from 10.1 to 12.4 and syllables per word from 1.49 to 1.58. That is enough to register on a formula and roughly what a teacher would call “a slightly longer question”, not “a harder question”.
The delivered level sits under the human banks
MMLU, the graduate-and-professional bank, averages 10.51 on the same formula. ARC-Challenge, written for grades three to nine, averages 8.39. A generated set requested at grade 12 comes back at 7.97: 0.42 below the grade-3-to-9 science bank and 2.54 below MMLU.
That gap is not proof the items are easy. Short questions score low on Flesch-Kincaid by construction, and MMLU stems are often longer than twelve words. It is proof that the number the teacher typed is not the number the paper reads at, and that the shortfall is systematic rather than random.
What the dial did not change
| grade 4 | grade 8 | grade 12 | |
|---|---|---|---|
| key appears word for word in the passage | 21 (35%) | 19 (32%) | 15 (25%) |
| 80% or more of the key's content words sit inside one sentence | 56 (93%) | 51 (85%) | 49 (82%) |
| syllables per word in the stem | 1.49 | 1.52 | 1.58 |
| stem words with three syllables or more | 9% | 12% | 14% |
| stem words that also appear in the passage | 37% | 34% | 30% |
The last three rows are the vocabulary side, and they barely move. One extra syllable-heavy word in twenty between grade 4 and grade 12, three points fewer stem words borrowed from the passage. The paper still talks about the same things in nearly the same words.
The first two rows are the ones that matter for marking. A key copied word for word out of the passage drops from 35% to 25%, which is real progress, and a key whose content words are 80% or more inside one sentence drops from 93% to 82%. So at grade 12, four items in five can still be answered by finding one sentence and matching strings. If the point of asking for a higher grade was to make students infer, the dial moved that number by eleven points.
The items are different, the difficulty is not
The three sets are not the same six questions reworded. Comparing every grade-4 stem with the closest grade-12 stem on the same passage, the mean best-match Jaccard score over content words is 0.427, fourteen of sixty pairs reach 0.60, three reach 0.80, and exactly one stem is identical after normalising case and punctuation. The model does write new questions.
So the dial changes which sentences get tested and how the question is phrased, and leaves the reading level and the one-sentence answerability where they were. On this evidence it is a style control wearing a difficulty label. That matches what the difficulty labels did on 13 November, when items tagged hard were answered without the passage as often as items tagged easy.
The Chinese arm is flatter still
| asked for | items | characters per stem, mean | median |
|---|---|---|---|
| primary, grade 4 | 24 | 18.6 | 17.0 |
| junior, grade 8 | 24 | 20.8 | 19.5 |
| senior, grade 12 | 24 | 20.4 | 20.5 |
Characters per stem go 18.6, 20.8, 20.4: the junior and senior settings produce stems of the same length, and the character-level similarity between primary and senior stems averages 0.262, with two of twenty-four pairs above 0.60. The senior setting is not a harder version of the primary setting; it is a different draw from the same distribution.
There is no Flesch-Kincaid equivalent in this test, so the Chinese arm only carries length and similarity. What it adds is a check that the flat result is not an artefact of an English readability formula: the same dial, on a language where the formula cannot apply, produces papers that differ in wording and not in counted difficulty.
The check to copy over
Export the items as JSON with the same shape used here, runs[].items[] carrying stem, options and a zero-based answer, and run this. It prints each run's mean grade and the gap against what you asked for, then lists the items whose key is lifted straight out of the passage.
import re, sys, json
VOW = "aeiouy"
def syllable(w):
w = re.sub(r"[^a-z]", "", w.lower())
if not w:
return 0
n, prev = 0, False
for ch in w:
v = ch in VOW
if v and not prev:
n += 1
prev = v
if w.endswith("e") and not w.endswith(("le", "ee")) and n > 1:
n -= 1
return max(1, n)
def fk(text):
words = re.findall(r"[A-Za-z][A-Za-z'-]*", text)
if not words:
return None
sents = max(1, len(re.findall(r"[.!?](\s|$)", text.strip())))
syl = sum(syllable(w) for w in words)
return 0.39 * (len(words) / sents) + 11.8 * (syl / len(words)) - 15.59
def flat(s):
return " " + re.sub(r"[^a-z0-9 ]+", " ", s.lower()).strip() + " "
def report(path):
d = json.load(open(path))
for run in d["runs"]:
grades = [fk(i["stem"]) for i in run["items"]]
grades = [g for g in grades if g is not None]
if not grades:
continue
mean = sum(grades) / len(grades)
print("%-22s n=%d FK mean %.2f gap %+.2f" % (run["label"], len(grades), mean, mean - run["grade"]))
for run in d["runs"]:
p = d["passages"]["en"][run["label"].split("|")[0]]
for i in run["items"]:
key = flat(i["options"][i["answer"]])
if len(key.split()) >= 3 and key.strip() in flat(p):
print("%-22s verbatim key: %s" % (run["label"], key.strip()[:60]))
report(sys.argv[1])
On the run above, the grade-4 block prints a gap of +1.86, the grade-8 block +(-1.25) and the grade-12 block -4.03. Two thresholds worth keeping: a gap beyond two grades in either direction means the number you typed is not doing what you think, and a verbatim-key rate above 25% means a chunk of the paper can be answered by string matching.
What this does not measure
One model, one temperature, ten English passages and six Chinese ones. Every passage is expository non-fiction, so the ceilings here apply to that genre and not to a poem or a lab report. Flesch-Kincaid on a ten-word question is unstable: the formula treats each stem as one sentence, so a clause added anywhere raises the score, and the counter mis-splits some words. Nobody sat these papers; there is no observed difficulty, only a computed reading level and a string test. The one-sentence test uses a mechanical definition, 80% of the key's content words inside a single sentence, and a stricter threshold would move the 82% down and the 93% down with it.
What is safe to act on: a grade request of 12 returned stems at 7.97, below the 8.39 of a bank written for grades three to nine, and 82% of those stems still have their key inside one sentence. The dial is worth typing, because it does make the questions longer and it does cut the copied-key rate from 35% to 25%; it is not worth trusting as a reading-level control. If the grade has to be right, control the passage and cap the stem length, then measure. How many sentences a ten-item set actually covers is in the coverage count; what happened when items were labelled hard and easy is in the label calibration; the option wording that gives an answer away is in the cue audit; and the free tools these prompts go into are in nine free test makers. Two numbers to carry: 2.11 grades delivered for 8 asked, and 82% of grade-12 keys still inside a single sentence.
The dial below moved the stems 2.11 grades. The key held up better than the items did: 188 items answered blind by two models, 376 rewrites, no answer moved, and 74 items where a wrong option can be defended with a passage sentence: the key recheck.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.