AI Test Maker From a Table: 60 Copied Cells, 18 Unit Mix-Ups

🔧 AI Tools 2026-11-22 7 min read

A numeric table should produce items that make students calculate. Instead every key comes back as a cell copied word for word, and the wrong options are other rows' cells, so the item tests row-finding only.

📜 Table of Contents

Give a test-maker a data table and ask for one item per row. What comes back is predictable: "For Nanjing A, what was water use in 2024?" with 41.2, 44.9, 30.4 and 38.8 as the options. Ten tables, sixty items, and every single key is one cell of the table copied as printed. Not one item asks the student to work anything out.

That is the first number, and it is not a fluke of one prompt. The tables here have three numeric columns each, 180 cells in total, and the 60 items touch 60 of them, one per item. A table exercise with no exercise in it: the student who can find a row has the mark.

How it was run

Ten tables, six rows each, three numeric columns, built for this test: irrigation water use by plot and year, canteen meals by counter and month, weather stations, library circulation, assembly lines, bus routes, water works, solar farms, outpatient clinics, orchards. 180 cells. I wrote the numbers, so the table is the ground truth by construction; what is measured is the model's behaviour on a table like this, not anybody's real factory.

Two prompts on the same ten tables. The first asks for exactly six items, item 1 about row 1 and so on, four options, the key checkable from the table alone. The second asks for the same shape but every item must make the student do one calculation, the key must not be a figure printed in the table, and the model must write out its arithmetic. Generator deepseek-flash at temperature 0.8, ten calls per arm; first arm 17.8 s and 21,077 tokens, second arm 265.5 s and 23,668 tokens. Then each item is answered blind, table shown and key hidden, at temperature 0: 25,532 tokens in the first arm, 22,056 in the second.

First arm: sixty items, sixty copied cells

what was countedof 60 items
the key is one cell of the table, copied as printed60
the item asks the student to work anything out0
all three distractors come from other rows of the same column40
the key sits in the row the prompt asked for60
blind answer agrees with the key60

The row targeting held completely: 60 of 60 keys come from the row the prompt named, so the model does read the row number. What it does not do is combine two cells. And the distractors are not invented either: in 40 of the 60 items all three wrong options are figures lifted from other rows of the same column, which makes the item answerable by pairing the row name in the stem with the one figure that belongs to that row. The correct option sat first 23 times, second 16, third 15, fourth 6, the same first-option habit measured on passages in the answer position audit.

Second arm: it calculates, and the arithmetic is right

Ask for a calculation and you get one. All 60 items combine two columns; 59 of the 60 keys are figures that appear nowhere in the table (the sixtieth collides with an unrelated cell: 0.55 − 0.31 = 0.24, and 0.24 is also the turbidity of Works 1). I re-computed every item from the table by hand and by script: 60 of 60 keys are arithmetically correct, to the rounding the stem asked for.

So the failure is not in the sums. It is in what is being summed.

Eighteen numbers with no unit

tableitemarithmetic in the itemkey
weather stationsmean of annual rainfall and frost-free days(731.4 + 231) / 2481.2
weather stationssum of annual rainfall and frost-free days967.3 + 2621229.3
water worksmean of turbidity and chlorine(0.24 + 0.62) / 20.43
outpatient clinicwaiting minutes plus return-visit rate24.5 + 18.242.7
orchardgrade-A share minus sugar content79.6 − 14.165.5

The rule I applied by hand: the two figures in the arithmetic carry different units and are added, subtracted, averaged or turned into a percentage change. That marks 18 of the 60 items: three of the six weather items, five of six at the water works, four of six at the solar farms, four of six at the clinic, two of six in the orchard. One more item adds units per day to defects per day, which is arithmetically defensible and still nonsense to ask a student; it is not in the 18.

What the model never looks at

Six of the ten tables carry a percentage column: saving, plate waste, overdue rate, first-pass yield, on-time rate, availability. Across both arms, sixty items, not one of those six columns is used. The mean temperature column at the weather stations is not used either, although it is the only column there you can sensibly compare with another station. The model takes column one and column two and stops.

operation that mixes two unitsitems
weather stations: rainfall (mm) with frost-free days3 of 6
water works: turbidity (NTU) with chlorine (mg/L) and daily supply5 of 6
solar farms: capacity (MW) with generation (GWh)4 of 6
outpatient clinic: waiting minutes with return-visit rate4 of 6
orchard: grade-A share with sugar content (Brix)2 of 6
total18 of 60

The usual check cannot see any of this

Both arms were re-answered blind: table in the prompt, key withheld, same model at temperature 0. 60 of 60 agreed in the first arm and 60 of 60 in the second. A "re-answer it and compare" pass therefore certifies every one of the 18 unit-mixing items as fine, because a second model reading the same table does the same arithmetic and reaches the same meaningless number.

The prose control points the same way. Four expository passages, six sentences each, one item per sentence, same prompt shape: 23 of 24 keys consist of 80% or more words lifted from the sentence the item was supposed to be about, mean share 0.97. With a passage the copying is a whole clause; with a table it is a single number, which is why the table version looks cleaner and is worse.

None of this is a second look at the key. It is the same finding as 188 blind answers and 376 rewrites that moved nothing, one layer down: the item does not ask the student to do the thing the table was chosen to teach.

A check you can copy

Ask the model to write down which columns each item draws on and which operation it performs, then screen the pairs for units. Two fields, one dictionary:

import re
# numeric column slot -> unit, per table (slots are 1-based, label column excluded)
UNITS = {
    "weather":    {1: "mm", 2: "C", 3: "day"},
    "waterworks": {1: "NTU", 2: "mg/L", 3: "10k m3/day"},
    "solar":      {1: "MW", 2: "GWh", 3: "%"},
    "clinic":     {1: "count", 2: "min", 3: "%"},
    "orchard":    {1: "kg", 2: "Brix", 3: "%"},
    # irrigation, canteen, library, factory, bus: slots 1 and 2 are the same kind of quantity
}

def op_of(stem, calc):
    s, e = stem.lower(), str(calc).replace(" ", "")
    if "percentage" in s or "percent of" in s or "*100" in e:
        return "pct_change"
    if re.search(r"\([^()]*\+[^()]*\)/\d", e):
        return "mean"
    if "-" in e: return "minus"
    if "+" in e: return "plus"
    if "/" in e: return "ratio"
    return "none"

def unit_mix(table, cols, stem, calc):
    # cols: slots the arithmetic draws on (match the numbers in the written calculation back to the table)
    units = {UNITS[table][c] for c in cols}
    return len(units) > 1 and op_of(stem, calc) in ("plus", "minus", "mean", "pct_change")

On these 60 items the function returns 18 names, all of them the ones in the table above, and no false positives among the 42 arithmetic items that stay inside one kind of quantity.

What this does not measure

The tables are constructed, the figures are mine, and none of it is a real school, factory or farm; a different set of numbers could change how often the model reaches for the wrong pair of columns. One model family, temperature 0.8 for generation and 0 for the blind pass. The 18 is my judgement against a stated rule, and a looser rule (any arithmetic between two different units, division included, which pulls in rainfall per frost-free day and yield per Brix but also ordinary rates like riders per kilometre) would put the count at 36. Nobody sat these items, so there is no observed classroom error rate. And the blind pass shares the generator's blind spots, which is exactly why it agrees.

What is safe to act on: from a numeric table this model writes lookup items by default and calculation items only when told to, so the prompt has to say which one you want. When told to calculate it gets the arithmetic right 60 times out of 60, and it averages millimetres with days 18 times out of 60, so the review worth doing is on units and on the column you wanted tested, not on the sums. Two numbers to carry: 60 of 60 keys copied from a cell when the prompt is plain, and 18 of 60 items whose calculated answer has no unit. The nine free makers these prompts go into are in the free test maker comparison; how much of a source ten items cover is in the coverage count; why a re-answer pass finds nothing is in the key recheck; and the checks that do catch this are in the six-check gate. What the free tiers of the grading tools cost next to these is in the grading quota audit.

Related Articles
2026-08-18
AI Essay Writer Free 2026: 7 Tools That Research and Outline (Not Cheat)
2026-08-17
Best AI PDF Summarizer in Boston 2026: 7 Tools for Students, Biotech and Kendall Square
2026-08-15
Best AI Transcription for Podcasters 2026: 7 Tools From Raw Audio to Show Notes
2026-08-14
AI Email Copywriting Tools 2026: 6 That Sound Human, Not Robotic
2026-07-25
Best Free AI Customer Service Chatbots in 2026
2026-08-13
AI Essay Writer Free 2026: 7 Tools Compared for Students Who Want Sources, Not Hallucinations

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment