Summary Tool Test: 78 Numbers per Paper
The summary reads well and the sample size, the price and the date are gone. Recall of numbers tracks the budget almost linearly, so a shorter summary is a number-losing summary.
📜 Table of Contents
Summary tools are sold on prose. The demo shows a clean paragraph that captures the argument, and the argument is usually captured. What nobody shows is the column of figures that disappeared, and figures are the part a reader opens the document for: the sample size, the price, the date, the percentage that made the paper worth citing.
So I measured that part alone. Eight arXiv papers, pulled as HTML on 17 November 2026, stripped to body text, 1,803 sentences and 623 distinct numbers. A sentence-budget summariser was pointed at each one and asked for 5 per cent and 10 per cent of the original characters. Then the same budget was spent a second way, with a numbers-first pass that reserves room for any sentence introducing a numeral not yet covered. Every summary was scored on one metric: what share of the document's distinct numbers survived.
What survived
| arXiv | chars | sentences | distinct numbers | sentences with a number | 5% plain | 5% numbers-first | 10% plain | 10% numbers-first |
|---|---|---|---|---|---|---|---|---|
| 2401.00001 | 26,279 | 183 | 80 | 65 | 31.2% | 32.5% | 40.0% | 42.5% |
| 2401.00368 | 28,831 | 182 | 54 | 84 | 7.4% | 11.1% | 20.4% | 27.8% |
| 2401.00741 | 32,613 | 203 | 58 | 91 | 6.9% | 10.3% | 12.1% | 25.9% |
| 2401.01234 | 41,968 | 250 | 60 | 109 | 16.7% | 18.3% | 16.7% | 25.0% |
| 2401.02954 | 65,872 | 407 | 163 | 199 | 4.9% | 7.4% | 17.2% | 27.0% |
| 2401.03770 | 40,824 | 222 | 37 | 60 | 16.2% | 21.6% | 32.4% | 48.6% |
| 2401.04088 | 22,791 | 140 | 90 | 83 | 2.2% | 4.4% | 4.4% | 10.0% |
| 2401.05371 | 29,283 | 216 | 81 | 87 | 14.8% | 16.0% | 30.9% | 34.6% |
| mean | 36,058 | 225 | 78 | 97 | 12.5% | 15.2% | 21.8% | 30.2% |
At a 5 per cent budget, a plain centrality summariser keeps 12.5 per cent of the numbers on average. That is roughly 10 numbers out of 78. The best paper in the set reached 31.2 per cent, the worst 2.2 per cent, and the worst case is not an outlier so much as a warning: 2401.04088 is 22,791 characters with 90 distinct numbers, an unusually dense document, and 5 per cent of it is 1,139 characters, which is five sentences. Five sentences cannot hold 90 numbers.
Doubling the budget to 10 per cent nearly doubles the number recall, from 12.5 to 21.8 per cent. That is the honest shape of the trade: recall tracks budget almost linearly, so if numbers matter you are choosing a ratio, not a tool.
The numbers-first pass helps, and not nearly enough
The second policy is the obvious fix. Keep picking the most central sentences, but before the budget closes, check which numerals are still missing and pull in the sentences that introduce them, allowing 15 per cent overshoot. It works, in the sense that the number never goes down: 12.5 to 15.2 per cent at 5 per cent budget, 21.8 to 30.2 per cent at 10 per cent, with the gain largest in the mid-range papers (2401.00741 goes from 12.1 to 25.9 per cent at 10 per cent).
It cannot work much better, and the reason is a single measurement. Sentences containing at least one number are 42.1 per cent of the text on average across these eight papers, and as high as 52.1 per cent in the densest one. Numbers are not sprinkled through the prose; they are half the prose. A policy that promises to carry every number has signed up to carry nearly half the document, which is not a summary any more.
That reframes the design question. If a reader needs the figures, the right move is not a better picker inside one 5 per cent block. It is two artefacts: a short prose summary with the numbers explicitly out of scope, and a separate numbers block that carries every numeral with its sentence fragment. The summary stays at 5 per cent, the block costs about 45 per cent of the source, and the reader who needs the table can find it. What fails is the third option, a 5 per cent summary that is quietly expected to keep the numbers.
Run it on the tool you are paying for
The script below is the one that produced the table. It is extractive, standard library only, and it needs a plain text file.
#!/usr/bin/env python3
# numrecall.py -> python3 numrecall.py paper.txt 5
import re, sys, math, collections
NUM = re.compile(r"\d[\d,\.]*\d|\d")
STOP = set("the a an of and or to in for on with that this is are was were be been by as at from it its "
"which we our their than then these those such not can may also used using use show shows shown".split())
def sentences(t):
ps = re.split(r"(?<=[.!?])\s+(?=[A-Z0-9(])", t)
return [p.strip() for p in ps if 40 <= len(p.strip()) <= 400]
def toks(s):
return [w for w in re.findall(r"[a-z][a-z\-]+", s.lower()) if w not in STOP and len(w) > 2]
def numbers(s):
return set(NUM.findall(s))
def centrality(sents):
df, tfs = collections.Counter(), []
for s in sents:
w = toks(s); tfs.append(collections.Counter(w))
for k in set(w):
df[k] += 1
idf = {k: math.log(len(sents) / (1 + v)) for k, v in df.items()}
return [sum(tf * idf[w] for w, tf in c.items()) / math.sqrt(sum(c.values()) or 1) for c in tfs]
def pick(sents, scores, budget, guard=False, slack=1.15):
order = sorted(range(len(sents)), key=lambda i: -scores[i])
kept, used = [], 0
for i in order:
if used + len(sents[i]) <= budget:
kept.append(i); used += len(sents[i])
if guard:
covered = set()
for i in kept:
covered |= numbers(sents[i])
for i in order:
ns = numbers(sents[i])
if i in kept or not ns or ns <= covered:
continue
if used + len(sents[i]) > budget * slack:
continue
kept.append(i); used += len(sents[i]); covered |= ns
return kept
def recall(sents, kept, allnums):
got = set()
for i in kept:
got |= numbers(sents[i])
return 100.0 * len(got & allnums) / len(allnums)
t = re.sub(r"\s+", " ", open(sys.argv[1], encoding="utf-8").read())
pct = float(sys.argv[2]) if len(sys.argv) > 2 else 10.0
sents = sentences(t)
sc = centrality(sents)
allnums = set()
for s in sents:
allnums |= numbers(s)
nb = [i for i, s in enumerate(sents) if numbers(s)]
print("chars %d sentences %d numbers %d numeric sentences %d (%.1f%% of text)"
% (len(t), len(sents), len(allnums), len(nb), 100.0*sum(len(sents[i]) for i in nb)/len(t)))
b = pick(sents, sc, len(t)*pct/100.0)
g = pick(sents, sc, len(t)*pct/100.0, guard=True)
print("budget %.0f%%: plain %d sentences, recall %.1f%%" % (pct, len(b), recall(sents, b, allnums)))
print("budget %.0f%%: numbers-first %d sentences, recall %.1f%%" % (pct, len(g), recall(sents, g, allnums)))
On 2401.02954 at a 5 per cent budget it prints 13 sentences and 4.9 per cent recall, then 15 sentences and 7.4 per cent with the numbers-first pass. Same numbers as the table row, which is the point of shipping the script: you can check the claim instead of trusting it.
To score a commercial summariser, paste its output into a file, strip the markup, and compare it against the source's number set with the same function: intersect the numbers in the summary with the numbers in the document, divide by the document's count. Do that on five of your own documents before you buy, because the number that matters is the one for your material, and a marketing PDF and a clinical report fail differently.
Where this measurement stops
Three limits. First, this baseline is extractive: it picks sentences, so every number it keeps is in its original context. A generative summariser can restate a number it half-remembers, which scores the same under this metric while being wrong, so run the same recall check and then read the output for numbers that changed. Second, the metric counts number tokens, not facts: 12 and 12 per cent and 2012 are three different claims and one token family, so a perfect token score is not a perfect document. Third, papers are number-dense by nature. A news article or a contract will score differently, and the only way to know is the script.
The practical use is narrower than it sounds, and more useful. Before you promise a client a one-page summary of a 40-page report, run the numbers check on a sample. If recall at your budget is 10 per cent, say so and add the numbers block. The alternative is a summary that looks complete, reads well, and quietly drops the figure the whole report was about.
Earlier on this site: 1,296 sentences in, 10 out, and the definition never survived, and seven tools that read 100-page documents.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.