Summary Tool Test: 1,296 Sentences In, 10 Out

A ten-sentence summary that opens on a stray citation marker, keeps a wiki edit button, and never defines the subject. The lede was welded to the infobox at extraction, and the ranker only picks long sentences.

💡 What You Will Learn

A ten-sentence summary that opens on a stray citation marker, keeps a wiki edit button, and never defines the subject. The lede was welded to the infobox at extraction, and the ranker only picks long

📜 Table of Contents

• 1. How the test was run
• 2. Four algorithms on one document
• 3. Why the definition sentence never got picked
• 4. Deleting 41.6% of the text changed nothing
• 5. Three chunks instead of one pass: 11 numbers become 22
• 6. Three errors you will hit reproducing this
• 7. The PDF route, and when to switch to an LLM
• Related

The ten-sentence summary opens with "[66] These are trained to generate step-by-step analysis before producing final answers." No subject. Somewhere in the middle there is a fragment starting with "because", then "Stereotyping [edit]" — a leftover wiki edit button. Across ten sentences, not one explains what a large language model is.

Nothing was installed wrong. The input arrives broken: on extraction, the article's lede definition was glued to the infobox, producing a 47-word sentence stitched together with pipe characters. LSA-style algorithms score sentences by length and term frequency, and all ten selected sentences run 23 to 29 words, against a median of 8 words across those 1,296 sentences.

1. How the test was run

Source: the English Wikipedia article "Large language model", HTML captured on 2 November 2026, 1,033 KB. Machine: 2 vCPU Intel Xeon Platinum, 1 GB RAM, Python 3.11.6, trafilatura 2.3.1, sumy 0.13.0, pypdf 6.21.4, numpy 2.4.6, nltk 3.10.4.

Extraction used trafilatura.extract with defaults, sentence splitting used sumy's bundled nltk tokenizer, and every summarizer was asked for 10 sentences. The stack runs offline, needs no API key and costs nothing — which is what most people searching for a summary tool will actually end up with.

Measured extraction: 93,521 characters, 13,601 words, 1,296 sentences, 0.49 to 0.73 seconds across two runs. Turning include_tables off removed only 153 characters, so the page's tables never reach the body text either way.

2. Four algorithms on one document

Numbers kept is the share of the document's 1,681 numeric tokens that survive into the summary; none of the four cleared 1%. Scaling was measured too: at 100 sentences LSA takes 0.16 s, LexRank 0.12 s, TextRank 0.07 s; at 300 sentences, 0.52, 1.44 and 0.93 s. At 1,296 sentences LexRank's 13.97 s is 116 times its 100-sentence time for 13 times the input — growth is clearly non-linear.

3. Why the definition sentence never got picked

In the extracted text the definition reads: "Large language model | Part of a series on | Machine learning and data mining | A large language model (LLM) is a machine learning model ... trained on a vast amount of text." The infobox heading and navigation got welded into the same sentence: 47 words, sliced by pipe characters.

All ten sit above the 90th percentile. The algorithm picks long sentences, and the definition lost its sentence shape at extraction time, so it never had a chance. By the time the summarizer takes over, the damage is already in the input.

4. Deleting 41.6% of the text changed nothing

Of the 93,521 extracted characters, everything from character 54,604 onward is the reference list: 38,917 characters, 41.6% of the document, holding 1,283 of the 1,681 numeric tokens. The body keeps 398. The obvious move is to trim it before summarizing, so that was measured:

Four point six times faster, byte-identical output. The reference list was never selected and never distorted the selection, so trimming bought time and nothing else. Worth recording, because a lot of guides lead with "clean the references before summarizing" — on this document that cleaning had zero effect on what came out.

5. Three chunks instead of one pass: 11 numbers become 22

One pass over the full text produced 1,834 characters and 11 numbers. Three chunks of 432 sentences, four sentences each, took 2.2 seconds and produced 1,863 characters with 22 numbers. Nearly the same length, twice the coverage, and the only lever in this test that moved the result. The cost: broken context at chunk boundaries, duplicate sentences to remove by hand, and citation-template residue leaking into the last chunk.

6. Three errors you will hit reproducing this

pip install trafilatura sumy pypdf numpy
python -c "import nltk; nltk.download('punkt_tab')"

Error strings, verbatim:

1. LSA without numpy: ValueError: LSA summarizer requires NumPy. Please, install it by command 'pip install numpy'.
2. Missing tokenizer data: LookupError: Resource 'punkt_tab' not found. — sumy then suggests python -c "import nltk; nltk.download('punkt_tab')".
3. Chinese text: ValueError: Chinese tokenizer requires jieba. Please, install it by command 'pip install jieba'.

The second one cost an extra round: nltk.download hung for seven minutes reaching GitHub from this machine, so punkt_tab.zip came from jsdelivr and was unpacked into /root/nltk_data/tokenizers. Wherever the official nltk data origin is blocked, that detour is mandatory.

7. The PDF route, and when to switch to an LLM

arXiv 2307.06435 as a PDF: 1.81 MB, 47 pages, 277,262 characters, 42,704 words, 5,335 numeric tokens, extracted by pypdf in 2.92 seconds, no page below 100 characters. The number count is misleading — tables come out as rows of interleaved spaces and line breaks, and a summarizer turns that into loose digits with no labels. Documents with many tables are better converted to Markdown first.

The trade-off: extractive summarizers never rewrite, and 8 of the 10 output sentences could be traced back to the source by their first 60 characters, so they are verifiable and will not invent a number — they omit. Cross-section synthesis and plain language need an LLM, which brings fabrication risk. The workable order is extractive first for a verifiable skeleton, then an LLM rewrite on top of it, then every number checked against the source. For documents of a few dozen pages, drop O(n²) algorithms like KL outright; on a 1 GB box LSA was the only one returning in seconds.

Related

PDF to Markdown extraction compared: Docling, Marker and MinerU
Chunking and retrieval pipelines: rebuilding RAG after LangChain 1.x
Whole-book summarizers: AI book summary generators
Meeting-recording summaries: local options for Zoom