Clause Maps Cut Contract Review Tokens 83% and Lost 8 of 32 Clauses

🔧 AI Tools 2026-11-18 7 min read

A clause-retrieval layer looks free in a demo because the demo contract has a heading that names the clause. Real contracts write the cap as will not be liable under any circumstances, the keyword list finds nothing, and the call quietly answers from the first two sections.

📜 Table of Contents

Two ways to ask a model about a contract. One pastes the whole document, the other picks the sections that mention the clause and asks about those. On seventy real clause questions the second used 83% fewer input tokens and got five fewer clauses right. Both numbers matter, and the second one is the one nobody puts in the demo.

The test set is CUAD, five hundred and ten public contracts with forty-one clause categories marked sentence by sentence. I took fourteen contracts between 8,852 and 40,590 characters, median 27,025, and asked five clause questions per contract: governing law, audit rights, cap on liability, insurance, non-compete. Seventy questions, thirty-two of them with the clause genuinely present.

What each architecture sends

The whole-contract version sends the full text, an average of 26,002 characters, with one instruction: does this contract contain a governing law clause, answer as JSON, quote the sentence.

The mapped version splits the contract into blocks of about 900 characters at section breaks, scores each block by how often that clause's keyword list appears, keeps the top five blocks plus the block after each, and sends roughly 3,864 characters. Same question, same output format, temperature 0, one call per clause.

Accuracy against token count

whole contractkeyword-mapped excerpts
clauses judged correct66 / 7061 / 70
accuracy94.3%87.1%
clauses present, found29 / 3224 / 32
false positives1 / 381 / 38
mean prompt tokens per question5,344906
prompt tokens per contract (5 clauses)26,7214,528
mean latency0.9 s0.8 s

The accuracy gap is small enough to look like noise and it is not: twenty-nine of thirty-two present clauses found, against twenty-four. The token gap is the one you notice on an invoice.

What it costs per thousand contracts

Measured prompt and completion tokens, priced at the deepseek-flash rates on the official pricing page: $0.30 per million input tokens and $1.20 per million output at peak, half of that off-peak.

per 1,000 contracts, 5 clauses eachwhole contractmapped excerpts
peak hours, $0.30 in / $1.20 out per million tokens$8.20$1.52
off-peak, $0.15 in / $0.60 out per million tokens$4.10$0.76

Five clause questions per contract is a shallow review. Run forty instead and the whole-contract column lands near $65 per thousand at peak, the mapped column near $12. Neither number decides the architecture. The eight clauses in the next section do.

The eight clauses the map lost

Every miss has the same shape: the keyword prefilter found nothing, the caller fell back to the first two blocks of the document, and the clause sat somewhere else. Not one miss came from the model misreading a section it was given.

Three examples, quoted from the CUAD labels:

An agency agreement caps damages as "In no event will the Company be liable for incidental or consequential damages." No word "cap", no "exceed", no "limitation of liability". The keyword list had nothing to hit.

A strategic alliance says "This Agreement shall be governed by the internal laws State of New York." The list held "governed by the laws", not "governed by the internal laws". One adjective, one missed clause, on the easiest clause type in the set.

A web services agreement limits liability as "i-on will not be liable under any circumstances for any lost profits". That phrasing sat in the limitation-of-liability list, which this test did not run, and not in the cap list, which it did.

Keyword lists built from clause names describe how a taxonomy labels clauses. Contracts describe the same clause as what the parties may not recover. The gap between those two vocabularies is the whole failure mode.

Where the architecture holds

clause typecontracts with itwhole contractmapped excerpts
Governing Law14 of 141413
Audit Rights8 of 1476
Cap On Liability6 of 1441
Insurance4 of 1444
Non-Compete0 of 14--

Governing law and insurance survived the map: short, phrased in their own words, in one place. Cap on liability collapsed from four of six to one of six, because a cap is usually a carve-out inside a longer damages paragraph and reads as a remedy rather than a limit.

Non-compete came up empty in this sample, which says nothing about the method and something about sampling fourteen contracts. Read that row as untested rather than as a win.

The quotes were fine

Of the twenty-nine clauses the whole-contract run detected, all twenty-nine returned a quoted sentence matching a CUAD gold span, median token overlap 1.0. The mapped run: twenty-four of twenty-four. Detection fails before citation does, the opposite of what most reviews warn about.

One false positive appeared in both runs, same contract, same clause type. The model quoted a sentence that does contain the word insurance, and CUAD labels no insurance clause there. That is a labelling judgement about what counts as a clause, not a fabrication. Some of your own disagreements will have the same shape.

A cascade that keeps the accuracy

Send the mapped excerpts first. When the model answers "present", take it. When it answers "absent", resend the whole contract and take that answer. Nothing else changes.

whole contractmapped excerptsmap, escalate on "absent"
clauses judged correct66 / 7061 / 7066 / 70
clauses present, found29 / 3224 / 3229 / 32
prompt tokens, all 70 questions374,09263,385296,037

Same seventy clauses right as the expensive path, 20.9% fewer tokens. The saving is small because cap-on-liability is absent from most contracts, so the escalation fires often. A question set weighted toward governing law and parties would save far more; one made only of damages clauses would save almost nothing.

Rules worth copying

Skip the map under about 3,000 tokens. At that size the whole text costs less than the code that trims it.

Write keyword lists from remedy language, not clause names: "shall not be liable", "in no event", "will not exceed", "shall have the right to audit", "shall be governed by the laws of". Then run the list over twenty of your own contracts and count the clauses it fails to hit. In this sample a four-term list per clause type missed eight of thirty-two; a longer list raises recall and dilutes the ranking, which is why escalating on "absent" beats tuning keywords.

Log the retrieved blocks with every answer. When a clause is missed, the log tells you within a second whether the prefilter or the model dropped it, and those two failures need opposite fixes.

The code, as run

import re

def blocks(text):
    parts = re.split(r"(?<=[.;:])\s*
|
\s*
|
(?=\s*\(?[0-9ivxIVX]+[.)]\s)|
(?=[A-Z][A-Za-z ]{2,38}\.?\s*$)", text)
    out, buf = [], ""
    for p in parts:
        buf += " " + p if buf else p
        if len(buf) > 900:
            out.append(buf.strip()); buf = ""
    if buf.strip():
        out.append(buf.strip())
    return [b for b in out if len(b) > 40]

def excerpts(text, keywords, k=5):
    bs = blocks(text)
    scored = sorted(((sum(b.lower().count(w) for w in keywords), i)
                     for i, b in enumerate(bs)), reverse=True)
    keep = set()
    for score, i in scored[:k]:
        if score == 0:
            break
        keep.update((i, i + 1))
    if not keep:
        keep = {0, 1}
    return "

".join(bs[i] for i in sorted(keep) if i < len(bs))

The prompt asked for one JSON object: whether the clause is present, and one verbatim sentence from the text supplied. Temperature 0 and a 300-token output cap kept completions at 2,161 tokens across seventy answers.

Limits of this test

Fourteen contracts, one model, one temperature, five of CUAD's forty-one categories. The contracts are US public filings from 1999 to 2014, which is why the typography is dated and the phrasing wordy; modern template contracts would likely score higher on both paths. Each question asked about one clause in isolation, so the run says nothing about extracting all forty-one categories at once, where one batch call per contract would change both the token math and the accuracy.

The labels are also strict in a way worth knowing before quoting these numbers: CUAD marks a clause present only where a reviewer highlighted a span, so a contract that mentions insurance without an obligation to carry it counts as absent, and a model that notices the mention counts as wrong.

Two older pieces in this cluster are worth reading next. The page-level map of what a review workflow should hand a model, clause by clause, is the clause page map. The free options and where each one stops are in five free contract review tools. For the non-technical version of the same workflow, AI for lawyers covers intake, privilege, and what to keep off a vendor's server.

Take away one line: keyword retrieval cut the bill by 83% and cost five clauses out of seventy; escalating on "absent" gave the same seventy answers as the expensive path for 21% less.

Related Articles
2026-08-30
Best AI Bookkeeping Tool for Freelance Artists 2026: 6 Tools for Invoices and Taxes
2026-08-18
AI Voice Cloning Free 2026: 6 Tools That Clone Ethically (With Consent)
2026-08-17
AI Quiz Generator From Notes Free 2026: 6 Tools That Turn Notes Into Tests
2026-08-18
Best AI Travel Planner in Auckland 2026: 6 Tools for NZ Road Trips and Campervan Tours
2026-07-26
Anker AI Note Taking Earbuds Review
2026-08-21
Best AI Form Builder in 2026: Typeform vs Jotform vs Tally Compared for Lead Generation

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment