Self-Hosted AI Tutor: 47 of 60 Right With BM25 Retrieval

🔧 AI Tools 2026-11-18 6 min read

A tutoring assistant that answers from model memory sounds fluent, quotes nothing, and is wrong two times out of three. Nothing in the reply tells the student which kind of answer they got, and a source line cannot be added after the fact.

📜 Table of Contents

Same sixty questions, same model, same temperature. With four retrieved passages in front of it, the model answered 47 of 60 correctly. Asked from memory, 21. The second number is the one that matters for a tutoring product, because a student cannot tell which kind of answer they just got.

This is the smallest AI learning platform that is worth calling one: a keyword index over the course material and a prompt that forbids answering from anywhere else. Two thousand and sixty-seven passages, forty-eight articles, ten thousand five hundred and seventy questions in the source set, sixty sampled for the test.

The parts, and what they cost to build

Corpus: the SQuAD dev split, Wikipedia paragraphs with a gold answer span and a gold paragraph for every question. Median paragraph is 700 characters, which is a workable unit for retrieval and small enough to fit four per prompt.

Index: BM25 written in plain Python, no dependencies. 2,067 passages, 22,892 distinct terms, 0.32 seconds to build and 4.9 milliseconds per query on one core. A 300-student course costs nothing to index and nothing to re-index when the reading list changes.

Generation: temperature 0, one call per question, four passages labelled [1] to [4], instruction to answer on line one with a short copied phrase and cite the passage number on line two, or write NOT_IN_CONTEXT. Scoring: exact match and token F1 against the gold spans, plus a second call that judges whether the answer states the same fact as any acceptable answer.

What retrieval buys

passages retrieved, answers citedmodel memory, no passages
exact match61.7%15.0%
token F10.7480.303
judged correct47 / 6021 / 60
answers withheld10 / 600 / 60
input tokens per question72838
output tokens per question8.33.0
latency1.4 s1.3 s

Retrieval lifts exact match from 15% to 62%, and the difference is not only accuracy: ten answers were withheld instead of guessed. Seven of those ten were questions whose gold passage never reached the top four, so abstaining was the correct behaviour, not a failure.

Among the fifty questions answered, F1 was 0.891 and 47 were judged correct. Of the three misses, one was a question whose passage never arrived; the model answered anyway and got it wrong, which is exactly the failure the abstention line is there to prevent.

Where the retrieval loses

Gold passage ranked first for 78.3% of questions, in the top four for 86.7%. Eight misses out of sixty, and they cluster where BM25 always loses: questions that name a relationship rather than a word. One asked about a marriage described as happy and successful; the gold paragraph calls it a clerical marriage, and shares no term with the question except marriage. Lexical search has no way in.

Four guesses out of five land in the top slot, which is why the naive index is worth shipping before anyone reaches for embeddings. A dense retriever would very likely fix the clerical-marriage case and add a hosting dependency, a re-embedding step every time the syllabus changes, and a bill. The measurement here can tell you which of those you actually need: keep eight misses per sixty questions as the baseline a vector index has to beat.

Citations, the part teachers check

Of the fifty answers given, forty-seven cited the passage that held the gold span. Three cited a neighbouring passage that was retrieved but did not contain the answer. In a classroom tool this is the only metric a teacher will verify, and 94% is good enough to put a source line under every answer, provided the source line is clickable and the passage is one paragraph, not one book.

Money, honestly

per 1,000 questions, at $0.30 in / $1.20 out per million tokenscost
retrieval, peak hours$0.228
retrieval, off-peak$0.114
model memory, peak hours$0.015

Per thousand questions, retrieval costs fifteen times more than memory. Per thousand correct answers it still costs more: 29 cents against 4.3 cents, because the closed-book run gets a third of the answers right for almost nothing. If the only metric were cost per correct answer, retrieval loses.

What the money buys is the ten withheld answers and the source line under the other fifty. A tutor that is wrong two thirds of the time and never says so is cheaper and unusable. At 0.228 dollars per thousand questions, a class of two hundred students asking five questions a week costs about a cent a week; the bill never becomes the argument.

The prompt that does the work

Plain instruction, no chain of thought, no examples:

Answer from the passages only. Line 1: the answer phrase, as short as possible, copied from a passage. Line 2: the passage number in brackets. If the answer is not in the passages, line 1 is exactly NOT_IN_CONTEXT.

The two lines matter. Asking for a copied phrase forces the model to locate the answer rather than paraphrase around it, which raises exact match and makes the citation checkable. Asking for the passage number makes the answer auditable without a second model. Asking for NOT_IN_CONTEXT as a first-class output gives the system a way to say the reading list does not cover this, which is the most useful thing a tutor can say to a student.

The index, as run

import math, re

STOP = set("a an the of in on at to for and or is are was were be it its this with as by from".split())

def tokens(s):
    return [w for w in re.findall(r"[a-z0-9]+", s.lower()) if w not in STOP]

class BM25:
    def __init__(self, docs, k1=1.5, b=0.75):
        self.k1, self.b = k1, b
        self.d = [tokens(x) for x in docs]
        self.len = [len(t) for t in self.d]
        self.avg = sum(self.len) / len(self.len)
        self.tf = [{w: t.count(w) for w in set(t)} for t in self.d]
        self.df = {}
        for t in self.d:
            for w in set(t):
                self.df[w] = self.df.get(w, 0) + 1
        self.N = len(self.d)

    def top(self, query, k=4):
        qt = tokens(query)
        out = []
        for i, tf in enumerate(self.tf):
            s = 0.0
            for w in qt:
                if w in tf:
                    idf = math.log(1 + (self.N - self.df[w] + 0.5) / (self.df[w] + 0.5))
                    s += idf * tf[w] * (self.k1 + 1) / (tf[w] + self.k1 * (1 - self.b + self.b * self.len[i] / self.avg))
            out.append((s, i))
        out.sort(reverse=True)
        return [i for _, i in out[:k]]

Retrieval is a linear scan, on purpose: 4.9 milliseconds is not a bottleneck for a class, and a scan has no index to corrupt. Swap in a vector store when the reading list passes a few thousand pages, not before.

Limits

Sixty questions, one model, one retriever. SQuAD questions were written by humans looking at the passage, so they share vocabulary with the answer paragraph, which flatters lexical search; real student questions are messier and recall@4 would drop. The corpus is Wikipedia prose, not lecture notes, PDFs with formulas, or slides, where paragraph boundaries are worse and clean text extraction is half the work.

The judge is the same model family as the answerer, so agreement is not independent evidence. Exact match and F1 are the harder numbers here and they point the same way. No reranker, no query rewriting, no chunk overlap beyond the paragraph: each of those would move recall and each costs a component to run.

The same lesson, measured on contracts instead of course readings, is in clause maps versus whole documents: keyword retrieval cut tokens 83% and lost eight clauses of thirty-two, because the prefilter searched for the wrong vocabulary. The existing walkthrough of free platforms, and where self-hosting stops being free, is the free AI learning platform list. If the thing you actually need is question generation rather than answering, start from the quiz maker that costs nothing.

Take away one line: four BM25 passages and one instruction about NOT_IN_CONTEXT took the same model from 21 correct out of 60 to 47, for 0.23 dollars per thousand questions.

Related Articles
2026-08-28
Best AI Landscape Design Tool in 2026: 6 Free Tools to Plan Your Backyard Before You Dig
2026-07-24
Best AI Document Processing Tools 2026 Extract Analyze Automate
2026-11-22
Best OCR API for Accounts Payable: $10 vs $100 per 1,000
2026-07-24
AI Image Generator Free Online No Sign Up
2026-09-27
Best AI SOAP Notes Tool for Physical Therapists in Private Practice (2026)
2026-08-22
Best AI Productivity App in Chiang Mai 2026: 6 Tools for Digital Nomads

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment