Reranker Guide 2026: The RAG Upgrade That Improves Retrieval Without Retraining
Top-k retrieval puts the right passage at rank 9 and the model never sees it. A reranker fixes the ordering in a few lines of code, no re-indexing. This guide covers the two-stage setup, model choice, parameters, evaluation and pitfalls.
📜 Table of Contents
Reranker Guide 2026: Better RAG Retrieval Without Retraining
If the correct passage lands at rank 9, the model never sees it. Changing embedding models means re-indexing everything; adding a reranker means changing a few lines. Recall stays the same, ordering gets fixed.
Why vector search misranks
A bi-encoder embeds query and document separately, so similarity measures topic overlap, not answerability. A glossary entry defining "refund" can outrank the actual refund policy.
| Type | Encoding | Cost | Accuracy | Precomputable |
|---|---|---|---|---|
| Bi-encoder | separate | low | medium | yes |
| Cross-encoder | joint | high | high | no |
They are two stages of one pipeline, not alternatives.
Two-stage flow
Retrieve 50-100 candidates with vectors, score each pair with a reranker, keep the top 3-8 for the prompt. Recall optimises for "nothing missed"; reranking optimises for "nothing wrong on top".
Model options
| Model | Deployment | Notes |
|---|---|---|
| bge-reranker-v2-m3 | self-hosted | multilingual, solid Chinese, runs on CPU in small batches |
| bge-reranker-base/large | self-hosted | lighter when VRAM is tight |
| Cohere Rerank | hosted API | no ops, per-call pricing |
| Jina Reranker | hosted or self-hosted | good for long documents |
| Qwen3-Reranker | self-hosted | larger models when you have GPUs |
| LLM-as-judge scoring | any LLM | most expensive, small candidate sets only |
Prefer self-hosting: reranker calls scale with query volume, so per-call pricing adds up fast.
Minimal code
from FlagEmbedding import FlagReranker
reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)
def rerank(query, candidates, top_n=5):
scores = reranker.compute_score([[query, c["text"]] for c in candidates],
normalize=True, batch_size=8)
ranked = sorted(zip(scores, candidates), key=lambda x: x[0], reverse=True)
return [c for _, c in ranked[:top_n]]
Parameters
| Parameter | Start at | Effect |
|---|---|---|
| recall_k | 50-100 | too small leaves nothing to rank; too large adds latency |
| top_n | 3-8 | more crowds the context |
| batch_size | 8-32 | VRAM vs throughput |
| max_length | 512 | longer text needs chunking first |
Tune in that order: widen recall, then narrow top_n, then consider a score threshold.
Proving it works
Offline: 50-100 real queries with labelled answer passages, compare recall@5, nDCG@10 and MRR before and after. Online: citation hit rate, follow-up rate, and a 30-sample manual review.
Pitfalls
- Treating scores as probabilities.
- Duplicate chunks surviving reranking; dedupe first.
- Ignoring latency: 50 candidates is tens of ms on GPU but hundreds on CPU.
- Truncation at max_length silently dropping the answer.
FAQ
Is a GPU required? No. Small and mid-size models run on CPU at lower throughput.
Does it conflict with hybrid retrieval? No, combine BM25 plus vectors for recall, then rerank once.
Should I fine-tune? Get a baseline with a general model first; fine-tune only with labelled data and very domain-specific vocabulary.
How many to keep? Start at 5. Use 8 for multi-passage answers, 3 for single-fact questions.
Model names and pricing change; check official model cards and pricing pages.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.