Hybrid Retrieval & Reranking

What it is

Hybrid retrieval runs two searches for every query, a keyword search (BM25) and an embedding search, and merges the results. Keyword search is exact: it finds 'Palletfly' or 'GKE' when those literal tokens appear. Embedding search is semantic: it finds 'container orchestration' when the document says 'Kubernetes'. Each misses what the other catches.

Reranking is a second pass over the merged candidates. A cross-encoder model reads the query and each candidate chunk together and produces a relevance score that is far more accurate than comparing two independent vectors. It is too slow to run over the whole corpus, so you use it on the top 20 or so candidates from the first stage.

Together they are the standard upgrade path from naive RAG. This site still runs naive cosine top-k because the corpus is tiny and clean; the upgrade is on the roadmap for the voice agent, where questions are messier because they are spoken.

Why it's needed

Pure embedding retrieval fails on exact terms. Product codes, people's names, acronyms and version numbers all have weak semantic signal, and the embedding model may never have seen them. A user asking 'what is dbr.palletfly.com?' gets a vague match. BM25 finds the exact token instantly.

Pure keyword retrieval fails on paraphrase, which is most real questions. Hybrid gives you both, and reranking fixes the ordering so the chunk that actually answers the question sits first, where the LLM pays the most attention. In most published and practical benchmarks hybrid plus rerank is the single biggest retrieval improvement available without changing the data.

How it works

• Index the chunks twice: a BM25 index over tokenised text and an embedding index over vectors.
• At query time, get the top N from each (N around 20). You now have two ranked lists that partly overlap.
• Merge with reciprocal rank fusion: each chunk's score is the sum over lists of 1 / (60 + rank). It needs no score normalisation and rewards chunks that appear high in both lists.
• Take the fused top 20 and rerank with a cross-encoder such as ms-marco-MiniLM-L-6-v2, which scores (query, chunk) pairs directly.
• Send the reranked top 3 to 5 to the LLM, highest score first.
• Measure recall at k before and after on your test set. If hybrid does not move the number, your corpus may not have the exact-term problem and you can skip the complexity.

Code example

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder

chunks = open("knowledge_base.txt", encoding="utf-8").read().split("\n\n")
chunks = [c.strip() for c in chunks if c.strip()]

# Stage 1a: keyword index
bm25 = BM25Okapi([c.lower().split() for c in chunks])
# Stage 1b: embedding index
embedder = SentenceTransformer("all-MiniLM-L6-v2")
vectors = embedder.encode(chunks, normalize_embeddings=True)
# Stage 2: reranker reads (query, chunk) pairs together
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def hybrid_search(query: str, n_candidates=20, k=5):
    kw_rank = np.argsort(bm25.get_scores(query.lower().split()))[::-1][:n_candidates]
    qv = embedder.encode([query], normalize_embeddings=True)[0]
    vec_rank = np.argsort(vectors @ qv)[::-1][:n_candidates]

    fused: dict[int, float] = {}                 # reciprocal rank fusion
    for ranking in (kw_rank, vec_rank):
        for rank, idx in enumerate(ranking):
            fused[idx] = fused.get(idx, 0.0) + 1.0 / (60 + rank)
    candidates = sorted(fused, key=fused.get, reverse=True)[:n_candidates]

    scores = reranker.predict([(query, chunks[i]) for i in candidates])
    best = np.argsort(scores)[::-1][:k]
    return [chunks[candidates[i]] for i in best]

for c in hybrid_search("what is dbr.palletfly.com"):
    print(c[:100])

Where I used this

Use cases

• RAG over documents full of names, codes, versions and acronyms
• Enterprise search where users mix exact identifiers with natural language
• Voice assistants, where transcription errors make exact-term recovery valuable
• Any RAG system whose evaluation shows the right chunk ranked 4th to 10th
• E-commerce search blending SKU matching with descriptive queries

Common mistakes

• Normalising and adding raw BM25 and cosine scores. They live on different scales; use rank fusion instead.
• Reranking the whole corpus. Cross-encoders cost a forward pass per pair; keep them to the top few dozen candidates.
• Skipping tokenisation details for BM25. Lowercasing and splitting on whitespace is fine for English prose but breaks on code, URLs and hyphenated names; test on your own data.
• Adding hybrid search before building an evaluation set, so you cannot tell whether it helped.
• Ignoring latency. A reranker adds tens of milliseconds on CPU per query; fine for chat, worth measuring for voice.

Interview questions