Hybrid search and reranking

The big idea

Vector search finds meaning: "money back" matches "refund". But it is fuzzy about exact strings: error codes, product names, part numbers and rare words. Keyword search with BM25 is the opposite: precise on exact terms, blind to paraphrase. Each fails where the other succeeds, so production retrieval usually runs both and merges the results: hybrid searchRetrieval that runs keyword search (such as BM25) and vector search for the same query and merges the two result lists..

Merging gives a good candidate list, maybe 20 to 100 chunks, but its order is still rough, and only the top few will fit in the prompt. A rerankerA model that scores how well each retrieved candidate answers the query, reading query and candidate together, and reorders the candidates by that score. reads the query together with each candidate and scores how well that candidate answers it, a much more careful judgement than comparing two vectors. It is too slow to run over the whole corpus, but fast enough for the top candidates.

Think of hiring. A recruiter screens thousands of CVs quickly with keywords and gut feel, two recruiters' shortlists are merged, and then a hiring manager actually interviews the top 20. The screening must not throw out the right person, and the interview must put them first.

How it works

Why both searches

• Keyword wins: "E1042", "SKU 88-410", a person's name, a function name, a rare technical term. Embeddings see these as a few meaningless fragments, and near-identical codes look alike ( Semantic search basics ).
• Vector wins: paraphrases and vague questions. "My package hasn't shown up" matches a page titled "Track your parcel".

Both indexes are built from the same chunks, so the pipeline from Document ingestion and chunking feeds them together.

Reciprocal rank fusion

The two searches produce scores on unrelated scales: BM25 scores are unbounded, cosine similarities sit in a narrow band. Adding them directly lets one dominate. Reciprocal rank fusionA way to merge ranked lists: each item scores the sum of 1 / (k + rank) over the lists it appears in, so only ranks matter, not raw scores. ignores raw scores and uses only positions:

With the commonly used $k=60$, a chunk ranked 1st in one list and 10th in the other scores $1/61+1/70=0.0307$, beating a chunk ranked 2nd in one list and absent from the other, $1/62=0.0161$. Appearing high in both lists is rewarded; a single strong vote counts less. A smaller $k$ gives more weight to top positions. RRF needs no tuning and no score calibration, which is why it is the default in many search engines. Weighted score blending can do better when tuned carefully, but it is sensitive to outliers and needs re-tuning when either search changes.

Bi-encoders and cross-encoders

Vector search uses a bi-encoderA model that embeds the query and each document separately into vectors, so document vectors can be computed in advance and compared with a query by a fast distance.: query and document are embedded separately, so document vectors can be computed in advance and compared in microseconds. The cost is that the two texts never see each other; everything about a document must be squeezed into one vector before anyone knows the question.

A cross-encoderA model that reads the query and a candidate document together in one pass and outputs a relevance score, more accurate than comparing separate embeddings but too slow to run over a whole corpus. reads the query and the candidate together in one pass, with attention between every query token and every document token, and outputs a relevance score. It notices that "after the 14-day window" changes which refund paragraph matters. It is far more accurate, and far slower: one forward pass per candidate per query. So it runs on the top 20 to 100 candidates only. An LLM prompted to judge relevance can play the same role, slower and more expensive still.

The reranker cannot find what retrieval missed

Reranking only reorders the candidates it is given. If the right chunk is not in the first-stage list, no reranker can rescue it. So tune stage 1 for recall (is the answer somewhere in the top 50?) and stage 2 for precision (is it in the top 5?). Measure both separately ( Evaluating RAG ).

Try it

The rankings and reranker scores are hand-made to illustrate typical behaviour.

Try this:

1. Pick "keyword only". The "14-day free trial" chunk ranks first because it matches "14-day" exactly, and the chunk that answers the question sits at position 6: it never reaches the model.
2. Pick "hybrid (RRF)". The trial chunk drops to 4th because vector search ranked it last, but the answer is still 6th. Fusion fixed one error, not the important one. Try k = 1 and see the trial chunk climb back to 2nd: small k trusts top positions more.
3. Turn the reranker on with 8 candidates. Reading the query and each chunk together, it moves the answer to 1st. Now lower "candidates reranked" to 5. The answer is outside the reranked set and stays 6th: a reranker cannot rescue what the first stage ranks too low.

Going deeper

Rank fusion versus score blending, in code

# Fusing a keyword ranking and a vector ranking: reciprocal rank fusion vs score blending.
bm25 = {"E": 7.9, "A": 6.1, "G": 5.8, "C": 4.0, "F": 3.2, "B": 2.9, "D": 1.1, "H": 0.4}         # unbounded
cosine = {"A": 0.83, "G": 0.81, "D": 0.79, "C": 0.78, "B": 0.78, "F": 0.74, "H": 0.71, "E": 0.52}  # squeezed

def ranks(scores):
    return {d: r for r, d in enumerate(sorted(scores, key=scores.get, reverse=True), start=1)}

def rrf(*rankings, k=60):
    """Each list votes 1 / (k + rank); raw scores are ignored, so scales do not matter."""
    fused = {}
    for r in rankings:
        for d, rank in r.items():
            fused[d] = fused.get(d, 0) + 1 / (k + rank)
    return sorted(fused, key=lambda d: (-fused[d], d))      # ties broken by name

def blend(a, b, w=0.5):
    """Min-max normalise each score list, then mix. Sensitive to outliers and to w."""
    norm = lambda s: {d: (v - min(s.values())) / (max(s.values()) - min(s.values())) for d, v in s.items()}
    na, nb = norm(a), norm(b)
    return sorted(a, key=lambda d: w * na[d] + (1 - w) * nb[d], reverse=True)

print("keyword  :", "".join(ranks(bm25)))
print("vector   :", "".join(ranks(cosine)))
print("RRF k=60 :", "".join(rrf(ranks(bm25), ranks(cosine))))
print("RRF k=1  :", "".join(rrf(ranks(bm25), ranks(cosine), k=1)))
for w in (0.3, 0.5, 0.7):
    print(f"blend w={w}:", "".join(blend(bm25, cosine, w)))

keyword  : EAGCFBDH
vector   : AGDCBFHE
RRF k=60 : AGCEDBFH
RRF k=1  : AEGCDBFH
blend w=0.3: AGCBDFHE
blend w=0.5: AGCBFEDH
blend w=0.7: AGECBFDH

RRF reproduces the lab's fused order from ranks alone. Blending normalised scores gives a different order for every weight, and BM25's outlier top score for the trial chunk (E) drags it up as the keyword weight grows. Blending can be tuned to beat RRF on a given dataset, but RRF is the robust default.

Choosing how many to rerank

Reranking cost grows linearly with the number of candidates, and so does latency, though candidates can be scored in parallel batches. Measure first-stage recall at 20, 50 and 100: if recall@50 is already close to recall@100, reranking 50 is enough. A typical result is a sharp improvement in the top 5 for a few tens of milliseconds to a few hundred, depending on the model and hardware.

Score thresholds

Rerankers output a relevance score you can threshold: if even the best candidate scores low, the corpus probably does not contain the answer, and the system can say so instead of passing weak context to the model. Calibrate the threshold on your evaluation set.

Trade-offs

Common mistakes

• Adding BM25 scores and cosine similarities directly, letting one scale swamp the other.
• Reranking too few candidates, so the right chunk never gets a chance.
• Judging the reranker without checking first-stage recall.
• Running a cross-encoder over the whole corpus instead of the top candidates.
• Keeping only one index after migrating, and losing exact-match queries such as error codes.

In the wild

Check yourself

Recap