# Hybrid search didn't help. It still fixed my retrieval.

> I added hybrid search and a cross-encoder reranker to my RAG service, wrote up what improved, then ran the ablation I'd been avoiding. The pipeline had contributed nothing directly — but its 512-token limit forced a chunk-size change that did everything.

Published: 2026-08-12
Author: Vishal Bhargava
Canonical URL: https://vbhargava.org/writing/hybrid-search-ablation/
Tags: rag, retrieval, evals, inference-cost, LLM
Series: ritam

---
I added hybrid search and a cross-encoder reranker to my RAG service, measured it, and wrote up four findings. Then I ran the one comparison I'd been putting off, and three of the four didn't survive.

The pipeline I'd spent weeks on had contributed nothing measurable. Every gain, every regression, and the entire cost reduction belonged to a second variable — one I hadn't chosen so much as inherited, because adding the reranker is what forced it.

That second part is why I'm writing this up rather than quietly reverting.

> **Note:**
> This carries on from [the first post](/writing/llm-lab-rag-v1/), but you don't need to have read it — I explain the terms as they come up.
> That post was a walkthrough of the smallest RAG system that still behaves like a real service. This one is less about the pipeline and more about the harder question underneath it: how do you actually know whether a change you made helped?

## Where this picks up

The last post was January. It's August.

Some of that gap is honest and some of it isn't. I moved from Berlin to Dubai, and the tail of a move is longer than the move — residency, a driving licence, a one-year-old adjusting to a new flat, a stretch of health stuff I hadn't planned for. I'd built the whole of v1 in about twenty days and drew exactly the wrong conclusion from that: that this was my speed. It wasn't. It was a burst.

What actually happened is that the building never stopped, it just slowed to a few hours on a good week — thirty-odd commits since February. The publishing stopped. Those are different failures and I'd rather be honest about which one I committed.

The project also got renamed. `llm_lab` was accurate when it was a lab; it stopped being accurate somewhere around the point it grew a regression gate. It's [ritam](https://github.com/bhvishal9/ritam) now.

## The system, and how to read its numbers

The first post described a service that answers questions from a set of documents, and the shape hasn't changed: documents get cut into **chunks**, each chunk becomes an embedding (A list of numbers representing a piece of text, produced by a model, positioned so that text with similar meaning lands nearby. Comparing embeddings is how the system finds relevant chunks without matching words literally.) in a vector database ([Qdrant](https://qdrant.tech/), now, rather than a JSON file), and a question retrieves the nearest ones, which go into the prompt (The full block of text sent to the model: the question, the retrieved chunks, and the instruction to answer only from those chunks. Everything in it is billed as input.) for the model to answer from.

What's new since v1 is that I can now tell whether a change to that pipeline helped — a harder problem, it turns out, than building the pipeline was. It's a 78-question eval set over a deliberately fictional corpus (Made-up facts, on purpose: seven documents about a duck civil war. On real-world topics the model can answer from what it absorbed in pretraining, so a correct answer tells you nothing about whether your retrieval worked. Fiction forces every correct answer to come through the retrieval path.), and the questions come in three groups that almost everything below depends on:

- **factual** — the answer sits in one document.
- **multi-hop** — the answer needs two documents combined, and both have to come back or the question fails.
- **out-of-scope** — the corpus genuinely can't answer it, and the right behaviour is to say so.

Three numbers run through the rest of this: recall@3 (Of the questions that do have an answer in the corpus, how often the right document came back in the three chunks returned. Three is the top-k — return more and you have a better chance of catching the answer, and a proportionally larger prompt to pay for. Higher is better.), abstention (Of the questions the corpus genuinely cannot answer, how often the system correctly returned nothing at all, rather than confidently retrieving something irrelevant and answering from it. Higher is better.), and coverage (How often anything at all came back, across every question. Low coverage means the system has gone quiet — which flatters abstention and wrecks recall at the same time, so it's the number that catches a system for refusing too much.).

The first two pull against each other. Loosen the filter and you answer more, including things you should have refused; tighten it and you refuse more, including things you could have answered. Most of this post is about that trade, and about how easy it is to think you've improved it when you haven't. A regression gate (An automated check that re-runs the eval set on every code change and refuses the change if any metric dropped below the recorded baseline. A unit test for quality rather than correctness — with the caveat that it can only compare against a number you previously agreed to.) runs the set in CI on every push and fails the build if any of the three drops.

## Why I wanted hybrid search

In June I swept 33 configurations — two embedding models, three chunk sizes, a range of similarity thresholds. 2,574 queries, $0.84 of API spend, no errored rows.

The result that mattered wasn't the winner. It was that **there was no good operating point at all.** Nothing in 33 cells reached recall above 0.8 and abstention above 0.3 together. The best compromise got recall 0.841 at abstention 0.133 — which reads fine until you translate it: for 87% of the questions my corpus genuinely couldn't answer, the system confidently retrieved something anyway.

The reason is in the scores. I was using one cutoff on a similarity score to decide what counts as relevant, but relevant and irrelevant chunks share a band, roughly 0.74 to 0.88. One cutoff can't separate two overlapping distributions. That's architectural, not a tuning problem.

The standard answer is to stop treating the retrieval score as the relevance score: retrieve generously, then re-score the shortlist with a cross-encoder (Also called a reranker. Vector search compares two texts that were each summarised into an embedding separately and never seen side by side — fast, and lossy. A cross-encoder reads the question and the chunk together as one input and scores the match directly. Far more accurate, and far too slow to run across a whole index, so it goes on a shortlist.) — a model that reads the question and the chunk *together*.

So: that, plus **BM25 (A classic keyword-ranking algorithm. It scores a document by how often the query's words appear in it, weighted so that rare words count for much more than common ones.)** keyword matching alongside the vector search — the hybrid (Running two different retrievers over the same corpus and combining their results: vector search, which matches on meaning, and keyword search, which matches on exact words. Each catches things the other misses — product codes and proper nouns for keyword search, paraphrases for vectors.) part, meant to catch exact terms vectors are bad at — with the two lists merged by RRF (Reciprocal Rank Fusion — combines several ranked lists by scoring each result on its rank position in each list, so anything ranking well in both rises to the top.). Textbook.

## The reranker came with a constraint attached

Before I ran a single eval, a review of the implementation turned up something I'd otherwise have found much later and much more expensively: the cross-encoder only reads about 512 tokens (The units models actually process — roughly a short word or a fragment of one. Every model has a hard ceiling on how many it can read at once, and text past that ceiling is simply cut off.) — roughly 2,000 characters.

The model I'd picked, `jina-reranker-v1-turbo-en`, advertises an 8,000-token context window. The library I was calling it through pins the tokeniser's truncation to 512. My chunks were 2,500, 5,000 and 10,000 characters. So *every* candidate on *every* query was being silently clipped before scoring, and the largest chunks were having roughly 80% of their content thrown away.

I checked it directly, by putting the same answer at the end of chunks of increasing length and reading the score the reranker gave back:

```
answer at END of  1500 chars -> score -2.178
answer at END of  2500 chars -> score -3.323
answer at END of  5000 chars -> score -4.068
answer at END of 10000 chars -> score -4.068   <- identical: stopped reading
```

Identical scores at 5,000 and 10,000 characters, because by then it's reading the same first 512 tokens either way. Anything past the cut may as well not exist.

Which has a consequence I didn't like: chunk size stops being a free parameter and becomes bounded by the reranker's window. Shrink the chunks to fit, or score a different unit than you retrieve. I shrank them — 2500 down to 1500 — because it was a flag change rather than a project.

And that is how a second variable walks into an experiment without ever feeling like one. I didn't change chunk size absent-mindedly. I changed it *because* I was adding the reranker, so it felt like part of installing the thing rather than an independent intervention. Which is exactly why it took me weeks to see it as one.

## The results looked good

They did, genuinely. Against the June baseline:

| Config | Recall@3 | Factual | Multi-hop | Abstention | $/query |
|---|---|---|---|---|---|
| baseline (chunk 2500) | 0.841 | 0.933 | 0.611 | 0.133 | $0.000420 |
| hybrid, chunk 1500, loose cutoff | 0.730 | **0.956** | 0.167 | **0.400** | $0.000204 |
| hybrid, chunk 1500, mid cutoff | 0.810 | 0.978 | 0.389 | 0.133 | $0.000247 |
| hybrid, chunk 1500, no filtering | 0.841 | **1.000** | 0.444 | 0.000 | $0.000304 |

Factual recall of 0.956 while correctly refusing 40% of the unanswerable questions — the region the June sweep said didn't exist. Cost down 28% at identical recall. I wrote up four findings, and I was pleased with them.

Multi-hop had collapsed, from 0.611 to somewhere between 0.167 and 0.444, and I explained that too: a cross-encoder scores each chunk on its own and never sees the result set as a whole, so whichever document matches the question best produces several strong chunks and takes every slot. The evidence was right there — on one run, 11 of 18 multi-hop questions came back with chunks from a single document, and only 3 of 18 got both documents the answer needed.

```
need = [duck_wars_extended_history.md, isoprene_planetary_survey.md]
 got = [duck_wars_extended_history.md, duck_wars_extended_history.md,
        duck_wars_extended_history.md]
```

Clean diagnosis. Confident write-up. Three variables in the experiment.

## The ablation

An **ablation** is the boring, essential experiment where you remove exactly one thing and re-run everything else unchanged. It's the only way to know which part of a change was responsible for the result. If you alter three things at once and the numbers move, you've learned that *something* worked — which feels like knowledge and isn't.

I knew mine was confounded. I'd written "the outstanding ablation is dense-only at chunk 1500" into my own notes and then not run it, because the results were good and it cost an evening. That's the whole failure mode, and it's worth saying plainly: ablations don't get skipped when the numbers look bad. They get skipped when the numbers look good.

So I checked out the code from before the hybrid work, set the chunk size to 1500, rebuilt the index from scratch, and ran the same 78 questions. Same corpus, same embedding model, same top-3. The only difference from the hybrid run is the pipeline itself.

| | recall | factual | multi-hop | coverage |
|---|---|---|---|---|
| plain vector search, chunk 1500 | 0.841 | 1.000 | 0.444 | 1.000 |
| hybrid + reranker, chunk 1500 | 0.841 | 1.000 | 0.444 | 1.000 |

Not close. The same numbers. With filtering off, both pipelines put the same documents in the top three on all 63 answerable questions. And across the rest of the range, plain vector search was marginally *ahead*.

![Each point is one run at a different filter setting. The two pipelines start on the same point and the simpler one stays slightly above — it answers a little more at any given level of caution.](ablation.png)

Three of my four findings were wrong. Reranking hadn't opened up that high-abstention region — halving the chunk size had. Reranking hadn't broken multi-hop — plain vector search at the same chunk size breaks it identically. The cost saving wasn't the pipeline either: plain vector search at chunk 1500 costs $0.000312 per query against the baseline's $0.000420, a 26% reduction with no pipeline change whatsoever.

Meanwhile the hybrid pipeline was adding about 160 ms of CPU-bound reranking to every query, and a 150 MB model to a service that scales to zero and therefore downloads it again on every cold start.

## What chunk size was actually doing

Truncation is half of it, and the probe above tells that story. The failure breakdown says the same thing from the other direction: of 11 factual misses in the 2500-character run, 10 returned *zero* chunks rather than wrong ones. Retrieval had found the right candidates. The reranker scored them below the cutoff because it never read that far into them.

The other half is the one I keep thinking about.

![Same eval set, same everything except how finely the documents were cut. The two buckets move in opposite directions by almost exactly the same amount.](chunk-size.png)

Smaller chunks fix the last three factual questions and break three multi-hop ones. Fifty-three of sixty-three answered either way — the identical total, because the two effects are the identical size. That's why overall recall sits at 0.841 in both rows while the system's behaviour changes underneath it.

If I'd been tracking aggregate recall, chunk size would have looked like a no-op. Splitting the questions into buckets is the only reason I can see the trade at all.

It's also worth being honest about what three questions can and can't tell you. On eighteen multi-hop rows, three is a 17-point swing, and I wouldn't defend the precise size of that trade-off from this corpus. What I would defend is its direction and its mechanism, which the failure breakdown explains independently of the score. The numbers in this post are the smallest part of it; the method is the part I'd argue for.

The mechanism, once the buckets tell you where to look, is embarrassingly simple. Smaller chunks mean more chunks per document. I return the top 3 with nothing stopping all three from coming out of the same file — and the better a document matches, the more of its chunks crowd the shortlist. Multi-hop questions here always need two distinct documents. So it isn't a retrieval-strategy problem at all. It's a selection problem, and capping how many chunks any one document can contribute fixes it directly, at no cost per query.

## A note on the 26%

Every query in the eval set records what it cost, so that saving isn't an estimate — and splitting the bill shows it had nothing to do with retrieval either. The model wrote the same amount of answer both times. It read a third less, because I'd halved the chunks going into the prompt.

Two things follow that are worth carrying out of here. Cost per query is trivially minimised by answering fewer questions — a system that refuses everything is nearly free — so the number is meaningless without recall beside it, which is why every table above carries both. And the reranker's 160 ms of CPU and its 150 MB cold-start download appear nowhere in that dollar column at all. The component with the weakest case for existing was invisible to the metric I was judging everything else by.

## So I'm keeping hybrid search

Which reads as sunk cost, so let me put the argument down properly.

The ablation is a null result on a test my corpus can't run. Seven documents, 52 chunks, and a shortlist of 9 — about 17% of everything I have. A reranker earns its keep by reordering a shortlist drawn from a much larger pool; mine already holds nearly everything relevant, so there's little left to fix. BM25's entire signal is which words are rare, and across seven documents there's no rarity to measure. So the claim is narrow — **hybrid search didn't pay for itself on this corpus at this scale**, not "hybrid search doesn't work."

But read that back, because there's something worse in it than a null result. If my corpus can't exhibit the failure hybrid search exists to fix, then I built the fix before I had any test capable of judging it — a bigger error than the confounded experiment, and one no amount of careful measurement afterwards could have caught. The measurement was fine. The thing being measured was never in scope. *Find a query your retrieval genuinely fails on, then go looking for what fixes it.* I did it backwards and spent a quarter finding out.

It stays anyway, and not only out of stubbornness: the reranker's 512-token window is what turned chunk size from a default I'd stopped seeing into a live parameter, so every gain I'm now crediting to chunking exists because it forced me there. But it stays labelled — an unproven bet with a stated test. Build a corpus where plain vector search genuinely fails, re-run this comparison, and if it still adds nothing, it comes out.

Though "stays" flatters it. There's no flag to turn it off: `query` prefetches both vectors, fuses them and calls the reranker on every request, with the threshold and model name as the only knobs I exposed. I wired it in as the only path rather than one of two, which is what building-before-testing looks like by the time it reaches the code.

## The part I'd keep

The interesting output here isn't a finding about chunk sizes. It's that I ran an experiment with three variables in it, produced a confident and internally coherent explanation, and was wrong — and the only reason I know is that I'd built the instrument before I needed it.

The write-ups in the repo now carry their corrections inline, with the wrong conclusions left standing and annotated rather than quietly edited out. That's deliberate. A results directory that only contains things that worked is marketing.

Most RAG write-ups I read change three things and credit the interesting one. I did exactly that, in public, with a straight face, in my own notes. The eval set caught it. That is what the eval set is for.

Next is the half of the bill I can't honestly attribute per query at all — GPU-seconds, cold starts, what it costs to run a model rather than rent one. That's the part I understand least, so that's where I'm going.

---

*The code, the eval harness, the full result write-ups and the corrections are all in [ritam](https://github.com/bhvishal9/ritam). If you're running retrieval in production and your evals report a single aggregate number, split it into buckets — I'd like to hear what falls out.*