How to Add a Reranking Layer to Your Enterprise RAG Pipeline: The 5 Retrieval Precision Decisions That Separate Relevant Chunks from High-Recall Noise

Why retrieval by similarity score is not the same as retrieval by relevance

Vector similarity retrieval works by encoding the query and all document chunks into a shared embedding space, then returning the chunks whose embeddings are closest to the query embedding by cosine distance. This approach scales to millions of documents and has the recall characteristics that make it a good first-stage retriever. What it does not do well is rank the returned chunks by how much they actually help answer the specific question asked.

The problem is that embedding models are trained on general text similarity, not question-answering relevance. Two chunks can have nearly identical cosine similarity to a query while having very different answer utility. A chunk that mentions the topic three times in passing may score above a chunk that directly answers the question, because embedding similarity captures surface co-occurrence as much as semantic relevance. In a top-10 retrieval, this means the most useful chunk may be at position 5, and the LLM context window starts with four chunks that are topically adjacent but not actually helpful.

Rerankers solve this by doing a second pass over the retrieved candidates using a model that jointly encodes the query and each chunk together, rather than comparing independent embeddings. A cross-encoder reads the query and a chunk as a single sequence and produces a relevance score for that specific pair. This is computationally more expensive than embedding lookup, which is why reranking runs only over the top-K retrieved candidates rather than the full corpus. The trade-off is that it produces a much more accurate relevance ranking: the cross-encoder can recognize that a chunk directly addresses the question being asked, even when its embedding distance to the query is not the lowest in the set.

A retriever is designed to make sure the right chunk is somewhere in the top-50. A reranker takes that candidate list and sorts it by how much each chunk actually answers the question. Both jobs matter, and neither replaces the other.

The 5 reranking approaches compared: where each fits and where each fails

Reranking is not a single technique. There are five distinct approaches to improving retrieval precision, each with a different cost-quality profile. Most production failures come from choosing the wrong approach for the retrieval baseline in place, or adding reranking before measuring whether the base retriever's recall is high enough to make reranking worthwhile.

The most common reranking mistake in enterprise RAG is adding a cross-encoder before measuring base retriever recall. Reranking can only promote a relevant chunk to position 1 if that chunk was retrieved in the first place. A retriever with recall@20 of 50% means that half the time the correct chunk is not in the set the reranker sees. In that situation, reranking improves the ranking of what was retrieved but cannot fix the underlying recall gap. Measure recall@20 or recall@50 before deploying a reranker; if it is below 70%, address the retrieval configuration first.

Why reranking fails when recall is low and latency budget is ignored

Two conditions kill reranking in production pipelines, and both are predictable from the design phase. The first is deploying a reranker on top of a single-method retriever with low recall. If you retrieve the top-10 chunks by cosine similarity and the correct chunk is at position 12 or 15, the reranker never sees it. The reranker will produce a good ranking of the ten chunks it was given, but the answer quality will not improve because the right chunk was filtered out before the reranker ran. The fix is to over-retrieve before reranking: retrieve the top 50 or 100 candidates, rerank those, and pass only the top 3 or 5 to the LLM. This is the standard retrieve-large, rerank-to-small pattern, and it requires that your retrieval index and query infrastructure can handle the higher candidate volume without unacceptable latency.

The second condition is mismatched latency budgets. Cross-encoders are computationally intensive because they process each query-chunk pair jointly rather than comparing pre-computed vectors. A cross-encoder reranking 100 candidates adds 200-500 ms at p99 on standard CPU inference. For a synchronous user-facing search interface with a 300 ms latency budget, that is a non-starter. For an asynchronous document processing pipeline with a 5-second budget, it is trivial. Teams that add reranking without modeling the latency impact under production load find that the p99 latency spike under concurrent queries exceeds their SLA before the reranker's precision benefits are realized. Measure latency impact on your actual hardware at your peak concurrent query rate before deciding whether a cross-encoder, a lightweight bi-encoder reranker, or a managed API reranker (Cohere Rerank, Jina Reranker) fits the budget.

A third underappreciated failure mode is running a reranker on a multilingual corpus with an English-trained model. A cross-encoder trained on English MS MARCO query-passage pairs will systematically deprioritize relevant chunks in French, German, or Japanese, even when the query is in the same language as the chunk. If your enterprise corpus includes multiple languages, verify the reranker's benchmark performance on each language before deployment. BAAI/bge-reranker-v2-m3 and multilingual variants of Cohere Rerank address this, but the tradeoff is higher latency compared to the lightweight English-only models.

The reranking deployment checklist

Each item below should be confirmed before a reranking layer goes to production. Most reranking failures in enterprise RAG pipelines trace back to skipping the measurement steps in favor of moving quickly to deployment.

Your retriever's job is to make sure the right chunk is somewhere in the top-50. Your reranker's job is to make sure it ends up at position 1. Both jobs matter, and neither substitutes for the other.

What to do this week

01 Measure your current retriever's recall@50 on a representative query sample

Pull 200 representative queries from your production logs, run them against your retrieval index, and check how often the ground-truth chunk appears in the top-50 results. This single measurement tells you whether your pipeline's answer quality problem is a retrieval recall gap or a ranking precision gap. If recall@50 is below 70%, adding a reranker will not fix the issue and you should work on the retriever first. If recall@50 is above 70%, a reranker will almost certainly improve precision@5, and you can move to evaluating which model fits your latency budget.

02 Benchmark a lightweight cross-encoder on your corpus before spending time on fine-tuning

BAAI/bge-reranker-base and ms-marco-MiniLM-L6-v2 are both freely available, run on standard GPU infrastructure, and perform well on English enterprise corpora without domain fine-tuning. Before investing in labeled training data for a domain-specific reranker, run one of these on your labeled query set and measure the precision@5 gain. If the out-of-the-box precision improvement meets your accuracy requirement, fine-tuning is unnecessary. If the gain is insufficient and the failure cases cluster around domain-specific terminology, that is when fine-tuning on your own query-passage pairs becomes worth the investment.

03 Set up the retrieve-large, rerank-to-small pattern this sprint

Switch your retrieval call from top-K to top-50 or top-100, route those candidates through a cross-encoder, and pass only the top-3 or top-5 to your generation step. Measure p50 and p99 latency on a load test at your expected peak concurrency before merging to production. The pattern is well-documented in LangChain, LlamaIndex, and Haystack; the implementation work is typically a half-day. The latency measurement at load is where most teams discover they need a smaller model or a managed API with autoscaling.

04 Add a precision@5 metric to your RAG evaluation suite alongside your existing metrics

Most RAG evaluation suites measure faithfulness and answer relevancy at the generation level. These metrics can score high even when the retriever is passing mediocre chunks, because a capable LLM will produce a coherent answer from whatever it is given. Adding precision@5 at the retrieval level, measured against a ground-truth labeled set, surfaces retrieval quality problems that generation-level metrics miss. Run this evaluation weekly on a fresh sample and track it as a primary pipeline health metric alongside answer quality scores.