10decoders
What re-ranking does and why vector similarity alone is insufficient
Vector similarity retrieval works by comparing a query embedding to chunk embeddings and returning the closest matches by cosine distance. The embedding model compresses the meaning of a query or chunk into a fixed-dimension vector. That compression is lossy. Two chunks can be close in vector space because they share topic vocabulary while addressing completely different aspects of that topic. A query about "model evaluation metrics for classification" will retrieve chunks about evaluation metrics for regression, evaluation metrics for clustering, and general discussions of classification, because all of them share vocabulary with the query embedding. The LLM receives this mix and must either synthesize an answer from partially relevant context or acknowledge it doesn't have enough information.
Re-ranking addresses this by treating the retrieved set as candidates and applying a second scoring pass that evaluates each candidate in direct relation to the query, rather than in isolation. A cross-encoder re-ranker takes the query and a candidate chunk as a pair, processes them jointly through a model, and outputs a relevance score for that specific pair. This joint encoding is more expensive than separate embeddings but captures the interaction between query and passage in ways that bi-encoder similarity cannot. The result is a re-ordered candidate list where the top-k positions hold chunks that are specifically relevant to the query, not just topically adjacent.
The improvement from re-ranking scales with the quality gap between vector similarity and true relevance. For broad, conversational queries against well-structured corpora, vector similarity is often adequate and re-ranking adds latency without meaningful quality gain. For precise, domain-specific queries against large heterogeneous corpora, the quality gap between vector similarity and re-ranking relevance is large, and the improvement in answer quality justifies the latency cost. The decision to add re-ranking, and which approach to use, depends on measuring that gap for your specific query distribution and corpus.
"Vector similarity finds chunks that are about the topic. Re-ranking finds chunks that answer the question. For most enterprise use cases, those two things are not the same list."
Four re-ranking approaches and where each one fits your pipeline
Four approaches cover the realistic decision space: no re-ranking (baseline), cross-encoder re-ranking, late-interaction models (ColBERT-style), and LLM-based re-ranking. Each trades latency, accuracy, cost, and implementation complexity differently. The comparison below covers the key dimensions for each, along with the conditions under which the approach is or isn't worth its overhead.
| Approach | How it works | Latency added (P99) | Relevance gain vs retrieval-only | Implementation complexity | Best fit | Suitability |
|---|---|---|---|---|---|---|
| No re-ranking | Retrieved candidates passed directly to the LLM in vector similarity order. Top-k chunks form the context window without any re-scoring | None | Baseline | None | Short uniform corpora, conversational queries with broad acceptable answers, latency-critical applications where <50ms total retrieval is required | Insufficient for precision-critical use cases |
| Cross-encoder re-ranking | A small BERT-class model encodes each (query, chunk) pair jointly and outputs a scalar relevance score. Candidates are re-ordered by score. The model sees the interaction between query and chunk rather than each in isolation. Available as managed APIs (Cohere Rerank, Jina Rerank) or self-hosted models (ms-marco-MiniLM series) | 80–120ms for 20–50 candidates | 28–31% improvement on domain-specific corpora | Medium | Domain-specific corpora, precision-critical queries (legal, medical, financial), knowledge bases with high topical density where many chunks are adjacent to the query topic. The practical default for most enterprise RAG use cases that can tolerate the latency | Recommended default |
| Late-interaction (ColBERT-style) | Query and passage are encoded separately into token-level embeddings rather than a single vector. Relevance is computed as the maximum similarity between each query token and each passage token (MaxSim). More accurate than bi-encoder similarity, faster than cross-encoders. Requires vector stores with late-interaction support or a dedicated retrieval engine | 20–40ms for indexed retrieval | 15–22% improvement, less than cross-encoder on small candidate sets | High: requires index infrastructure changes | High-volume applications where cross-encoder latency is prohibitive. Large corpora where the token-level resolution of ColBERT's similarity function catches term-level relevance patterns that full-passage embeddings miss. Requires a vector store that natively supports the late-interaction scoring format | High-volume, latency-constrained |
| LLM-based re-ranking | The LLM itself scores or ranks retrieved candidates, either by prompting it to rate each chunk's relevance to the query or by using log-probabilities of a "relevant/not relevant" token as the ranking signal. Highest accuracy of any approach on the query types the LLM understands well. Very high latency and cost at scale | 500ms–3s+ depending on candidate count and model | 35–45% improvement on complex multi-hop queries | High: substantial latency and cost increase | Offline batch pipelines where latency is not a constraint, high-value queries that justify the cost (executive briefings, legal research, compliance reviews), or hybrid pipelines where the LLM re-ranks only when cross-encoder scores are inconclusive | Not suitable for real-time at scale |
Not sure whether re-ranking is worth adding to your RAG pipeline?
10decoders runs two-week RAG architecture assessments that measure the quality gap between your current retrieval and re-ranked retrieval against your actual query distribution, and implement the approach that fits your latency and accuracy requirements.
Book a Free AI Assessment →How to measure whether re-ranking is worth adding to your pipeline
Adding re-ranking is not always the right call. Before implementing it, measure the quality gap it would close on your actual queries. The measurement process uses your representative query set, the annotated subset with expected relevant chunks, and a context relevance scoring pass. Run your existing pipeline and record the retrieval order for each query. Calculate NDCG (Normalized Discounted Cumulative Gain) or context relevance scores for the top-5 results. Then run the same queries through a cross-encoder re-ranker on the same retrieved candidates and calculate the same scores for the re-ranked order. If the improvement is under 5%, vector similarity is already capturing relevance well enough for your query distribution. If the improvement is 15% or more, re-ranking will have a visible positive impact on answer quality.
The measurement also tells you which query types benefit most from re-ranking and which don't. Broad informational queries often show little improvement because the top similarity results are already the most relevant. Precision queries, queries with negation, multi-part questions, and queries where the relevant answer appears in a dense technical passage surrounded by other dense technical content all tend to show larger improvements from re-ranking. Knowing which query types drive the improvement shapes how you deploy re-ranking: a pipeline that applies it selectively to high-precision query categories, rather than uniformly, can capture most of the quality gain at a fraction of the latency cost.
Latency measurement requires the same care as quality measurement. Run the re-ranking step on the candidate set sizes you'll use in production: if your pipeline retrieves 20 candidates for re-ranking to top-5, measure with 20 candidates. Latency scales roughly linearly with candidate count for cross-encoders, so the measurement at production candidate count gives you the real P99 budget impact rather than the optimistic number from testing with 5 candidates. Compare that against your total response time budget and decide whether the quality gain justifies the cost on each query category.
Retrieval-Only Pipeline
Vector similarity retrieval returns k candidates in cosine distance order. The top-k chunks go directly into the LLM context window. Retrieval quality issues attributed to chunk size, embedding model, or prompt rather than candidate ordering. Teams observe that the LLM sometimes gives incomplete or partially relevant answers without identifying the root cause as a retrieval ordering problem. Quality improvement efforts focus on prompt engineering and chunk size tuning because those are the most visible levers. The candidate ordering problem remains unexamined because no measurement of NDCG or context relevance is being tracked by query type.
Cross-Encoder Re-Ranking Added
Representative query set evaluated to measure the quality gap between retrieval-only and re-ranked ordering. Cross-encoder re-ranker added between retrieval and LLM context assembly. Initial retrieval set expanded (typically 3–4× the final k) to give the re-ranker more candidates to work with. Re-ranker evaluated on both quality improvement and latency impact. Deployed via managed API for the initial implementation to avoid infrastructure overhead. Context relevance scores tracked pre- and post-re-ranking as part of ongoing retrieval quality monitoring. Query types where re-ranking provides the most improvement identified for potential selective application.
Adaptive Re-Ranking Architecture
Re-ranking applied selectively based on query classification: precision-critical query categories receive cross-encoder re-ranking; broad informational queries skip it to reduce latency. Re-ranker model tuned or fine-tuned on domain-specific query-passage pairs from production logs where ground truth relevance can be established. NDCG and context relevance monitored continuously by query category. Re-ranker performance benchmarked quarterly against the initial baseline to detect drift as the corpus evolves. Candidate pool size dynamically adjusted based on query category and retrieval confidence scores. Late-interaction approaches evaluated as query volume grows and cross-encoder latency approaches the response time budget limit.
Re-ranking implementation readiness checklist
"The teams with the best retrieval quality aren't the ones who picked the best embedding model. They're the ones who measured where their pipeline was losing relevance and fixed that specific step."
What to do this week
01 Measure your current retrieval NDCG before deciding anything about re-ranking
Pull 40–60 queries from your representative query set, the ones where you have annotated ground truth for at least 3–5 relevant chunks. Run your pipeline and collect the top-10 retrieved chunks for each query. Score NDCG@5: given the ranked list, how much of the relevant content appears in the top positions versus buried at positions 6–10? If your NDCG@5 is above 0.85, vector similarity is working well and re-ranking is unlikely to move the needle. Below 0.70, there's meaningful ordering quality being left on the table. Below 0.55, the retrieval ordering is poor enough that many of your LLM answers are being generated from mediocre context, and re-ranking should be a near-term priority.
02 Run a 2-hour cross-encoder comparison on your annotated query pairs
Take your annotated query-chunk pairs (50–100 is enough) and run them through two cross-encoders: a general-domain model from Hugging Face (cross-encoder/ms-marco-MiniLM-L-6-v2 is a reasonable starting point) and a managed API re-ranker (Cohere Rerank or equivalent). Score both against your annotations. If one significantly outperforms the other on your specific query types, that determines your implementation path: managed API if the hosted model wins, self-hosted if the open-source model is competitive. If performance is close, factor in latency and cost at your production query volume to make the final call.
03 Add re-ranker score logging to your pipeline this sprint
Even if you're not adding re-ranking yet, you can add the instrumentation now. Log the top-k retrieval similarity scores alongside each query and the final LLM response. When you eventually add re-ranking, the pre-implementation logs give you a baseline to compare against. Without the baseline, you can't measure whether the re-ranker is actually improving quality in production. Two lines of logging code added now saves a month of backfilling later when someone asks whether the re-ranking investment made a difference.
04 Identify your precision-critical query categories before deploying re-ranking uniformly
Look at your query distribution and classify queries by how much retrieval precision matters. Queries asking for specific facts, procedures, policy clauses, or calculations require precise context to generate a correct answer. General explanations and summaries are more tolerant of partially relevant context. Deploy re-ranking on the precision-critical category first: you get the quality gain where it matters most, at a fraction of the total query volume, keeping the latency impact low. Expand to broader query categories only after validating that the latency budget holds at the initial deployment scope.
Let 10decoders assess your RAG retrieval quality
We measure NDCG and context relevance across your query distribution, identify the pipeline steps producing the largest quality gaps, and implement re-ranking with monitoring built in from day one.