If you've built a RAG pipeline and found the results just okay, not great, the fix might not be a better embedding model. It might be reranking. Reranking is one of the most underused yet highest-impact techniques for improving retrieval quality in RAG systems. It sits between the retrieval step and the generation step, acting as a precision filter that ensures your language model sees the most relevant context possible. Despite its simplicity, adding a reranker can dramatically improve answer quality, reduce hallucinations, and make better use of limited context windows. This post breaks down what reranking actually does, how it compares to standard RAG retrieval, when it helps, and when you can skip it.

How standard RAG retrieval works

In a typical RAG pipeline, the process looks like this:

Bi-encoders are the workhorses of this retrieval step. They encode the query and each document independently into fixed-size vectors, then compare them using a fast similarity metric. Because document embeddings can be precomputed and indexed, this approach scales well to millions of documents. But there's a fundamental limitation. Because the query and document are encoded separately, their tokens never interact during encoding. It's like summarizing two essays independently and then comparing the summaries. You lose nuance, and the similarity scores can be misleading. A chunk that uses similar vocabulary might rank higher than one that actually answers the question. This is why retrieval results are often "close enough" but not precise.

What reranking does differently

Reranking introduces a second pass over the retrieved results. After the initial retrieval returns, say, the top 50 or 100 candidates, a reranker evaluates each one more carefully and reorders them by true relevance to the query. The most common approach uses a cross-encoder model. Unlike bi-encoders, a cross-encoder processes the query and document together as a single input. This means every token in the query can attend to every token in the document, and vice versa. The result is a much richer understanding of whether a document actually answers the question. Think of it this way:

The trade-off is speed. Cross-encoders are significantly slower because they can't precompute document representations. Every query-document pair requires a full forward pass through the model. This is why reranking is used as a second stage, not as the primary retrieval method.

The two-stage retrieval architecture

The standard pattern in production systems is a two-stage pipeline: Stage 1, initial retrieval (bi-encoder): Cast a wide net. Retrieve the top 50 to 100 candidates quickly using vector similarity search. The goal here is recall, making sure relevant documents are in the candidate set. Stage 2, reranking (cross-encoder): Narrow the focus. Score each candidate against the query using a cross-encoder and select the top 5 to 10 most relevant results. The goal here is precision, making sure only the best context reaches the LLM. This architecture gives you the best of both worlds: the speed and scalability of bi-encoders for initial filtering, and the accuracy of cross-encoders for final selection.

Beyond cross-encoders

Cross-encoders aren't the only option for reranking. The landscape has evolved considerably. Late interaction models (ColBERT) sit between bi-encoders and cross-encoders. They encode the query and document separately into per-token embeddings (not pooled into a single vector), then compute relevance through token-level interactions. This offers better accuracy than bi-encoders while being faster than cross-encoders. LLM-based rerankers use large language models themselves to score relevance. You can prompt an LLM with a query and a document and ask it to rate relevance on a scale, or compare pairs of documents. Intercom's engineering team deployed an LLM-based reranker in production and found it effective enough that they used it to guide the training of a custom, smaller reranker model. Purpose-built reranking models from providers like Cohere, Jina, and NVIDIA offer hosted APIs optimized for production use. Cohere's Rerank 4, for example, supports over 100 languages and offers a speed-optimized "Nimble" variant. Open-source alternatives like BGE-reranker-v2-m3 and Qwen3-Reranker provide strong baselines you can self-host.

When reranking helps

Reranking tends to have the biggest impact in these scenarios:

When you can skip it

Reranking isn't always worth the added complexity and latency:

The numbers

Benchmarks consistently show meaningful improvements from reranking. Research from MIT demonstrated that two-stage retrieval with cross-encoder reranking improved RAG accuracy by up to 40% compared to single-stage vector search across multiple benchmarks. In a benchmark of 8 reranking models on Amazon reviews, the best reranker lifted Hit@1 from 62.67% to 83.00%, a gain of over 20 percentage points. That said, the marginal gains are narrowing as embedding models improve. Recent studies show gains in the range of 4 to 5 percentage points on some tasks, like LitSearch, suggesting that the value of reranking depends heavily on your specific retrieval quality baseline.

Practical considerations for implementation

If you're adding reranking to your pipeline, here are a few things to keep in mind: Choose your initial retrieval size carefully. Too few candidates and you risk missing relevant documents before reranking even starts. Too many and you increase latency without proportional quality gains. A common starting point is top-50 to top-100 for initial retrieval, reranked down to top-5 to top-10. Profile latency end to end. Cross-encoder reranking on 100 documents typically adds tens to hundreds of milliseconds depending on the model and hardware. For many applications this is fine, but measure it in your specific setup. Consider model size trade-offs. Smaller reranking models (like MiniLM-based cross-encoders) are fast but less accurate. Larger models (like those based on Mistral or Qwen) are more accurate but slower. Match the model to your latency budget. Evaluate on your own data. Public benchmarks are useful directionally, but reranking performance varies significantly across domains. Build a small evaluation set from your actual queries and documents, and measure whether reranking improves your specific metrics before committing to it in production.

The bottom line

RAG and reranking aren't competing approaches. Reranking is a refinement layer that sits inside the RAG pipeline and makes retrieval dramatically more precise. The two-stage architecture of fast initial retrieval followed by careful reranking has become the standard pattern in production RAG systems for good reason: it delivers noticeably better results with relatively modest implementation effort. If your RAG pipeline produces answers that are "pretty good" but not consistently great, reranking is likely the highest-leverage improvement you can make.

References