There is a moment in most RAG projects where someone drops in a cross-encoder reranker, watches nDCG@10 climb six points, and declares the retrieval problem solved. Two weeks later, that same person is staring at a P99 latency graph trying to work out what went wrong.
Both observations are correct. Rerankers do improve retrieval quality, often dramatically. They also insert a second transformer into the critical path, and unlike the embedding model, a cross-encoder cannot precompute anything.
This post is about holding both facts at once: keeping the quality and paying less for it. The framing throughout is budget allocation. You have a latency target, every stage spends against it, and the job is to spend where the spending buys the most quality.
The arithmetic that makes rerankers expensive
Start with why the cost profile differs, because everything else follows from it.
A bi-encoder embeds the query and the document independently, then scores them by cosine similarity between the two resulting vectors.
The critical property is that the document encoder never sees the query. You run it once, at ingest time, across all ten million documents. At query time you pay for one forward pass plus an approximate nearest neighbor (ANN) lookup. Retrieval costs O(1) transformer inferences regardless of corpus size.
A cross-encoder scores the pair jointly. Query and document are concatenated into a single sequence and pushed through the model together, producing one relevance score.
Because that score depends on both arguments at once, nothing can be cached ahead of time. Score 100 candidates, run 100 forward passes. Retrieval now costs O(K) transformer inferences, where K is the candidate count.
That is the entire story. The reranker is more accurate precisely because attention runs across query and document tokens simultaneously; the model can see that “capital” in the query and “capital” in the document mean different things in this context. You are paying for cross-attention, and cross-attention is the part that cannot be cached.
The second-order effect is worth internalizing as well. Attention is quadratic in sequence length, so a reranker call on a 512-token chunk is more than twice the cost of a 256-token chunk. Chunk size is a reranker latency knob, not only a retrieval quality knob.
Step zero: instrument before you touch anything
Every RAG request decomposes cleanly into six stages: query embedding, ANN lookup, reranking, context assembly, time to first token, and generation. Total latency is just their sum, which means any one of them can be the problem and only measurement will tell you which.
Emit a structured span per stage, and record a distribution rather than an average:
{
“embed_ms”: 9,
“ann_ms”: 14,
“retrieved_k”: 100,
“rerank_ms”: 63,
“rerank_pairs”: 100,
“kept_n”: 8,
“context_tokens”: 2400,
“ttft_ms”: 210,
“gen_ms”: 640,
“total_ms”: 936
}
Two things that averages will hide from you:
Amdahl’s law applies to your intuition, not just your code. If reranking accounts for 12% of wall time, making it infinitely fast caps out at a 12% improvement. Halving a stage that owns 40% of the budget beats eliminating a stage that owns 5%. Engineers tend to optimize the component they find interesting; measure the one that is actually large.
P50 and P99 usually have different bottlenecks. Median latency is dominated by raw compute. Tail latency is dominated by queueing, batch formation, and the one request carrying a 1,500-token chunk that blocked the whole batch. A change that improves P50 can degrade P99. Track both and set your SLO on the tail, since that is what users actually feel.
The highest-leverage knob: K
Reranking cost scales linearly in candidate count. You process K candidates in batches of size B, so the reranker runs roughly K divided by B batches, each costing whatever a full batch costs at your sequence length. Moving from K=200 to K=50 is a 4× reduction in reranker work. No new model, no new hardware, one configuration change.
The obvious question is how much recall this costs you. Usually less than you would guess, for a specific reason: recall@K for a competent dense retriever saturates quickly. The documents sitting at ranks 100–200 in your ANN results are, for the overwhelming majority of queries, genuinely irrelevant. The reranker dutifully scores every one of them and discards every one of them. You are paying full price for candidates that were never going to survive.
Do not guess at the crossover point. Build the sweep once, on your own evaluation set.
The shape you are looking for is a recall curve that flattens while the latency curve keeps climbing. Take the elbow. On most corpora it lands somewhere between 50 and 100, but “most corpora” is not yours, which is the whole reason to run the sweep.
One nuance: recall@K from the dense stage is a ceiling on final quality. The reranker can reorder what it is given; it cannot conjure a document the ANN stage missed. The right target is therefore not “minimize K” but “find the smallest K at which dense recall has already saturated.”
The cascade principle
Zoom out and the architecture writes itself as three tiers, each cheaper per item than the one below it and each handling far more items:
- Tier 1: ANN + BM25. Runs against all ten million documents. Costs on the order of a millisecond per doc-equivalent, and hands down the top 100.
- Tier 2: Cross-encoder. Runs on those 100 candidates only. Roughly half a millisecond per pair, and hands down the top 8.
- Tier 3: LLM. Runs on those 8 passages. Priced in dollars per token, which is why it never sees the other 9,999,992.
Cost per item rises at each tier; item count falls faster. This is not a RAG invention. It is how search engines have worked for decades, how a query planner works, and how a CPU cache hierarchy works. Cheap filters feed expensive judges.
The design rule that falls out of it: each stage exists to hand the next stage a smaller, better set. If a stage is not measurably improving the input to the stage below it, delete it.
Improve the candidate set before the reranker sees it
If cutting K hurts recall, the fix is usually a better retriever rather than a bigger K.
Dense retrieval is systematically weak on exact-match content, especially in rare domain jargon like Stock Keeping Units (SKUs), error codes, person names, version numbers. Those are exactly the tokens that embeddings smooth away, and exactly what BM25 handles natively. Running both and fusing the results produces a stronger top-50 than either method alone, which means you can afford a smaller K at equal recall.
Reciprocal Rank Fusion is the usual choice because it requires no score calibration between the two systems. The rule is simple: a document’s fused score is the sum, across every retriever, of one divided by (k plus that document’s rank in that retriever’s list), with k conventionally set to 60. A document ranked first everywhere scores highest; a document ranked 200th in one list contributes almost nothing from it. Because it operates on ranks, you never have to reconcile a cosine similarity bounded between -1 and 1 against an unbounded BM25 score. Weighted linear fusion also works, but then you own the normalization problem, and it drifts whenever you change either retriever.
Note what this buys you in budget terms. BM25 is cheap and runs parallel to the dense path, so it adds close to zero wall-clock latency while shrinking the input to the expensive stage.
Serve inference as a service, not as an import
A pattern that shows up constantly in prototypes that became production systems:
from sentence_transformers import CrossEncoder
model = CrossEncoder(“…”)
scores = model.predict([(query, d) for d in docs])
This is fine in a notebook. In a web application it means your API process now owns GPU memory management, model loading, batch formation, concurrency limits, and inference scheduling, all while serving HTTP. Your autoscaling policy is now coupling request concurrency to GPU memory. Every deploy reloads the model. Two replicas means two full copies of the weights.
Split it apart. Hugging Face Text Embeddings Inference (TEI) exists for precisely this: a purpose-built server for embedding and reranking models with dynamic batching, optimized kernels, and metrics endpoints.
docker run –gpus all -p 8080:80 –pull always
ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
–model-id Qwen/Qwen3-Embedding-0.6B
The same server handles cross-encoders through /rerank:
curl http://localhost:8080/rerank -X POST
-H ‘Content-Type: application/json’
-d ‘{
“query”: “what is retrieval augmented generation”,
“texts”: [
“RAG combines retrieval with generation.”,
“The weather will be sunny tomorrow.”,
“Vector databases store embeddings.”
]}’
Run two separate deployments, one for embedding and one for reranking, because their load profiles have nothing in common.
The embedding service sees high request rates, short sequences, and bursty ingest traffic. It is throughput-shaped. The reranker sees low request rates, but every request is a batch of K pairs at full chunk length. It is latency-shaped, and its work per request is K times larger.
Scaling them together means over-provisioning one to satisfy the other:
Embedding TEI → 3 replicas (ingest bursts)
Reranker TEI → 5 replicas (query-path latency)
LLM (vLLM) → 2 replicas (KV-cache bound)
There is no principled reason those three numbers should be equal.
Batching: the throughput/latency tradeoff, stated precisely
Larger batches use the GPU better. Larger batches also mean waiting for requests to accumulate. These pull in opposite directions, and you have to pick a side.
The complication specific to text is that a batch of n requests is not a fixed amount of work, because sequence lengths vary wildly:
Request A → 50 tokens
Request B → 100 tokens
Request C → 800 tokens
Request D → 1500 tokens
Pad to longest, and request D dictates the cost of the entire batch while requests A through C burn compute on padding. This is why token-budget batching beats request-count batching, and why TEI exposes MAX_BATCH_TOKENS rather than a batch-size integer. You are capping work, not items.
A practical tuning loop:
- Fix the Service Level Objective (SLO) first. For example, “P95 end-to-end under 300 ms.” That is your constraint, and everything else is negotiable against it.
- Raise MAX_BATCH_TOKENS until throughput stops improving or P95 breaches, whichever comes first.
- Set MAX_CONCURRENT_REQUESTS to fail fast rather than queue unboundedly. A request that queues for four seconds and then succeeds is worse than one that returns 503 in five milliseconds and gets retried elsewhere.
Maximizing throughput is the right objective for a batch ingest job. It is the wrong objective for an interactive query path, and conflating the two is how you end up with a well-utilized GPU and unhappy users.
The cheapest inference is the one you skip
Document embeddings are an offline artifact. If any part of your query path is embedding documents, that is a bug rather than a tuning opportunity. Chunk, embed, and index all belong at ingest time.
Query embeddings cache well, because production traffic is typically Zipfian (meaning query frequency drops off sharply with rank): a small set of queries accounts for a large share of volume. Key the cache carefully:
sha256(model_id + revision + normalization + preprocessing_version + query)
The revision field matters more than most people expect, which brings us to the failure mode nobody plans for.
Pin the revision, and treat embeddings as schema
This is not a fully specified dependency:
Qwen/Qwen3-Embedding-0.6B
This is:
{
“embedding_model”: “Qwen/Qwen3-Embedding-0.6B”,
“revision”: “abc123def”,
“dimension”: 1024,
“metric”: “cosine”,
“normalized”: true,
“query_prefix”: “query: “
}
Store that alongside the index and validate it at startup. The reasoning: an embedding model defines the geometry of your vector space. Change the model, and the vectors in your index and the vectors coming out of your query encoder are describing different spaces. Cosine similarity still returns a number, a perfectly plausible-looking float, and retrieval quality quietly collapses. No exception, no alert, no stack trace. Just worse answers.
Changing the embedding model means rebuilding the index and re-running your evaluation. Version it like a database migration, because that is what it is.
Do not hand the LLM everything that survived
The reranker’s output is a score distribution, and score distributions carry information you should be using:
doc_1 0.94 keep
doc_2 0.91 keep
doc_3 0.89 keep
doc_4 0.43 drop
doc_5 0.31 drop
There is a cliff between 0.89 and 0.43. Documents below it are not marginally useful; they are off-topic. Passing them along costs you three ways: prefill compute, money per token, and attention dilution. The last of those is the most underrated. Long contexts stuffed with weak passages measurably degrade answer quality, because the model has more plausible-looking material to be wrong about.
Apply a score threshold, deduplicate near-identical chunks (RAG corpora are full of boilerplate), and cap the token budget.
The prefill arithmetic is worth seeing explicitly. Twenty chunks of 500 tokens is 10,000 tokens of context. Eight chunks of 300 tokens is 2,400.
Prefill is compute-bound and roughly linear in input length at these scales. That 4× reduction lands directly on TTFT, the single number users perceive as “is it fast.” Retrieval quality and generation latency turn out to be the same lever pulled from different ends.
Putting it together
Traced end to end, a single request walks through seven components, and each one is a separately deployable service:
- API Gateway → RAG Service. The gateway terminates the request; the RAG service owns orchestration and nothing else. No model weights live in this process.
- Embed TEI. The RAG service sends the query text out for embedding and gets a vector back.
- Qdrant and BM25, in parallel. The vector goes to the ANN index; the raw query string goes to the lexical index at the same time. Neither waits on the other.
- Fusion. The two ranked lists are merged with RRF into a single top-100 candidate set.
- Rerank TEI. Those 100 candidates are scored as query-document pairs and cut to the top 8.
- Context Builder. Score threshold, deduplication, token cap. The only stage here with no GPU behind it, and one of the highest-leverage.
- LLM (vLLM). Receives the assembled context and the original query, and generates.
Every stage scales on its own axis. Reranker saturating? Add reranker replicas. Ingest backlog? Add embedding replicas. Neither forces the other.
The order of operations
If you inherit a slow RAG system, work in this sequence. It is ordered by expected payoff per hour spent.
- Profile. Per-stage spans, P50 and P99. Everything below this line is guesswork without it.
- Delete work. Are documents being embedded at query time? Are you reranking 200 candidates to keep 5? Are you sending 10k tokens of context to answer a yes/no question? These are free wins.
- Sweep the candidate count. Recall and latency on the same axis. Take the elbow.
- Strengthen the candidate set. Hybrid retrieval and better chunking let you hold recall at a smaller K.
- Tune serving. Dedicated TEI deployments, token-budget batching measured against your actual SLO.
- Scale the real bottleneck. One service, not all of them.
The first four steps involve no new hardware and no model swaps. That is the point.
The takeaway
Adding a reranker is usually the right call. Retrieval quality is the ceiling on RAG answer quality, and cross-encoders raise that ceiling in a way no amount of prompt engineering will.
The mistake is budgeting for it as though cross-attention were free.
The largest wins in production RAG rarely come from finding a faster reranker. They come from not running inference you never needed: a smaller candidate set, a stronger retriever feeding it, a compact context feeding the LLM, and each model served as an independently scalable component rather than an import statement inside your web process.
Expensive computation is not the enemy. Expensive computation in the wrong place is.
Data Scientist
Building and researching end-to-end machine learning and LLM systems, from model training to deployment.
