Advanced RAG, Part 2: Post-Retrieval Optimization

Fixing the search after it happens

Pre-retrieval optimization (Lesson 6) improves what gets searched for. Post-retrieval optimization improves what happens to the results after the initial similarity search returns them, and before they reach the language model.

Re-ranking

The initial top-k search is fast but approximate: it is optimized to search millions of chunks quickly, not to make the single most accurate relevance judgment. Re-ranking takes a larger initial candidate set (say, the top 50) and runs a slower, more accurate model, called a cross-encoder, over each candidate paired directly with the question, to re-score and reorder them. Only the top few after re-ranking are actually passed to the language model. This two-stage pattern, cheap broad search followed by expensive precise re-scoring, shows up constantly in information retrieval because it gets both speed and accuracy instead of trading one for the other.

Context compression and filtering

Even a correctly-ranked chunk often contains irrelevant sentences alongside the useful ones. Context compression uses a model to extract only the sentences within a retrieved chunk that actually bear on the question, discarding the rest, which shortens the prompt and reduces the chance that the language model latches onto an irrelevant detail. This matters because of the "lost in the middle" effect: language models pay measurably less attention to information buried in the middle of a long prompt than to information near the start or end, so a shorter, denser context is not just cheaper, it is more reliably read.

Together, pre- and post-retrieval optimization turn naive RAG's one-shot search into a small pipeline of its own, still linear, still one chunk store, but tuned at both ends. Modular RAG, next, takes the further step of making that pipeline's stages swappable rather than fixed.