Post-Retrieval Re-Ranking
The Cross-Encoder. Fixing the mistakes of the vector store.
1. Bi-Encoders vs Cross-Encoders
Standard Vector Search uses Bi-Encoders. It encodes the Query independently. It encodes the Document independently. Then it measures the angle. Pros: Fast (millions of docs). Cons: Lossy. It misses subtle interactions between query and doc.
A Cross-Encoder takes both the query and the document as a single input: Model(Query, Doc).
It outputs a score (0 to 1) of "How relevant is this?"
Pros: Extremely accurate.
Cons: Slow. Expensive. You cannot index this ahead of time.
2. The Two-Stage Pipeline
We combine them.
- Stage 1 (Retrieval): Use Vector Search (Bi-Encoder) to get the top 50 candidates. (Fast, coarse).
- Stage 2 (Re-Ranking): Use a Cross-Encoder (like BGE-Reranker or Cohere Rerank) to score those 50.
- Result: Take the top 5 from the Cross-Encoder.
3. Why It Matters
"Apple" means fruit. "Apple" means company. A Vector store puts them close. A Cross-Encoder sees the full sentence "I want to buy a Granny Smith Apple" and pushes the "Company" documents to the bottom.
4. Summary
Re-ranking is the single most effective way to boost precision in a RAG pipeline. It acts as a "Quality Filter" before the expensive LLM generation step.
Key Intuition: "Index fast. Rank slow."