Conversational RAG & History

1. The Context Problem

User: "Who is the CEO of Apple?" AI: "Tim Cook." User: "How old is he?"

If you search for "How old is he?", the vector store has no idea who "he" is. It retrieves random documents about men. This is why naïve RAG fails in chat.

2. History-Aware Retrieval (Query Rewriting)

We need to rewrite the prompt to be Standalone. Pass the Chat History + New Question to an LLM.

• History: "User: Who is CEO of Apple? AI: Tim Cook."
• Question: "How old is he?"
• Standalone: "How old is Tim Cook?"

Now, we search for "How old is Tim Cook?" and we get the correct answer.

from langchain.chains import create_history_aware_retriever

history_aware_retriever = create_history_aware_retriever(
  llm=llm,
  retriever=retriever,
  prompt=rephrase_prompt
)

3. Common Failure Modes in RAG

Even with all these tools, systems fail.

1. Stale Context: The vector store has the 2021 manual. The world refers to the 2024 version.
2. Over-Retrieval: You retrieve 20 docs. The answer is in the 1st one. The other 19 confuse the model (Distraction).
3. Retrieval-Generation Mismatch: The retriever finds "Pricing Page." The model is asked "Write a poem about prices." The model ignores facts to be creative.

4. Final Thoughts: Retrieval Defines Truth

In RAG systems, the model answers only what retrieval allows it to see. If the retriever is blind, the model is hallucinating. Retrieval is not a support component—it is the system’s epistemology.

Key Intuition: "Garbage in, Hallucination out."