RAG Evaluation Basics

1. The "Looks Good to Me" Trap

You act as the QA. You ask 5 questions. The bot answers 4 correctly. You ship it. In production, users ask 5,000 questions. The bot fails 2,000 of them. Anecdotal Testing is not Evaluation.

2. Offline Evaluation (The Exam)

Before you deploy, run a test suite. You need a Gold Dataset: A CSV of (Question, Ground_Truth_Answer, Ground_Truth_Document_ID).

Metrics:

1. Retrieval Metrics:
   • Recall@5: Is the correct Document ID in the top 5 results?
   • MRR (Mean Reciprocal Rank): How high up was it?
2. Generation Metrics:
   • Semantic Similarity: Is the generated answer effectively the same as the Ground Truth answer? (Use an LLM to judge this).

3. Online Evaluation (The Feedback Form)

Users are the best testers. Add a "Thumbs Up / Thumbs Down" button. Log every Thumbs Down to LangSmith. Review them weekly. These are your "Hard Negatives." Add them to your Offline Dataset. This creates a Data Flywheel.

4. LLM-as-a-Judge

We can't manually read 5,000 logs. We use a strong model (GPT-4) to grade the weak model (GPT-3.5). Prompt: "You are a judge. Verify if the Student Answer matches the Ground Truth. Score 1 to 5."

5. Summary

Evaluation is not a one-time event. It is a Continuous Integration (CI) pipeline. Code has Unit Tests. AI has Evals.

Key Intuition: "If you don't measure it, you can't improve it."