RAG Evaluation Basics
Beyond 'vibes'. Measuring what matters.
1. The "Looks Good to Me" Trap
You act as the QA. You ask 5 questions. The bot answers 4 correctly. You ship it. In production, users ask 5,000 questions. The bot fails 2,000 of them. Anecdotal Testing is not Evaluation.
2. Offline Evaluation (The Exam)
Before you deploy, run a test suite.
You need a Gold Dataset: A CSV of (Question, Ground_Truth_Answer, Ground_Truth_Document_ID).
Metrics:
- Retrieval Metrics:
- Recall@5: Is the correct Document ID in the top 5 results?
- MRR (Mean Reciprocal Rank): How high up was it?
- Generation Metrics:
- Semantic Similarity: Is the generated answer effectively the same as the Ground Truth answer? (Use an LLM to judge this).
3. Online Evaluation (The Feedback Form)
Users are the best testers. Add a "Thumbs Up / Thumbs Down" button. Log every Thumbs Down to LangSmith. Review them weekly. These are your "Hard Negatives." Add them to your Offline Dataset. This creates a Data Flywheel.
4. LLM-as-a-Judge
We can't manually read 5,000 logs. We use a strong model (GPT-4) to grade the weak model (GPT-3.5). Prompt: "You are a judge. Verify if the Student Answer matches the Ground Truth. Score 1 to 5."
5. Summary
Evaluation is not a one-time event. It is a Continuous Integration (CI) pipeline. Code has Unit Tests. AI has Evals.
Key Intuition: "If you don't measure it, you can't improve it."