Faithfulness & Relevance

1. The RAG Triad

RAG evaluation usually comes down to checking the relationships between 3 nodes: Query, Context (Docs), and Answer.

We care about two main edges:

1. Faithfulness: Answer ↔ Context
2. Relevance: Answer ↔ Query

2. Faithfulness (Did we hallucinate?)

"Faithfulness" measures: Is the Answer purely derived from the Context? If the Context says "Revenue is $10M" and the Answer says "Revenue is$12M", faithfulness is 0. Even if "Revenue is $12M" is true in the real world... if it's not in the context, it is a Faithfulness failure (External Hallucination).

Detection: Ask an evaluator LLM: "List all claims in the Answer. Verify if each claim is supported by the Context."

3. Answer Relevance (Did we help?)

"Relevance" measures: Does the Answer actually address the Query? Query: "What is the capital of France?" Answer: "France is a beautiful country in Europe." Faithfulness: High (Assuming context supports it). Relevance: Low (Didn't answer the question).

Detection: Ask an evaluator LLM: "Does the Answer resolve the intent of the Query?"

4. Context Relevance (Did we retrieve junk?)

A third metric. Query ↔ Context. Did we retrieve 5 documents about "Pizza" when the user asked about "Burgers"? Low Context Relevance confuses the LLM and lowers Faithfulness.

5. Summary

A good RAG answer is both Faithful (Grounded) and Relevant (Useful). You need high scores on both. One is not enough.

Key Intuition: "Tell the truth (Faithfulness), and tell me what I asked for (Relevance)."