Evaluating RAG Systems: Metrics That Actually Matter

"It seems better" is not an evaluation

Every technique in this tier is a claim that it improves something. Testing that claim needs metrics that separate retrieval quality from generation quality, the same separation CRAG (Lesson 14) makes structurally, because a RAG system can fail at either stage independently, and a metric that only looks at the final answer cannot tell you which one broke.

Retrieval-side metrics

Context precision asks: of the chunks that were retrieved, how many were actually relevant? Low precision means the system is retrieving noise alongside signal. Context recall asks the opposite question: of the chunks that were actually relevant somewhere in the knowledge base, how many did retrieval actually find? Low recall means relevant material exists but never reaches the model at all, a failure invisible from the answer alone if the model happens to already know part of the answer from its own training.

Generation-side metrics

Faithfulness (also called groundedness) asks: is every claim in the generated answer actually supported by the retrieved context, or did the model add something from its own memory that was not in the evidence? This is the metric most directly aimed at hallucination within a RAG system specifically. Answer relevance asks the simpler question of whether the generated answer actually addresses the question asked, independent of whether it is faithful to the context.

Why measuring both sides matters

A system can have perfect context precision and recall and still produce an unfaithful answer, because the model ignored good evidence and used its own memory instead. A system can also have a perfectly faithful, well-written answer built entirely from irrelevant context, because it faithfully summarized the wrong chunks. Only measuring both retrieval-side and generation-side metrics together tells you which stage to fix when a RAG system's answers are wrong.