Learn the industry-standard metrics for RAG evaluation, including the RAG Triad, retrieval precision/recall, MRR, NDCG, groundedness, faithfulness, LLM judges, and continuous monitoring.
Evaluation Metrics in RAG: What to Measure and Why It Matters
The trap most teams fall into
I once watched a team launch a RAG system that looked perfect in demos. Fluent answers. Fast responses. No complaints for a week. Then a customer asked a slightly different question, and the system invented a policy that didn’t exist. The team had no idea because they’d measured “look and feel” but not groundedness or retrieval quality. That’s the trap.
A RAG system can retrieve the right documents and still answer badly. It can also produce a fluent answer while ignoring the context entirely. Good evaluation is the only way to know which part of the pipeline is broken.
What RAG evaluation actually measures
RAG evaluation should answer five questions:
Did we retrieve the right information?
Did the model use that information?
Did the answer actually solve the user’s question?
Did the system stay safe and grounded?
How much did it cost in latency and tokens?
The key point is that RAG evaluation is not one metric. It is a layered measurement system that scores retrieval, generation, and system health separately.
The RAG Triad
Modern RAG evaluation is often organized as the RAG Triad: Context Relevance, Groundedness, and Answer Relevance. This is the cleanest way to explain the architecture to both beginners and experienced teams.
[ User Query ]
╱ ╲
Context ╱ ╲ Answer
Relevance ╱ ╲ Relevance
▼ ▼
[ Retrieved Context ] ───> [ Final Answer ]
Groundedness
Context Relevance asks whether retrieved chunks actually relate to the user’s intent.
Groundedness asks whether each claim is supported by the retrieved context.
Answer Relevance asks whether the final answer directly addresses the question.
If any one of these fails, the system risks drifting into hallucination or task mismatch.
The evaluation split
You cannot evaluate a RAG pipeline with one broad brush. Production systems split testing into two distinct environments.
Offline evaluation
Offline evaluation happens before deployment and uses a golden dataset with questions, retrieved context, and ground-truth answers. This is where you measure metrics that require labels, such as Precision@K, Recall@K, MRR, and NDCG.
Online evaluation
Online evaluation happens on live traffic where you usually do not have ground-truth answers. This is where you measure groundedness, faithfulness, latency, token cost, and drift using judge models or NLI-style validators.
Offline vs online
That split matters because the wrong metric in the wrong environment leads to false confidence.
Core retrieval metrics
Retrieval metrics tell you whether the right evidence reached the model.
Retrieval math metrics
When rank matters, simple hit counts are not enough.
MRR and NDCG code:
import math
from typing import List
def calculate_mrr(relevant_ranks: List[int]) -> float:
if not relevant_ranks:
return 0.0
return sum(1.0 / rank for rank in relevant_ranks) / len(relevant_ranks)
def calculate_ndcg(relevant_scores: List[float], k: int = 10) -> float:
if not relevant_scores:
return 0.0
dcg = 0.0
for i, score in enumerate(relevant_scores[:k]):
if score > 0:
dcg += score / math.log2(i + 2)
ideal_scores = sorted(relevant_scores, reverse=True)
idcg = 0.0
for i, score in enumerate(ideal_scores[:k]):
if score > 0:
idcg += score / math.log2(i + 2)
return dcg / idcg if idcg > 0 else 0.0
relevant_ranks = [1, 3, 5]
print(f”MRR: {calculate_mrr(relevant_ranks):.3f}”)
scores = [3, 2, 0, 1, 0]
print(f”NDCG@5: {calculate_ndcg(scores, k=5):.3f}”)
Core response metrics:
Response metrics measure what the LLM actually did with the context.
A response can be correct by luck but still not be grounded. That makes it fragile.
Context sufficiency versus groundedness:
These terms are related, but they are not the same.
The judge pattern:
In production, teams usually avoid brittle keyword matching and use a judge model or semantic evaluator. A practical path is to use a framework like RAGAS, which provides built-in metrics for faithfulness, answer relevance, context precision, and context recall.
import os
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(api_key=os.environ.get(“OPENAI_API_KEY”))
async def evaluate_groundedness(context: str, answer: str) -> float:
judge_system_prompt = “””
You are a strict verification engine.
Evaluate whether the statement is fully supported by the context.
Return exactly one float from 0.0 to 1.0:
– 1.0 = fully supported
– 0.5 = partially supported
– 0.0 = unsupported or contradictory
“””
user_payload = f”CONTEXT:\n{context}\n\nSTATEMENT:\n{answer}”
response = await client.chat.completions.create(
model=”gpt-4o-mini”,
messages=[
{“role”: “system”, “content”: judge_system_prompt},
{“role”: “user”, “content”: user_payload}
],
temperature=0.0
)
return float(response.choices[0].message.content.strip())
LLM judge calibration note:
LLM judges need calibration. A judge model may be biased toward fluency over factual accuracy, or it may over-penalize style differences. The recommended practice is to validate judge scores against human ratings on a small subset of 50 to 100 examples before trusting the judge in production.
That calibration step matters because an uncalibrated judge can be confidently wrong.
Why RAGAS matters
RAGAS is useful because it gives teams a ready-made framework for evaluating RAG quality without building everything from scratch. It is especially helpful for faithfulness, context recall, context precision, and answer relevance, which are the core fields most teams need to monitor.
A practical workflow is to start with RAGAS for baseline scoring, then add custom judge logic for domain-specific edge cases.
Production-grade evaluation pipeline:
from typing import List, Dict, Optional
import time
import json
class RAGEvaluator:
def __init__(self, llm_judge_client=None):
self.judge = llm_judge_client
self.results = []
def evaluate_live_turn(
self,
query: str,
retrieved_chunks: List[str],
generated_answer: str
) -> Dict:
start_time = time.time()
context_relevance = self._judge_score(“context_relevance”, query, retrieved_chunks)
groundedness = self._judge_score(“groundedness”, generated_answer, retrieved_chunks)
answer_relevance = self._judge_score(“answer_relevance”, query, generated_answer)
latency = time.time() – start_time
return {
“query”: query,
“metrics”: {
“context_relevance”: context_relevance,
“groundedness”: groundedness,
“answer_relevance”: answer_relevance
},
“system”: {
“latency_seconds”: round(latency, 4)
},
“status”: “PASS” if (groundedness > 0.8 and answer_relevance > 0.8) else “FAIL”
}
def _judge_score(self, metric_type: str, primary, secondary) -> float:
if metric_type == “context_relevance”:
return 0.95 if secondary else 0.0
if metric_type == “groundedness”:
return 0.92
if metric_type == “answer_relevance”:
return 0.93
return 0.0
evaluator = RAGEvaluator()
report = evaluator.evaluate_live_turn(
query=”What is the internal casual leave policy?”,
retrieved_chunks=[“Casual leave is capped at 15 days annually.”, “Manager log required.”],
generated_answer=”Employees receive 15 days of casual leave per year.”
)
print(json.dumps(report, indent=2))
Important note on the code:
The code above is a structural pattern. In real production systems, teams plug in a live judge model, a semantic evaluator, or an NLI-based scorer instead of fixed placeholder values. The point is to show the architecture of the evaluation loop, not to pretend that string matching is enough.
Metric selection matrix:
If you want to know which metric to use, start here.
Evaluation itself has a cost, especially when you use judge models.
def estimate_evaluation_cost(eval_runs: int, tokens_per_run: int, cost_per_1k_tokens: float = 0.002) -> dict:
total_tokens = eval_runs * tokens_per_run
total_cost = total_tokens * cost_per_1k_tokens / 1000
return {
“eval_runs”: eval_runs,
“tokens_per_run”: tokens_per_run,
“total_tokens”: total_tokens,
“estimated_cost_usd”: round(total_cost, 3)
}
print(estimate_evaluation_cost(1000, 500))
This matters because teams often forget that evaluating 10,000 live turns with an LLM judge is still a budget line.
Generating an evaluation dataset:
A serious RAG system needs a labeled test set.
from typing import List, Dict
def generate_eval_set(source_docs: List[str], num_samples: int = 100) -> List[Dict]:
eval_set = []
for doc in source_docs[:num_samples]:
eval_set.append({
“question”: f”What does this document say about {doc[:50]}?”,
“ground_truth”: doc,
“relevant_chunk”: doc
})
return eval_set
source_docs = [
“Employees get 15 days of leave per year.”,
“Expenses require manager approval.”,
“Remote work is allowed only on approved days.”
]
print(generate_eval_set(source_docs, num_samples=3))
In practice, teams usually generate questions from chunks, then validate them before adding them to the test set.
Test set sizing guidance:
Complete evaluation pipeline:
from typing import List, Dict, Optional
import time
import json
class RAGEvaluator:
def __init__(self, llm_judge_client=None):
self.judge = llm_judge_client
self.results = []
def evaluate_single(
self,
query: str,
retrieved_docs: List[str],
generated_answer: str,
ground_truth: Optional[str] = None
) -> Dict:
start_time = time.time()
precision = self._calculate_precision(retrieved_docs, query)
recall = self._calculate_recall(retrieved_docs, ground_truth) if ground_truth else None
context_sufficiency = self._check_context_sufficiency(retrieved_docs, query)
answer_relevance = self._check_relevance(generated_answer, query)
groundedness = self._check_groundedness(generated_answer, retrieved_docs)
faithfulness = self._check_faithfulness(generated_answer, retrieved_docs)
correctness = self._check_correctness(generated_answer, ground_truth) if ground_truth else None
latency = time.time() – start_time
token_count = self._estimate_tokens(query, retrieved_docs, generated_answer)
return {
“query”: query,
“retrieval”: {
“precision”: precision,
“recall”: recall,
“context_sufficiency”: context_sufficiency
},
“response”: {
“relevance”: answer_relevance,
“groundedness”: groundedness,
“faithfulness”: faithfulness,
“correctness”: correctness
},
“system”: {
“latency_seconds”: latency,
“estimated_tokens”: token_count
},
“passed”: self._check_thresholds(precision, groundedness, faithfulness)
}
def evaluate_batch(self, test_cases: List[Dict]) -> Dict:
results = []
for case in test_cases:
result = self.evaluate_single(
case[“query”],
case[“retrieved_docs”],
case[“generated_answer”],
case.get(“ground_truth”)
)
results.append(result)
agg = {
“avg_precision”: sum(r[“retrieval”][“precision”] for r in results) / len(results),
“avg_groundedness”: sum(r[“response”][“groundedness”] for r in results) / len(results),
“avg_faithfulness”: sum(r[“response”][“faithfulness”] for r in results) / len(results),
“avg_latency”: sum(r[“system”][“latency_seconds”] for r in results) / len(results),
“pass_rate”: sum(1 for r in results if r[“passed”]) / len(results)
}
return {“results”: results, “aggregate”: agg}
def _calculate_precision(self, docs, query):
terms = set(query.lower().split())
if not docs:
return 0.0
relevant = sum(1 for doc in docs if any(t in doc.lower() for t in terms))
return relevant / len(docs)
def _calculate_recall(self, docs, ground_truth):
if not docs or not ground_truth:
return 0.0
gt_terms = set(ground_truth.lower().split())
doc_text = ” “.join(docs).lower()
matched = sum(1 for t in gt_terms if t in doc_text)
return matched / len(gt_terms) if gt_terms else 0.0
def _check_context_sufficiency(self, docs, query):
if not docs:
return 0.0
joined = ” “.join(docs).lower()
return 1.0 if any(t.lower() in joined for t in query.split()) else 0.0
def _check_relevance(self, answer, query):
return 1.0 if any(t.lower() in answer.lower() for t in query.split()) else 0.0
def _check_groundedness(self, answer, docs):
joined = ” “.join(docs).lower()
return 1.0 if any(term in joined for term in answer.lower().split()[:8]) else 0.0
def _check_faithfulness(self, answer, docs):
return self._check_groundedness(answer, docs)
def _check_correctness(self, answer, ground_truth):
if not ground_truth:
return None
return 1.0 if answer.strip().lower() == ground_truth.strip().lower() else 0.0
def _estimate_tokens(self, query, docs, answer):
return len(query.split()) + sum(len(d.split()) for d in docs) + len(answer.split())
def _check_thresholds(self, precision, groundedness, faithfulness):
return precision > 0.7 and groundedness > 0.8 and faithfulness > 0.8
evaluator = RAGEvaluator()
result = evaluator.evaluate_single(
query=”What is the leave policy?”,
retrieved_docs=[“Employees get 15 days of leave.”, “Leave requires approval.”],
generated_answer=”Employees get 15 days of leave with manager approval.”,
ground_truth=”Employees get 15 days of leave per year.”
)
print(json.dumps(result, indent=2))
Continuous monitoring:
Evaluation should not stop after launch.
class RAGMonitor:
def __init__(self, window: int = 1000):
self.window = window
self.metrics = []
def log_evaluation(self, eval_result: Dict):
self.metrics.append(eval_result)
if len(self.metrics) > self.window:
self.metrics.pop(0)
def get_drift_alert(self) -> Dict:
recent = self.metrics[-min(100, len(self.metrics)):]
old = self.metrics[-min(200, len(self.metrics)):-100]
if not old or not recent:
return {“alert”: False}
avg_recent = sum(m[“response”][“groundedness”] for m in recent) / len(recent)
avg_old = sum(m[“response”][“groundedness”] for m in old) / len(old)
if avg_recent < avg_old – 0.1:
return {“alert”: True, “metric”: “groundedness”, “drop”: avg_old – avg_recent}
return {“alert”: False}
Retrieval-generation correlation analysis:
This is one of the fastest ways to understand where the system is failing.
from typing import List, Dict
def analyze_pipeline_performance(eval_results: List[Dict]) -> Dict:
good_retrieval_bad_generation = 0
bad_retrieval_good_generation = 0
both_good = 0
both_bad = 0
for result in eval_results:
retrieval_ok = result[“retrieval”][“precision”] > 0.7
generation_ok = result[“response”][“groundedness”] > 0.8
if retrieval_ok and not generation_ok:
good_retrieval_bad_generation += 1
elif not retrieval_ok and generation_ok:
bad_retrieval_good_generation += 1
elif retrieval_ok and generation_ok:
both_good += 1
else:
both_bad += 1
total = len(eval_results) if eval_results else 1
return {
“retrieval_problem”: bad_retrieval_good_generation / total,
“generation_problem”: good_retrieval_bad_generation / total,
“both_good”: both_good / total,
“both_bad”: both_bad / total,
“recommendation”: (
“Fix retrieval” if bad_retrieval_good_generation > 0.3 else
“Fix generation” if good_retrieval_bad_generation > 0.3 else
“Both need work” if both_bad > 0.3 else
“System is healthy”
)
}
Why semantic evaluation matters ?
A naive keyword-based evaluator can fail badly in real semantic settings. That is why the standard practice is to use a semantic judge, an NLI verifier, or a framework such as RAGAS instead of raw token overlap. The judge should score meaning, not just word matching.
LLM judge calibration note:
LLM judges need calibration. Validate judge scores against human ratings on a small subset of 50 to 100 examples before trusting them in production. Otherwise, the judge may reward fluent but unsupported answers.
Cost of evaluation code
def estimate_evaluation_cost(eval_runs: int, tokens_per_run: int, cost_per_1k_tokens: float = 0.002) -> dict:
total_tokens = eval_runs * tokens_per_run
total_cost = total_tokens * cost_per_1k_tokens / 1000
return {
“eval_runs”: eval_runs,
“tokens_per_run”: tokens_per_run,
“total_tokens”: total_tokens,
“estimated_cost_usd”: round(total_cost, 3)
}
print(estimate_evaluation_cost(1000, 500))
That gives teams a real answer to the question: “How much will evaluation cost?”
It also matters for p95 latency and token cost. Over-retrieving to compensate for weak evaluation creates token bloat and slows the entire system.
Evaluation approach comparison
Quick-reference card
Measure retrieval and response separately.
Add the RAG Triad: context relevance, groundedness, answer relevance.
Use Precision@K, Recall@K, MRR, and NDCG offline.
Use groundedness and faithfulness online.
Calibrate LLM judges before production use.
Track latency, token count, and cost continuously.
Treat evaluation as a monitoring system, not a one-time test.
Glossary
Context Relevance: Whether retrieved chunks match the query intent.
Groundedness: Whether the answer is supported by retrieved context.
Answer Relevance: Whether the answer directly addresses the user’s question.
MRR: Metric that rewards putting the first relevant document near the top.
NDCG: Ranking metric that discounts relevant documents based on position.
Faithfulness: Whether the answer stays faithful to the source.
Context Sufficiency: Whether retrieved evidence is enough to answer.
LLM Judge: A separate model used to evaluate outputs.
RAGAS: A framework for evaluating RAG with standard metrics.
RAG evaluation is not a single score. It is a layered measurement system that tells you whether your retriever is finding the right evidence, whether your generator is using it correctly, and whether the system is still safe, grounded, and affordable at scale. If you only measure “does it look good?”, you are not evaluating RAG. You are guessing. The best teams measure retrieval quality, groundedness, answer relevance, ranking quality, cost, latency, and drift together — then use those metrics to improve the pipeline continuously.