How to Evaluate LangGraph Agents with Ragas: The Complete Beginner’s Guide

Learn how to evaluate LangGraph agents with Ragas. Step-by-step guide covering Faithfulness, Answer Relevancy, Context Precision, the bridge function, common errors, and agent-specific metrics.

Table of Contents

1. Why Your LangGraph Agent Needs an Inspector

2. What is Ragas? 

3. The RAG Triad: The Three Things Every Agent Must Get Right

4. The Problem: LangGraph State vs Ragas Format

5. Step-by-Step: Building a Simple Agent with Evaluation

6. The Bridge: Converting LangGraph State to Ragas Format

7. Running Your First Evaluation

8. Common Beginner Mistakes and Fixes

9. Agent-Specific Metrics (Beyond Basic RAG)

10. Production Tips

11. Limitations of Ragas

12. FAQs

13. The Bottom Line

1. Why Your LangGraph Agent Needs an Inspector ?

You’ve built your LangGraph agent. It’s working. But how do you know if it’s actually working well? How do you know if it’s hallucinating, retrieving the wrong documents, or giving answers that don’t even answer the question?

This is where Ragas comes in. Think of Ragas as an inspector who grades your agent’s work.

The Detective Analogy :

Remember the detective from the previous article? The one with the messy desk and the assistant who hands them the right files?

Now imagine you have a supervisor (the Inspector) who grades the detective’s work:

1. Did you open the right books? → Context Precision

2. Did you make stuff up? → Faithfulness

3. Did you actually answer what was asked? → Answer Relevancy

Why Manual Grading Doesn’t Scale ?

You could manually check 10 conversations. But when your agent handles 1,000 conversations a day, you need automated evaluation.

Ragas gives you a quantitative “scorecard” from 0 to 1 for every metric. You can track improvements over time, compare different agent configurations, and catch failures before they reach production.

2. What is Ragas? 

Ragas (Retrieval-Augmented Generation Assessment) is an open-source Python framework that scores RAG behavior using LLM-as-a-judge metrics.

The Simple Explanation :

Ragas doesn’t know the right answer. Instead, it uses an LLM (like GPT-4 or a local model) to judge your agent’s output against following key metrics:

What Ragas Measures:

The Big Three :

For most LangGraph agents, start with these three metrics:

1. Faithfulness catches hallucinations. It checks if every claim in your agent’s answer is supported by the retrieved documents. Score below 0.7 means your agent is making things up.

2. Answer Relevancy catches dodging. It checks if the answer actually addresses the question—even if it’s factually correct. Score below 0.7 means your agent is off-topic.

3. Context Precision catches noise. It checks if your retrieval node is pulling relevant documents or just filling the context with junk.

3. The RAG Triad: The Three Things Every Agent Must Get Right 

Here’s the visual anchor that ties everything together:

       [ Question ]
       /          \
Answer Relevancy   Context Precision
     /                  \
[ Answer ]  --------  [ Context ]
       Faithfulness

Understanding Each Metric :

Faithfulness (Groundedness) -> “Did you make stuff up?”

How it works: Your agent’s answer is broken down into claims. Each claim is checked against the retrieved context. If a claim is not in the context, it’s a hallucination.

Score Interpretation:

• >0.85: Your agent is trustworthy

• 0.70-0.85: Some hallucinations, needs work

• <0.70: Your agent is hallucinating badly. Fix retrieval or prompt engineering

Answer Relevancy ->“Did you actually answer the question?”

How it works: The answer is compared to the original question. If the answer addresses the question, score is high. If it’s off-topic or generic, score is low.

Score Interpretation:

• >0.85: Answers are on-point

• 0.70-0.85: Sometimes misses the mark

• <0.70: Agent is dodging questions or going off on tangents

Context Precision -> “Did you retrieve the right documents?”

How it works: Retrieval results are checked. Are the retrieved chunks relevant to the query? Precision measures if the top-ranked chunks are the right ones.

Score Interpretation:

• >0.85: Retrieval is solid

• 0.70-0.85: Some irrelevant chunks are getting through

• <0.70: Retrieval is noisy. Fix your embeddings or chunking strategy

4. The Problem: LangGraph State vs Ragas Format 

This is where every beginner gets stuck.

What LangGraph Gives You

LangGraph stores everything in a State object with messages (a list of BaseMessage objects):

class AgentState(TypedDict):
    messages: List[BaseMessage]  # All conversation history
    # ... your custom fields

What Ragas Expects :

Ragas expects a flat dictionary with specific fields:

{
    "user_input": ["What is LangGraph?"],           # List of strings
    "retrieved_contexts": [["doc1", "doc2"]],       # List of lists of strings
    "response": ["LangGraph is a framework..."],    # List of strings
    "reference": ["LangGraph is used for stateful LLM apps."]  # Optional
}

The Mismatch

LangGraph gives you messages. Ragas needs user_input, response, and contexts. You need to bridge the two.

This is where beginners pull their hair out. Without the bridge, you get errors about missing columns or wrong data types.

5. Step-by-Step: Building a Simple Agent with Evaluation 

Let’s build a simple LangGraph agent and evaluate it with Ragas.

Prerequisites

pip install langgraph langchain-openai ragas datasets

Step 1: Define Your State

from typing import List, TypedDict
from langchain_core.messages import BaseMessage

class GraphState(TypedDict):
    question: str                     # User's original question
    contexts: List[str]               # Retrieved documents
    messages: List[BaseMessage]       # All conversation history

Step 2: Define Your Nodes

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def retrieval_node(state: GraphState):
    """Simulates retrieving documents from a vector database."""
    question = state["question"]
    
    # Mock retrieval - in production, this would query a vector DB
    if "langgraph" in question.lower():
        docs = ["LangGraph is a library for building stateful, multi-actor LLM applications."]
    else:
        docs = ["Random document that doesn't answer the question."]
    
    return {"contexts": docs}

def generation_node(state: GraphState):
    """Generates the final answer using the retrieved context."""
    question = state["question"]
    contexts = "\n".join(state["contexts"])
    
    prompt = f"Answer the question: '{question}' using ONLY this context:\n{contexts}"
    response = llm.invoke([HumanMessage(content=prompt)])
    
    return {"messages": [response]}

Step 3: Build the Graph

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(GraphState)
workflow.add_node("retrieve", retrieval_node)
workflow.add_node("generate", generation_node)
workflow.add_edge(START, "retrieve")
workflow.add_edge("retrieve", "generate")
workflow.add_edge("generate", END)

graph = workflow.compile()

6. The Bridge: Converting LangGraph State to Ragas Format 

This is the most important function you’ll write. It extracts data from your LangGraph state and formats it for Ragas. 

What Exactly Is the “Bridge”?

Think of LangGraph and Ragas as two different countries that speak different languages.

• LangGraph speaks in State and Messages. Everything lives inside a state object: a list of HumanMessage, AIMessage, and ToolMessage objects, plus any custom fields you added (like contexts or question).
• Ragas speaks in a flat evaluation dictionary. It expects simple lists of strings with very specific keys: user_input, response, retrieved_contexts, and optionally reference.

The bridge is the small piece of code that translates LangGraph’s language into Ragas’ language.

Without this translation, Ragas simply cannot understand your agent’s output. You’ll get errors about missing columns, wrong data types, or mysterious MESSAGE_COERCION_FAILURE messages.

Why Do You Need It?

LangGraph is designed for running agents. It carefully tracks conversation history, tool calls, and intermediate state so the agent can continue reasoning.

Ragas is designed for grading agents. It doesn’t care about the full conversation history or the internal message objects. It only needs four clean pieces of information:

1. What did the user ask? (user_input)
2. What documents did the agent retrieve? (retrieved_contexts)
3. What final answer did the agent give? (response)
4. (Optional) What is the correct answer? (reference)

Because these two systems have completely different goals, you must convert the rich LangGraph state into the simple format Ragas expects.

When Do You Need the Bridge?

You need the bridge every time you want to run a Ragas evaluation on a LangGraph agent. Typical situations:

• After the agent finishes a single turn and you want to score it
• After a full multi-turn conversation (you evaluate the final answer against the original question)
• When you are building an automated evaluation pipeline that runs after every code change
• When you want to compare two different versions of the same agent

You do not need the bridge if you are only using Ragas with a simple LangChain chain (no LangGraph state).

How to Use It in LangGraph ?

Option 1: The Official Ragas Helper (Recommended)

Ragas 2026 has built-in support for LangGraph via convert_to_ragas_messages:

from ragas.integrations.langgraph import convert_to_ragas_messages

# Run your graph
final_state = graph.invoke({"question": "What is LangGraph?", "contexts": [], "messages": []})

# Convert using official helper
ragas_data = convert_to_ragas_messages(
    messages=final_state["messages"],
    user_input=final_state["question"],
    contexts=final_state["contexts"]
)

# Now you have a format Ragas understands!

Option 2: The Custom Bridge (Full Control)

If you need more control or want to understand what’s happening under the hood:

def convert_langgraph_to_ragas(final_state: dict, original_question: str, ground_truth: str = None) -> dict:
    """
    Converts LangGraph state to Ragas evaluation format.
    
    This is the bridge that every beginner needs.
    """
    # Extract the final answer from the last message
    messages = final_state.get("messages", [])
    if messages:
        # IMPORTANT: Extract .content from BaseMessage objects
        final_answer = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])
    else:
        final_answer = ""
    
    # Extract retrieved contexts (guarantee they're strings)
    contexts = final_state.get("contexts", [])
    # Ensure each context is a string
    contexts = [str(c) for c in contexts]
    
    # Format for Ragas
    eval_dict = {
        "user_input": [original_question],
        "retrieved_contexts": [contexts],  # List of lists
        "response": [final_answer],
    }
    
    if ground_truth:
        eval_dict["reference"] = [ground_truth]
    
    return eval_dict

Using the Bridge

# Run your graph
initial_state = {"question": "What is LangGraph?", "contexts": [], "messages": []}
final_state = graph.invoke(initial_state)

# Convert to Ragas format
eval_data = convert_langgraph_to_ragas(
    final_state=final_state,
    original_question="What is LangGraph?",
    ground_truth="LangGraph is a library for building stateful, multi-actor LLM applications."
)

# Create dataset
from datasets import Dataset
ragas_dataset = Dataset.from_dict(eval_data)

Important: The contexts Must Be Strings!

If your contexts are lists of dictionaries, the Dataset.from_dict() will fail with a PyArrow schema error. Use this to fix it:

# If contexts are documents with .page_content
contexts = [doc.page_content for doc in retrieved_docs]

# If contexts are anything else
contexts = [str(c) for c in contexts]

7. Running Your First Evaluation 

Now that you have your data in the right format, let’s run the evaluation.

Step 1: Set Up the Judge Model

Ragas uses an LLM to judge your agent’s output. By default, it uses OpenAI. You can customize it:

from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

# Option 1: Use GPT-4o-mini (recommended for quality)
judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

# Option 2: Use a larger model for better scoring
judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))

# Option 3: Use a local model (less reliable, but cheaper)
# from langchain_community.llms import Ollama
# judge_llm = LangchainLLMWrapper(Ollama(model="llama3:70b"))

Step 2: Run Evaluation

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

metrics = [faithfulness, answer_relevancy, context_precision]

scores = evaluate(
    dataset=ragas_dataset,
    metrics=metrics,
    llm=judge_llm  # Optional: override the default judge
)

print(scores.to_pandas())

Example Output:

   user_input  faithfulness  answer_relevancy  context_precision
0  What is LangGraph?          0.95             0.88                0.90

Interpreting Your Scores :

8. Common Beginner Mistakes and Fixes :

Mistake 1: Forgetting to Save Contexts in State

The Problem: Your retrieval node doesn’t save contexts to the state, so Ragas has nothing to evaluate.

The Fix:

def retrieval_node(state: GraphState):
    # ... retrieve documents
    return {"contexts": retrieved_docs}  # SAVE IT!

Mistake 2: Wrong Data Format for Ragas

The Problem: Ragas expects retrieved_contexts as a list of lists of strings. You pass a list of strings.

The Fix:

# ❌ Wrong
ragas_input = {
    "retrieved_contexts": ["doc1", "doc2"]  # List of strings
}

# ✅ Correct
ragas_input = {
    "retrieved_contexts": [["doc1", "doc2"]]  # List of lists
}

Mistake 3: Evaluating with Local LLMs

The Problem: Ragas uses LLM-as-a-judge. Small local models often return NaN scores because they can’t follow the complex evaluation prompt.

The Fix: Use a larger model for evaluation. GPT-4o-mini works well.

from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

scores = evaluate(
    dataset=ragas_dataset,
    metrics=[faithfulness, answer_relevancy],
    llm=judge_llm  # Override the default judge
)

Mistake 4: Not Using Ground Truth

The Problem: You skip reference (ground truth) but want to measure context precision.

The Fix: Context Precision requires ground truth. If you don’t have it, stick to Faithfulness and Answer Relevancy.

# Faithfulness and Answer Relevancy don't need ground truth
metrics = [faithfulness, answer_relevancy]

# Context Precision needs reference
metrics = [faithfulness, answer_relevancy, context_precision]
# Then include "reference" in your eval data

Mistake 5: MESSAGE_COERCION_FAILURE (The Most Common Error)

The Problem: You pass a BaseMessage object (like HumanMessage or AIMessage) to Ragas instead of its .content string.

The Fix: Always extract .content before passing to Ragas.

# ❌ Wrong: Passing BaseMessage object
final_answer = state["messages"][-1]  # This is a BaseMessage object

# ✅ Correct: Passing .content string
final_answer = state["messages"][-1].content  # This is the actual text

# What the error looks like:
# ValueError: Message dict must contain 'role' and 'content' keys
# The error happens because Ragas sees the BaseMessage object and tries to serialize it

How to fix it in your bridge function:

def convert_langgraph_to_ragas(final_state: dict, original_question: str) -> dict:
    messages = final_state.get("messages", [])
    if messages:
        # ✅ ALWAYS extract .content
        final_answer = messages[-1].content if hasattr(messages[-1], 'content') else str(messages[-1])
    else:
        final_answer = ""
    
    # ... rest of function

Mistake 6: PyArrow Schema Error with Dataset.from_dict()

The Problem: Dataset.from_dict() fails because retrieved_contexts contains elements that aren’t strings.

# ❌ Wrong: contexts is a list of dictionaries
contexts = [{"text": "doc1", "score": 0.9}, {"text": "doc2", "score": 0.8}]
ragas_input = {"retrieved_contexts": [contexts]}  # PyArrow error!

# ✅ Correct: contexts is a list of strings
contexts = ["doc1", "doc2"]
ragas_input = {"retrieved_contexts": [contexts]}  # Works!

Fix it:

# Convert documents to strings
contexts = [doc.page_content for doc in retrieved_docs]
# or
contexts = [str(doc) for doc in retrieved_docs]

9. Agent-Specific Metrics (Beyond Basic RAG)

Ragas 2026 now supports metrics specifically for full agents (tool-using, multi-turn):

Tool Call Accuracy

What it checks: Did your agent call the right tools at the right time?

When to use: When your agent has multiple tools and needs to select the correct one.

Agent Goal Accuracy

What it checks: Did your agent achieve the user’s stated goal?

When to use: For task-oriented agents (e.g., “Book a flight” or “Order a pizza”).

Topic Adherence

What it checks: Did your agent stay on topic or go off on tangents?

When to use: For customer support or domain-specific agents.

Multi-Turn Evaluation (MultiTurnSample)

What it checks: How does your agent perform across a conversation with multiple turns?

When to use: For conversational agents that need to remember context across exchanges.

10. Production Tips :

Tip 1: Build a Quality Test Dataset

For reliable evaluation, build at least 100-200 curated examples with ground truth. Focus on edge cases and common failure modes.

Tip 2: Use a Real Vector Database

In production, replace the mock retrieval with a real vector store:

# Example with Chroma
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
vector_store = Chroma(collection_name="docs", embedding_function=embeddings)

def retrieval_node(state: GraphState):
    docs = vector_store.similarity_search(state["question"], k=5)
    contexts = [doc.page_content for doc in docs]
    return {"contexts": contexts}

Tip 3: Track Metrics Over Time

import pandas as pd

def track_evaluation(scores):
    """Store scores for trend analysis."""
    df = pd.DataFrame(scores)
    df["timestamp"] = pd.Timestamp.now()
    df.to_csv("eval_history.csv", mode="a", header=False)

Tip 4: Set Quality Thresholds

# Alert if your agent is hallucinating too much
if scores["faithfulness"] < 0.8:
    print("⚠️ Agent faithfulness dropped below 0.8! Needs attention.")

Tip 5: Handle Multi-Turn and Agentic Behavior

• Original query vs rewritten query: Evaluate against the original user query

• Multiple retrieval passes: Combine all retrieved contexts into one list

• Agent refusal: Don’t penalize refusals if they’re correct behavior

11. Limitations of Ragas :

Ragas Uses LLMs as Judges

Ragas relies on LLMs to evaluate your agent’s output. This has three significant costs:

1. Cost: Running a judge model (even GPT-4o-mini) across 100 test cases costs money

2. Latency: Each metric requires multiple LLM calls, adding seconds to your evaluation

3. Bias: The judge model’s preferences can influence scores

Local Small Models Are Unreliable

Small local models (<7B parameters) often produce NaN scores or inconsistent results. For reliable scoring, use GPT-4o-mini or larger.

Timeliness

Ragas scores can vary between runs with the same dataset. This is inherent to LLM-as-judge evaluation. To minimize this:

• Use temperature=0

• Run multiple times and average scores

• Focus on trends rather than absolute numbers

Dataset Requirements

Ragas works best with:

• 50+ examples for meaningful signal (100-200 recommended)

• Diverse examples covering edge cases, not just easy ones

• Ground truth for Context Precision and Recall metrics

12. FAQs :

What’s the difference between Faithfulness and Answer Relevancy?

Faithfulness catches hallucinations (“Did you make stuff up?”). Answer Relevancy catches dodging (“Did you actually answer the question?”). An answer can be faithful to the context but completely irrelevant to the question.

Do I need ground truth for Ragas?

Not for Faithfulness and Answer Relevancy. Yes for Context Precision and Context Recall.

Why are my Ragas scores NaN?

You’re likely using a small local model as the judge. Use GPT-4o-mini or larger for evaluation.

What’s the correct format for retrieved_contexts?

retrieved_contexts must be a list of lists of strings. Each inner list is the contexts for one example.

# ✅ Correct format
{
    "retrieved_contexts": [
        ["doc1", "doc2"],  # Example 1's contexts
        ["doc3", "doc4"]   # Example 2's contexts
    ]
}

How do I handle agent loops in Ragas?

Run your agent to completion, then evaluate the final state with the original user question. Combine all retrieved contexts from each loop into a single list.

What’s a good score?

Can I use Ragas with local LLMs?

Yes, but use a capable model like Mixtral or Llama-3-70B. Small models (<7B) often produce unreliable scores.

What’s MESSAGE_COERCION_FAILURE?

It happens when you pass a BaseMessage object (like HumanMessage) instead of its .content string. Always extract .content before passing to Ragas.

Why is my Dataset.from_dict() failing?

Make sure retrieved_contexts is a list of lists of strings. If your contexts are documents, extract .page_content first.

13. The Bottom Line :

Your LangGraph agent needs an inspector. Ragas is that inspector.

The Bridge Function is Everything: LangGraph gives you messages, Ragas needs user_input, response, and contexts. Write the bridge function once and reuse it everywhere.

The Three Key Metrics: Faithfulness (hallucinations), Answer Relevancy (dodging), Context Precision (noise). Master these three and you’ll catch 90% of failures.

The Golden Rules:

1. Always extract .content from BaseMessage objects

2. Always convert contexts to strings

3. Always wrap retrieved_contexts in an extra list

4. Use GPT-4o-mini as the judge for reliable scores

Ragas turns your LangGraph agent from “it feels like it works” into “I can prove it works.”