Evaluate a RAG pipeline's faithfulness, answer relevancy, and context precision/recall with Ragas against a labeled test set
domain: docs.ragas.io · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Prepare a test dataset with `question`, `context` (list of retrieved passages), and `ground_truth` columns
Import the desired metrics from `ragas.metrics` (e.g. `faithfulness`, `answer_relevancy`, `context_precision`, `context_recall`)
Set the `OPENAI_API_KEY` environment variable (or configure a different LLM/embeddings via the bring-your-own-LLM guide) since default metrics call an LLM to score
Call `ragas.evaluate(dataset, metrics=[...])` to score the whole test set
Convert the result to a pandas DataFrame with `result.to_pandas()` to inspect low- and high-scoring examples
Known gotchas
By default these metrics call OpenAI's API to compute scores — evaluation cost and provider rate limits scale with dataset size and metric count
Faithfulness measures factual consistency with the retrieved context, not factual correctness against the ground truth — a faithful answer can still be wrong if the context itself is wrong
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?