Score RAG pipeline outputs with Ragas faithfulness and context precision metrics
domain: docs.ragas.io · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Install ragas and a supported LLM client (e.g., the OpenAI SDK) which Ragas uses as its judge model
Prepare an evaluation dataset as a list of dicts containing question, answer, contexts (list of retrieved chunks), and optionally ground_truth
Wrap the dataset using ragas.dataset_schema.EvaluationDataset or convert it to a Hugging Face Dataset object
Select metrics from ragas.metrics such as Faithfulness, AnswerRelevancy, ContextPrecision, and ContextRecall
Call ragas.evaluate(dataset, metrics=[...]) to run all selected metrics; Ragas makes LLM judge calls internally
Inspect the returned result object for per-metric scores and the aggregate ragas_score, and export to a dataframe for further analysis
Known gotchas
Faithfulness and AnswerRelevancy require a configured LLM judge; if no LLM is set Ragas falls back to a default that may not match your environment and produces unexpected scores
ContextRecall requires ground_truth in the dataset; omitting it causes that metric to silently return NaN rather than raising an error
Ragas makes multiple LLM calls per sample per metric — evaluating large datasets without concurrency limits can exhaust rate quotas quickly
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?