Build a RAG retrieval evaluation pipeline using RAGAS to measure faithfulness and answer relevancy
domain: docs.ragas.io · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Assemble a dataset of question, answer, contexts (list of retrieved chunks), and ground_truth strings as a Hugging Face Dataset or pandas DataFrame
Install ragas and import evaluate along with the desired metrics: faithfulness, answer_relevancy, context_recall, context_precision
Run result = evaluate(dataset, metrics=[faithfulness, answer_relevancy], llm=<llm_wrapper>, embeddings=<embeddings_wrapper>)
Inspect result.to_pandas() to identify per-sample failures — low faithfulness scores indicate hallucinations relative to the retrieved context
Iterate on chunk size, embedding model, or retrieval top-k by re-running the pipeline and comparing aggregate metric scores
Known gotchas
RAGAS metrics use an LLM judge internally — the quality of RAGAS scores is bounded by the judge model's capability; a weak judge model will produce unreliable faithfulness scores
context_recall requires a ground_truth string and uses the judge LLM to assess whether the ground truth is entailed by the retrieved contexts — it is not a pure embedding similarity metric
RAGAS API changed significantly between v0.1 and v0.2; the evaluate() function signature, metric import paths, and dataset schema differ between versions
Give your agent this knowledge — and 15,600+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?