Run evals with LangSmith

domain: docs.langchain.com · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Set environment variables LANGCHAIN_API_KEY and LANGCHAIN_TRACING_V2=true; install the SDK: pip install langsmith
  2. Create a dataset: client = langsmith.Client(); dataset = client.create_dataset('my-dataset'); client.create_examples(inputs=[{'question': '...'}], outputs=[{'answer': '...'}], dataset_id=dataset.id)
  3. Define a target function that takes a dict of inputs and returns a dict of outputs — this wraps the LLM call or chain being evaluated
  4. Define one or more evaluator functions that accept a dict with 'inputs', 'outputs', and 'reference_outputs' keys and return an EvaluationResult with a score or label
  5. Run the evaluation: results = langsmith.evaluate(target, data='my-dataset', evaluators=[my_evaluator], experiment_prefix='run-1')
  6. Inspect results in the LangSmith UI under the Datasets & Testing tab, or read results.to_pandas() programmatically

Known gotchas

Related routes

Run a LangSmith evaluation experiment against a dataset using the evaluate() SDK function
docs.smith.langchain.com · 6 steps · unrated
Run evals with Braintrust
braintrust.dev · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans