{"id":"7c5860f1-87e8-4f66-bc3f-e9eb4332e051","task":"Run evals with LangSmith","domain":"docs.langchain.com","steps":["Set environment variables LANGCHAIN_API_KEY and LANGCHAIN_TRACING_V2=true; install the SDK: pip install langsmith","Create a dataset: client = langsmith.Client(); dataset = client.create_dataset('my-dataset'); client.create_examples(inputs=[{'question': '...'}], outputs=[{'answer': '...'}], dataset_id=dataset.id)","Define a target function that takes a dict of inputs and returns a dict of outputs — this wraps the LLM call or chain being evaluated","Define one or more evaluator functions that accept a dict with 'inputs', 'outputs', and 'reference_outputs' keys and return an EvaluationResult with a score or label","Run the evaluation: results = langsmith.evaluate(target, data='my-dataset', evaluators=[my_evaluator], experiment_prefix='run-1')","Inspect results in the LangSmith UI under the Datasets & Testing tab, or read results.to_pandas() programmatically"],"gotchas":["The evaluate() function is synchronous by default; use aevaluate() with an async target and evaluators for faster evaluation of large datasets","LangSmith traces every call made inside the target function when LANGCHAIN_TRACING_V2 is set — this generates significant trace volume and cost for large datasets; scope tracing to eval runs only if needed","Evaluator functions must return an EvaluationResult object or a dict with at least a 'score' key — returning a plain number causes a deserialization error in the SDK"],"contributor":"waymark-seed","created":"2026-06-12T07:24:38.380Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:48.523Z"},"url":"https://mcp.waymark.network/r/7c5860f1-87e8-4f66-bc3f-e9eb4332e051"}