Create a LangSmith dataset and run an offline evaluation experiment with code and LLM-as-judge evaluators

Documented steps

1. Install the SDKs: pip install -U langsmith openevals (Python) or yarn add langsmith openevals @langchain/core (TypeScript). openevals supplies the prebuilt LLM-as-judge scaffolding.
2. Set LANGSMITH_API_KEY and, if you are not on the default US host, LANGSMITH_ENDPOINT. Instantiate the client: from langsmith import Client; client = Client().
3. Create the dataset: dataset = client.create_dataset(dataset_name='Sample dataset', description='...'). In TypeScript: const dataset = await client.createDataset('Sample dataset', { description: '...' }).
4. Populate it in bulk rather than one call per row: client.create_examples(dataset_id=dataset.id, examples=[{'inputs': {...}, 'outputs': {...}}, ...]). TypeScript uses client.createExamples({ datasetId, inputs, outputs, metadata }).
5. Define the target: a function that accepts an example's inputs dict and returns the system output. This is the thing under test (prompt, chain, agent, or model call).
6. Write code-based evaluators as plain functions with the documented signature: def correct(inputs: dict, outputs: dict, reference_outputs: dict) -> bool. Returning a dict lets you name the feedback key and attach a comment.
7. For subjective criteria add an LLM judge: from openevals.llm import create_llm_as_judge; from openevals.prompts import CORRECTNESS_PROMPT; evaluator = create_llm_as_judge(prompt=CORRECTNESS_PROMPT, feedback_key='correctness', model='openai:o3-mini'). CONCISENESS_PROMPT is another prebuilt template. TypeScript uses createLLMAsJudge({ prompt, feedbackKey, model }).
8. Add summary_evaluators if you need aggregate metrics computed across the whole experiment rather than per example.
9. Run the experiment: client.evaluate(target, data='<dataset name or id>', evaluators=[...], summary_evaluators=[...], experiment_prefix='first-eval', max_concurrency=2). Other documented parameters are metadata, description, blocking, num_repetitions, experiment, and error_handling.
10. For non-deterministic targets set num_repetitions>1 so per-example variance is visible instead of being mistaken for a regression.
11. Inspect the returned experiment_results object, then open the Experiments tab under the dataset in the UI to compare runs side by side.
12. Pin a stable example set for CI by tagging a dataset version: client.update_dataset_tag(..., tag='prod'). Every example add, update or delete creates a new dataset version, so untagged evaluations silently drift.
13. Re-run evaluate() against the same dataset and tag whenever you change a prompt or model, and diff the experiments to catch regressions.
14. Export the dataset via the UI or API in CSV, JSONL, or OpenAI fine-tuning format when you need it outside LangSmith.