Create a Langfuse dataset and dataset items, then run a dataset experiment and compare runs across prompt or model versions

Documented steps

1. Create the dataset. Python: `langfuse.create_dataset(name="<dataset_name>", description="My first dataset", metadata={"author": "Alice"})`. JS/TS: `await langfuse.api.datasets.create({ name: "<dataset_name>", description: "My first dataset", metadata: { author: "Alice" } })`.
2. Add items. Python: `langfuse.create_dataset_item(dataset_name="<dataset_name>", input={"text": "hello world"}, expected_output={"text": "hello world"})`. JS/TS: `await langfuse.dataset.createItem({ datasetName: "<dataset_name>", input: { text: "hello world" }, expectedOutput: { text: "hello world" } })`. input, expected_output, and metadata are optional and accept arbitrary JSON.
3. Fetch the dataset before running. Python: `dataset = langfuse.get_dataset(name="my-dataset")`. JS/TS: `const dataset = await langfuse.dataset.get("my-dataset")`.
4. Run the experiment bound to the dataset so Langfuse creates a DatasetRun and links one trace per item. Python: `result = dataset.run_experiment(name="Baseline Experiment v1", description="Monthly evaluation", task=lambda *, item, **kwargs: item.expected_output)`. JS/TS: `const result = await dataset.runExperiment({ name: "Baseline Experiment v1", task: async (item) => item.expectedOutput })`.
5. Define a real task function with the documented signature. Python: `def my_task(*, item, **kwargs): question = item["input"]; ...; return response`. JS/TS: `const myTask: ExperimentTask = async (item) => { const question = item.input; ...; return response; }`.
6. To run against arbitrary local data instead of a hosted dataset, use the standalone runner. Python: `langfuse.run_experiment(name="Geography Quiz", description="Testing basic functionality", data=local_data, task=my_task, evaluators=[...], run_evaluators=[...], max_concurrency=10)`. JS/TS: `await langfuse.experiment.run({ name, description, data: localData, task: myTask })`.
7. Attach item-level evaluators to score each output automatically. Python: `def accuracy_evaluator(*, input, output, expected_output, metadata, **kwargs): return Evaluation(name="accuracy", value=1.0, comment="...")`. Run-level: `def average_accuracy(*, item_results, **kwargs): return Evaluation(name="avg_accuracy", value=avg)`. JS/TS evaluators receive `{ input, output, expectedOutput }` and run-level receive `{ itemResults }`.
8. Print a quick summary from the result object: `print(result.format())` in Python, `console.log(await result.format())` in JS/TS.
9. Compare runs in the UI: open the dataset, list its DatasetRuns, and use the run-comparison view to diff outputs and scores across runs (for example gpt-4 versus gpt-4o, or prompt v3 versus v4). Each DatasetRunItem links a datasetItemId to the produced traceId.
10. Verified against current official docs on 2026-07-29: https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk , https://langfuse.com/docs/evaluation/experiments/datasets