Run an offline LLM evaluation in Braintrust with Eval(), datasets, autoevals scorers, and the bt CLI

Documented steps

1. Create a Braintrust account and generate an API key under Settings > API keys. Export it as BRAINTRUST_API_KEY. The bt CLI also auto-loads .env and .env.local.
2. Note your data plane host: https://api.braintrust.dev for US, https://api-eu.braintrust.dev for EU, or your custom URL if self-hosted.
3. Install the SDKs: pip install braintrust openai autoevals (Python) or npm install braintrust openai autoevals (TypeScript). The scorer package is named autoevals on both PyPI and npm.
4. Optionally install the CLI. macOS/Linux: curl -fsSL https://bt.dev/cli/install.sh | bash. Windows: powershell -ExecutionPolicy Bypass -c "irm https://github.com/braintrustdata/bt/releases/latest/download/bt-installer.ps1 | iex". The binary is bt and the eval subcommand is bt eval.
5. Create or reference a dataset. Python: dataset = braintrust.init_dataset(project='My App', name='Customer Support'). TypeScript: const dataset = initDataset('My App', { dataset: 'Customer Support' }). Both auto-create the project and dataset if they do not exist.
6. Insert records with dataset.insert(input=..., expected=..., metadata=...) (Python) or dataset.insert({ input, expected, metadata }) (TypeScript), then call dataset.flush() to guarantee persistence before evaluating.
7. Write the task function: it takes an input and returns the model output. Put your LLM call inside it so the eval measures the real code path.
8. Choose scorers. Import prebuilt ones from autoevals, for example from autoevals.llm import Factuality (Python) or import { Factuality } from 'autoevals' (TypeScript). Documented scorer families include LLM judges (Battle, ClosedQA, Humor, Factuality, Moderation, Security, Summarization, SQL, Translation), RAG scorers (context precision, context relevancy, context recall, faithfulness, answer relevancy, answer similarity, answer correctness), embedding similarity, and heuristics (Levenshtein, exact match, numeric diff, JSON diff and validity).
9. Add custom scorers where the prebuilt ones do not fit. A scorer receives input, output, expected, and metadata and must return either a number between 0 and 1 or an object of the form { name, score, metadata }.
10. Call the runner. Python: Eval('My project', data=init_dataset(project='My project', name='My dataset'), task=lambda input: call_model(input), scores=[Factuality], experiment_name='...', metadata={...}, max_concurrency=N, trial_count=N). TypeScript: Eval('My Project', { data: initDataset(...), task, scores: [Factuality], experimentName, metadata, maxConcurrency, trialCount }).
11. Execute either by running the file directly (python my_eval.py, tsx my_eval.eval.ts) or via the CLI. bt eval auto-discovers *.eval.ts and *.eval.py; bt eval tests/ and bt eval 'tests/**/*.eval.ts' also work.
12. Iterate cheaply with CLI flags: --first N or --sample N to shrink the run, --watch for re-run on save, --no-send-logs to keep results local, --filter <evaluator-name> to run one evaluator, --runner to pick tsx/bun/ts-node/python.
13. For CI, set BRAINTRUST_API_KEY as a pipeline secret and run bt eval tests/ --no-input --json so the job is non-interactive and machine readable.
14. Compare runs in the UI: open an experiment, go to Details > Comparisons to other experiments, and use Display > Columns > Order by regressions to sort by what got worse relative to a baseline.