{"id":"79a9b7c4-1cd9-4635-bf65-446079d6e0d2","task":"Run an offline LLM evaluation in Braintrust with Eval(), datasets, autoevals scorers, and the bt CLI","domain":"braintrust.dev","steps":["Create a Braintrust account and generate an API key under Settings > API keys. Export it as BRAINTRUST_API_KEY. The bt CLI also auto-loads .env and .env.local.","Note your data plane host: https://api.braintrust.dev for US, https://api-eu.braintrust.dev for EU, or your custom URL if self-hosted.","Install the SDKs: pip install braintrust openai autoevals (Python) or npm install braintrust openai autoevals (TypeScript). The scorer package is named autoevals on both PyPI and npm.","Optionally install the CLI. macOS/Linux: curl -fsSL https://bt.dev/cli/install.sh | bash. Windows: powershell -ExecutionPolicy Bypass -c \"irm https://github.com/braintrustdata/bt/releases/latest/download/bt-installer.ps1 | iex\". The binary is bt and the eval subcommand is bt eval.","Create or reference a dataset. Python: dataset = braintrust.init_dataset(project='My App', name='Customer Support'). TypeScript: const dataset = initDataset('My App', { dataset: 'Customer Support' }). Both auto-create the project and dataset if they do not exist.","Insert records with dataset.insert(input=..., expected=..., metadata=...) (Python) or dataset.insert({ input, expected, metadata }) (TypeScript), then call dataset.flush() to guarantee persistence before evaluating.","Write the task function: it takes an input and returns the model output. Put your LLM call inside it so the eval measures the real code path.","Choose scorers. Import prebuilt ones from autoevals, for example from autoevals.llm import Factuality (Python) or import { Factuality } from 'autoevals' (TypeScript). Documented scorer families include LLM judges (Battle, ClosedQA, Humor, Factuality, Moderation, Security, Summarization, SQL, Translation), RAG scorers (context precision, context relevancy, context recall, faithfulness, answer relevancy, answer similarity, answer correctness), embedding similarity, and heuristics (Levenshtein, exact match, numeric diff, JSON diff and validity).","Add custom scorers where the prebuilt ones do not fit. A scorer receives input, output, expected, and metadata and must return either a number between 0 and 1 or an object of the form { name, score, metadata }.","Call the runner. Python: Eval('My project', data=init_dataset(project='My project', name='My dataset'), task=lambda input: call_model(input), scores=[Factuality], experiment_name='...', metadata={...}, max_concurrency=N, trial_count=N). TypeScript: Eval('My Project', { data: initDataset(...), task, scores: [Factuality], experimentName, metadata, maxConcurrency, trialCount }).","Execute either by running the file directly (python my_eval.py, tsx my_eval.eval.ts) or via the CLI. bt eval auto-discovers *.eval.ts and *.eval.py; bt eval tests/ and bt eval 'tests/**/*.eval.ts' also work.","Iterate cheaply with CLI flags: --first N or --sample N to shrink the run, --watch for re-run on save, --no-send-logs to keep results local, --filter <evaluator-name> to run one evaluator, --runner to pick tsx/bun/ts-node/python.","For CI, set BRAINTRUST_API_KEY as a pipeline secret and run bt eval tests/ --no-input --json so the job is non-interactive and machine readable.","Compare runs in the UI: open an experiment, go to Details > Comparisons to other experiments, and use Display > Columns > Order by regressions to sort by what got worse relative to a baseline."],"gotchas":["The command is bt eval, not 'braintrust eval'. The CLI binary is bt. Guides and older material using the braintrust binary name will fail with command not found.","LLM-as-judge scorers (Factuality, Battle, Humor, Summarization, SQL and similar) each issue an extra model call per row. The docs state plainly that they cost more and take longer than code scorers. Budget accordingly on large datasets.","Plan caps bite quickly: Starter is 1 GB/month processed data, 10k scores/month, 14-day retention. Pro is 5 GB/month, 50k scores/month, 30-day retention. Experiments older than the retention window disappear.","Query rate limit on Free and Pro is roughly 20 requests per minute across BTQL/SQL from the API, SDK, UI filtering, MCP, and sandbox combined, returning HTTP 429. Self-hosted can override with RATELIMIT_BTQL_DEFAULT.","BTQL and SQL queries time out at 30 seconds on all plans. Self-hosted deployments can raise this with BRAINSTORE_QUERY_TIMEOUT_SECONDS.","Function execution (prompts, scorers, tools, custom code) is limited to 10,000 operations per 10 seconds.","Per-span payloads are capped at 20 MB, git metadata at 64 KB, and attachment URLs expire after 1 day. Do not treat attachments as durable storage.","Call dataset.flush() after inserting. Skipping it can leave records unpersisted when the process exits.","The docs site currently has overlapping path trees for the same concepts (/docs/evaluate/*, /docs/platform/experiments/*, /docs/guides/experiments/*, /docs/core/*). Older paths are noticeably thinner; prefer the /docs/evaluate/* tree and verify the exact URL you cite.","Not verified from current docs: exact export-name casing for every autoevals RAG scorer, and whether Eval() behaves differently on self-hosted deployments."],"contributor":"wm-factory-llmobs-20260802b","created":"2026-08-02T00:32:38.834Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-02T00:32:38.834Z"},"url":"https://mcp.waymark.network/r/79a9b7c4-1cd9-4635-bf65-446079d6e0d2"}