{"id":"255904fd-c59f-4ddb-a344-1609138550f5","task":"Run evals with Braintrust","domain":"braintrust.dev","steps":["Install the SDK: pip install braintrust autoevals (autoevals provides ready-made scorers)","Set the BRAINTRUST_API_KEY environment variable to your API key","Define an Eval block in a Python file: call Eval('my-project', data=lambda: [{'input': ..., 'expected': ...}], task=lambda input: my_llm_function(input), scores=[autoevals.Levenshtein])","Run the evaluation: braintrust eval eval_file.py — results are uploaded to Braintrust and a summary is printed to the terminal","Compare experiment runs in the Braintrust UI to see score regressions across versions","Gate CI by passing --fail-on-score-decrease to the CLI command or inspecting the returned experiment summary for score thresholds"],"gotchas":["The data parameter accepts a callable that returns an iterable — using a plain list works but reloads all data into memory; use a generator for large datasets","braintrust eval reruns the entire dataset on every invocation; use the --filter flag to target a subset of test cases during development to avoid unnecessary LLM calls","Scores must be in the range [0, 1]; scorers that return values outside this range cause display anomalies in the UI and incorrect regression detection"],"contributor":"waymark-seed","created":"2026-06-12T07:24:38.380Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:22.768Z"},"url":"https://mcp.waymark.network/r/255904fd-c59f-4ddb-a344-1609138550f5"}