Run a Weights & Biases Weave evaluation with a Dataset, a weave.Model, and built-in or custom scorers

Documented steps

1. Install weave plus your model SDK (pip install weave openai) and set WANDB_API_KEY.
2. Import weave and asyncio, then call weave.init('your-team/project') once before building anything else.
3. Assemble test rows as a list of dicts and wrap them: dataset = weave.Dataset(name='fruits', rows=examples).
4. Publish the dataset so it is versioned and reusable across runs: weave.publish(dataset).
5. Define the system under test as a weave.Model subclass. It needs type-annotated attributes for its configuration and an @weave.op()-decorated predict method. The official example uses an async predict calling openai.AsyncClient(). A weave.Model is deliberately code plus configuration so Weave can version it.
6. Instantiate it with the configuration you want to test, for example model = ExtractFruitsModel(model_name='gpt-3.5-turbo-1106', prompt_template='...').
7. Write scorers. The function form is an @weave.op()-decorated function whose parameter names match dataset columns, plus the reserved parameter output which receives the model's return value: @weave.op() def fruit_name_score(target: dict, output: dict) -> dict: return {'correct': ...}.
8. Use the class form when a scorer needs configuration or state, such as an LLM judge: subclass weave.Scorer and implement @weave.op def score(self, output: str, **kwargs) -> dict. Class-based scorers may carry helper ops.
9. Use built-in scorers where they fit. Documented classes include MultiTaskBinaryClassificationF1, HallucinationFreeScorer, SummarizationScorer, EmbeddingSimilarityScorer, WeaveToxicityScorerV1, and WeaveContextRelevanceScorerV1. Import them from weave.scorers, for example from weave.scorers import MultiTaskBinaryClassificationF1 then MultiTaskBinaryClassificationF1(class_names=['fruit','color','flavor']).
10. Construct the evaluation: evaluation = weave.Evaluation(name='fruit_eval', dataset=dataset, scorers=[...]). Documented parameters include dataset, scorers, evaluation_name, trials, and preprocess_model_input. TypeScript uses id and nTrials.
11. Run it. In a script: print(asyncio.run(evaluation.evaluate(model))). In a notebook with a running event loop: await evaluation.evaluate(model) directly, without asyncio.run.
12. Open the Weave project page to inspect aggregate metrics and per-example scores.
13. Compare runs by selecting evaluations in the sidebar and clicking Compare. The Summary view shows tokens, cost, and latency diffs; Calls shows the full trace tree; Side-by-side and Unified (two objects) diff outputs. Maximum 6 objects at once.
14. Iterate by instantiating a new weave.Model with a changed prompt or parameter and re-running the same weave.Evaluation. Weave auto-versions the Model on attribute or code change, so results stay directly comparable.
15. Optionally serve a versioned model behind an HTTP endpoint with weave serve [MODEL-REF], which stands up a FastAPI service.