{"id":"245ebb99-a435-4bdc-bc6b-8467cf80f7c0","task":"Build a promptfoo eval config to regression-test outputs across model versions","domain":"promptfoo.dev","steps":["Create a promptfooconfig.yaml defining prompts, providers (or targets) for each model/version to compare, and tests","For each test case, set vars for prompt variables and an assert array of assertion rules to check outputs","Use defaultTest to apply shared assertions across all test cases, and options.disableDefaultAsserts on individual tests to opt out where needed","Run the evaluation with promptfoo eval (optionally combining multiple config files with repeated -c flags)","Review the comparison results across providers/model versions in the generated report to catch regressions"],"gotchas":["defaultTest.assert is inherited by every test case unless a test explicitly disables it, which can cause double-counted or unexpectedly skipped assertions","Do not template secrets like API keys directly into the config's env block; values there are resolved at config-load time and can leak into exported results","Confirm whether your config uses providers or the newer targets field, since exactly one is required and current docs distinguish between them"],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/245ebb99-a435-4bdc-bc6b-8467cf80f7c0"}