Configure an LLM-as-a-judge evaluator in Langfuse to automatically score production traces (model, variable mapping, filters, sampling)

Documented steps

1. In the Langfuse UI, open Evaluations and choose or create an LLM-as-a-judge template defining the judge instructions and the expected structured output.
2. Configure the judge model under LLM Connections. Pick a model that supports structured output, since the evaluator parses the judge response into a score. You can pin a model per evaluator or fall back to the project default evaluation model.
3. Set up variable mapping to bind trace/observation fields to the judge prompt's placeholders such as `{{input}}`, `{{output}}`, `{{ground_truth}}`. Use JSONPath for nested fields, e.g. `$.choices[0].message.content`.
4. Use the live preview, which shows real data from the last 24 hours matching your filters, to sanity-check the mapping before saving. An empty preview usually means the filters match nothing, not that the mapping is wrong.
5. Choose the evaluation target: live production data (observations) for continuous scoring of new traces, or offline data (experiments) to score dataset experiment runs.
6. For live production evaluation, scope with target filters: observation type, trace name, trace tags, userId, sessionId, and metadata.
7. Set a sampling percentage (for example 5%) to bound judge cost and volume instead of scoring every production trace.
8. Ensure your instrumentation uses `propagate_attributes()` if you filter on trace-level attributes (userId, sessionId, version, tags) from an observation-level evaluator; otherwise those attributes are absent on the observation and the filter matches nothing.
9. Save the evaluator; it begins scoring new matching traces going forward. To backfill historical traces, select them in the Traces table and use Actions > Evaluate (requires Fast Mode enabled).
10. Verified against current official docs on 2026-07-29: https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge , https://langfuse.com/docs/evaluation/evaluation-methods/custom-scores