Set up LangSmith online evaluation, automation rules, annotation queues, and user feedback on production traces

Documented steps

1. Confirm the application is already tracing to LangSmith; online evaluation operates on incoming production runs.
2. In the UI go to Tracing, select the project receiving production traces, then click + New and choose New Automation.
3. Name the rule and define a filter using the same filter DSL as trace search, scoping to what you actually want scored (for example by tag, latency, error status, or metadata field).
4. Set the sampling rate as a float between 0 and 1. This is your primary cost control for LLM-judge scoring of live traffic; 0.05 scores 5 percent of matching runs.
5. Choose one or more actions: Run online evaluator (LLM-as-judge on live traces), Run custom code evaluator, Add to annotation queue, Add to dataset, Trigger webhook, or Trigger alert.
6. Decide per action whether matching traces get upgraded to extended 400-day retention. This is set per action, not once per rule, and it is a billing decision.
7. Optionally enable Apply to past runs with a backfill date range to score historical traces retroactively.
8. Save the rule, then open the Logs button under the Automations tab to confirm runs are being matched and processed without evaluator errors.
9. For human review, create the queue first: Annotation Queues in the left nav, + Annotation Queue, give it a name, description, and optionally a default dataset.
10. Configure the rubric on the queue with instructions plus + Add a feedback rubric to define the feedback keys reviewers score against, and set reviewer settings (all workspace members versus specific assignees, reservation timeout, number of reviewers per run).
11. Route runs into the queue by any of: the Add to Annotation Queue button in a trace's Details view, multi-select in the runs table, an automation rule action, or from the Datasets and Experiments view.
12. When comparing two experiment variants, select exactly two experiments, choose Annotate, then Add to Pairwise Annotation Queue. Reviewers judge A better / B better / Equal with hotkeys A, B, E.
13. Capture product-side feedback from your backend: client.create_feedback(key='user_feedback', score=1, trace_id=trace_id, comment='...') in Python, or await client.createFeedback(runId, 'feedback-key', { score: 1.0, comment: '...' }) in TypeScript. Passing trace_id makes creation background and non-blocking, which is what you want on a request path.
14. Close the loop by adding an Add to dataset action on a rule that filters for low-score or thumbs-down traces, so real production failures continuously grow your offline eval set.