Run an Opik evaluation with datasets, the evaluate() runner, and built-in LLM-as-judge metrics

Documented steps

1. Install and configure Opik first (pip install opik, then opik configure) so traces and experiments land in the right workspace.
2. Create a client: from opik import Opik; client = Opik().
3. Create or fetch the dataset: dataset = client.get_or_create_dataset('Example dataset', 'my-project'). Note that Opik 2.0 makes experiments and datasets project-scoped.
4. Populate it: dataset.insert([{'input': '...', 'expected': '...'}, ...]).
5. Define the task function under test. It receives one dataset item as a dict and must return a dict: def evaluation_task(x): result = your_llm_application(x['input']); return {'output': result}.
6. Import the metrics you want: from opik.evaluation.metrics import Hallucination, AnswerRelevance, Moderation.
7. Instantiate them: metrics = [Hallucination(), AnswerRelevance(), Moderation()]. Hallucination().score(input=..., output=..., context=[...]) returns 0 for no hallucination or 1 for detected. AnswerRelevance().score(input=..., output=..., context=[...]) requires context by default, disabled with require_context=False. Moderation().score(output=...) returns 0 (safe) to 1 (unsafe) across 10 content categories. All three expose an async ascore() and accept model= to override the judge.
8. Choose from the wider documented metric set as needed. Heuristic: BERTScore, ChrF, Contains, Corpus BLEU, Equals, GLEU, IsJson, JSDivergence, JSDistance, KLDivergence, Language Adherence, Levenshtein, Readability, RegexMatch, ROUGE, Sentence BLEU, Sentiment, Spearman Ranking, Tone. LLM-as-judge: Agent Task Completion Judge, Agent Tool Correctness Judge, Answer Relevance, Compliance Risk Judge, Context Precision, Context Recall, Dialogue Helpfulness Judge, G-Eval, Hallucination, LLM Juries Judge, Meaning Match, Moderation, Prompt Uncertainty Judge, QA Relevance Judge, Structured Output Compliance, Summarization Coherence Judge, Summarization Consistency Judge, Trajectory Accuracy, Usefulness. Conversation metrics: Conversational Coherence, Session Completeness Quality, User Frustration, Knowledge Retention, DegenerationC.
9. Import the runner: from opik import evaluate (or opik.evaluation.evaluate).
10. Call it: evaluate(dataset=dataset, task=evaluation_task, scoring_metrics=metrics, experiment_name='My experiment', project_name='my-project'). Other documented parameters are experiment_config, verbose, nb_samples, task_threads (default 16), prompt, prompts, scoring_key_mapping, dataset_item_ids, dataset_sampler, and trial_count.
11. If dataset keys do not match a metric's expected argument names, remap them with scoring_key_mapping rather than rewriting the dataset.
12. Smoke test with nb_samples set small before committing to a full run, and tune throughput with task_threads.
13. Use trial_count greater than 1 to measure judge variance on the same item rather than mistaking noise for a real score change.
14. Read the returned EvaluationResult and open the experiment in the Opik UI under the specified project.
15. If you want assertion-style pass/fail checks instead of scored metrics, Opik documents a separate Test Suites API (client.get_or_create_test_suite(...), opik.run_tests(...)) which is distinct from evaluate().