{"id":"5dbdfc90-20df-4e90-8971-25297991cfe9","task":"Run standard benchmark evaluations on a Hugging Face model using EleutherAI's lm-evaluation-harness (lm-eval CLI)","domain":"ml-ops","steps":["Install from source: git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness; cd lm-evaluation-harness; pip install -e .","List available benchmarks: lm-eval ls tasks (or lm-eval ls groups for grouped benchmarks like mmlu)","Run an evaluation: lm-eval run --model hf --model_args pretrained=<hf-model-id> --tasks hellaswag,arc_easy --num_fewshot 5","Persist outputs for review: add --output_path ./results/ --log_samples to save per-example inputs/outputs alongside aggregate scores","Optionally sanity-check a task config before a long run: lm-eval validate --tasks <task_name>","Inspect the JSON results written under --output_path, or push them to the Hub via --hf_hub_log_args"],"gotchas":["The CLI is now organized into subcommands (lm-eval run / lm-eval ls / lm-eval validate); the legacy flat form (lm-eval --model hf --tasks ...) still works but is auto-forwarded into `run`, so newer flags may require the explicit subcommand form","--limit truncates the number of examples evaluated per task and is documented as being 'for testing only' — using it will produce scores that don't match published benchmark numbers","Some tasks execute custom/remote dataset-loading code; running them requires explicitly passing --trust_remote_code and/or --confirm_run_unsafe_code"],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/5dbdfc90-20df-4e90-8971-25297991cfe9"}