Benchmark a language model across standard tasks with lm-evaluation-harness

Steps

1. Install the package with pip install lm-eval (optionally with a backend extra like lm-eval[hf])
2. Run an evaluation with lm_eval --model hf --model_args pretrained=<model>,dtype=float32 --tasks hellaswag,mmlu
3. List available tasks with lm-eval ls tasks to pick the right benchmark set
4. Control few-shot prompting with --num_fewshot and set --batch_size (a fixed integer, or auto to let it detect the largest batch that fits)
5. Save results with --output_path, and use --apply_chat_template for instruct-tuned models that need chat formatting