Dictionary entry Trending
evaluation harness
also eval harness or benchmark harness or model evaluation harness or AI evaluation harness
Definition of evaluation harness
-
: the software and configuration that run an AI model or agent through evaluation tasks under repeatable conditions, including how prompts are formatted, tools and memory are provided, budgets are enforced, outputs are recorded, and results are scored
- The lab reran every model with the same evaluation harness before comparing their benchmark scores.
- Changing the evaluation harness gave the agent persistent notes and produced a much higher result without changing the model's weights.
Origin & history
The phrase extends test harness, a longstanding software-engineering term for the machinery used to run tests and collect results. In machine learning, evaluation harnesses became common as projects standardized the tasks, prompts, model interfaces, generation settings, and scoring used to compare language models and agents.
Editorial references
Sources & further reading
- GPT-6 Astra — ARC-AGI Results↗ ARC Prize · 2026-09-02
- lm-evaluation-harness: A framework for few-shot evaluation of language models↗ EleutherAI
- Reasoning Is More Than the Model: Harness-Aware Evaluation of Agents on Verifiable Reasoning Tasks↗ OpenReview
- OpenAI’s AGI number came from a harness, not the model↗ TNW
Test yourself
Which of these is the meaning of evaluation harness?
Cite this entry
"Evaluation harness." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/evaluation-harness/. Accessed 25 Oct. 2026.