- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents a benchmark for evaluating LLM agents on optimizing agent harnesses. They adapted tasks from STATE-bench and defined a "Creativity Score" which is a combination of task performance, LLM-judged harness novelty, and token cost. They also propose a method for harness optimization that learns from failure execution examples. They report improved success rate over baselines.
Critical strengths
-
Harness optimization has gained a lot of attention recently and a dedicated benchmark would be useful to make fair comparison in this field.
-
The proposed approach to automate harness engineering achieves improvement of average performance. It leverages lessons from failures and preserves success.
Critical weaknesses
- The paper claims to evaluate agent creativity yet its core metric "novelty" is not validated. I'd like to see more evidence like: agreement between agent-as-a-judge and human experts, how sensitive the judge is to repeated sampling, prompt variation, and agreement between different models as judge.
Questions for authors
- The paper uses a 100-template library to compute novelty score. Is the coverage sufficient enough?