- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper introduces a new benchmark that adapts two existing benchmarks (Terminal-Bench and Harness-Bench) into goal, context, and constraint (permitted actions) variants to gauge how different information gaps affect model-harness pairings. The empirical results motivate model-harness evaluation across different conditions of task underspecification.
Critical strengths
- The empirical results demonstrate that a model’s performance can degrade depending on the harness it’s paired with.
- The benchmark isolates completion (ability to finish a task) from compliance (ability to follow rules) and demonstrates that when constraints are missing, agents frequently finish the task but violate rules in the process.
Critical weaknesses
- When an agent asks a clarifying question during an underspecified task, the benchmark does not actually provide an answer, but the run’s outcome is instead replaced with the agent’s average success rate on the task's fully specified version. This approach does not measure the agent’s actual continuation of the task and likely creates an overly optimistic estimate of clarification.
- The should report confidence intervals, especially when making comparative claims based on small pp differences (e.g., Codex has a 1.5 pp Success lead over OpenClaw, in Table 21) given the high variance inherent in binary success metrics evaluated over only three runs per configuration.
Questions for authors
- The authors should explain more in the main text how AM-Bench differs from UnderSpecBench and LHAW. What is “benign task underspecification” and how does it differ from UnderSpecBench?
- The underspecified condition definitions (Table 1) would be easier to interpret if a running example (e.g., from one of the benchmark tasks) was provided.
- The Task conditions (GA, GM, etc.) should be defined in the main text with examples, like in Appendix B.2. Otherwise, it is difficult to interpret Table 2 and Figures 3 and 4.