- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Heuristic Bench, a novel benchmark that evaluates agents based on executable code as policies. The work provides a detailed overview of the benchmark and its associated evaluation methods before using it to conduct a comparison between a large model set. The evaluation setup allows for interpretability of agent strategies and clear understanding of instances in which agents complete tasks in unintended ways.
Critical strengths
Heuristic Bench is comprehensive in its coverage, with 47 models across 45 tasks, and thus supports broad comparisons. The comparison between shielded and unshielded access provides valuable insight into various agents' reliance on access to existing solutions. Lastly, the work is extremely well-written and easy to follow.
Critical weaknesses
The paper claims to provide a novel way to assess agentic problem solving capabilities. However, the tasks studied are those that can be solved through executable code as a policy. Many tasks of interest may not be reducible in this way, and besides, modern-day agents are increasingly used to solve problems outside this paradigm. Explicit acknowledgement of the limits on the claims that the evaluation can support appears appropriate.
Questions for authors
- I'm uncertain about the specifics of the brief that is provided to agents. Section 3.2 mentions that the brief provides "the held-out evaluation target", with Appendix A.4 showing that the specific target seed block is provided in the instruction. Does this allow for the agent to iterate directly against what it is eventually evaluated on?
- For the main leaderboard, can you specify the access level provided for each task?
- Do you have further evidence that performance on Heuristic Bench translates to agentic problem-solving in domains where code policies may not be a natural fit?
3 comments
Author Response to Reviewers (Follow-up)
As agent models evolve rapidly, benchmarks should be adaptable and versatile. To illustrate these properties in Heuristic Bench, we evaluated five models released after submission and shortly before the rebuttal, namely Claude Fable 5.2, Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Carbon, and Grok 4.7, on the 13 core tasks (Appendix B.1).
We compared each new model with its predecessor. In the table below (also Table 10 in the revised manuscript), we computed mean z by first applying the fixed taskwise normalization to each run, then averaging repeats within each task and weighting all 13 tasks equally. Current mean z scores and ranks describe the existing models on these 13 tasks only, so they slightly differ from those in the primary ranking. Hypothetical ranks show the ordering after adding all five new models.
| New model | Earlier model | New / earlier model mean z | Delta z | New / earlier model ranking if inserted | Earlier model current ranking |
|---|---|---|---|---|---|
| Claude Fable 5.2 | Claude Fable 5.1 | +1.131 / +0.726 | +0.405 | 2 / 7 | 3 |
| Claude Opus 5.5 | Claude Opus 5 | +1.345 / +0.515 | +0.830 | 1 / 13 | 8 |
| GPT-6.1 Sol | GPT-5.6 Sol | +1.017 / +0.732 | +0.285 | 3 / 6 | 2 |
| Gemini 4 Carbon | Gemini 3.1 Pro | +0.610 / +0.219 | +0.391 | 10 / 21 | 16 |
| Grok 4.7 | Grok 4.6 | +0.819 / +0.401 | +0.418 | 5 / 17 | 12 |
All five new models achieved higher mean z scores and ranks than their predecessors. Most gains came from tasks with remaining headroom, notably Ms. Pac-Man and HalfCheetah, showing that our benchmark remains informative even for the latest models. We plan to keep our benchmark updated as new models are released.
The results reported here have also been added to Appendix D.11.
Thank you for your revisions and additional analyses; the unseen-seed analysis in particular is very helpful. On Q2, what were agents able to access in the 41 level0 tasks? Given that the shielding experiments showed this access could inflate scores, were the main leaderboard's winning policies audited for unintended strategies, such as copying opponent code? I agree that a deeper understanding of Q3 can be left to future work. For now, I maintain the original weak accept decision, and may revise this depending on the response.
Author Response to Reviewer ep3G
Thank you for acknowledging the coverage, shielding insights, and clarity (Strengths 1-3). We revised the manuscript (revisions are marked in blue) to clarify the methodology and scope, and added analyses to address your comments and suggestions, as follows.
Regarding (Weakness 1) the scope of evaluating agentic capabilities through executable policies.
Thank you for this valuable suggestion. We agree that the benchmark's scope and limits should be stated explicitly. We revised the abstract to clarify the targeted traits of agentic decision making and execution with feedback-driven strategy construction. We also revised Section 3.3 to explain how established environments are adapted for agent-driven policy authoring. The conclusion (Section 5) also now identifies extending the range of benchmarked capabilities to include those required by broader agent workflows as future work.
Regarding (Question 1) access to evaluation seeds in the task brief.
Thank you for checking this detail. We would like to clarify which seeds agents could access and how we checked performance on unseen seeds. This concern is closely related to reviewer qCbi's Weakness 3, so here we address both with the same clarification.
Each run received a task-specific brief with partial seed blocks (a seed block refers to a set of seeds used in scoring). This was designed to help agents conduct reproducible policy trials and test tools. We conducted a conservative scan and flagged possible use of the provided seed blocks in 64% of runs. As such, the score is a best-attempt statistic rather than an unseen-seed test. We clarified this in Appendices A.2 and A.4.
To assess the risk that the best-attempt statistic does not generalize to unseen seeds, we added reruns of champion policies scored on entirely unseen seeds in Appendix B.5. We computed z scores using the same score normalization as in the primary ranking. We have 1,615 reruns and refitted the primary ranking using the same completed observations on each seed block. It show a mean change of -0.006 z units (95% bootstrap interval [-0.010, -0.003]). We found the following. (1) The average change is therefore only 0.6% of the standard deviation used to normalize task scores, which is a small change relative to the score variation. (2) 98% of reruns on seed-dependent tasks had an absolute score change no larger than the max of original 95% confidence-interval half-width and 0.05 z units. (3) Comparing the two rankings gave a Kendall tau-b of 0.987 across 43 models. Only one model moved two places, which is the maximum change. The fully comparison of the 19 core models on the core tasks gave a Kendall tau-b of 0.988, with no move exceeding one place.
As a secondary check of our shielding ablation (Section 4.3), we found that the open-minus-shielded score gap increased by only 0.005 z units, about 0.7% of its original 0.699 z gap across all shielding pairs. These small aggregate changes support the stability of the main ranking and the shielding-ablation result in the matched replay comparisons.
Regarding (Question 2) task access levels in the main leaderboard.
Thank you for asking us to specify the access levels for the main leaderboard. We specified the access levels in the paragraph "Agent workspace and access levels" of Appendix A.3. The four level1 tasks are Hopper, Connect Four, Slime Volleyball, and Overcooked, and the remaining 41 tasks use level0. As already described there, both levels use the same tools, protocols for policy submission, execution and scoring, and baseline protections for network access, host filesystem access and process isolation.
Regarding (Question 3) relevance to domains where code policies are less natural.
Thank you for asking about the relevance to domains where code policies are less natural. We agree that examining broader relevance across domains is a useful direction. Positive associations with ALE-Bench, Agents' Last Exam, and Terminal-Bench 4.0 suggest related strengths across different evaluation demands. We expanded the discussion in Appendix D.9 to clarify what these existing comparisons suggest about broader relevance. Agreement is weaker for some other benchmarks, consistent with their emphasis on different traits. Reusable executable skills and tool composition could help extend policy authoring to domains where standalone code policies are less natural. Studying how the capabilities measured here carry over to other domains remains future work. We also revised the conclusion (Section 5) to identify these directions.