acceptodds
Under review as a conference paper at ICLR 2027

Heuristic Bench: Evaluating LLM Agents by Authoring Executable Policies

Abstract

Large language model (LLM) agents have shown strong capabilities in agentic problem solving. But assessing these capabilities across models in a human-interpretable and fine-grained way remains challenging. Inspired by a recent paradigm, Heuristic Learning, we introduce Heuristic Bench, a benchmark in which agents iteratively solve tasks by authoring executable code as policies that can be run and evaluated independently. As code, these policies are human-interpretable and inspectable, allowing readers to examine the strategies behind the scores and investigate potential shortcuts. Using this benchmark, we evaluate 47 models with varying coverage across 45 problem-solving tasks, with discrete and continuous action spaces in solo, competitive, and cooperative settings. Controlled execution, independent evaluation, and a wide range of metrics support reliable measurement of model performance. The benchmark's breadth allows us to rank models by their ability, and its analyses reveal fine-grained traits, such as performance-cost trade-offs across models and gains from additional attempts, uneven gains across model versions, imperfect yet still informative self-reported scores, and partial agreement in model rankings across tasks. Interestingly, policy inspection reveals that capable models can exploit unintended shortcuts, including copying opponent and task code to construct exact-simulation search policies. These findings highlight the importance of careful evaluation design to reveal model capabilities and inform how AI models are evaluated, used, and developed.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, heuristic learning, agent benchmarking.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ep3GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Heuristic Bench, a novel benchmark that evaluates agents based on executable code as policies. The work provides a detailed overview of the benchmark and its associated evaluation methods before using it to conduct a comparison between a large model set. The evaluation setup allows for interpretability of agent strategies and clear understanding of instances in which agents complete tasks in unintended ways.

Critical strengths

Heuristic Bench is comprehensive in its coverage, with 47 models across 45 tasks, and thus supports broad comparisons. The comparison between shielded and unshielded access provides valuable insight into various agents' reliance on access to existing solutions. Lastly, the work is extremely well-written and easy to follow.

Critical weaknesses

The paper claims to provide a novel way to assess agentic problem solving capabilities. However, the tasks studied are those that can be solved through executable code as a policy. Many tasks of interest may not be reducible in this way, and besides, modern-day agents are increasingly used to solve problems outside this paradigm. Explicit acknowledgement of the limits on the claims that the evaluation can support appears appropriate.

Questions for authors
  • I'm uncertain about the specifics of the brief that is provided to agents. Section 3.2 mentions that the brief provides "the held-out evaluation target", with Appendix A.4 showing that the specific target seed block is provided in the instruction. Does this allow for the agent to iterate directly against what it is eventually evaluated on?
  • For the main leaderboard, can you specify the access level provided for each task?
  • Do you have further evidence that performance on Heuristic Bench translates to agentic problem-solving in domains where code policies may not be a natural fit?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer ep3G

Thank you for acknowledging the coverage, shielding insights, and clarity (Strengths 1-3). We revised the manuscript (revisions are marked in blue) to clarify the methodology and scope, and added analyses to address your comments and suggestions, as follows.

Regarding (Weakness 1) the scope of evaluating agentic capabilities through executable policies.

Thank you for this valuable suggestion. We agree that the benchmark's scope and limits should be stated explicitly. We revised the abstract to clarify the targeted traits of agentic decision making and execution with feedback-driven strategy construction. We also revised Section 3.3 to explain how established environments are adapted for agent-driven policy authoring. The conclusion (Section 5) also now identifies extending the range of benchmarked capabilities to include those required by broader agent workflows as future work.

Regarding (Question 1) access to evaluation seeds in the task brief.

Thank you for checking this detail. We would like to clarify which seeds agents could access and how we checked performance on unseen seeds. This concern is closely related to reviewer qCbi's Weakness 3, so here we address both with the same clarification.

Each run received a task-specific brief with partial seed blocks (a seed block refers to a set of seeds used in scoring). This was designed to help agents conduct reproducible policy trials and test tools. We conducted a conservative scan and flagged possible use of the provided seed blocks in 64% of runs. As such, the score is a best-attempt statistic rather than an unseen-seed test. We clarified this in Appendices A.2 and A.4.

To assess the risk that the best-attempt statistic does not generalize to unseen seeds, we added reruns of champion policies scored on entirely unseen seeds in Appendix B.5. We computed z scores using the same score normalization as in the primary ranking. We have 1,615 reruns and refitted the primary ranking using the same completed observations on each seed block. It show a mean change of -0.006 z units (95% bootstrap interval [-0.010, -0.003]). We found the following. (1) The average change is therefore only 0.6% of the standard deviation used to normalize task scores, which is a small change relative to the score variation. (2) 98% of reruns on seed-dependent tasks had an absolute score change no larger than the max of original 95% confidence-interval half-width and 0.05 z units. (3) Comparing the two rankings gave a Kendall tau-b of 0.987 across 43 models. Only one model moved two places, which is the maximum change. The fully comparison of the 19 core models on the core tasks gave a Kendall tau-b of 0.988, with no move exceeding one place.

As a secondary check of our shielding ablation (Section 4.3), we found that the open-minus-shielded score gap increased by only 0.005 z units, about 0.7% of its original 0.699 z gap across all shielding pairs. These small aggregate changes support the stability of the main ranking and the shielding-ablation result in the matched replay comparisons.

Regarding (Question 2) task access levels in the main leaderboard.

Thank you for asking us to specify the access levels for the main leaderboard. We specified the access levels in the paragraph "Agent workspace and access levels" of Appendix A.3. The four level1 tasks are Hopper, Connect Four, Slime Volleyball, and Overcooked, and the remaining 41 tasks use level0. As already described there, both levels use the same tools, protocols for policy submission, execution and scoring, and baseline protections for network access, host filesystem access and process isolation.

Regarding (Question 3) relevance to domains where code policies are less natural.

Thank you for asking about the relevance to domains where code policies are less natural. We agree that examining broader relevance across domains is a useful direction. Positive associations with ALE-Bench, Agents' Last Exam, and Terminal-Bench 4.0 suggest related strengths across different evaluation demands. We expanded the discussion in Appendix D.9 to clarify what these existing comparisons suggest about broader relevance. Agreement is weaker for some other benchmarks, consistent with their emphasis on different traits. Reusable executable skills and tool composition could help extend policy authoring to domains where standalone code policies are less natural. Studying how the capabilities measured here carry over to other domains remains future work. We also revised the conclusion (Section 5) to identify these directions.

AuthorsOpenReview ↗

Author Response to Reviewers (Follow-up)

As agent models evolve rapidly, benchmarks should be adaptable and versatile. To illustrate these properties in Heuristic Bench, we evaluated five models released after submission and shortly before the rebuttal, namely Claude Fable 5.2, Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Carbon, and Grok 4.7, on the 13 core tasks (Appendix B.1).

We compared each new model with its predecessor. In the table below (also Table 10 in the revised manuscript), we computed mean z by first applying the fixed taskwise normalization to each run, then averaging repeats within each task and weighting all 13 tasks equally. Current mean z scores and ranks describe the existing models on these 13 tasks only, so they slightly differ from those in the primary ranking. Hypothetical ranks show the ordering after adding all five new models.

New modelEarlier modelNew / earlier model mean zDelta zNew / earlier model ranking if insertedEarlier model current ranking
Claude Fable 5.2Claude Fable 5.1+1.131 / +0.726+0.4052 / 73
Claude Opus 5.5Claude Opus 5+1.345 / +0.515+0.8301 / 138
GPT-6.1 SolGPT-5.6 Sol+1.017 / +0.732+0.2853 / 62
Gemini 4 CarbonGemini 3.1 Pro+0.610 / +0.219+0.39110 / 2116
Grok 4.7Grok 4.6+0.819 / +0.401+0.4185 / 1712

All five new models achieved higher mean z scores and ranks than their predecessors. Most gains came from tasks with remaining headroom, notably Ms. Pac-Man and HalfCheetah, showing that our benchmark remains informative even for the latest models. We plan to keep our benchmark updated as new models are released.

The results reported here have also been added to Appendix D.11.

Thank you for your revisions and additional analyses; the unseen-seed analysis in particular is very helpful. On Q2, what were agents able to access in the 41 level0 tasks? Given that the shielding experiments showed this access could inflate scores, were the main leaderboard's winning policies audited for unintended strategies, such as copying opponent code? I agree that a deeper understanding of Q3 can be left to future work. For now, I maintain the original weak accept decision, and may revise this depending on the response.

Reviewer_eivEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an intriguing benchmark that requires agents submit code policies to play games such as Tic-Tac-Toe, Hopper etc. The paper benchmarks many model+agent harnesses and shows that the models can somewhat play these games with this constraint though they often still fall short of simple baselines like PPO-trained hopper policy. The provided results and the insight into sandbox are useful for development and evaluation of agents. While the provided tasks are rather simplistic, the benchmark does provide a more interpretable means of understanding agents as game strategists. My main gripe with the paper is that there is no way to understand/analyze the code policies systemically. So, while the code policies are available, it requires a human to understand these long policies that are often over-engineered to play these pixel games. Thus the need for a code policy doesn't seem to provide a clear benefit.

Critical strengths
  • Clear benchmark with details into sandboxing and task design
  • Tested numerous models and their official harness
Critical weaknesses
  • Why is there uneven trial count per task? 13/45 get n=3, but rest get n=1? None-core models only got 1 trial per task. This seems to be not ideal as one of the main metrics is z-score. So the other models have a much noisier estimate. Also it is unclear which of these 13 tasks are deemed 'representative'.
  • No ablation examining effectiveness of harness. Some of the recent work has shown that different models work with different provider harness. Providing this would be helpful to avoid any issues caused by some harnesses providing more/better tooling or models being overfit to a specific harness.
  • No systematic metric or way to understand the code policy besides human analysis. This makes the code policy constraint more for bookkeeping rather than something that can help our understanding immediately. This is compounded by the fact that policies are really detailed since they are used to play games.
Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer eivE (Part 1)

Thank you for acknowledging the benchmark's usefulness, clear sandbox and task design, and broad evaluation (Summary, Strengths 1-2). We revised the manuscript (revisions are marked in blue) to clarify the methodology and scope, and added analyses to address your comments and suggestions, as follows.

Regarding (Summary) performance relative to trained RL baselines.

Thank you for noting the performance relative to trained RL baselines. We revised in Section 4.4 to state explicitly that different paradigms make these comparisons useful for calibration, rather than for establishing equal-budget superiority.

Regarding (Weakness 1) unequal trial counts, estimate precision, and core-task selection.

Thank you for raising this concern. This connects closely to reviewer qCbi's Weaknesses 2 and 4 about ranking scope and weighting, so here we address both with the same clarification.

The benchmark compares agent configurations under common task interfaces and measurement rules. Our decisions on task coverage and repetition were made to prioritize important models and tasks while limiting the total budget. For the primary benchmark, each of the 19 core models was allocated three runs per task on the 13 core tasks and one run per task on the remaining 32 tasks. Each of the other 28 models was allocated one run per task on the 13 core tasks. Runs for ablation studies do not count in the primary model ranking. When computing each model's overall ranking score, we give equal weight to runs. Tasks with more repeated runs therefore contribute more to that score. We revised the "Coverage and core selection" paragraph of Appendix B.1 to describe this.

We added a sensitivity analysis to check how this weighting affects the ranking: Comparing the 19 core models' relative order in the primary ranking with a new ranking that removes the extra influence of repeated tasks (first averaging repeats within each task and then giving all 45 tasks equal weight, while using z scores from the same score normalization as in the primary ranking). The two rankings had Kendall tau-b = 0.942, and no core model moved more than two places within this group. This supports the stability of the core-model ordering under equal task weights. Most non-core models appear in the lower half of the full ranking (Table 5), also supporting our decision to prioritize the core models for more extensive evaluation. We added this analysis in Appendix B.3.

Regarding (Weakness 2) harness effects on model performance.

Thank you for your suggestion. This concern is closely related to reviewer qCbi's Question 3, so here we address both with the same clarification.

We paired models with harnesses that workable for the tasks at the time of the writing to keep the integration effort manageable, as clarified in Appendix B.1. We agree that harness choice is an additional comparison dimension. As a preliminary investigation, we compared GPT-5.6 Terra under Codex and Claude Fable 5.1 under Claude Code with each model under OpenCode on Hopper and Slime Volleyball, with matched high reasoning settings and time and attempt budgets. We computed z scores using the same score normalization as in the primary ranking.

The average native-minus-OpenCode difference was -0.303 z units (95% bootstrap interval [-0.676, 0.088]). The GPT-minus-Claude gaps were -1.043 under native harnesses and -1.640 under OpenCode. In magnitude, the average harness difference (0.303 z units) was about 29% of the model gap under native harnesses (1.043 z units) and 18% under OpenCode (1.640 z units). Thus, Claude Fable 5.1 remained ahead of GPT-5.6 Terra under both harness setups, preserving their order in the primary model ranking.

We want to emphasize that these results by no means establish harness equivalence. In fact, switching from native harnesses to OpenCode also changed the serving path to vendor-pinned OpenRouter, and we observed different failure patterns across deployments. Broader comparisons require matched settings across model-harness pairings and remain future work. We added Appendix B.4 to report the experimental details and analysis.

AuthorsOpenReview ↗

Author Response to Reviewer eivE (Part 2)

Regarding (Weakness 3) systematic policy analysis beyond individual human inspection.

Thank you for this valuable suggestion. This connects closely to reviewer qCbi's Question 5 requesting a broader policy audit, so here we address both with the same clarification.

Our existing policy examples (Figure 6, discussed in Section 4.4) and code-length analysis (Appendix D.4) already illustrate the potential for direct human inspection. Executable code also allows LLMs to assist people in analyzing larger policy collections. As an initial step, we added an exploratory analysis of 87 policies from five models across six tasks in Appendix D.2 and Section 4.4. We used Claude Opus 5.5 to propose a rubric of 14 traits (strategies in the code) and apply it to the sampled policies, with authoring-model identities and scores hidden to limit attribution bias.

The analysis identified task-specific strategies. We also compared models using Pearson correlations between their 14-trait as feature pooled across tasks, and using hierarchical clustering on these correlations. These exploratory findings show how LLM assistance can organize patterns across policies to support human judgment. However, given the complexity of such analysis, broader audits and more structured human-AI collaborations in such analysis remain future work.

AuthorsOpenReview ↗

Author Response to Reviewers (Follow-up)

As agent models evolve rapidly, benchmarks should be adaptable and versatile. To illustrate these properties in Heuristic Bench, we evaluated five models released after submission and shortly before the rebuttal, namely Claude Fable 5.2, Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Carbon, and Grok 4.7, on the 13 core tasks (Appendix B.1).

We compared each new model with its predecessor. In the table below (also Table 10 in the revised manuscript), we computed mean z by first applying the fixed taskwise normalization to each run, then averaging repeats within each task and weighting all 13 tasks equally. Current mean z scores and ranks describe the existing models on these 13 tasks only, so they slightly differ from those in the primary ranking. Hypothetical ranks show the ordering after adding all five new models.

New modelEarlier modelNew / earlier model mean zDelta zNew / earlier model ranking if insertedEarlier model current ranking
Claude Fable 5.2Claude Fable 5.1+1.131 / +0.726+0.4052 / 73
Claude Opus 5.5Claude Opus 5+1.345 / +0.515+0.8301 / 138
GPT-6.1 SolGPT-5.6 Sol+1.017 / +0.732+0.2853 / 62
Gemini 4 CarbonGemini 3.1 Pro+0.610 / +0.219+0.39110 / 2116
Grok 4.7Grok 4.6+0.819 / +0.401+0.4185 / 1712

All five new models achieved higher mean z scores and ranks than their predecessors. Most gains came from tasks with remaining headroom, notably Ms. Pac-Man and HalfCheetah, showing that our benchmark remains informative even for the latest models. We plan to keep our benchmark updated as new models are released.

The results reported here have also been added to Appendix D.11.

Reviewer_qCbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HeuristicBench, a benchmark that evaluates LLM agents by asking them to iteratively author executable code policies. Agents test and revise their policies using execution feedback, and the resulting policies are independently evaluated and retained for inspection. The authors evaluate 47 models with varying coverage across 45 tasks spanning discrete and continuous action spaces in solo, competitive, and cooperative settings. The results show performance and cost trade-offs, gains from additional attempts, and uneven improvements across model versions and tasks. Separate shielding experiments show that access to existing solutions can substantially increase scores, with policy inspection revealing copied opponent weights and trained controllers. The main contribution is an evaluation framework that combines quantitative comparison with inspectable policy code and behavioral evidence, allowing researchers to examine both what agents achieve and how they achieve it.

Critical strengths
  1. The benchmark combines quantitative evaluation with inspectable executable policies. This allows researchers to examine the strategies behind measured performance and identify behaviors that aggregate scores alone may conceal. It provides a useful basis for understanding how agents solve problems.
  2. The evaluation covers a broad range of models and tasks across different action spaces and interaction settings. This breadth helps reveal differences in model capabilities that may be missed by narrower benchmarks, while the cost and iterative improvement analyses make the comparison more informative.
  3. The paper examines how access to existing solutions affects measured performance. Section 4.3 and Appendix C trace high unshielded scores to copied opponent weights and trained controllers, demonstrating why access controls matter when assessing capable agents. This offers useful guidance for designing more reliable agent benchmarks.
Critical weaknesses
  1. The task selection process is insufficiently documented. Section 3.3 gives broad selection principles, but neither it nor Appendix A.1 explains the candidate pool, exclusion criteria, or any manual screening process. It is therefore difficult to assess how representative the final 45 tasks are or whether exploratory results influenced their inclusion.
  2. The interpretation of the ranking as a comparison of model capabilities is broader than the experimental controls support. Section 4.1 compares models under different harnesses and reasoning settings, while the aggregation also reflects unequal task coverage and repetition. The results characterize performance under these configurations, but do not isolate model capability from the surrounding evaluation setup.
  3. Section 3.2 separates authoring and evaluation seeds, but Appendix A.4 permits previewing target seeds, and the same measurements select and score the champion. Without a separate final test, generalization to unseen episodes remains unclear.
  4. Rankings weight runs equally despite unequal task coverage and repetition. Extra runs on selected tasks change the core models’ task weights, leaving ranking stability under shared coverage and equal task weights unverified.
Questions for authors
  1. The benchmark mainly builds an evaluation framework with corresponding evaluation metrics, while the 45 tasks largely reuse established environments. Beyond adapting these environments to a unified interface, are there newly constructed tasks or data? What aspects of agent capability does this task collection assess that the existing suites do not?
  2. Section 3.1 states that proposed lessons are collected after a run and used in future runs following human review. What information do these lessons contain, and do models receive the same lessons for a given task?
  3. Have any models been tested across multiple harnesses under matched reasoning settings? How large are those differences relative to the gaps between models?
  4. How would policy authoring compare with direct LLM action selection under matched resource budgets? This comparison could clarify when executable policies offer a practical advantage.
  5. Could policy inspection be extended to a systematic audit across models and tasks? Estimating how frequently solution reuse occurs would help establish how broadly the showcased cases apply.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer qCbi (Part 1)

Thank you for acknowledging the inspectable policies, breadth, and shielding analysis (Strengths 1, 2, and 3). We revised the manuscript (revisions are marked in blue) to clarify the methodology and scope, and added analyses to address your comments and suggestions, as follows.

Regarding (Weakness 1) task selection and (Question 1) novelty and assessed capabilities.

Thank you for asking us to clarify the task selection and the contribution of adapting existing environments. As outlined in Section 3.3, we selected available tasks that place diverse demands on agents, have informative difficulty (neither too easy nor too hard for models), and can be evaluated within feasible time budgets (excluding tasks too lengthy or difficult for the current budget). The task profiles show heterogeneous strengths and headroom (Figure 12 in Appendix B.2, and Appendix D.8), which confirms the validity of our selection rationale.

Many of the tasks were adapted from established RL environments intentionally for feasibility. We also note that constructing executable policies through agent-driven experimentation (the capability our benchmark aims to reveal) represents a different paradigm from conventional RL training and requires nontrivial adaptation of a wide range of tasks (Table 3) to unify these tasks. As with all benchmarks, extending the task set to include tasks requiring longer budgets could be a direction for future work.

We revised Section 3.3 and Appendix A.1 to reflect the discussion above.

Regarding (Weakness 2) ranking scope and (Weakness 4) task coverage and weighting sensitivity.

Thank you for raising these concerns. These concerns are closely related to reviewer eivE's Weakness 1, so here we address both with the same clarification.

The benchmark compares agent configurations under common task interfaces and measurement rules. Our decisions on task coverage and repetition were made to prioritize important models and tasks while limiting the total budget. For the primary benchmark, each of the 19 core models was allocated three runs per task on the 13 core tasks and one run per task on the remaining 32 tasks. Each of the other 28 models was allocated one run per task on the 13 core tasks. Runs for ablation studies do not count in the primary model ranking. When computing each model's overall ranking score, we give equal weight to runs. Tasks with more repeated runs therefore contribute more to that score. We revised the "Coverage and core selection" paragraph of Appendix B.1 to describe this.

We added a sensitivity analysis to check how this weighting affects the ranking: Comparing the 19 core models' relative order in the primary ranking with a new ranking that removes the extra influence of repeated tasks (first averaging repeats within each task and then giving all 45 tasks equal weight, while using z scores from the same score normalization as in the primary ranking). The two rankings had Kendall tau-b = 0.942, and no core model moved more than two places within this group. This supports the stability of the core-model ordering under equal task weights. Most non-core models appear in the lower half of the full ranking (Table 5), also supporting our decision to prioritize the core models for more extensive evaluation. We added this analysis in Appendix B.3.

AuthorsOpenReview ↗

Author Response to Reviewer qCbi (Part 2)

Regarding (Weakness 3) target-seed previews and generalization to unseen seeds.

Thank you for this conern. As it is closely related to reviewer ep3G's Question 1, here we address both with the same clarification.

Each run received a task-specific brief with partial seed blocks (a seed block refers to a set of seeds used in scoring). This was designed to help agents conduct reproducible policy trials and test tools. We conducted a conservative scan and flagged possible use of the provided seed blocks in 64% of runs. As such, the score is a best-attempt statistic rather than an unseen-seed test. We clarified this in Appendices A.2 and A.4.

To assess the risk that the best-attempt statistic does not generalize to unseen seeds, we added reruns of champion policies scored on entirely unseen seeds in Appendix B.5. We computed z scores using the same score normalization as in the primary ranking. We have 1,615 reruns and refitted the primary ranking using the same completed observations on each seed block. It show a mean change of -0.006 z units (95% bootstrap interval [-0.010, -0.003]). We found the following. (1) The average change is therefore only 0.6% of the standard deviation used to normalize task scores, which is a small change relative to the score variation. (2) 98% of reruns on seed-dependent tasks had an absolute score change no larger than the max of original 95% confidence-interval half-width and 0.05 z units. (3) Comparing the two rankings gave a Kendall tau-b of 0.987 across 43 models. Only one model moved two places, which is the maximum change. The fully comparison of the 19 core models on the core tasks gave a Kendall tau-b of 0.988, with no move exceeding one place.

As a secondary check of our shielding ablation (Section 4.3), we found that the open-minus-shielded score gap increased by only 0.005 z units, about 0.7% of its original 0.699 z gap across all shielding pairs. These small aggregate changes support the stability of the main ranking and the shielding-ablation result in the matched replay comparisons.

Regarding (Question 2) lesson contents and consistency across models.

All runs in the main comparison used the same fixed set of 56 common lessons harvested beforehand, without updating this set between runs. We revised Section 3.2 to state this rule and distinguish supplied guidance from notes generated during successive attempts within a single run. In Appendix A.5, we added four complete examples of these lessons to show what the guidance looks like. We use these lessons as practical search support rather than as a standalone method for improving performance.

Regarding (Question 3) harness effects relative to model differences under matched settings.

Thank you for suggesting this comparison. This question is closely related to reviewer eivE's Weakness 2, so here we address both with the same clarification.

We paired models with harnesses that workable for the tasks at the time of the writing to keep the integration effort manageable, as clarified in Appendix B.1. We agree that harness choice is an additional comparison dimension. As a preliminary investigation, we compared GPT-5.6 Terra under Codex and Claude Fable 5.1 under Claude Code with each model under OpenCode on Hopper and Slime Volleyball, with matched high reasoning settings and time and attempt budgets. We computed z scores using the same score normalization as in the primary ranking.

The average native-minus-OpenCode difference was -0.303 z units (95% bootstrap interval [-0.676, 0.088]). The GPT-minus-Claude gaps were -1.043 under native harnesses and -1.640 under OpenCode. In magnitude, the average harness difference (0.303 z units) was about 29% of the model gap under native harnesses (1.043 z units) and 18% under OpenCode (1.640 z units). Thus, Claude Fable 5.1 remained ahead of GPT-5.6 Terra under both harness setups, preserving their order in the primary model ranking.

We want to emphasize that these results by no means establish harness equivalence. In fact, switching from native harnesses to OpenCode also changed the serving path to vendor-pinned OpenRouter, and we observed different failure patterns across deployments. Broader comparisons require matched settings across model-harness pairings and remain future work. We added Appendix B.4 to report the experimental details and analysis.

AuthorsOpenReview ↗

Author Response to Reviewer qCbi (Part 3)

Regarding (Question 4) policy authoring versus direct action selection under matched budgets.

Thank you for suggesting this complementary comparison. Policy authoring in our benchmark uses LLM inference to construct code that subsequently acts without further LLM calls. Direct action selection follows a different paradigm from policy authoring, using LLM inference for each decision. A full comparison would require aligning design choices such as strategy learning, context construction, and deployment duration across these two paradigms.

This is a valuable direction. To provide initial insights, we added an exploratory zero-shot comparison in Appendix D.10 and summarized it in Section 4.4, to examine what a straightforward adaptation achieves. In one Slime Volleyball match per controller, a frozen code policy authored by an LLM completed 3,000 steps in 0.27 s with neither side scoring (a reasonable outcome against the strong neural-network opponent), while Gemini 3.7 Flash and DeepSeek V4 Pro both lost 0-5, reaching the five-loss stopping condition. The two models required median decision times of 2.831 s and 0.917 s, respectively. The simulator paused during inference, so slow responses did not cause missed actions.

This illustration shows why direct action selection needs dedicated design beyond replacing code execution with per-step LLM calls. It also shows a practical advantage of executable policies when many rapid decisions are required. Once authored, the code executes without per-step LLM calls. We therefore view a full comparison with matched development and deployment budgets as a valuable complementary study for future work.

Regarding (Question 5) systematic policy audits and the frequency of solution reuse.

Thank you for this valuable suggestion. We agree that systematic inspection is a promising research direction. This also connects closely to reviewer eivE's Weakness 3 about systematic policy understanding, so here we address both with the same clarification.

Our existing policy examples (Figure 6, discussed in Section 4.4) and code-length analysis (Appendix D.4) already illustrate the potential for direct human inspection. Executable code also allows LLMs to assist people in analyzing larger policy collections. As an initial step, we added an exploratory analysis of 87 policies from five models across six tasks in Appendix D.2 and Section 4.4. We used Claude Opus 5.5 to propose a rubric of 14 traits (strategies in the code) and apply it to the sampled policies, with authoring-model identities and scores hidden to limit attribution bias.

The analysis identified task-specific strategies. We also compared models using Pearson correlations between their 14-trait as feature pooled across tasks, and using hierarchical clustering on these correlations. These exploratory findings show how LLM assistance can organize patterns across policies to support human judgment. However, given the complexity of such analysis, broader audits and more structured human-AI collaborations in such analysis remain future work.

AuthorsOpenReview ↗

Author Response to Reviewers (Follow-up)

As agent models evolve rapidly, benchmarks should be adaptable and versatile. To illustrate these properties in Heuristic Bench, we evaluated five models released after submission and shortly before the rebuttal, namely Claude Fable 5.2, Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Carbon, and Grok 4.7, on the 13 core tasks (Appendix B.1).

We compared each new model with its predecessor. In the table below (also Table 10 in the revised manuscript), we computed mean z by first applying the fixed taskwise normalization to each run, then averaging repeats within each task and weighting all 13 tasks equally. Current mean z scores and ranks describe the existing models on these 13 tasks only, so they slightly differ from those in the primary ranking. Hypothetical ranks show the ordering after adding all five new models.

New modelEarlier modelNew / earlier model mean zDelta zNew / earlier model ranking if insertedEarlier model current ranking
Claude Fable 5.2Claude Fable 5.1+1.131 / +0.726+0.4052 / 73
Claude Opus 5.5Claude Opus 5+1.345 / +0.515+0.8301 / 138
GPT-6.1 SolGPT-5.6 Sol+1.017 / +0.732+0.2853 / 62
Gemini 4 CarbonGemini 3.1 Pro+0.610 / +0.219+0.39110 / 2116
Grok 4.7Grok 4.6+0.819 / +0.401+0.4185 / 1712

All five new models achieved higher mean z scores and ranks than their predecessors. Most gains came from tasks with remaining headroom, notably Ms. Pac-Man and HalfCheetah, showing that our benchmark remains informative even for the latest models. We plan to keep our benchmark updated as new models are released.

The results reported here have also been added to Appendix D.11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.