acceptodds
Under review as a conference paper at ICLR 2027

Choosing How Coding Agents Continue: A Study on BigCodeBench

Abstract

After producing a program, a coding agent can stop, revise it, request a review, or start again. We ask whether information from the first attempt helps choose among these actions. On an eligible subset of BigCodeBench, we train a routing policy on exploratory executions and freeze it before evaluation on unseen tasks. Compared with a fixed strategy, regeneration by the stronger model, the observed gain was 2.5 percentage points (95% CI: -1.0 to 5.5). It was not statistically conclusive and fell below the preregistered practical target. A post-hoc analysis locates the net gain in stopping and stronger-model review. That review action performed poorly across the full evaluation set, but produced no execution errors where the policy selected it. All its execution errors coincided with output-limit hits; review and regeneration used different effective output allowances. These descriptive associations do not establish why the selections worked or whether they will recur. We report state-acquisition, continuation and learning costs separately. The task-level comparison exposes differences hidden by aggregate action rankings while leaving the generalisable benefit of adaptive continuation unresolved.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, code generation, agent routing, post-generation state, reproducible evaluation, cost accounting

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_hwFsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The work examines how the result of a coding agent’s first attempt might influence the decision to repair the existing solution or solve the task again, possibly using a different agent. Empirical studies train different continuation policies on exploratory tasks from a subset from BigCodeBench, and then evaluate on unseen tasks. The result is no practical effect of continuation policy on hidden test pass rate, compared to a fixed (model regeneration) policy.

Critical strengths
  • The study separates policy learning from evaluation by ensuring that the continuation policy never observed hidden test outcomes at decision time, and also preregisters the continuation policy, baseline, and tasks.
Critical weaknesses
  • There’s an asymmetry in the budget allocation between the baseline regeneration (B-solo) and review (B-expert) that confounds the experimental design. B-Solo was given a budget that included the cost of the first-pass generation, while B-expert was restricted to only the continuation budget. B-expert suffered execution errors caused by hitting this lower budget ceiling. As noted in the paper, this asymmetry prevents isolating whether review performed poorly due to intrinsic difficulty of reviewing code or due to token constraints.
  • As the authors conclude, a prompt-plus-program ablation that omits execution feedback is needed to isolate whether information from the first coding attempt in the form of a visible-check feedback helps the continuation policy choose the next best action. The current design of the post-generation policy (P2-post) mixes executed check outcomes with other types of info into a single feature representation.
Questions for authors
  • Pg. 2: where is the P1 Table? It would be helpful to display at least a stylized version.
  • The central comparison of continuation policy (P2-Post) vs. regeneration strategy (B-solo) yields no practical difference. What can we conclude about the generalizable benefit of the continuation policy?
Reviewer_pUG9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether first-attempt state helps choose among stopping, revision, review, and regeneration in coding agents. A frozen policy is evaluated on held-out BigCodeBench tasks, yielding an inconclusive gain over a fixed regeneration strategy, followed by analyses of selected actions, output limits, and cost.

Critical strengths
  1. Exploration and confirmation are separated, and both the policy and fixed comparator are frozen before evaluation. The paired analysis and interpretation of the inconclusive result are careful, making the primary comparison credible for its stated protocol.
  2. The task-level decomposition reveals action differences that aggregate rankings conceal. The paper also discloses budget asymmetry and the cost-accounting correction. This makes the study useful for designing and interpreting future continuation experiments.
Critical weaknesses
  1. The most interesting review pattern remains entangled with output budget. Section 4 gives B-solo an additional allowance based on first-pass cost. In Sec. 5.5, all 49 B-expert execution errors coincide with output-limit hits. The authors correctly avoid a causal claim, but the records do not distinguish selecting programs that benefit from review from selecting programs whose review fits the tighter limit. This limits what can be learned from the selected-review pattern beyond the specific budget protocol.
  2. The empirical conclusion remains narrow. The selected router has negative exploration CV gain in Table 12, while Table 8's synthetic results show substantial loss of available gain as the feature menu grows. The held-out result therefore does not separate limited action-selection signal from limitations of the learning procedure. The authors acknowledge this.
Questions for authors
  1. Do the existing records support any analysis separating the selected-review pattern from available output allowance or program length? Please clarify what can be concluded without new model calls.
  2. Do the existing exploration analyses provide evidence about learning limitations beyond the selected router's CV gain, such as training-size sensitivity or stability of the selected actions? This would help assess what the fitted-policy result teaches us about continuation selection.
Reviewer_EC7WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether information from an initial program helps choose between stopping, revising, requesting review, and regenerating. The learned policy achieves 74.0% on 200 held-out tasks, compared with 71.5% for fixed regeneration, but the confidence interval includes no improvement.

Critical strengths

The separation between policy selection and evaluation is sound, and the authors are appropriately candid about the uncertainty.

Critical weaknesses

The main problem is that the experimental design leaves the motivating question largely unresolved. Review and regeneration receive different output allowances: the median limits are 644 and 965 tokens, respectively, and all 49 execution errors in the review condition coincide with hitting its limit. This does not invalidate the comparison of the exact policies tested, but it makes the action rankings difficult to interpret beyond this particular configuration.

Questions for authors

In AI USE STATEMENT: "The author used Claude (Anthropic) and Codex (OpenAI) to develop the research question and experimental design, write and review code and tests, implement the synthetic-study generator, check statistical statements, search and verify references, analyse saved results, and draft and revise the manuscript. Reviews by another assistant were part of this process, not independent replications. The author directed the work, approved expenditure and takes responsibility for the submitted claims. Language models are also the objects of study." Can the authors disclose how much human work were put in this work?

Reviewer_mvnYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how a coding agent can adaptively select subsequent actions based on the current task state after producing initial code. The authors design six candidate actions and propose a routing policy based on logistic regression. In the experiments, the authors select 403 tasks from BigCodeBench. The experimental results show that the proposed P2-post routing policy achieves a success rate of 74.0%, an improvement of 2.5 percentage points over the best fixed strategy B-solo (71.5%). However, this gain is not statistically significant and fails to meet the pre-specified practical target. Further analysis reveals that the gains of the router mainly come from stopping directly and strong-model review. Nevertheless, the experiments are limited by the small size of training data, unequal budgets across actions and a single model combination. Overall, this paper presents an exploratory empirical analysis of the subsequent action selection problem for coding agents, yet it does not sufficiently demonstrate that learned routing policies can stably outperform the fixed strategy of using a stronger model.

Critical strengths
  1. Valuable research problem. The paper investigates an important and practical problem: how coding agents should adaptively select continuation actions based on the current task state, rather than relying on a fixed strategy. This problem is relevant to improving the effectiveness and efficiency of coding agents.

  2. Well-designed experimental evaluation. The paper adopts a reasonably rigorous experimental protocol, with separate exploration and confirmation sets, policy freezing before evaluation, and statistical significance testing. These design choices help reduce data leakage and selection bias, making the empirical findings more credible.

  3. Promising research direction. Although the proposed routing method does not achieve a statistically significant improvement, the results suggest that different continuation actions may be advantageous for different tasks. This observation highlights the potential of task-adaptive routing strategies and provides a useful foundation for future research.

Critical weaknesses
  1. Limited generalizability. The proposed routing policy is trained on task-specific features and action outcomes from a particular model pair. Since the relative effectiveness of continuation actions may vary across models and task distributions, the learned policy may not transfer well to other settings.

  2. Limited methodological novelty and inconclusive effectiveness. The proposed method relies primarily on manually designed features and logistic regression models to predict the success probability. While the problem itself is meaningful, the methodological design is relatively straightforward and offers limited algorithmic novelty. Moreover, the observed improvement is not statistically significant, making it difficult to establish the effectiveness of the proposed approach.

  3. Insufficient budget-aware routing and potentially confounded comparisons. The routing policy does not explicitly optimize the trade-off between success probability and computational cost. This makes it difficult to determine whether the observed performance differences arise from the continuation strategies themselves or from differences in available resources.

  4. Limited comparison with alternative routing strategies. The evaluation mainly compares the proposed logistic regression-based router against fixed-action strategies and a simple lookup-based policy. More competitive routing baselines, such as directly prompting an LLM to select the next action based on the current task state, are not considered. Such comparisons would help establish whether a separately trained routing model provides meaningful benefits over existing agent decision-making approaches.

Questions for authors

B-expert performs substantially worse than the other continuation actions. Could this be due to overly restrictive budget constraints, and might these constraints affect the fairness of the comparison across different actions?

Interpretability: Since the router is based on logistic regression, it should offer some degree of interpretability. Could the authors analyze the feature importance derived from the learned models to explain which factors most strongly influence the routing decisions?

My other points in the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.