- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper compares two ways to spend inference budget after a failed first attempt at code generation. Resampling: Best-of-N, with the benchmark tests deciding acceptance. Repair with execution feedback: an always-repair policy, plus REx as a third comparator. The paper compares policies using a solve-within-token-budget curve. Each call is charged its reported prompt plus completion tokens, and the curve is summarized by its normalized area. The experiments use BigCodeBench, BigCodeBench-Hard, and HumanEval+ with four open-weight models. The main findings include that repair leads on the two demanding benchmarks, and it disappears when the failure text is replaced by a content-free notice, and reverses when the program is kept without the failure text.
Critical strengths
-
Well-motivated, cost-aware evaluation of repair vs resampling.
-
Evaluation on withheld tests strengthens the evidence for repair.
-
Targeted ablations clarify the role of execution feedback.
Critical weaknesses
-
The cost model counts tokens, not provider cost. Prompt and completion tokens are summed with equal weight. In practice, input tokens are typically priced several times lower than output tokens (e.g. GPT6 input 10, output 50). By the paper’s own account, the repair premium is almost entirely input tokens.
-
The restart does not reflect how restarts work in real coding agents. The introduction and related work motivate the study with self-debugging and repository-level coding agents, and the conclusion frames its allocation rule for a coding system. But every experiment is single-function generation, and the restart is a single fresh completion that never sees execution feedback. In a real agent, restarting means launching a new full agent that runs the tests and repairs its own code. The relevant question then becomes whether the budget should extend the current failing trajectory, or start a new trajectory with its own test-and-repair loop. The paper does not test this.
-
Thinking is disabled for all models, including hybrid reasoning models. Current code systems mostly run with reasoning enabled, so transfer is unclear.
-
A teaser figure early in the paper would make the work much easier to follow. The experimental setup in Section 4.1 runs for nearly a full page of dense prose. It's better to condense this section and add a figure before.
Questions for authors
-
Since the additional token consumption of repair is primarily in the prompt, how sensitive are the conclusions to assigning different prices to input and output tokens? Could the authors recompute the solve-within-budget curves using C=N_{in}+5N_{out} or sweeping over input/output pricing ratios?
-
Could the authors evaluate this comparison on Roo Eval[1] or any code agent benchmark, contrasting continuation of a failing agent trajectory with restarting a fresh agent from a clean workspace and context, where the restarted agent is itself allowed to execute tests and iteratively repair its code? Given the limited rebuttal period, a study on the Python subset would provide useful evidence.
-
Could the authors repeat a comparison with reasoning enabled, keeping the model and task set fixed and including all billed reasoning tokens in the budget? How does enabling reasoning affect the first-attempt failure rate, the conditional advantage of repair after failure, and the budget crossover?
[1] Roo Code Evals by Roomote.
Response to Reviewer f5bG
Q1: Sensitivity to input/output pricing (e.g., C=Nin+5⋅Nout). We ran this for r∈{1,3,5,8} on all eight BigCodeBench and BigCodeBench-Hard cells, with and without caching (Table A5). 72–95% of a repair call's extra tokens are in the prompt, so weighting output tokens more does not shrink the lead, and no cell changes sign. At r=5 without caching (last column of Table A5):
Q2: Continuing a failing trajectory vs restarting a fresh agent that itself tests and repairs. We ran the closest function-level analogue of this contrast, under one 12k-token budget and held-out scoring (Table A6). A "restart" opens a new trajectory from a clean context. That trajectory runs the visible tests and repairs its own code (one or two turns) before the next restart. Gemma-4, BigCodeBench, n=196, AUC relative to resampling:
Every policy whose trajectories receive execution feedback leads resampling, whether the budget extends the current trajectory or starts a new one. This is consistent with the paper's feedback ablation (Table 6):
Extending this contrast to repository-level agents such as Roo Code Evals, where a restart also re-explores the workspace, is a natural next step, and the revised Conclusion names it as future work.
Q3: Reasoning enabled, with all billed reasoning tokens. Table A4 has the full numbers; please also see our response to Reviewer ZRiH.
Q4: Teaser figure and Section 4.1. We added Figure 1 on page 2, which shows the shared first attempt, the repair chain versus fresh samples, and the charged costs. Section 4.1 is condensed to three short paragraphs, and the details are in Appendix B.3.