acceptodds
Under review as a conference paper at ICLR 2027

On the Value of Iterative Repair in Code Generation under a Token Budget

Abstract

When a language model is asked to write a program and its first attempt fails the tests, the system spends further model calls to recover, and it does so in one of two ways: it either draws additional independent attempts from the same prompt, which is the resampling family behind pass@ evaluation and Best-of- selection, or it feeds the failing program and its test output back to the model for repair, which is the family behind self-debugging agents. Practice is split between the two and the published evidence points in both directions, since repair has been reported both as a decisive improvement and as a poor deal compared with simply drawing more samples. Which of the two is worth its price, and what decides the sign? In this paper, we first define the solve-within-budget curve of an inference policy under the provider's own token accounting and take its normalized area as the quantity to compare, so that a longer repair prompt is charged for what it costs. Then, we record two elementary facts that organize the measurement: the two families can differ only on episodes whose first attempt fails, so the benchmark-level contrast is a first-attempt failure rate times a conditional advantage, and a repair turn is charged for carrying the previous program, so repair cannot lead below the price of a first repair call. Finally, we measure both policy families on BigCodeBench, its hard subset and HumanEval+ across four model families, and we repeat the comparison with neither family capped in the number of calls it may buy. The experimental results show that repair leads resampling on demanding benchmarks once the budget affords a first repair call, that at the price of one further fresh call the two are indistinguishable, that the lead survives uncapping, that it rides on the execution feedback and persists when the graded tests are withheld from every prompt, and that the extra solutions carry a premium in tokens per solved task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code generation, iterative repair, test-time compute, token budget, execution feedback

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_f5bGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper compares two ways to spend inference budget after a failed first attempt at code generation. Resampling: Best-of-N, with the benchmark tests deciding acceptance. Repair with execution feedback: an always-repair policy, plus REx as a third comparator. The paper compares policies using a solve-within-token-budget curve. Each call is charged its reported prompt plus completion tokens, and the curve is summarized by its normalized area. The experiments use BigCodeBench, BigCodeBench-Hard, and HumanEval+ with four open-weight models. The main findings include that repair leads on the two demanding benchmarks, and it disappears when the failure text is replaced by a content-free notice, and reverses when the program is kept without the failure text.

Critical strengths
  1. Well-motivated, cost-aware evaluation of repair vs resampling.

  2. Evaluation on withheld tests strengthens the evidence for repair.

  3. Targeted ablations clarify the role of execution feedback.

Critical weaknesses
  1. The cost model counts tokens, not provider cost. Prompt and completion tokens are summed with equal weight. In practice, input tokens are typically priced several times lower than output tokens (e.g. GPT6 input 10, output 50). By the paper’s own account, the repair premium is almost entirely input tokens.

  2. The restart does not reflect how restarts work in real coding agents. The introduction and related work motivate the study with self-debugging and repository-level coding agents, and the conclusion frames its allocation rule for a coding system. But every experiment is single-function generation, and the restart is a single fresh completion that never sees execution feedback. In a real agent, restarting means launching a new full agent that runs the tests and repairs its own code. The relevant question then becomes whether the budget should extend the current failing trajectory, or start a new trajectory with its own test-and-repair loop. The paper does not test this.

  3. Thinking is disabled for all models, including hybrid reasoning models. Current code systems mostly run with reasoning enabled, so transfer is unclear.

  4. A teaser figure early in the paper would make the work much easier to follow. The experimental setup in Section 4.1 runs for nearly a full page of dense prose. It's better to condense this section and add a figure before.

Questions for authors
  1. Since the additional token consumption of repair is primarily in the prompt, how sensitive are the conclusions to assigning different prices to input and output tokens? Could the authors recompute the solve-within-budget curves using C=N_{in}+5N_{out} or sweeping over input/output pricing ratios?

  2. Could the authors evaluate this comparison on Roo Eval[1] or any code agent benchmark, contrasting continuation of a failing agent trajectory with restarting a fresh agent from a clean workspace and context, where the restarted agent is itself allowed to execute tests and iteratively repair its code? Given the limited rebuttal period, a study on the Python subset would provide useful evidence.

  3. Could the authors repeat a comparison with reasoning enabled, keeping the model and task set fixed and including all billed reasoning tokens in the budget? How does enabling reasoning affect the first-attempt failure rate, the conditional advantage of repair after failure, and the budget crossover?

[1] Roo Code Evals by Roomote.

1 comment
AuthorsOpenReview ↗

Response to Reviewer f5bG

Q1: Sensitivity to input/output pricing (e.g., C=Nin+5⋅NoutC = N_{\mathrm{in}} + 5 \cdot N_{\mathrm{out}}). We ran this for r∈{1,3,5,8}r \in \lbrace 1, 3, 5, 8 \rbrace on all eight BigCodeBench and BigCodeBench-Hard cells, with and without caching (Table A5). 72–95% of a repair call's extra tokens are in the prompt, so weighting output tokens more does not shrink the lead, and no cell changes sign. At r=5r = 5 without caching (last column of Table A5):

  • BigCodeBench: DeepSeek +0.059 [+0.010, +0.109], Qwen +0.051 [+0.010, +0.092], Gemma +0.083 [+0.039, +0.130], Ministral +0.063 [+0.016, +0.110].
  • BigCodeBench-Hard: Gemma +0.108 [+0.050, +0.166], Ministral +0.112 [+0.042, +0.187].

Q2: Continuing a failing trajectory vs restarting a fresh agent that itself tests and repairs. We ran the closest function-level analogue of this contrast, under one 12k-token budget and held-out scoring (Table A6). A "restart" opens a new trajectory from a clean context. That trajectory runs the visible tests and repairs its own code (one or two turns) before the next restart. Gemma-4, BigCodeBench, n=196n = 196, AUC relative to resampling:

Policy− Best-of-NN
Continue the failing trajectory (always-repair)+0.060 [+0.027, +0.096]
Restart; each new trajectory repairs once+0.050 [+0.023, +0.082]
Restart; each new trajectory repairs twice+0.056 [+0.024, +0.090]

Every policy whose trajectories receive execution feedback leads resampling, whether the budget extends the current trajectory or starts a new one. This is consistent with the paper's feedback ablation (Table 6):

  • Replacing the program and the failure text by a content-free notice removes the advantage on two families and shrinks it to a quarter on the third.
  • Keeping the program but removing the failure text reverses the advantage.

Extending this contrast to repository-level agents such as Roo Code Evals, where a restart also re-explores the workspace, is a natural next step, and the revised Conclusion names it as future work.

Q3: Reasoning enabled, with all billed reasoning tokens. Table A4 has the full numbers; please also see our response to Reviewer ZRiH.

  • First-attempt failure: 0.47 on BigCodeBench and 0.70 on BigCodeBench-Hard.
  • Conditional advantage after a failure: repair solves 0.37 / 0.30 of episodes, against 0.12 / 0.08 for a fresh sample.
  • Area lead over Best-of-3: +0.078 and +0.140.
  • Crossover (Appendix D): with reasoning, a repair call costs about twice a fresh call (median ratio 1.8 and 2.0). The repair curve takes the lead at 1.25 times the median first-attempt cost, about 1.9k and 3.1k tokens. This is just above the cheapest first-plus-repair cost of any task (1.5k and 1.4k), as Proposition 2 requires.

Q4: Teaser figure and Section 4.1. We added Figure 1 on page 2, which shows the shared first attempt, the repair chain versus fresh samples, and the charged costs. Section 4.1 is condensed to three short paragraphs, and the details are in Appendix B.3.

Reviewer_SPoPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper compares iterative code repair using execution feedback with independent resampling under a token budget. It evaluates solve-within-budget curves and their average over a budget grid, charging both prompt and completion tokens. Two propositions characterize how first-attempt failures and repair costs constrain the comparison. Experiments across four model families on BigCodeBench, its Hard subset, and HumanEval+ report repair gains on the more demanding benchmarks within the evaluated call limits. Held-out test evaluations and feedback ablations further examine these gains, while cost analysis shows that repair achieves additional successes at a higher token cost per solved task.

Critical strengths

Important research question. Comparing independent resampling, such as Best-of-N, with sequential repair, such as self-debugging, under a token budget is a practically relevant question for allocating inference-time computation.

Critical weaknesses

Limited theoretical contribution. Section 3 introduces the solve-within-budget curve, which provides an interesting evaluation perspective. However, the propositions offer little theoretical insight. In particular, the authors invoke Proposition 2 to argue that comparison at a single budget is insufficient. Its small budget result merely states that resampling cannot be worse when repair is unaffordable but additional resampling may still be possible. This follows directly from feasibility, while the eventual advantage of repair is assumed rather than derived. The proposition therefore provides limited support for the authors’ argument.

The experiments are too restricted to support the broader conclusions. The main evaluation permits only three calls, without adequately justifying this restriction for the intended applications. This can prevent cheaper resampling from using remaining tokens even when additional calls are affordable. It also substantially restricts REx [1]: the original work evaluates up to 300 LLM calls per problem (64 on ARC), whereas here REx makes only one repair-selection decision between two fresh candidates. This setting does not adequately evaluate its repeated adaptive allocation between fresh sampling and repair. The additional experiment still collects only 12 fresh samples and six repair turns, making the “uncapped” characterization misleading unless these limits are shown to be nonbinding. The results therefore demonstrate an empirical advantage within the evaluated call limits, but provide insufficient evidence for broader conclusions about budget allocation.

Ambiguous notation. The costs (c0)(c_0) and (cr)(c_r) depend on the task, but this dependence is omitted from the notation, making expressions such as $(\min_t(c_0+c_r)) $ confusing. Please make the task dependence explicit, for example by writing (min⁡tc0(t)+cr(t))(\min_t{c_0(t)+c_r(t)}).

[1]: Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code Repair with LLMs gives an Exploration-Exploitation Tradeoff. Advances in Neural Information Processing Systems (NeurIPS), 2024.

Questions for authors
  1. What justifies the three-call limit in the reference protocol?
  2. Why is REx restricted to two fresh generations followed by one repair? Does the comparison hold when repeated scheduling is allowed under the same token budget?
  3. What supports the repair-switching rule in Section 5 beyond Proposition 2’s feasibility lower bound?
2 comments
AuthorsOpenReview ↗

Response to Reviewer SPoP

Q1: What justifies the three-call limit? It is a reference protocol that matches the two families in calls as well as in tokens. Two experiments relax it.

(a) The budget-adaptive collection (Table 3) draws 12 fresh samples and 6 repair turns per task.

  • Whole-range lead: +0.108 [+0.061, +0.155] for DeepSeek and +0.105 [+0.054, +0.158] for Gemma.
  • At the top of the grid, an episode can afford essentially all 12 fresh samples, and resampling never overtakes; its marginal return is near zero by the fifth sample (Figure 2, right).
  • We now call this collection "budget-adaptive" throughout rather than "uncapped".

(b) To remove the sample limit as well, the new Table A6 runs every policy to a 12k-token budget with no limit other than that budget. A 40-call safety cap is never reached. On episodes whose first attempt fails, Best-of-NN uses 15.7 calls on average (up to 35) and still trails every repair-based policy.

Q2: REx with repeated scheduling under the same token budget. We implemented REx as in Tang et al. (2024). Before every call it Thompson-samples over all programs generated so far, using Beta(1+κh,1+κ(1−h)+N)\mathrm{Beta}(1 + \kappa h, 1 + \kappa (1 - h) + N) with κ=20\kappa = 20, where NN counts the program's refinements, and refines the chosen program. The results are on Gemma-4, BigCodeBench, n=196n = 196, scored on held-out tests:

Policy (run to a 12k budget)Calls / episode (all; first attempt failed)AUC − Best-of-NN
Best-of-NN7.1; 15.7—
Always-repair3.2; 6.3+0.060 [+0.027, +0.096]
REx, re-scheduled at every call3.2; 6.4+0.058 [+0.026, +0.092]
REx + fresh-sample arm3.6; 7.2+0.055 [+0.023, +0.090]

The repair-versus-resampling comparison therefore holds when repeated scheduling is allowed. Every policy stops at the first program that passes the visible tests; on episodes whose first attempt fails, REx schedules 6.4 calls on average and up to 15, so its depth is set by the token budget alone.

Q3: Support for the switching rule beyond Proposition 2. Proposition 2 tells us where the comparison must be measured: below min⁡t{c0(t)+cr(t)}\min_t \lbrace c_0(t) + c_r(t) \rbrace a single-budget comparison is decided by the budget, not by the policies, so the curve has to be swept. The revision states this role explicitly, and the switching rule in the Conclusion is set by the measured crossover (Table 4). That crossover lies at 1,500–2,000 tokens, about three times the cost of a fresh call (2.8–4.1 across families). Above it, resampling does not overtake at any budget of the budget-adaptive collection (Table 3).

Notation. The revision writes c0(t)c_0(t), cb(t)c_b(t), cr(t)c_r(t) and min⁡t{c0(t)+cr(t)}\min_t \lbrace c_0(t) + c_r(t) \rbrace throughout.

Thank you for the additional experiments and clarifications. Increasing the call and token budgets partially addresses my concerns. However, the 12k-token evaluation remains limited compared with the substantially larger search budgets explored in related test-time scaling work [1]. Since the relative performance of repair and resampling varies across models and benchmarks, the current evaluation remains insufficient to support broader conclusions.

The clarification of the theoretical results does not fully address my concerns. While Propositions 2 formalize relevant observations, they offer limited insight beyond their assumptions and do not substantially strengthen the paper’s motivation or conclusions.

For these reasons, I maintain my score.

[1] Inoue et al. “Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search.” NeurIPS, 2025.

Reviewer_ZRiHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the tradeoff between resampling and iterative repair when an LLM fails on a code-generation task. It compares the two approaches under the same token budget and measures their performance after a failed first attempt. The experiments use BigCodeBench, BigCodeBench-Hard, and HumanEval+ with four model families. The paper finds that iterative repair outperforms resampling on the more demanding benchmarks, and its ablations suggest that execution feedback drives this advantage.

Critical strengths

There are many prior papers on repair vs resampling budget and this paper addresses some issues that previous papers overlooked such as comparing against same token budget and testing with some hidden tests that LLM does not get to see during repair or resampling. This experiment setup also shares the first attempt among different setups which makes the main comparison easier to interpret.

Critical weaknesses
  • The technical contribution beyond prior work is limited. Repair versus resampling is a well-studied question, and prior work already compares these methods under token budgets. The additional measurements are useful, but the experimental setting is too narrow to support a broad practical recommendation.

  • A major limitation is the lack of reasoning-enabled experiments. All four model families are run with thinking disabled and a 900-token completion cap. Reasoning could substantially change both methods' success rates and token costs, and therefore which method is preferable under a fixed budget. The authors acknowledge this limitation, but the paper does not establish whether its main recommendation holds with reasoning enabled.

  • The cost comparison does not account for different input/output prices or prompt caching. The paper adds input and output tokens with equal weight. Repair carries extra input containing the previous program and failure feedback, so different token prices could change its cost relative to resampling. Caching could also affect both methods. The raw-token comparison is useful, but it does not establish which method is better under equal monetary budgets.

  • The paper does not examine how repair and resampling should be compared in an agentic coding workflow. The experiments use function-level code-generation benchmarks rather than terminal-based repository tasks. In an agentic setting, a fresh attempt may involve exploration, tool calls, and multiple edits, while repair may reuse an existing workspace. It is unclear whether the reported tradeoff extends to these settings.

  • The main experiments still assume that the full test suite is already available for the code selection. The held-out experiment is only done as an ablation.

Questions for authors
  • With reasoning enabled and its tokens included in the budget, which method performs better?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZRiH

Q: With reasoning enabled and its tokens included in the budget, which method performs better? Repair. We re-collected DeepSeek-V4-Flash with thinking enabled on the same task draws as Table 1, with a 16,000-token completion cap and every billed completion token charged, including all reasoning tokens (Table A4). A reasoning call costs several times a non-reasoning one. The median first attempt costs 1.3k tokens on BigCodeBench and 2.0k on BigCodeBench-Hard. The grid is therefore expressed in multiples of the first-attempt cost.

BigCodeBenchBigCodeBench-Hard
First-attempt failure0.470.70
After a failure, next call solves: repair / fresh0.37 / 0.120.30 / 0.08
Repair − Best-of-3+0.078 [+0.041, +0.116]+0.140 [+0.074, +0.208]
Repair − Best-of-6 (twice the calls for resampling)+0.077 [+0.040, +0.116]+0.099 [+0.040, +0.160]

After a failed first attempt, an independent reasoning resample solves only 8–12% of episodes, while a repair turn solves 30–37%: reasoning does not make a fresh sample a substitute for execution feedback.

Q: Input/output prices and prompt caching. We recomputed the main matrix with C=Nin+r⋅NoutC = N_{\mathrm{in}} + r \cdot N_{\mathrm{out}} for r∈{1,3,5,8}r \in \lbrace 1, 3, 5, 8 \rbrace, with and without prompt caching (Table A5). With caching, the task statement shared with the first call is billed at 10%. With r=1r = 1 and no caching, the table reproduces Table 1 exactly.

Across the eight BigCodeBench and BigCodeBench-Hard cells, 72–95% of the extra tokens of a repair call are prompt tokens. Pricing output tokens higher therefore does not shrink the lead, and no cell changes sign. At r=5r = 5 without caching (last column of Table A5):

  • BigCodeBench: DeepSeek +0.059 [+0.010, +0.109], Qwen +0.051 [+0.010, +0.092], Gemma +0.083 [+0.039, +0.130], Ministral +0.063 [+0.016, +0.110].
  • BigCodeBench-Hard: Gemma +0.108 [+0.050, +0.166], Ministral +0.112 [+0.042, +0.187].

Q: The main experiments assume the full test suite is available for selection. The held-out protocol removes exactly this assumption (Table 5, main text). Each policy sees only the visible half of each task's tests, both for repair feedback and for deciding when to stop. The accepted program is then scored on the other half, which no prompt ever contains. Under this protocol repair is ahead in all six cells, and the pooled contrast excludes zero: +0.066 [+0.046, +0.087], or +0.069 [+0.045, +0.090] when the six cells are resampled as clusters. We keep the full-suite protocol as the reference because it matches how these benchmarks are reported; the held-out protocol shows that the conclusion does not depend on it. The new Table A6 also uses this protocol throughout.

Q: Agentic workflows. Please see our answer to Reviewer f5bG (Q2), where restarts that run their own test-and-repair loop are compared under one budget.

Q: Contribution beyond prior work, and breadth of the recommendation. Olausson et al. (2024) and Tang et al. (2024) compare repair and sampling at matched budgets. Our contribution is to explain when each wins and why:

  • the contrast factorizes into a first-attempt failure rate and a conditional advantage (Proposition 1), which determines on which benchmarks a difference can appear at all;
  • the crossover is bounded below by the price of a repair call (Proposition 2) and measured on four families (Table 4);
  • the advantage is carried by the execution feedback, which a content-free notice removes (Table 6);
  • the advantage holds on withheld tests (Table 5).

The revision adds reasoning, provider pricing, and repeated REx scheduling (Appendix D). The recommendation in the Conclusion is stated for function-level code generation with executable tests, the setting these experiments cover.

Reviewer_ZRiHOpenReview ↗

Response

Thank you for the additional results. Some of my concerns are addressed, but I still feel the contribution is very narrow. Keeping the score the same.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading. The reviews raise four common questions: does the result hold (i) with reasoning enabled and all reasoning tokens billed, (ii) under provider prices with prompt caching, (iii) when no policy is limited to three calls and REx can schedule repeatedly, and (iv) when a restart is itself a trajectory that tests and repairs. We ran new experiments for each question on the same task draws as the paper. Unless stated otherwise, all intervals are paired 95% bootstrap intervals over tasks.

QuestionNew evidence (revised PDF)Result
Reasoning on, reasoning tokens billed (ZRiH, f5bG)Table A4: DeepSeek-V4-Flash, thinking enabled, 16k capRepair − Best-of-3: +0.078 [+0.041, +0.116] on BigCodeBench, +0.140 [+0.074, +0.208] on BigCodeBench-Hard; also ahead of Best-of-6
Output-weighted prices, caching (ZRiH, f5bG)Table A5: cost Nin+r⋅NoutN_{\mathrm{in}} + r \cdot N_{\mathrm{out}}, r∈{1,3,5,8}r \in \lbrace 1, 3, 5, 8 \rbrace, with and without caching, BigCodeBench and BigCodeBench-HardNo cell changes sign; the lead does not shrink as rr grows, because 72–95% of a repair call's extra tokens are prompt tokens
Policies run to the budget, repeated REx (SPoP)Table A6: every policy runs to a 12k-token budget; REx re-schedules over all programs before every callResampling uses more than twice as many calls and still trails always-repair by +0.060 [+0.027, +0.096] and REx by +0.058 [+0.026, +0.092]
Restart as a new trajectory that repairs (f5bG)Table A6: restarts whose new trajectories test and repair their own codeBoth restart variants also lead resampling (+0.050 [+0.023, +0.082], +0.056 [+0.024, +0.090])

Revisions to the PDF:

  • Figure 1 on page 2 shows the protocol.
  • Section 4.1 is condensed, with the details moved to Appendix B.3.
  • Proposition 2 makes the task dependence c0(t)c_0(t), cr(t)c_r(t), cb(t)c_b(t) explicit, and its role in the analysis is stated.
  • A new paragraph in Section 4.2 and a new Appendix D (Tables A4–A6) report the experiments above.
  • The earlier 12-sample collection is consistently called "budget-adaptive" rather than "uncapped".
  • The Conclusion states the recommendation for function-level generation, the setting we measure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.