acceptodds
Under review as a conference paper at ICLR 2027

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Abstract

Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce **CodeTriage**, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with **21%** fewer program revisions and **12%** fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automated program repair, large language models, learning to search, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iKyxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method CodeTriage which ranks the next procedure of fixing and testing in the LLM-based coding repair pipeline. It leverages failure feedback and history to predict solve distance of each proposed edit, saving tokens and time compared to implementing each edits into real programs. The experiments indicate that CodeTriage reduces the number of repair expansions by 21% and output token consumption by 12% compared to the strongest baselines. Moreover, it generalizes to 11 unseen repair LLMs without retraining, achieving an average AUSC improvement of 0.90 percentage points, demonstrating its search efficiency and cross-model generalization.

Critical strengths
  1. This simple yet effective method can reduce 21% search expansion and 12% output token compared to the SOTA baselines.
  2. This 0.6B LLM-based scoring model could be adapted to 11 unseen LLMs without retraining, and achieves the highest average success.
  3. Clear motivation and presentation.
Critical weaknesses
  1. Main experiment (section 4) only takes 3-branch and 4-depth pre-generated candidate trees (mentioned in Appendix A) for evaluation. These small-size trees would fit the 0.6B model context limit; however, I wonder whether it is similar to the real repository-level repair scenario. I would like to see clarification on choosing these small-size trees, or please add experiments with candidate trees of more depth and branches, because some similar work, SWE-Search, for example, considers search depths of up to 20 in repository-level edits.

  2. The global ranking baseline's performance is approaching CodeTriage's ( in table 12), with only a 3.4% gap between these two methods. However, this improvement is not statistically resolved with only 3 seeds. I suggest evaluating both methods with more training seeds to identify the robustness of this improvement. Besides, adding comparison on output tokens like in section 5.1 would help compare comprehensively. On top of that, adding some analysis on the improvement gain among these methods would help clarify when and why distributional solve-distance prediction provides benefits.

  3. The main experiment only measures the existing methods but does not report the mean expansion of the oracle search policy that knows the best search&repair trajectory in advance. This could help quantify the gap between current methods and oracle performance.

Questions for authors
  1. The model is trained to predict the shortest solve distance, but the evaluation setting is which model could achieve success with the lowest number of search expansions. A shorter solve trajectory does not necessarily mean lower search effort (the former is about depth, the latter is about depth and branch). Could the authors further justify the alignment between the supervision target and the search cost objective?

  2. Appendix J proved that involving the censored observations into training data could improve the probability calibration. However, since the major objective is to minimize the search expansion, could the authors compare the mean expansion count of models trained with and without censored observations as well?

  3. Why not provide the artifact of CodeTriage and the related evaluation process? Please add the anonymous link or explain.

3 comments

Thank you for the detailed review. You asked for more seeds, an oracle, and a censoring comparison, and we ran all of them. We also ran a new experiment on when to stop searching.

Regarding the global ranker (2nd weakness): The global ranker is also our contribution: it is CodeTriage with a single score per edit instead of a distribution (Table 12). It keeps the two design choices that AB-MCTS and REx lack: ranking all pending edits on one learned scale, and depth-aware targets (Eq. 2). Its strong result therefore supports the paper's main claim, that comparing edits across programs improves search.

Five seeds. As you suggested, we trained two more seeds of each model. CodeTriage needs 2.3% fewer expansions than the global ranker:

Five seeds per modelMean expansionsFewer expansions on
CodeTriage11.814 of 5 seeds
Global ranker12.091 of 5 seeds

Difficult repairs that need several steps. Here, CodeTriage's savings over the global ranker are largest: where the shortest fix takes three steps, they are four times the average, and CodeTriage wins on all five seeds.

Why the distribution helps. CodeTriage learns from each edit's own outcome: how many expansions it is from a fix. That number means the same thing in every tree, so predictions for edits from different programs are directly comparable, even though CodeTriage never trains on such pairs. The global ranker learns only which of two edits is better, and only from pairs that a fixed mix of search policies happened to sample (App. F.4). So CodeTriage orders edits from different programs correctly more often (Table 13).

Deciding when to stop. Unlike a single score, the distribution also gives the probability that an edit reaches a fix within the remaining depth. With it, search can stop once no pending edit is likely to succeed. We fit the threshold on validation problems and applied it once to the test split. At the same cost of about 1 point of success, this rule saves nearly twice as many expansions as the global ranker's score, and more on every seed:

Stopping signalExpansions savedWasted search cut on problems with no fix
CodeTriage probability24%30%
Global ranker score13%16%

Regarding the oracle (3rd weakness): CodeTriage closes 2.6 times as much of the gap to the oracle as AB-MCTS, the strongest baseline. We will add the oracle to Table 7:

PolicyMean expansions, test splitShare of the random-to-oracle gap closed
Random16.540%
AB-MCTS14.5913.5%
CodeTriage11.5435%
Oracle2.12100%

The remaining gap keeps our released corpus far from saturated: predicting repair outcomes before generating code is still an open problem.

Regarding the training target and search cost (1st question): Both are counted in expansions, and an expansion costs the same whether it goes deeper or branches wider. Solve distance is the fewest expansions from an edit to a fix. CodeTriage ranks all pending edits, across programs and depths, by this predicted distance (Eq. 2), so each expansion goes to the edit predicted closest to a fix.

  • Exact predictions give the fewest possible expansions, the oracle's cost. Search then expands only a shortest repair path, with no wasted expansion in depth or breadth (App. F.4). We will add this as a proposition. Prior code-search scorers, such as the critic in CodeRL and the selection rules in AB-MCTS and REx, offer no such guarantee.
  • A target that models branching directly costs more. Training on the chance that random search below an edit finds a fix needs 16% more expansions than CodeTriage (Table 12).

Regarding censoring and search cost (2nd question): Censoring improves calibration at no search cost, within seed noise:

Training data, three seeds eachMean expansionsCalibration error (App. J)
With censored edits (CodeTriage)11.900.068
Without censored edits11.730.172

The stopping rule above uses this calibrated probability.

Regarding tree size (1st weakness): The 0.6B model's context does not limit tree size. CodeTriage reads a proposed edit and the feedback on its parent program, never the whole tree (App. C). Instead, tree size is set by evaluation: we generate and run every program in each tree, so every search policy is compared on the same programs. Each added level of depth roughly triples that cost, and the main corpus already runs 4.0 million programs (App. A). This setting matches accepted work:

  • Three edits per program and depth 4 are typical for full-program repair search. CodeTree (NAACL 2025) branches three ways to depth 3, and most Self-Debugging (ICLR 2024) repairs end within three turns.
  • SWE-Search (ICLR 2025) counts depth in agent actions, not program revisions. Its depth of 20 includes actions such as searching code, and it caps each search at 100 iterations, compared with up to 120 programs in our trees.
  • AB-MCTS (NeurIPS 2025) and REx (NeurIPS 2024) also evaluate on self-contained problems. CodeTriage reads only what repository-level repair also produces.

Regarding artifacts (3rd question): Yes. We release all 12.2M executed programs as repair trees, so any search policy can be replayed on exactly our trees. As our Reproducibility Statement lists, the paper specifies the search setting, loss, priority rule, metrics, baselines and their tuning, data construction, and hyperparameters (Sec. 2 to 4, App. A, C, and D). We are happy to answer specific questions about any part.

In summary, the 0.6B model's context does not limit tree size. The distribution's stopping rule saves nearly twice as much as the global ranker's, and more on every seed. CodeTriage also closes 2.6 times as much of the gap to the oracle as the strongest baseline. We believe these answers address your concerns.

Thanks for the response. I maintain my score for the following reasons.

  1. Limited evaluation on realistic repository-level repair tasks. The evaluation is still limited to self-contained programming problems and does not demonstrate CodeTriage's effectiveness in repository-level code repair scenarios.

  2. Lack of accessible artifacts. The authors claim to have released the 12.2M executed programs as repair trees. However, I could not find the related anonymous links to download the dataset, and the link to the CodeTriage model.

Reviewer_HnAZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CodeTriage uses a trained 0.6B model to prioritize proposed code repairs before implementation and testing. From execution feedback, failure diagnoses, and edit descriptions, it predicts a distribution over the shortest successful repair distance and ranks candidates across the frontier. On held-out solvable repair trees, it reduces expansions by approximately 21% and proposer output tokens by 12% relative to the strongest baseline for each metric. Without retraining, it transfers to eleven unseen repair models, improving the mean normalized area under the success curve (AUSC) by 0.90 percentage points.

Critical strengths
  1. The paper explores a promising direction for improving code-repair efficiency by selecting useful edits before generating and testing revised programs.
  2. Large-scale repair-tree experiments and evaluations with eleven unseen repair models demonstrate improvements in search efficiency and transfer across models.
Critical weaknesses
  1. The methodological contribution beyond learned global ranking remains insufficiently established. Global scalar ranking achieves 12.20 mean expansions, compared with 11.79 for CodeTriage. Appendix F.4 acknowledges that this difference is not statistically resolved with three seeds, leaving the additional benefit of distributional prediction uncertain. The allocation of model capacity also lacks a cost-matched comparison. Although the backbone sweep evaluates selector quality, it does not establish the relative efficiency of using a larger model for planning and selection. A smaller model could then implement sufficiently explicit edits. This alternative need not be tested to validate the fixed-proposer result, but it exposes an unanswered question about the architecture's overall efficiency.
  2. The evaluation provides limited evidence of practical gains in real software repair. Experiments use Python programming problems with branching factor three and maximum repair depth four. They do not evaluate repository-level repair on benchmarks such as SWE-bench or SWE-bench Pro, leaving applicability to cross-file changes and project dependencies untested. Independent held-out testing of final patches is also not reported. The main benchmark retains only solvable roots, although the cross-proposer evaluation usefully removes this restriction. Appendix K measures scorer throughput and estimates FLOPs, but does not report end-to-end latency or amortize the necessary data-generation and training costs. Repository-scale repair is acknowledged as future work, yet these boundaries still limit the demonstrated practical significance of the approach.
Questions for authors
  1. Given the unresolved gap in Appendix F.4, what evidence supports the additional search benefit of distributional prediction over global scalar ranking?
  2. After accounting for data generation, training, validation, and scorer inference, how many repair tasks are needed to recover the upfront cost?
  3. Do any existing evaluations test repository-level repair or validate final patches against tests withheld during repair?
4 comments

Thank you for the review. For your first question, we ran a new experiment: the distribution's probability tells search when to stop. We also trained two more seeds of each model.

Regarding what distributional prediction adds (1st weakness, 1st question): The global ranker is also our contribution: it is CodeTriage with a single score per edit instead of a distribution (Table 12). It keeps the two design choices that AB-MCTS and REx lack: ranking all pending edits on one learned scale, and depth-aware targets (Eq. 2). Its strong result therefore supports the paper's main claim, that comparing edits across programs improves search. On top of it, the distribution gives the probability that an edit reaches a fix within the remaining depth, which a single score cannot.

Deciding when to stop. With this probability, search can stop once no pending edit is likely to succeed. We fit the stopping threshold on validation problems and applied it once to the held-out test split. At the same cost of about 1 point of success, this rule saves nearly twice as many expansions as the global ranker's score, and more on every seed:

Stopping signalExpansions savedWasted search cut on problems with no fix
CodeTriage probability24%30%
Global ranker score13%16%

Difficult repairs that need several steps. Here, CodeTriage's savings over the global ranker are largest: when the shortest fix takes three steps, they are four times the average, and CodeTriage wins on all five seeds. Overall, it needs 2.3% fewer expansions:

Five seeds per modelMean expansionsFewer expansions on
CodeTriage11.814 of 5 seeds
Global ranker12.091 of 5 seeds

Regarding model capacity allocation (1st weakness): The paper covers this trade-off: the backbone sweep gives each scorer's expansions (Sec. 5.4, Fig. 5b), and App. K gives its compute. Together they show that a larger scorer needs fewer expansions but costs more compute than it saves. A 14B scorer saves 0.77 expansions per search over our 0.6B scorer, but its per-search scoring costs as much compute as about 12 expansions. A larger planner with a smaller implementer changes what each expansion costs, not which edit to expand next, so it can be combined with CodeTriage (Sec. 6).

Regarding amortization and latency (2nd weakness, 2nd question): About 1.2M repair tasks recover the cost of building the training data, and that cost is paid once. One frozen scorer serves 11 unseen repair LLMs without retraining (Table 16), and we release the corpus for reuse.

  • The training data costs as much as about 2.4M expansions. That is the compute to generate and test the training and validation trees. On the training repair LLM, CodeTriage saves about 1.9 expansions per task over AB-MCTS (NeurIPS 2025), after subtracting its own scoring (Table 1 and App. K). This leaves out the test runs that each saved expansion also skips.
  • Training a scorer takes about 25 A100 GPU-hours.
  • Scoring costs under 5% of the repair LLM's compute (App. K). Scoring three edits takes an amortized 8.3 ms on one H100 (Table 18), while each expansion decodes about 1,150 output tokens and runs the full test suite.

Regarding scope (2nd weakness, 3rd question): A complete repair system has many components, and CodeTriage focuses on one of them: which proposed edit to expand next. Like the strongest baselines, AB-MCTS (NeurIPS 2025) and REx (NeurIPS 2024), we evaluate on self-contained problems.

  • Three edits per program and depth 4 are typical for full-program repair search. CodeTree (NAACL 2025) branches three ways to depth 3, and most Self-Debugging (ICLR 2024) repairs end within three turns.
  • CodeTriage reads only what repository-level repair also produces. Its inputs are test output, a failure diagnosis, the edit text, and four numeric features (App. C).

In summary, the distribution's stopping rule saves nearly twice as much as the global ranker's score. CodeTriage's search saving is largest on repairs that need several steps. Both hold on every seed, and the upfront cost is paid once. We would value your view on whether these answers address your concerns.

Thank you for the helpful response. It addresses several points in my review, but two of my original concerns remain unclear.

My question about model size concerned the division of work between selection and code generation. The scorer-size experiment keeps the code generator fixed. I was asking about a different setup: a larger model for planning and selection, with a smaller model implementing the selected edit. Since the large model already diagnoses the failure and proposes a repair, why also use it to implement the change? I understand that this alternative could be combined with CodeTriage, but I am still unclear about the reasoning behind the current design.

My concern about repository-level evaluation also remains. Transfer across eleven repair models is useful evidence, but all evaluations still use self-contained programming problems. Similar input formats do not establish that the policy will remain effective with cross-file changes and project dependencies. Are any results available on SWE-bench, SWE-bench Pro, or other repository-level benchmarks?

Thank you for clarifying your question about model size. We start there.

1. Planning and implementation: The paper studies which proposed edit to implement next. For this, it holds the repair LLM fixed.

  • A fixed repair LLM isolates the search policy. Every policy sees the same generated programs. Only the order of expansions differs (Sec. 2).
  • Selection does not need a large model. A 14B scorer saves 0.77 expansions per search over our 0.6B scorer. But its scoring costs as much compute as about 12 expansions (Sec. 5.4 and App. K).

2. Repository-level experiments: Repository-level repair involves fundamentally different challenges, including code localization, context retrieval, and issue reproduction. It requires a complete coding-assistant framework and is outside the scope of the current paper. The problem we address is highly challenging, far from being solved, and well studied in multiple papers that refine code with execution feedback: AB-MCTS (NeurIPS 2025), SFS (ICLR 2025), ORPS (ICML 2025), RethinkMCTS (EMNLP 2025), and ReVeal (ICLR 2026).

To summarize, the paper holds the repair LLM fixed to isolate the search policy. A larger selector costs more than it saves. Self-contained repair is also still far from solved.

We hope these answers resolve your two remaining concerns.

Before the discussion period closes, we would like to know whether our reply answers your questions. If so, please consider this in your final evaluation. If anything is still unclear, we are glad to clarify.

Reviewer_B7RuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a small, fine-tuned policy that re-ranks repair candidates in program repair. This re-ranking allows fewer expansions and thus quicker repairs. The paper evaluates this policy on pre-computed search trees using different proposer coding models, which works because the policy does not change the tree, just changes the ordering. The paper compares the policy models with other repair reordering methods, evaluates transfer to other models, and performs relevant ablations.

Critical strengths
  • Proposing small, specialized models to aid larger models in being more efficient is relevant.
  • The approach is conceptually simple, easy to understand and easy to implement.
  • The evaluation is thorough; I appreciate the evaluation protocol on complete trees which allows for many experiments on a limited budget, the transfer experiment, and the ablations on training data. I appreciate that the main model used (.6B) is not strictly the best model, but the model that best captures the point of using a small policy.
  • The paper is clearly written and the results are properly visualized.
Critical weaknesses
  • The results, even though consistent, are modest.
  • There are no comparisons with frontier models or self-driven agentic loops. Whereas not grounds for rejection, it helps to compare the delta between small, specialized approaches and frontier agents.
Questions for authors
  • How do frontier models perform on the repair task (performance, cost)?
  • How do the proposer models perform under a self-driven agentic loop?
2 comments

Thank you for the review.

Magnitude of gains from CodeTriage (1st weakness): CodeTriage reduces the number of expansions by 21% compared with the strongest baseline (AB-MCTS, NeurIPS 2025), and reduces output tokens by 12% compared with the most token-efficient baseline (REx, NeurIPS 2024; Tables 7 and 8). On the same trees, this is by far the largest step over the previous best:

PolicyMean expansionsFewer expansions than the row above
Random search16.54—
REx (NeurIPS 2024)14.9210%
AB-MCTS (NeurIPS 2025)14.592%
CodeTriage11.5421%

Regarding frontier models (2nd weakness, 1st question): CodeTriage's cost stays fixed as the repair LLM grows: scoring costs under 5% of the training repair LLM's compute, and under 0.5% for dense 27B to 31B models (App. K). We tested 12 open-weight repair LLMs from six model families, including Gemma-4-31B, Nemotron-3-Super, and Qwen3-Next-80B. CodeTriage leads on all 11 unseen ones without retraining (Table 16).

Regarding a self-driven agentic loop (2nd weakness, 2nd question): CodeTriage needs 32% fewer expansions and 13% fewer output tokens than either self-driven loop in our protocol: depth-first search, which always revises the newest program, and revising the program that passes the most tests (Tables 7 and 8). It also beats depth-first search on all 12 repair LLMs (Fig. 6). Agents that keep several candidate repairs face the choice CodeTriage makes: which candidate to revise next.

In short, CodeTriage needs 21% fewer expansions than the strongest baseline and beats depth-first search on all 12 repair LLMs, while scoring costs under 5% of the repair LLM's compute. We believe these answers support your view that small, specialized models can make larger models more efficient.

As the discussion period comes to an end, does our reply address the concerns behind your current assessment? If so, please take this into account in your final evaluation. If any point remains open, we are happy to address it.

Comments on the paper

1 comment

We thank the reviewers. They find CodeTriage simple yet effective (iKyx), its evaluation thorough (B7Ru), and its direction promising (HnAZ).

Three points stand out from the discussion:

1. Domain: The problem we address is highly challenging, far from being solved, and well studied in multiple papers that refine code with execution feedback: AB-MCTS (NeurIPS 2025), REx (NeurIPS 2024), SFS (ICLR 2025), ORPS (ICML 2025), RethinkMCTS (EMNLP 2025), ReVeal (ICLR 2026), RLEF, µCode, and CTRL (ICML 2025), and CURE and EG-CFG (NeurIPS 2025). In the corpus we release, 65% of the gap between random search and an oracle remains open.

2. Size of the gains: CodeTriage needs 21% fewer expansions than the strongest baseline, a bigger step than all prior progress over random search. It closes 35% of this gap, against 13.5% before.

3. Transfer and cost: All three reviewers value the transfer to 11 unseen repair LLMs. One frozen 0.6B model leads on all of them without retraining. Its scoring costs under 5% of the repair LLM's compute.

Revisions. The revised PDF marks new text in blue. It adds:

  • an oracle row in Table 7
  • a proposition that our priority rule attains the oracle cost under exact predictions (Proposition 1, App. F.4)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.