- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method CodeTriage which ranks the next procedure of fixing and testing in the LLM-based coding repair pipeline. It leverages failure feedback and history to predict solve distance of each proposed edit, saving tokens and time compared to implementing each edits into real programs. The experiments indicate that CodeTriage reduces the number of repair expansions by 21% and output token consumption by 12% compared to the strongest baselines. Moreover, it generalizes to 11 unseen repair LLMs without retraining, achieving an average AUSC improvement of 0.90 percentage points, demonstrating its search efficiency and cross-model generalization.
Critical strengths
- This simple yet effective method can reduce 21% search expansion and 12% output token compared to the SOTA baselines.
- This 0.6B LLM-based scoring model could be adapted to 11 unseen LLMs without retraining, and achieves the highest average success.
- Clear motivation and presentation.
Critical weaknesses
-
Main experiment (section 4) only takes 3-branch and 4-depth pre-generated candidate trees (mentioned in Appendix A) for evaluation. These small-size trees would fit the 0.6B model context limit; however, I wonder whether it is similar to the real repository-level repair scenario. I would like to see clarification on choosing these small-size trees, or please add experiments with candidate trees of more depth and branches, because some similar work, SWE-Search, for example, considers search depths of up to 20 in repository-level edits.
-
The global ranking baseline's performance is approaching CodeTriage's ( in table 12), with only a 3.4% gap between these two methods. However, this improvement is not statistically resolved with only 3 seeds. I suggest evaluating both methods with more training seeds to identify the robustness of this improvement. Besides, adding comparison on output tokens like in section 5.1 would help compare comprehensively. On top of that, adding some analysis on the improvement gain among these methods would help clarify when and why distributional solve-distance prediction provides benefits.
-
The main experiment only measures the existing methods but does not report the mean expansion of the oracle search policy that knows the best search&repair trajectory in advance. This could help quantify the gap between current methods and oracle performance.
Questions for authors
-
The model is trained to predict the shortest solve distance, but the evaluation setting is which model could achieve success with the lowest number of search expansions. A shorter solve trajectory does not necessarily mean lower search effort (the former is about depth, the latter is about depth and branch). Could the authors further justify the alignment between the supervision target and the search cost objective?
-
Appendix J proved that involving the censored observations into training data could improve the probability calibration. However, since the major objective is to minimize the search expansion, could the authors compare the mean expansion count of models trained with and without censored observations as well?
-
Why not provide the artifact of CodeTriage and the related evaluation process? Please add the anonymous link or explain.
3 comments
Regarding tree size (1st weakness): The 0.6B model's context does not limit tree size. CodeTriage reads a proposed edit and the feedback on its parent program, never the whole tree (App. C). Instead, tree size is set by evaluation: we generate and run every program in each tree, so every search policy is compared on the same programs. Each added level of depth roughly triples that cost, and the main corpus already runs 4.0 million programs (App. A). This setting matches accepted work:
- Three edits per program and depth 4 are typical for full-program repair search. CodeTree (NAACL 2025) branches three ways to depth 3, and most Self-Debugging (ICLR 2024) repairs end within three turns.
- SWE-Search (ICLR 2025) counts depth in agent actions, not program revisions. Its depth of 20 includes actions such as searching code, and it caps each search at 100 iterations, compared with up to 120 programs in our trees.
- AB-MCTS (NeurIPS 2025) and REx (NeurIPS 2024) also evaluate on self-contained problems. CodeTriage reads only what repository-level repair also produces.
Regarding artifacts (3rd question): Yes. We release all 12.2M executed programs as repair trees, so any search policy can be replayed on exactly our trees. As our Reproducibility Statement lists, the paper specifies the search setting, loss, priority rule, metrics, baselines and their tuning, data construction, and hyperparameters (Sec. 2 to 4, App. A, C, and D). We are happy to answer specific questions about any part.
In summary, the 0.6B model's context does not limit tree size. The distribution's stopping rule saves nearly twice as much as the global ranker's, and more on every seed. CodeTriage also closes 2.6 times as much of the gap to the oracle as the strongest baseline. We believe these answers address your concerns.
Thanks for the response. I maintain my score for the following reasons.
-
Limited evaluation on realistic repository-level repair tasks. The evaluation is still limited to self-contained programming problems and does not demonstrate CodeTriage's effectiveness in repository-level code repair scenarios.
-
Lack of accessible artifacts. The authors claim to have released the 12.2M executed programs as repair trees. However, I could not find the related anonymous links to download the dataset, and the link to the CodeTriage model.
Thank you for the detailed review. You asked for more seeds, an oracle, and a censoring comparison, and we ran all of them. We also ran a new experiment on when to stop searching.
Regarding the global ranker (2nd weakness): The global ranker is also our contribution: it is CodeTriage with a single score per edit instead of a distribution (Table 12). It keeps the two design choices that AB-MCTS and REx lack: ranking all pending edits on one learned scale, and depth-aware targets (Eq. 2). Its strong result therefore supports the paper's main claim, that comparing edits across programs improves search.
Five seeds. As you suggested, we trained two more seeds of each model. CodeTriage needs 2.3% fewer expansions than the global ranker:
Difficult repairs that need several steps. Here, CodeTriage's savings over the global ranker are largest: where the shortest fix takes three steps, they are four times the average, and CodeTriage wins on all five seeds.
Why the distribution helps. CodeTriage learns from each edit's own outcome: how many expansions it is from a fix. That number means the same thing in every tree, so predictions for edits from different programs are directly comparable, even though CodeTriage never trains on such pairs. The global ranker learns only which of two edits is better, and only from pairs that a fixed mix of search policies happened to sample (App. F.4). So CodeTriage orders edits from different programs correctly more often (Table 13).
Deciding when to stop. Unlike a single score, the distribution also gives the probability that an edit reaches a fix within the remaining depth. With it, search can stop once no pending edit is likely to succeed. We fit the threshold on validation problems and applied it once to the test split. At the same cost of about 1 point of success, this rule saves nearly twice as many expansions as the global ranker's score, and more on every seed:
Regarding the oracle (3rd weakness): CodeTriage closes 2.6 times as much of the gap to the oracle as AB-MCTS, the strongest baseline. We will add the oracle to Table 7:
The remaining gap keeps our released corpus far from saturated: predicting repair outcomes before generating code is still an open problem.
Regarding the training target and search cost (1st question): Both are counted in expansions, and an expansion costs the same whether it goes deeper or branches wider. Solve distance is the fewest expansions from an edit to a fix. CodeTriage ranks all pending edits, across programs and depths, by this predicted distance (Eq. 2), so each expansion goes to the edit predicted closest to a fix.
Regarding censoring and search cost (2nd question): Censoring improves calibration at no search cost, within seed noise:
The stopping rule above uses this calibrated probability.