acceptodds
Under review as a conference paper at ICLR 2027

LeanRepair: Token-Efficient External Control for Local Code Repair

Abstract

Local coding agents keep source code private and give users direct control over deployment, but multi-candidate test-time reasoning incurs severe token costs that strain local serving budgets: under standard branch-and-merge search, every unguided generation, aggregation, and replayed context token burns inference computation without guaranteeing progress. We introduce LeanRepair, a deterministic external controller designed to improve the token efficiency of local code repair while maintaining competitive pass rates, requiring no training and no additional language-model calls. LeanRepair addresses two complementary sources of token expenditure: (1) context inflation, bounded by maintaining compact external edit-geometry state rather than replaying verbose histories, and (2) unproductive search persistence, mitigated via trajectory-aware early stopping that prunes remaining branch expansions when edit trajectories oscillate or stagnate. Across six repair benchmarks and four instruction-tuned backbones (24 cells), LeanRepair reduces token expenditure compared to Graph-of-Thoughts by a median of 23.9% across cells (and 23.0% pooled across the five non-QuixBugs benchmarks, reaching 82.4% on QuixBugs where baseline search suffers pathologically long loops), while achieving a competitive 75.4% aggregate pass rate (vs. 72.9% for GoT; 15 wins, 3 ties, 6 losses). Our findings demonstrate that lightweight, non-parametric external control can substantially reduce test-time token waste in local code repair without relying on expensive model-based reflection or learned value functions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vD2pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LeanRepair, a lightweight external controller for token-efficient iterative code repair. It uses edit-trajectory statistics to guide revisions, detect repeated failures, and stop unproductive search without additional model calls. Across six benchmarks and four backbones, LeanRepair reduces token usage relative to GoT in 23/24 settings (23.9% median reduction) while achieving a comparable or slightly higher aggregate pass rate (75.4% vs. 72.9%).

Critical strengths
  1. A simple and intuitive mechanism for reducing test-time compute.
  2. Broad empirical evidence for the headline efficiency claim, though the benchmarks are out-of-date by 2026 standards.
Critical weaknesses
  1. The evaluation focuses on relatively outdated code-repair tasks and benchmarks. Most of the benchmarks are based on short, self-contained programming problems or synthetic mutations, which may not adequately reflect modern repository-level coding agents. This limits the practical relevance of the empirical conclusions.
  2. The evaluation is largely restricted to small language models. The main experiments use models in roughly the 4B–9B range. Although the appendix includes Qwen3-14B, it remains unclear whether the proposed controller provides similar benefits for substantially stronger modern coding models, whose repair dynamics may differ considerably. In fact, the 14B results already suggest that the advantage can diminish as model capacity increases.
  • Minor: There is an unresolved citation placeholder (“??”) on Line 35.
Questions for authors

See weaknesses

1 comment

Thank you for your review, and for recognizing LeanRepair's simple, intuitive mechanism for reducing test-time compute and the broad empirical evidence for its efficiency claim. Below we quote each of your comments and respond to it. Changes in the revised PDF are marked in blue.


Weakness 1.

The evaluation focuses on relatively outdated code-repair tasks and benchmarks. Most of the benchmarks are based on short, self-contained programming problems or synthetic mutations, which may not adequately reflect modern repository-level coding agents. This limits the practical relevance of the empirical conclusions.

Response. Thank you for your comment. LeanRepair reduces tokens on every benchmark we evaluate. This includes QuixBugs, which contains real, human-written defects. The revision now states the repository-level scope explicitly.

  • Focus of the study. The paper asks how much of a multi-candidate search backend's token spend an external controller can remove without losing pass rate. Function-level repair answers this with exact ground truth and deterministic tests. These make possible analyses such as the counterfactual stopping audit and the stagnation/oscillation ablation.
  • Coverage. The six benchmarks span easy functions (HumanEval, MBPP), semantic reasoning (CRUXEval), competition problems (APPS), and library-heavy tasks (BigCodeBench). LeanRepair reduces tokens on all of them, with a median reduction of 23.9% across 24 cells.
  • Real defects. On QuixBugs, under matched runs, LeanRepair uses 26.2% fewer tokens than GoT and reaches a higher pass rate (89.0% vs. 86.5%).
  • Repository-level agents. These are named as a limitation (Sec. 6, App. C.11). The controller's inputs, successive program versions and execution feedback, are also available in repository-level agents, which makes them a natural next step.

Weakness 2.

The evaluation is largely restricted to small language models. The main experiments use models in roughly the 4B–9B range. Although the appendix includes Qwen3-14B, it remains unclear whether the proposed controller provides similar benefits for substantially stronger modern coding models, whose repair dynamics may differ considerably. In fact, the 14B results already suggest that the advantage can diminish as model capacity increases.

Response. Thank you for your comment. LeanRepair's advantage holds at larger scale. At Qwen3-14B it has the highest average pass rate of all five methods (84.5%) and uses 22% fewer tokens than GoT.

  • Model range. The 4–9B range reflects the target setting, local serving on a single consumer GPU, where token budgets are tightest. Within it we covered four model families.
  • 14B per benchmark (Table 10). LeanRepair has the best or tied-best pass rate on three of four benchmarks: QuixBugs, HumanEval, and MBPP. On BigCodeBench it is within one task of GoT (73.3% vs. 76.7% on 30 tasks) at 31% fewer tokens.
  • Across scales. At 1.7B, 4B, and 14B, LeanRepair has the highest average pass rate and uses 22–32% fewer tokens than GoT.

Evaluating even larger and code-specialized models is a natural next step, which we list as future work.

Note on one Table 1 cell. While answering other reviewers, we found that in the Qwen3-8B / QuixBugs cell, the baselines had been run with Qwen3's thinking mode while LeanRepair had not. We will correct that cell and withdraw the QuixBugs-specific 82.4% figure. The headline result is unchanged: the median reduction across 24 cells remains 23.9%. Details are in the general response.


Minor.

Minor: There is an unresolved citation placeholder ("??") on Line 35.

Response. Thank you for your comment. The "??" came from a missing BibTeX pass. The recompiled PDF has no unresolved references.


Thank you again for your review. We hope the clarifications and the added scope discussion address your concerns, and we would be glad to answer any further questions.

Reviewer_L24pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LeanRepair, a lightweight external controller for token-efficient local code repair. It combines edit-geometry guidance, a recurrence buffer, and trajectory-aware early stopping to reduce redundant repair iterations without additional LLM calls or training.

Critical strengths
  1. Lightweight and straightforward design. LeanRepair introduces a simple, deterministic mechanism for controlling iterative code repair without requiring additional LLM calls or model training. Its controller is largely model-agnostic and can potentially be incorporated into existing iterative repair frameworks with relatively low implementation overhead.
  2. Empirical investigation of token efficiency across multiple settings. The paper evaluates its approach across multiple model families and benchmark settings, reporting consistent token reductions relative to GoT. The authors also provide component ablations, stopping-policy comparisons, and additional analyses of model scaling and quantization, which help characterize the behavior of the proposed controller.
Critical weaknesses
  1. Insufficient justification for the evaluation scope and limited generalizability. LeanRepair is largely model-agnostic, yet the main evaluation focuses on relatively small, general-purpose instruction-tuned models (3.8B–9B). Although Appendix C.8 includes Qwen3-14B, the evidence remains limited on whether the gains extend to stronger code-specialized models. The results also suggest that the relative benefit can depend on model and task characteristics; for example, LeanRepair does not outperform Self-Refine on Qwen3-14B/BigCodeBench. Moreover, most tasks are constructed through controlled mutations rather than naturally occurring repository level defects. These limitations weaken the evidence that the reported efficiency gains generalize broadly or translate directly to realistic local code-repair settings. Additionally, while the paper highlights its suitability for local deployment, it fails to report practical runtime metrics, including inference latency and computational overhead.
  2. Limited methodological novelty and insufficiently controlled evidence for incremental benefits. LeanRepair combines established ideas in feedback-guided repair, recurrence detection, and adaptive early stopping, with edit-geometry heuristics as its main methodological distinction. However, the experiments do not sufficiently isolate the benefits of these components over simpler alternatives. The main GoT comparison conflates the effects of guidance, recurrence handling, and early stopping, while the stopping-enhanced GoT comparisons in Appendix C.7 cover only six of 24 configurations and show mixed accuracy–efficiency trade-offs. The component ablations also use separate stochastic runs, and the guidance/stopping studies differ from the main evaluation in decoding and generation settings. Finally, Appendix C.3 reports substantial seed variation in GoT and unexpectedly large variation in the supposedly deterministic CoT and Self-Refine baselines. These issues make it difficult to establish the specific and reproducible contribution of LeanRepair's design.
Questions for authors
  1. Evaluation scope and practical benefits. Since LeanRepair is model-agnostic, what motivates the focus on relatively small, general-purpose instruction-tuned models in the main evaluation? Given the mixed results on Qwen3-14B, could the authors provide stronger evidence that the gains extend to stronger code-specialized models? Also, how do the reported token savings translate into practical deployment benefits such as inference latency or computational cost?
  2. Baseline fairness and reproducibility. Appendix C.7 compares LeanRepair with stopping-enhanced GoT, but only on six of the 24 main configurations, with mixed accuracy–efficiency results. Could the authors provide broader matched comparisons to isolate the benefit of LeanRepair's additional mechanisms? In addition, Appendix C.3 shows large seed-dependent changes in GoT token usage, while CoT and Self-Refine also change substantially despite being described as greedy baselines. Could the authors explain these discrepancies and assess their impact on the reported efficiency gains?
  3. Independent component contributions. What is the methodological advance of LeanRepair beyond combining feedback guidance, recurrence detection, and early stopping? In particular, can the authors provide matched ablations under the main experimental settings to establish the independent value of edit-geometry guidance and the recurrence buffer over simpler heuristics and stopping policies? This is especially important given that fixed generic guidance achieves lower token consumption than state-dependent guidance in Appendix C.5.
5 comments

Thank you for your detailed and constructive review. We also appreciate that you recognized LeanRepair's lightweight, model-agnostic design, which can be added to existing iterative repair frameworks with little effort. And we appreciate that you recognized the consistent token reductions relative to GoT, supported by the component ablations, stopping-policy comparisons, and scale and quantization analyses. Below we quote each of your comments and respond to it. Changes in the revised PDF are marked in blue.


Weakness 1.

Insufficient justification for the evaluation scope and limited generalizability. LeanRepair is largely model-agnostic, yet the main evaluation focuses on relatively small, general-purpose instruction-tuned models (3.8B–9B). Although Appendix C.8 includes Qwen3-14B, the evidence remains limited on whether the gains extend to stronger code-specialized models. The results also suggest that the relative benefit can depend on model and task characteristics; for example, LeanRepair does not outperform Self-Refine on Qwen3-14B/BigCodeBench. Moreover, most tasks are constructed through controlled mutations rather than naturally occurring repository level defects. These limitations weaken the evidence that the reported efficiency gains generalize broadly or translate directly to realistic local code-repair settings. Additionally, while the paper highlights its suitability for local deployment, it fails to report practical runtime metrics, including inference latency and computational overhead.

Response. Thank you for your comment. LeanRepair's gains extend beyond the main setting. At Qwen3-14B it has the highest average pass rate of all five methods, with 22% fewer tokens than GoT. On real, human-written defects it also keeps a clear lead.

  • Model range. The paper targets local deployment, where token budgets are tightest. Models of 4–9B fit in 16-bit weights on a single 24 GB consumer GPU, with room left for the KV cache of b=3b{=}3 parallel branches. Within this range we covered four model families, and tokens are reduced in 23 of 24 cells.
  • Stronger models (Table 10). At 14B, LeanRepair has the best or tied-best pass rate on three of the four benchmarks: QuixBugs 92.0%, HumanEval 95.0%, and MBPP 77.5%. On BigCodeBench it is within one task of GoT (73.3% vs. 76.7% on 30 tasks) at 31% fewer tokens. Self-Refine scores higher on BigCodeBench alone, but averaged over the four benchmarks LeanRepair's pass rate is the highest (84.5% vs. 84.2%). At 1.7B, 4B, and 14B, LeanRepair has the highest average pass rate and uses 22–32% fewer tokens than GoT.
  • Code-specialized models. The controller reads only program strings and execution feedback, so it applies to code-specialized backbones unchanged. We list this evaluation as future work.
  • Real defects. QuixBugs contains real, human-written defects. Under matched runs, LeanRepair uses 26.2% fewer tokens than GoT there, with a higher pass rate (89.0% vs. 86.5%). Repository-level repair is named as a limitation (Sec. 6, App. C.11). The controller's inputs, successive program versions and execution feedback, are also available in repository-level agents.
  • Runtime. The controller makes no model call. Per attempt, it computes one edit distance, one similarity ratio, at most K=7K{=}7 Jaccard comparisons, and a sign count over 4 numbers, all on the CPU. Early stopping removes whole generation and aggregation calls, which are the most expensive part of an episode. The token savings therefore mean less GPU time per repair, and we report measured end-to-end time per task in App. C.10.

Weakness 2.

Limited methodological novelty and insufficiently controlled evidence for incremental benefits. LeanRepair combines established ideas in feedback-guided repair, recurrence detection, and adaptive early stopping, with edit-geometry heuristics as its main methodological distinction. However, the experiments do not sufficiently isolate the benefits of these components over simpler alternatives. The main GoT comparison conflates the effects of guidance, recurrence handling, and early stopping, while the stopping-enhanced GoT comparisons in Appendix C.7 cover only six of 24 configurations and show mixed accuracy–efficiency trade-offs. The component ablations also use separate stochastic runs, and the guidance/stopping studies differ from the main evaluation in decoding and generation settings. Finally, Appendix C.3 reports substantial seed variation in GoT and unexpectedly large variation in the supposedly deterministic CoT and Self-Refine baselines. These issues make it difficult to establish the specific and reproducible contribution of LeanRepair's design.

Response. Thank you for your comment. Following your suggestion, we added a matched comparison in which every configuration runs under one harness (expanded App. C.7). Full LeanRepair has the lowest token cost of all eight configurations, and improves on GoT by +4.1 pp at 31% fewer tokens.

  • Setup. The comparison covers Llama-3-8B and Qwen3-8B on HumanEval, MBPP, and QuixBugs (6 cells, 210 episodes). Every row uses the same tasks, seeds, and generation settings. The harness follows Table 9 (Panel B).
  • Results. Pooled over the six cells:
Pass ↑Tok/task ↓
GoT79.73,783
GoT + fixed cap (J=5J{=}5)77.82,778
GoT + oscillation stopping80.12,905
GoT + fixed generic instruction81.03,581
GoT + random template instruction83.83,455
GoT + state-dependent guidance84.43,370
+ recurrence buffer83.63,771
LeanRepair (full)83.82,604
  • Gains.
    • Over GoT: +4.1 pp with 31% fewer tokens.
    • Over stopping-enhanced GoT: +3.7 pp with 10% fewer tokens, winning 4 of 6 cells and tying 1.
    • On Gemma-2-9B and Phi-3.5 (QuixBugs, same harness): 92.0% vs. 88.0% for GoT, with 40% and 28% fewer tokens.
  • Complementarity. Stopping alone reduces tokens without improving accuracy, and guidance alone does the reverse. Only their combination does both. This separates each mechanism's contribution, which the main GoT comparison could not.

The novelty lies in three things: control at zero model cost, the oscillation asymmetry (Table 8), and the complementarity shown above (details in Question 3). Every row of the table shares one harness, so differences between rows reflect the controller change rather than a change in protocol. The seed variation in App. C.3 is explained in Question 2.

Question 1.

Evaluation scope and practical benefits. Since LeanRepair is model-agnostic, what motivates the focus on relatively small, general-purpose instruction-tuned models in the main evaluation? Given the mixed results on Qwen3-14B, could the authors provide stronger evidence that the gains extend to stronger code-specialized models? Also, how do the reported token savings translate into practical deployment benefits such as inference latency or computational cost?

Response. Thank you for your question. Weakness 1 gives the details; in brief:

  • The 4–9B focus reflects local deployment on a single GPU.
  • At 14B, LeanRepair has the highest average pass rate, uses 22% fewer tokens than GoT, and is best or tied-best on three of four benchmarks.
  • Because the controller adds no model call, its token savings mean less GPU time per repair. We report this as measured end-to-end time in App. C.10.

Question 2.

Baseline fairness and reproducibility. Appendix C.7 compares LeanRepair with stopping-enhanced GoT, but only on six of the 24 main configurations, with mixed accuracy–efficiency results. Could the authors provide broader matched comparisons to isolate the benefit of LeanRepair's additional mechanisms? In addition, Appendix C.3 shows large seed-dependent changes in GoT token usage, while CoT and Self-Refine also change substantially despite being described as greedy baselines. Could the authors explain these discrepancies and assess their impact on the reported efficiency gains?

Response. Thank you for your question. The broader matched comparison appears under Weakness 2. On seed variation:

  • Determinism. The greedy baselines are deterministic: the seed-2028 and seed-2029 rows of Self-Refine and CoT are identical on every backbone.
  • Matched results. Under these reruns, LeanRepair uses 26.2% fewer tokens than GoT, reduces tokens in 7 of 8 cells, and reaches a higher pass rate (89.0% vs. 86.5%).
  • Source of the variation. The variation comes from seed 2027, copied from Table 1. There, the four Qwen3/QuixBugs baselines had run with Qwen3's thinking mode on, while LeanRepair had it off.

We will replace those cells with matched runs and withdraw the QuixBugs-specific figures (82.4% and 95.7%). The 24-cell median reduction is unchanged at 23.9%, and the five non-QuixBugs benchmarks are unaffected.


Question 3.

Independent component contributions. What is the methodological advance of LeanRepair beyond combining feedback guidance, recurrence detection, and early stopping? In particular, can the authors provide matched ablations under the main experimental settings to establish the independent value of edit-geometry guidance and the recurrence buffer over simpler heuristics and stopping policies? This is especially important given that fixed generic guidance achieves lower token consumption than state-dependent guidance in Appendix C.5.

Response. Thank you for your question.

  • Methodological advance. First, stopping, steering, and repeat detection are computed from program diffs and execution feedback alone. This requires no model calls, no training, and no calibration data, and keeps the prompt at Θ(J)\Theta(J). Second, the oscillation asymmetry: stagnation-only stopping saves 0% of tokens, while oscillation-only stopping recovers the full saving at the same pass rate (Table 8). Third, the combined controller is the only configuration that improves both accuracy and cost.
  • Independent value (Weakness 2 table). State-dependent guidance is the most accurate guidance policy: 84.4%, vs. 81.0% for a fixed generic instruction and 79.7% for none. Oscillation stopping beats a fixed cap by +2.3 pp at similar cost. The recurrence buffer matters most when guidance alone misleads the model. On Gemma it raises pass rate from 84% to 92% (Table 6 and the matched runs); on the other backbones its effect is small and mixed.
  • App. C.5. In the larger matched comparison, state-dependent guidance is both more accurate (+3.4 pp) and cheaper (3,370 vs. 3,581 tokens/task) than fixed guidance. The single-seed 5.4% gap in App. C.5 is therefore not consistent across settings. In both studies, state-dependent guidance gives the highest pass rate, which is why LeanRepair uses it by default.

Thank you again for your review. We hope the matched comparisons and clarifications address your concerns, and we would be glad to answer any further questions.

Reviewer_wAypOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes LeanRepair, a training-free external controller for token-efficient code repair with local language models. It uses edit trajectories and recent failure history to guide revisions and terminate unproductive search. Experiments across code repair benchmarks show reduced token consumption compared to structured-search baselines while maintaining comparable repair success rates.

Critical strengths

The method reduces token consumption without additional model calls or training. Its lightweight design is practical for resource-constrained local code repair. The evaluation covers multiple benchmarks and model families, with generally consistent token savings and comparable repair success rates. The ablation studies also provide evidence for the contribution of the proposed control mechanisms.

Critical weaknesses

The proposed controller combines heuristic rules for edit guidance, recurrence detection, and early stopping. While this is a practical approach, the individual components are relatively straightforward, and the broader methodological contribution remains unclear. The theoretical analysis establishes the benefit of bounded context over full-history replay, but this does not directly explain the improvements over GoT, which already uses bounded prompts.

The experiments demonstrate token savings, but the evidence supporting the proposed stopping mechanism is less conclusive. The analysis of whether early stopping discards potentially successful trajectories is based on a small number of cases. Some substantial differences in token consumption across repeated runs of greedy baselines also require clarification. The manuscript contains unresolved citations and inconsistent method names and experimental descriptions, which raise additional concerns about its completeness.

Questions for authors

Could the authors clarify the discrepancies in token consumption across repeated runs of greedy baselines and provide further evidence that the stopping criterion reliably identifies unproductive trajectories? Please also address the concerns raised in the weaknesses above.

3 comments

Thank you for your careful review, and for recognizing that LeanRepair reduces token consumption without additional model calls or training, that its lightweight design is practical for resource-constrained local code repair, and that the evaluation shows consistent token savings across multiple benchmarks and model families. Below we quote each of your comments and respond to it. Changes in the revised PDF are marked in blue.


Weakness 1.

The proposed controller combines heuristic rules for edit guidance, recurrence detection, and early stopping. While this is a practical approach, the individual components are relatively straightforward, and the broader methodological contribution remains unclear.

Response. Thank you for your comment. The components are simple by design, and their combination is what delivers the gains. LeanRepair reduces tokens in 23 of 24 cells (median −23.9% vs. GoT) without any extra model call. The contribution has three parts:

  • Control at zero model cost. Every control decision is computed from program diffs and execution feedback alone: whether to continue, which instruction to give next, and whether a candidate repeats an earlier failure. Control adds no model tokens, no training, and no calibration data, and the prompt stays Θ(J)\Theta(J). Model-based reflection, learned value functions, and calibrated stopping each pay one of these costs.
  • The oscillation asymmetry. Stagnation-only stopping saves 0% of tokens, while oscillation-only stopping recovers the full 14.6% saving with no loss in pass rate (Tables 2 and 8). Failing repair loops do not stall on near-identical edits; they alternate between conflicting patches. This tells practitioners which single cheap detector to add to an existing search loop.
  • Complementary components. In a new matched comparison under one harness (App. C.7), stopping alone makes GoT 23% cheaper, and guidance alone raises its pass rate by 4.7 pp. Only the full controller delivers both: +4.1 pp at 31% fewer tokens.

Weakness 2.

The theoretical analysis establishes the benefit of bounded context over full-history replay, but this does not directly explain the improvements over GoT, which already uses bounded prompts.

Response. Thank you for your comment. We agree, and Sec. 3.1 now says so explicitly after Proposition 1. Proposition 1 explains LeanRepair's advantage over full-history methods such as multi-turn Self-Refine and Reflexion, whose prompt cost grows as Θ(J2)\Theta(J^2). The gain over GoT comes from two other mechanisms:

  • Output-side pruning. Stopping at attempt j∗j^* removes up to b(J−j∗)b(J-j^*) generation calls and their aggregation calls.
  • Guided generation. Guidance steers which candidates are generated, which raises pass rate.

Bounded state limits input tokens and early stopping limits output tokens. This is why LeanRepair saves tokens against both families of baselines.

Weakness 3.

The experiments demonstrate token savings, but the evidence supporting the proposed stopping mechanism is less conclusive. The analysis of whether early stopping discards potentially successful trajectories is based on a small number of cases.

Response. Thank you for your comment. Beyond the counterfactual audit, two matched studies support the stopping rule. In both, it preserves pooled pass rate while saving 15–23% of tokens.

  • Counterfactual audit (App. C.7). None of the 7 oscillation-triggered episodes succeeded when allowed to continue to J=8J{=}8, and the only late solver was correctly not stopped.
  • Within LeanRepair (Table 8; 4 backbones × 25 tasks). Oscillation stopping matches the no-stopping pass rate on every backbone (92.0% pooled) and cuts tokens by 14.6%. A fixed cap at the same attempt loses a solvable task.
  • Added to plain GoT (App. C.7; 6 cells, 210 episodes). Oscillation stopping cuts GoT tokens by 23% with no loss in pooled pass rate (79.7% → 80.1%). A fixed cap loses up to 17.5 pp.

We agree that the counterfactual audit itself is small (9 unresolved episodes), and App. C.7 now states its size.


Weakness 4.

Some substantial differences in token consumption across repeated runs of greedy baselines also require clarification.

Response. Thank you for your comment. The greedy baselines are deterministic, and under matched runs LeanRepair keeps a clear lead over GoT.

  • Determinism. In Table 5, the seed-2028 and seed-2029 rows of Self-Refine and CoT are identical on every backbone.
  • Matched results (seeds 2028/2029, 8 QuixBugs cells). LeanRepair uses 26.2% fewer tokens than GoT, reduces tokens in 7 of 8 cells, and reaches a higher pass rate (89.0% vs. 86.5%).
  • Source of the variation. All of it comes from seed 2027, which is copied from Table 1. There, the four Qwen3/QuixBugs baselines had been run with Qwen3's thinking mode on, while LeanRepair had it off.

We will replace those cells with matched runs, correct the explanation in App. C.3, and withdraw the QuixBugs-specific figures (82.4% and 95.7%). The headline result is unchanged: the median token reduction over all 24 cells remains 23.9%, and the five non-QuixBugs benchmarks are unaffected.


Weakness 5.

The manuscript contains unresolved citations and inconsistent method names and experimental descriptions, which raise additional concerns about its completeness.

Response. Thank you for your comment. The "??" came from a missing BibTeX pass, and the recompiled PDF has no unresolved references. The method is now called LeanRepair throughout the text, algorithm, and tables. We are also reconciling the remaining experimental descriptions; for example, Sec. 4 will match the 1.7B–14B range of Table 10.


Question.

Could the authors clarify the discrepancies in token consumption across repeated runs of greedy baselines and provide further evidence that the stopping criterion reliably identifies unproductive trajectories? Please also address the concerns raised in the weaknesses above.

Response. Thank you for your question. Weakness 4 above explains the greedy-baseline discrepancy, and Weakness 3 gives the additional stopping evidence. Across two matched studies (310 episodes), oscillation stopping preserved pooled pass rate while saving 15–23% of tokens.


Thank you again for your review. We hope the matched comparisons and clarifications address your concerns, and we would be glad to answer any further questions.

Thank you for the responses and additional experiments. The new results address some of my concerns regarding experimental validity, and I will raise my soundness score accordingly. However, my concerns regarding the methodological contribution and the broader applicability of the proposed approach remain. I will therefore maintain my original overall rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.