- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MARS, a training-free controller for repository-level program repair. MARS leaves the host pipeline's localization and patch generation unchanged and uses execution results, LLM verification, reviewer scores, and resource consumption to decide whether to verify, refine, expand, or stop. MARS includes Incumbent-Challenger Control (IC) to check whether a new patch should replace the current one, and Candidate Commitment (CC) to keep accumulated patches available for final selection. On SWE-bench Lite and Verified, MARS improves pass@1 over AdaRepair-Mem by 2.0 and 2.8 percentage points, respectively, while reducing average cost by 31% and 22%. This paper also reports transfer to three backbones and three other host pipelines.
Critical strengths
- The paper addresses a practical problem. A controller that can be added to an existing repair pipeline without retraining or changing the patch generator is useful.
- The paper includes several useful analyses beyond the main results, such as tracking where correct patches are lost during search and comparing with simple fixed-budget variants.
- The evaluation covers two benchmarks, three backbones, and three additional hosts, and reports cost together with accuracy.
Critical weaknesses
- It is not clear what MARS adds over existing adaptive search methods. DARS and SWE-Search already use execution feedback to decide which repair paths to explore on SWE-bench, and because they search over a tree, earlier states are kept rather than overwritten. Section 2.2 acknowledges this, which weakens the main motivation for IC. However, DARS only appears in Table 1 with a Lite number taken from prior work, and SWE-Search is not evaluated at all. Most other rows in Table 1 also appear to be taken from prior reports with different models. Table 7 shows that MARS does better than fixed search schedules, which is useful, but it does not show that MARS does better than existing adaptive search. A comparison with DARS or SWE-Search using the same backbone and a similar budget would help show what MARS adds beyond early stopping and keeping earlier patches.
- Key parts of the controller are not described clearly enough to reproduce the method. The promotion rule γ decides whether a new patch replaces the current best one, which is the core of IC. However, Section 3.3 and Appendix A.3 only describe its purpose and list some cases in which replacement does not happen; they do not give a complete rule for when replacement happens. Execution evidence is defined in Appendix A.1, but how it affects the action rules in Appendix A.2 is unclear, despite the improvement reported in Table 10. The released code does not fill this gap. The ExpeRepair adapter mainly contains two verification prompts rather than the integration needed to reproduce the experiments.
- The reported gains are small and may fall within the random variation across runs. Compared with AdaRepair-Mem, MARS solves only 6 more issues on Lite and 14 more on Verified. With the other two backbones, the gains are 1.4 and 1.0 points. Section 4.1 says that each run is executed independently and that localization is not seeded, so results can change from run to run. However, no repeated-run results or variance estimates are reported.
Table 8 further shows that removing CC changes the number of issues with at least one correct patch from 342 to 329. Since CC is applied only after search stops, this difference cannot be explained by commitment itself. This makes the independent-run comparison difficult to interpret. The authors should reuse the same localization results across methods and report results over several runs.
- The contribution of CC is smaller than the main ablation suggests. In Table 2, the w/o CC variant forwards only one patch to the selector, which changes both the set of available candidates and the amount of downstream validation. The authors acknowledge this and provide a matched control in Table 11, where the gap drops from 4.6 to 1.2 points. In addition, the matched control is run independently, so its search may produce different patches from MARS. Since CC only acts after search stops, a cleaner test would be to apply both commitment rules to the same search trajectories. This would show whether the remaining 1.2-point gap comes from CC rather than from differences between runs.
- The cross-pipeline results need clearer experimental details. The Default rows in Table 3 are identical to the corresponding rows in Table 1. It is therefore unclear whether these baselines were rerun with MARS in the same environment. Appendix B.1 also states that the additional hosts do not provide the reproduction artifacts needed for ESR and RSR. The paper does not explain which execution, reviewer, and verifier signals each host provides, how missing signals are handled, or whether the extra verification calls are included in the reported costs. Without these details, it is difficult to tell whether the same controller transfers across hosts and whether the comparisons isolate its effect.
Questions for authors
- When exactly does a new patch replace the current incumbent? Please give the complete rule for γ, and explain how execution evidence affects the decisions to verify, refine, or stop.
- Can the authors repeat the main experiments several times, using the same retrieval and localization outputs for all methods? For CC, can both commitment rules be applied to the same search trajectories?
- How does MARS compare with an existing adaptive search method, such as DARS or SWE-Search, when both use the same backbone and a similar inference budget?
- Were the Default rows in Table 3 rerun in the same environment as MARS? For each host, which evidence signals does MARS use, which extra model calls does it make, and are these calls included in the reported cost?
- Which review and verification steps are inherited from ExpeRepair, and which are added by MARS? Are the host's memory modules enabled, and is their configuration identical across ExpeRepair, MARS, and the ablation runs? If the memory configuration differs, can the authors provide a comparison with memory held fixed?
7 comments
Table R3-2. Signal ladder and candidate lifecycle on the fixed Verified 200-issue subset; IC and CC enabled.
| AS policy | Pass@1 (%) | Cost ($) | Ever-correct | Final-pool correct | Resolved | Pool loss | Selector loss |
|---|---|---|---|---|---|---|---|
| AS disabled | 61.5 | 1.53 | 129 | 127 | 123 | 2 | 4 |
| Budget only | 60.5 | 1.23 | 128 | 125 | 121 | 3 | 4 |
| Verifier + budget | 63.0 | 1.34 | 133 | 130 | 126 | 3 | 4 |
| Review + verifier + budget, execution treated as fixed input | 64.5 | 1.37 | 136 | 132 | 129 | 4 | 3 |
| Full MARS | 67.0 | 1.31 | 140 | 138 | 134 | 2 | 4 |
The full policy resolves five more issues (+2.5 pp) at 4.4% lower cost than the review/verifier configuration with fixed execution. Relative to disabled AS, it gains +5.5 pp while reducing cost from $1.53 to $1.31. These are configuration-level comparisons with IC/CC enabled, rather than isolated effects of a single execution branch. The fixed-execution intervention freezes AS's execution view at its initial observation while retaining IC's actual evidence interface.
The ladder also explains why a budget stop alone is insufficient. Budget-only control reduces expenditure but also loses resolutions relative to disabled AS, whereas the full policy improves both outcomes. Relative to fixed execution, the full policy increases Ever-correct and final-pool correctness while lowering cost. This pattern supports using evolving evidence to allocate search and preserve useful candidates, beyond terminating computation earlier.
Rule and integration checks. The rules pass 41/41 fixed cases (five replays each), 5,184/5,184 promotion inputs, and 15/15 state/cost checks. Together with the explicit promotion predicate, action precedence, and patch-bound verification, these checks make the state transitions reproducible beyond the two verification prompts. The revised Appendices A.2 and A.3 document the complete integration rules and check results.
Q2 (W3, W4). Paired repeats and CC comparisons on the same search trajectories
Can the authors repeat the main experiments several times, using the same retrieval and localization outputs for all methods? For CC, can both commitment rules be applied to the same search trajectories?
We report an additional three-run paired evaluation and a same-trajectory CC comparison under a fixed output allowance. The paired evaluation is separate from the independently executed runs in the original Table 1. On SWE-bench Lite and Verified, methods share actual retrieval/localization outputs within each issue and repeat. MARS has higher accuracy and lower cost in every run; Table R3-3 reports means ± sample SD. The CC experiment separately evaluates the k=3 commitment variant below.
Table R3-3. Additional three-run paired evaluation with fixed upstream outputs, mean ± sample SD.
| Method | Lite Pass@1 (%) | Lite cost ($) | Verified Pass@1 (%) | Verified cost ($) |
|---|---|---|---|---|
| ExpeRepair Default | 48.33 ± 0.67 | 1.893 ± 0.035 | 57.27 ± 0.70 | 1.740 ± 0.030 |
| AdaRepair-Mem | 51.11 ± 0.51 | 1.673 ± 0.025 | 63.27 ± 0.31 | 1.523 ± 0.025 |
| ExpeRepair + MARS | 53.00 ± 0.67 | 1.153 ± 0.025 | 65.93 ± 0.50 | 1.180 ± 0.020 |
MARS resolves 6, 5, and 6 more Lite issues and 14, 12, and 14 more Verified issues than AdaRepair-Mem. Its mean advantages are +1.89/+2.67 pp with 31.08%/22.54% lower cost. Relative to ExpeRepair, the mean gains are +4.67/+8.67 pp. Thus the observed advantage persists when upstream outputs are shared.
Table R3-3a. Mean paired differences and marginal 95% bootstrap confidence intervals (CIs). Differences are MARS minus the reference.
| Dataset | Reference | Accuracy difference (pp) | 95% bootstrap CI (pp) | Cost difference ($/issue) | 95% bootstrap CI ($/issue) |
|---|---|---|---|---|---|
| Lite | ExpeRepair | +4.67 | [−0.78, 10.11] | −0.740 | [−0.880, −0.604] |
| Lite | AdaRepair-Mem | +1.89 | [−3.22, 7.11] | −0.520 | [−0.638, −0.408] |
| Verified | ExpeRepair | +8.67 | [4.53, 12.93] | −0.560 | [−0.690, −0.452] |
| Verified | AdaRepair-Mem | +2.67 | [−1.20, 6.60] | −0.343 | [−0.440, −0.256] |
All four cost-difference CIs lie below zero, and the Verified accuracy CI against ExpeRepair is positive. The accuracy CIs against AdaRepair-Mem include zero. Across all three paired repeats, MARS achieves higher observed accuracy while reducing mean cost by 31.08%/22.54% on Lite/Verified, supporting a practical improvement in the accuracy–cost operating point.
Intervals use a paired two-way bootstrap with 20,000 resamples: draw issues uniformly with replacement to preserve each dataset's size, independently draw three run identifiers with replacement, and use the same draws for every method. Mean success-rate and cost differences use the Cartesian product of the sampled issues and runs; the 2.5th and 97.5th percentiles give the endpoints. Relative savings equal 1 − mean(MARS cost)/mean(reference cost), using unrounded means. These are marginal percentile intervals from paired resampling of issues and runs; SD summarizes run-to-run variation.
We thank the reviewer for recognizing the practical value of controlling an existing repair pipeline and for identifying specific gaps in reproducibility and attribution. We organize the response by the five questions, incorporating the corresponding weaknesses. Response tables are numbered R3-x. Accuracy changes are percentage points (pp), and costs are dollars per issue.
Q1 (W2). The complete promotion rule, execution-to-action path, and reproducible implementation
We provide a complete positive promotion rule and an ordered execution-to-action policy. The controller specification defines the following conditions explicitly.
The promotion predicate. Let execution be pass, fail, or unknown; review be 0, 1, 2, or missing; and verification be unqueried, pass, or refine. The first candidate initializes the incumbent. Every challenger is registered before promotion is considered. In the completed rule, γ equals 1 if and only if all the following conditions hold.
Otherwise γ = 0. A passing, review-2 challenger can replace a failing incumbent, while a tie between passing, review-2 candidates retains the incumbent. Unknown/missing evidence, duplicates, and post-verification candidates also retain it. Registration is separate from promotion: later candidates remain available for final selection, and exact duplicates retain their first evidence record. Once verification begins, its request and verdict bind to the incumbent's patch hash and that identity is fixed. Strong verification is invoked at most once per issue; an invalid or unparsable verdict maps to refine.
Execution and action selection. Candidate-attributable nonzero execution is fail. Missing observations, timeouts, and infrastructure errors are unknown, including infrastructure-corrupted observations with an apparent zero return code. An execution pass is evidence from the available test, not an official correctness label.
Table R3-1. Ordered action rules in the completed specification.
The first matching rule applies. A review-2 candidate triggers verification of the current incumbent. The main configuration allows three initial candidates, two refinement rounds, and two refinements per round. The $2.20 budget is checked after review/multi-candidate review, before refinement following declined verification, after each refinement, and after each round. A budget-triggered stop freezes the accumulated candidates and issues no further model calls. The threshold is soft because costs can exceed it between consecutive ledger checks.
Execution reaches control through promotion, review inputs, and reproduction-dependent fallback. It can change the incumbent, the review-supported set, or expansion availability; an execution pass alone is not a verified-success stop. CC preserves exact-deduplicated candidates in generation order, with the first-eight cap only on capped write-out paths.