acceptodds
Under review as a conference paper at ICLR 2027

MARS: Multi-Evidence Adaptive Search with State-Preserving Control for Repository-Level Program Repair

Abstract

Repository-level program repair aims to resolve real software issues under a bounded compute budget, requiring decisions about how much computation to allocate, which repair state to preserve, and what to commit. Existing staged repair pipelines largely follow fixed search schedules that repeatedly generate, execute, and refine candidate patches while underusing runtime evidence. We refer to this mismatch as Budget–Evidence Decoupling: computation is spent without conditioning on whether current evidence justifies further search, and later refinement may overwrite previously discovered useful repair states. We propose MARS, a training-free controller that formulates candidate search as a budget-aware sequential decision process using execution, verification, review, and resource signals. Specifically, Multi-Evidence Adaptive Search (AS) deterministically selects among verification, refinement, expansion, and termination according to the current repair state. MARS further introduces Incumbent–Challenger Control (IC), separating candidate admission from incumbent replacement, together with Candidate Commitment (CC), which freezes the accumulated candidate state for downstream final selection. These designs jointly condition compute allocation, state preservation, and commitment on runtime evidence while keeping valuable repair states recoverable. Experiments on SWE-bench Lite and SWE-bench Verified show that MARS improves pass@1 over the strongest same-backbone baseline by 2.0 and 2.8 percentage points, respectively, while reducing per-issue cost by 31% and 22%. We further demonstrate its transferability across three backbones and three additional host pipelines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
repository-level program repair, LLM agents, inference-time compute allocation, search control, SWE-bench, software engineering agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UZRjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MARS, a training-free controller for repository-level program repair. MARS leaves the host pipeline's localization and patch generation unchanged and uses execution results, LLM verification, reviewer scores, and resource consumption to decide whether to verify, refine, expand, or stop. MARS includes Incumbent-Challenger Control (IC) to check whether a new patch should replace the current one, and Candidate Commitment (CC) to keep accumulated patches available for final selection. On SWE-bench Lite and Verified, MARS improves pass@1 over AdaRepair-Mem by 2.0 and 2.8 percentage points, respectively, while reducing average cost by 31% and 22%. This paper also reports transfer to three backbones and three other host pipelines.

Critical strengths
  1. The paper addresses a practical problem. A controller that can be added to an existing repair pipeline without retraining or changing the patch generator is useful.
  2. The paper includes several useful analyses beyond the main results, such as tracking where correct patches are lost during search and comparing with simple fixed-budget variants.
  3. The evaluation covers two benchmarks, three backbones, and three additional hosts, and reports cost together with accuracy.
Critical weaknesses
  1. It is not clear what MARS adds over existing adaptive search methods. DARS and SWE-Search already use execution feedback to decide which repair paths to explore on SWE-bench, and because they search over a tree, earlier states are kept rather than overwritten. Section 2.2 acknowledges this, which weakens the main motivation for IC. However, DARS only appears in Table 1 with a Lite number taken from prior work, and SWE-Search is not evaluated at all. Most other rows in Table 1 also appear to be taken from prior reports with different models. Table 7 shows that MARS does better than fixed search schedules, which is useful, but it does not show that MARS does better than existing adaptive search. A comparison with DARS or SWE-Search using the same backbone and a similar budget would help show what MARS adds beyond early stopping and keeping earlier patches.
  2. Key parts of the controller are not described clearly enough to reproduce the method. The promotion rule γ decides whether a new patch replaces the current best one, which is the core of IC. However, Section 3.3 and Appendix A.3 only describe its purpose and list some cases in which replacement does not happen; they do not give a complete rule for when replacement happens. Execution evidence is defined in Appendix A.1, but how it affects the action rules in Appendix A.2 is unclear, despite the improvement reported in Table 10. The released code does not fill this gap. The ExpeRepair adapter mainly contains two verification prompts rather than the integration needed to reproduce the experiments.
  3. The reported gains are small and may fall within the random variation across runs. Compared with AdaRepair-Mem, MARS solves only 6 more issues on Lite and 14 more on Verified. With the other two backbones, the gains are 1.4 and 1.0 points. Section 4.1 says that each run is executed independently and that localization is not seeded, so results can change from run to run. However, no repeated-run results or variance estimates are reported.

Table 8 further shows that removing CC changes the number of issues with at least one correct patch from 342 to 329. Since CC is applied only after search stops, this difference cannot be explained by commitment itself. This makes the independent-run comparison difficult to interpret. The authors should reuse the same localization results across methods and report results over several runs.

  1. The contribution of CC is smaller than the main ablation suggests. In Table 2, the w/o CC variant forwards only one patch to the selector, which changes both the set of available candidates and the amount of downstream validation. The authors acknowledge this and provide a matched control in Table 11, where the gap drops from 4.6 to 1.2 points. In addition, the matched control is run independently, so its search may produce different patches from MARS. Since CC only acts after search stops, a cleaner test would be to apply both commitment rules to the same search trajectories. This would show whether the remaining 1.2-point gap comes from CC rather than from differences between runs.
  2. The cross-pipeline results need clearer experimental details. The Default rows in Table 3 are identical to the corresponding rows in Table 1. It is therefore unclear whether these baselines were rerun with MARS in the same environment. Appendix B.1 also states that the additional hosts do not provide the reproduction artifacts needed for ESR and RSR. The paper does not explain which execution, reviewer, and verifier signals each host provides, how missing signals are handled, or whether the extra verification calls are included in the reported costs. Without these details, it is difficult to tell whether the same controller transfers across hosts and whether the comparisons isolate its effect.
Questions for authors
  1. When exactly does a new patch replace the current incumbent? Please give the complete rule for γ, and explain how execution evidence affects the decisions to verify, refine, or stop.
  2. Can the authors repeat the main experiments several times, using the same retrieval and localization outputs for all methods? For CC, can both commitment rules be applied to the same search trajectories?
  3. How does MARS compare with an existing adaptive search method, such as DARS or SWE-Search, when both use the same backbone and a similar inference budget?
  4. Were the Default rows in Table 3 rerun in the same environment as MARS? For each host, which evidence signals does MARS use, which extra model calls does it make, and are these calls included in the reported cost?
  5. Which review and verification steps are inherited from ExpeRepair, and which are added by MARS? Are the host's memory modules enabled, and is their configuration identical across ExpeRepair, MARS, and the ablation runs? If the memory configuration differs, can the authors provide a comparison with memory held fixed?
7 comments

We thank the reviewer for recognizing the practical value of controlling an existing repair pipeline and for identifying specific gaps in reproducibility and attribution. We organize the response by the five questions, incorporating the corresponding weaknesses. Response tables are numbered R3-x. Accuracy changes are percentage points (pp), and costs are dollars per issue.

Q1 (W2). The complete promotion rule, execution-to-action path, and reproducible implementation

When exactly does a new patch replace the current incumbent? Please give the complete rule for γ, and explain how execution evidence affects the decisions to verify, refine, or stop.

We provide a complete positive promotion rule and an ordered execution-to-action policy. The controller specification defines the following conditions explicitly.

The promotion predicate. Let execution be pass, fail, or unknown; review be 0, 1, 2, or missing; and verification be unqueried, pass, or refine. The first candidate initializes the incumbent. Every challenger is registered before promotion is considered. In the completed rule, γ equals 1 if and only if all the following conditions hold.

  1. The challenger is not an exact duplicate of a candidate already registered.
  2. The issue's strong-verification procedure has not begun.
  3. Both candidates have the normal unqueried verification state.
  4. The incumbent has known execution evidence and a review score in {0, 1, 2}.
  5. The challenger has execution pass and review score 2.
  6. The incumbent has execution fail or review score below 2.

Otherwise γ = 0. A passing, review-2 challenger can replace a failing incumbent, while a tie between passing, review-2 candidates retains the incumbent. Unknown/missing evidence, duplicates, and post-verification candidates also retain it. Registration is separate from promotion: later candidates remain available for final selection, and exact duplicates retain their first evidence record. Once verification begins, its request and verdict bind to the incumbent's patch hash and that identity is fixed. Strong verification is invoked at most once per issue; an invalid or unparsable verdict maps to refine.

Execution and action selection. Candidate-attributable nonzero execution is fail. Missing observations, timeouts, and infrastructure errors are unknown, including infrastructure-corrupted observations with an apparent zero return code. An execution pass is evidence from the available test, not an official correctness label.

Table R3-1. Ordered action rules in the completed specification.

PriorityGuardAction
1At a configured checkpoint, cumulative cost ≥ $2.20Stop for budget
2No candidate is availableStop with no candidates
3Incumbent verification verdict is passStop for verified success
4The review-supported set is non-empty, the incumbent is unqueried, and no verification has begunVerify the incumbent
5The supported set is empty or the incumbent verdict is refine, and refinement depth < 2Refine the incumbent
6No usable reproduction is available and the host has remaining fallback allowanceExpand
7No preceding rule appliesStop as exhausted

The first matching rule applies. A review-2 candidate triggers verification of the current incumbent. The main configuration allows three initial candidates, two refinement rounds, and two refinements per round. The $2.20 budget is checked after review/multi-candidate review, before refinement following declined verification, after each refinement, and after each round. A budget-triggered stop freezes the accumulated candidates and issues no further model calls. The threshold is soft because costs can exceed it between consecutive ledger checks.

Execution reaches control through promotion, review inputs, and reproduction-dependent fallback. It can change the incumbent, the review-supported set, or expansion availability; an execution pass alone is not a verified-success stop. CC preserves exact-deduplicated candidates in generation order, with the first-eight cap only on capped write-out paths.

Table R3-2. Signal ladder and candidate lifecycle on the fixed Verified 200-issue subset; IC and CC enabled.

AS policyPass@1 (%)Cost ($)Ever-correctFinal-pool correctResolvedPool lossSelector loss
AS disabled61.51.5312912712324
Budget only60.51.2312812512134
Verifier + budget63.01.3413313012634
Review + verifier + budget, execution treated as fixed input64.51.3713613212943
Full MARS67.01.3114013813424

The full policy resolves five more issues (+2.5 pp) at 4.4% lower cost than the review/verifier configuration with fixed execution. Relative to disabled AS, it gains +5.5 pp while reducing cost from $1.53 to $1.31. These are configuration-level comparisons with IC/CC enabled, rather than isolated effects of a single execution branch. The fixed-execution intervention freezes AS's execution view at its initial observation while retaining IC's actual evidence interface.

The ladder also explains why a budget stop alone is insufficient. Budget-only control reduces expenditure but also loses resolutions relative to disabled AS, whereas the full policy improves both outcomes. Relative to fixed execution, the full policy increases Ever-correct and final-pool correctness while lowering cost. This pattern supports using evolving evidence to allocate search and preserve useful candidates, beyond terminating computation earlier.

Rule and integration checks. The rules pass 41/41 fixed cases (five replays each), 5,184/5,184 promotion inputs, and 15/15 state/cost checks. Together with the explicit promotion predicate, action precedence, and patch-bound verification, these checks make the state transitions reproducible beyond the two verification prompts. The revised Appendices A.2 and A.3 document the complete integration rules and check results.

Q2 (W3, W4). Paired repeats and CC comparisons on the same search trajectories

Can the authors repeat the main experiments several times, using the same retrieval and localization outputs for all methods? For CC, can both commitment rules be applied to the same search trajectories?

We report an additional three-run paired evaluation and a same-trajectory CC comparison under a fixed output allowance. The paired evaluation is separate from the independently executed runs in the original Table 1. On SWE-bench Lite and Verified, methods share actual retrieval/localization outputs within each issue and repeat. MARS has higher accuracy and lower cost in every run; Table R3-3 reports means ± sample SD. The CC experiment separately evaluates the k=3 commitment variant below.

Table R3-3. Additional three-run paired evaluation with fixed upstream outputs, mean ± sample SD.

MethodLite Pass@1 (%)Lite cost ($)Verified Pass@1 (%)Verified cost ($)
ExpeRepair Default48.33 ± 0.671.893 ± 0.03557.27 ± 0.701.740 ± 0.030
AdaRepair-Mem51.11 ± 0.511.673 ± 0.02563.27 ± 0.311.523 ± 0.025
ExpeRepair + MARS53.00 ± 0.671.153 ± 0.02565.93 ± 0.501.180 ± 0.020

MARS resolves 6, 5, and 6 more Lite issues and 14, 12, and 14 more Verified issues than AdaRepair-Mem. Its mean advantages are +1.89/+2.67 pp with 31.08%/22.54% lower cost. Relative to ExpeRepair, the mean gains are +4.67/+8.67 pp. Thus the observed advantage persists when upstream outputs are shared.

Table R3-3a. Mean paired differences and marginal 95% bootstrap confidence intervals (CIs). Differences are MARS minus the reference.

DatasetReferenceAccuracy difference (pp)95% bootstrap CI (pp)Cost difference ($/issue)95% bootstrap CI ($/issue)
LiteExpeRepair+4.67[−0.78, 10.11]−0.740[−0.880, −0.604]
LiteAdaRepair-Mem+1.89[−3.22, 7.11]−0.520[−0.638, −0.408]
VerifiedExpeRepair+8.67[4.53, 12.93]−0.560[−0.690, −0.452]
VerifiedAdaRepair-Mem+2.67[−1.20, 6.60]−0.343[−0.440, −0.256]

All four cost-difference CIs lie below zero, and the Verified accuracy CI against ExpeRepair is positive. The accuracy CIs against AdaRepair-Mem include zero. Across all three paired repeats, MARS achieves higher observed accuracy while reducing mean cost by 31.08%/22.54% on Lite/Verified, supporting a practical improvement in the accuracy–cost operating point.

Intervals use a paired two-way bootstrap with 20,000 resamples: draw issues uniformly with replacement to preserve each dataset's size, independently draw three run identifiers with replacement, and use the same draws for every method. Mean success-rate and cost differences use the Cartesian product of the sampled issues and runs; the 2.5th and 97.5th percentiles give the endpoints. Relative savings equal 1 − mean(MARS cost)/mean(reference cost), using unrounded means. These are marginal percentile intervals from paired resampling of issues and runs; SD summarizes run-to-run variation.

Reviewer_t76aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Repository-level program repair has been widely studied in recent years, and many coding agents have been proposed to improve repair performance. The paper claims that existing repair techniques largely follow fixed search schedules that repeatedly generate and refine candidate patches without fully leveraging runtime evidence. To solve this problem, this paper proposes MARS, a training-free controller that formulates patch search as a budget-aware sequential decision process using execution, verification, review, and resource signals. Basically, MARS deterministically selects next steps based on the current repair state, and then further separates candidate submissions from incumbent replacement and also freezes accumulated candidate state for final selection. The experimental results on SWE-bench Lite and SWE-bench Verified show that MARS outperforms the strongest baseline by 2 and 2.8 percentage points, respectively, with 20-30% cost reduction.

Critical strengths

The paper targets the important problem of repository-level program repair, which has been extensively studied in the literature and also widely adopted in practice.

The proposed solution does not require any training or fine-tuning and can be easily applied to most off-the-shelf program repair agents. Therefore, MARS has the potential to benefit developers worldwide.

Critical weaknesses

Overall the paper presentation is a bit hard to follow for human readers. For example, the paper introduces lots of notations and terms that are not quite intuitive. Also, the approach part would benefit from some concrete code examples.

The improvement over the baseline is a bit marginal. For example, MARS only outperforms the backbone agent AdaRepair-Mem by less than 3 percentage points on both SWE-bench Lite and Verified. Such marginal differences may also not be statistically significant.

The models and agents studied are a bit outdated. For example, the best models studied are GPT-4o and Claude 3 Opus, while now we already have GPT-6 and Opus 5.5, which are way stronger. Also, the modern mainstream coding agents, including Claude Code, Codex CLi, and OpenCode, are now widely adopted for program repair. It would be important to study such more widely used agents instead of some old domain-specific agents. Overall, it is not quite clear whether the current findings can generalize to more recent models and agents.

The benchmarks adopted are also a bit outdated. SWE-bench Lite and Verified have been largely saturated by more recent models and agents. In recent months, people have proposed a number of more challenging and newer benchmarks, including SWE-bench Pro and DeepSWE. It would be interesting to study how the proposed approach help improve the scores on those newer benchmarks.

Questions for authors

How would the current findings generalize to newer models, agents, and benchmarks?

4 comments

We thank the reviewer for recognizing the importance of repository-level repair and the practical value of a training-free controller. We address presentation, the size and uncertainty of the gains, and generalization separately. Response tables are numbered R2-x; accuracy changes are in percentage points (pp).

W1. Making the method easier to follow with a concrete code example

Overall the paper presentation is a bit hard to follow for human readers. For example, the paper introduces lots of notations and terms that are not quite intuitive. Also, the approach part would benefit from some concrete code examples.

The controller can be understood through three concrete questions: AS decides what to do next; IC decides which patch to make active; CC decides what to retain for final selection. We illustrate these decisions with a short code example before introducing the formal notation.

A small executable example makes these distinctions visible. Suppose None selects a default timeout, zero is valid, and negative values must raise an error. The following change fixes the zero-timeout bug while preserving negative-input validation.

 def resolve_timeout(timeout, default=30):
-    if not timeout:
+    if timeout is None:
         return default
     if timeout < 0:
         raise ValueError("timeout must be nonnegative")
     return timeout

Consider three generated candidates. A handles None and zero but drops the negative-input check; B implements the diff above; C returns max(timeout, 0). All three pass a reproduction that checks only resolve_timeout(0) == 0. Their broader behavior differs, showing why passing the generated reproduction is insufficient to establish correctness.

Table R2-1. Illustrative controller trace with fixed evidence inputs.

StepEvidence or operationController outcome
1Register A, execution pass, review 1Initialize incumbent A
2Register B, execution pass, review 2Promote B over A
3Register C, execution pass, review 2Keep B on equal support; retain C in the pool
4Supported candidates exist; B has not been verifiedVerify B
5B receives a verification passStop search
6Freeze accumulated candidatesForward [A, B, C] in generation order

This trace separates registration, promotion, and final selection: stronger evidence promotes B, equal support retains B, and CC preserves all three candidates. The evidence values are fixed teaching inputs. We have added this example to the method overview before the formal definitions in Section 3.1, with detailed boundary rules in Appendix A.

W2. The magnitude and statistical support of the improvement

The improvement over the baseline is a bit marginal. For example, MARS only outperforms the backbone agent AdaRepair-Mem by less than 3 percentage points on both SWE-bench Lite and Verified. Such marginal differences may also not be statistically significant.

The practical benefit combines higher observed accuracy with substantially lower cost. In an additional three-run paired evaluation, MARS improves on AdaRepair-Mem by +1.89/+2.67 pp on Lite/Verified while reducing mean cost by 31.08%/22.54%. This evaluation is separate from the independently executed runs in the original Table 1. MARS uses ExpeRepair as its host; AdaRepair-Mem is the separate memory-augmented baseline with the same main backbone pair.

Table R2-2. Additional three-run paired evaluation with fixed upstream outputs, mean ± sample SD.

MethodLite Pass@1 (%)Lite cost ($)Verified Pass@1 (%)Verified cost ($)
ExpeRepair Default48.33 ± 0.671.893 ± 0.03557.27 ± 0.701.740 ± 0.030
AdaRepair-Mem51.11 ± 0.511.673 ± 0.02563.27 ± 0.311.523 ± 0.025
ExpeRepair + MARS53.00 ± 0.671.153 ± 0.02565.93 ± 0.501.180 ± 0.020

Each run evaluates SWE-bench Lite and Verified, sharing actual retrieval and localization outputs across methods within each issue/repeat. MARS resolves 6, 5, and 6 more Lite issues and 14, 12, and 14 more Verified issues than AdaRepair-Mem: the advantage is positive in every run. Against ExpeRepair, the mean gains are +4.67/+8.67 pp with 39.08%/32.18% lower cost.

Table R2-2a. Mean paired differences and marginal 95% bootstrap confidence intervals (CIs). Differences are MARS minus the reference.

DatasetReferenceAccuracy difference (pp)95% bootstrap CI (pp)Cost difference ($/issue)95% bootstrap CI ($/issue)
LiteExpeRepair+4.67[−0.78, 10.11]−0.740[−0.880, −0.604]
LiteAdaRepair-Mem+1.89[−3.22, 7.11]−0.520[−0.638, −0.408]
VerifiedExpeRepair+8.67[4.53, 12.93]−0.560[−0.690, −0.452]
VerifiedAdaRepair-Mem+2.67[−1.20, 6.60]−0.343[−0.440, −0.256]

All four cost-difference CIs lie below zero, and the Verified accuracy CI against ExpeRepair is positive. The accuracy CIs against AdaRepair-Mem include zero. Across all three paired repeats, MARS achieves higher observed accuracy while reducing mean cost by 31.08%/22.54% on Lite/Verified, supporting a practical improvement in the accuracy–cost operating point.

This joint improvement matters when assessing a controller. The observed accuracy advantage persists across repeated runs with shared upstream outputs and accompanies substantial cost savings. The evidence therefore supports more effective use of the available inference budget.

Intervals use a paired two-way bootstrap with 20,000 resamples: draw issues uniformly with replacement to preserve each dataset's size, independently draw three run identifiers with replacement, and use the same draws for every method. Mean success-rate and cost differences use the Cartesian product of the sampled issues and runs; the 2.5th and 97.5th percentiles give the endpoints. Relative savings equal 1 − mean(MARS cost)/mean(reference cost), using unrounded means. These are marginal percentile intervals from paired resampling of issues and runs; SD summarizes run-to-run variation.

W3. Generalization to current models and mainstream coding agents

The models and agents studied are a bit outdated. For example, the best models studied are GPT-4o and Claude 3 Opus, while now we already have GPT-6 and Opus 5.5, which are way stronger. Also, the modern mainstream coding agents, including Claude Code, Codex CLi, and OpenCode, are now widely adopted for program repair. It would be important to study such more widely used agents instead of some old domain-specific agents. Overall, it is not quite clear whether the current findings can generalize to more recent models and agents.

MARS has been tested across three proposal backbones and four host pipelines. To clarify the main configuration, our experiments use Claude 3.5 Sonnet with o4-mini; Table 6 additionally evaluates DeepSeek R1 and Kimi 2.5. The corresponding accuracy gains over the stronger baseline are +2.8/+1.4/+1.0 pp.

We additionally compare deepseek-flash and a GPT configuration on SWE-rebench with ExpeRepair as the host. The GPT configuration uses gpt-6-astra for the main model and gpt-6.1-terra for review. MARS increases observed GPT success from 35.8% to 51.4% while reducing mean cost from $17.47 to $13.54 per issue, approximately 22.5%. Tables R2-3/3a in W4 report both model configurations and their difference intervals.

For host transfer, Table 3 reports accuracy gains of 2.2–3.4 pp with lower cost across AutoCodeRover, Agentless-1.5, and PatchPilot. These results complement the ExpeRepair experiments and support the evidence-interface design.

Claude Code and similar coding agents highlight the rapid progress of repository-level repair. MARS addresses a complementary question: how can a repair system use its inference budget more effectively through better search, candidate preservation, and commitment? Our additional results show that these decisions remain consequential with a stronger model configuration. On SWE-rebench, MARS raises the GPT-based host’s observed success rate from 35.8% to 51.4% while reducing mean cost from $17.47 to $13.54 per issue. Across three additional hosts, it improves accuracy by 2.2–3.4 percentage points with lower cost. These results support the continued value of evidence-guided control alongside advances in coding models and repair systems.

W4. Evaluation on newer and more challenging benchmarks

The benchmarks adopted are also a bit outdated. SWE-bench Lite and Verified have been largely saturated by more recent models and agents. In recent months, people have proposed a number of more challenging and newer benchmarks, including SWE-bench Pro and DeepSWE. It would be interesting to study how the proposed approach help improve the scores on those newer benchmarks.

In the additional SWE-rebench evaluation, MARS achieves higher observed success rates and lower mean costs under both model configurations. We keep ExpeRepair as the host and compare deepseek-flash and GPT configurations. The GPT configuration uses gpt-6-astra for the main model and gpt-6.1-terra for review.

Table R2-3. Additional model configurations on SWE-rebench.

Model configurationMethodPass@1 (%)Mean cost ($/issue)
deepseek-flashExpeRepair20.90.22
deepseek-flashExpeRepair + MARS35.80.17
gpt-6-astra / gpt-6.1-terraExpeRepair35.817.47
gpt-6-astra / gpt-6.1-terraExpeRepair + MARS51.413.54

Table R2-3a. Changes in success rate and cost relative to ExpeRepair.

Model configurationSuccess-rate difference (pp)95% CI for the difference (pp)Mean cost difference ($/issue)
deepseek-flash+14.9[4.9, 25.5]−0.05
gpt-6-astra / gpt-6.1-terra+15.5[−0.4, 31.5]−3.93

With deepseek-flash, MARS increases the success rate from 20.9% to 35.8%, a gain of 14.9 pp whose difference interval lies entirely above zero, while reducing mean cost from $0.22 to $0.17 per issue. These results directly demonstrate accuracy and cost benefits for the MARS system configuration on the additional benchmark.

With GPT, MARS increases the observed success rate from 35.8% to 51.4%, with a reported gain of 15.5 pp, while reducing mean cost from $17.47 to $13.54 per issue, approximately 22.5% based on the tabulated amounts. The success-rate gain has a positive point estimate, with an interval of [−0.4, 31.5] pp; both configurations show the same direction of higher observed success and lower mean cost.

These additional results extend the evaluation to SWE-rebench and two additional model configurations, supporting MARS's practical value beyond the original Lite/Verified settings. The deepseek-flash configuration provides an interval-supported success-rate gain, while GPT adds a positive observed result under another model configuration.

Q1. How would the findings generalize to newer models, agents, and benchmarks?

How would the current findings generalize to newer models, agents, and benchmarks?

The evidence now includes SWE-rebench with two additional model configurations, alongside the original cross-model and cross-host experiments. On SWE-rebench, both configurations show higher observed success and lower mean cost: deepseek-flash has a positive success-rate difference interval, while GPT provides another positive point estimate. These results support applicability beyond Lite/Verified and the original models. Host transfer is supported by AutoCodeRover, Agentless-1.5, and PatchPilot. Together, the additional model and benchmark results and the cross-host gains support the reuse of evidence-guided control across different repair settings.

Reviewer_g2HsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses search resource allocation and candidate patch preservation in repository-level program repair through MARS, a training-free deterministic controller. AS uses runtime evidence to decide whether to verify, refine, expand, or stop; IC separates candidate admission to the pool from replacement of the incumbent; and CC freezes the candidate set at search termination and passes it to the host system for selection.

Critical strengths

None

Critical weaknesses
  1. Independent runs do not hold upstream stochastic outputs fixed, leaving small gains and component attribution insufficiently reliable.

  2. Two core decisions of the deterministic controller are not fully operationalized, impeding reproduction and interpretation of the ablations.

  3. The LLMs are too old. I have no idea if these methods still work on newer ones.

Questions for authors

See above

4 comments

We thank the reviewer for the questions on reliability, reproducibility, and model transfer. We provide paired repeated-run results, a complete controller specification, and additional model evidence below. Accuracy differences are percentage points (pp); costs are dollars per issue.

W1. Reliability under upstream randomness and attribution of the components

Independent runs do not hold upstream stochastic outputs fixed, leaving small gains and component attribution insufficiently reliable.

MARS achieves higher accuracy and lower cost in all three paired runs. To address upstream randomness directly, we conducted an additional paired evaluation, separate from the independently executed runs in the original Table 1. Within each issue and repeat, methods share the actual retrieval and localization outputs. Each method was run three times on SWE-bench Lite and Verified. Table R1-1 reports the mean ± sample standard deviation across runs.

Table R1-1. Additional three-run paired evaluation with fixed upstream outputs, mean ± sample SD.

DatasetMethodPass@1 (%)Cost ($/issue)
LiteExpeRepair Default48.33 ± 0.671.893 ± 0.035
LiteAdaRepair-Mem51.11 ± 0.511.673 ± 0.025
LiteExpeRepair + MARS53.00 ± 0.671.153 ± 0.025
VerifiedExpeRepair Default57.27 ± 0.701.740 ± 0.030
VerifiedAdaRepair-Mem63.27 ± 0.311.523 ± 0.025
VerifiedExpeRepair + MARS65.93 ± 0.501.180 ± 0.020

Relative to AdaRepair-Mem, MARS resolves 6, 5, and 6 additional Lite issues and 14, 12, and 14 additional Verified issues. The mean gains are +1.89/+2.67 pp with 31.08%/22.54% lower cost. Relative to its ExpeRepair host, MARS gains +4.67/+8.67 pp while reducing cost by 39.08%/32.18%. ExpeRepair is the host; AdaRepair-Mem is the separate memory-augmented baseline.

Table R1-2. Mean paired differences and marginal 95% bootstrap confidence intervals (CIs). Differences are MARS minus the reference.

DatasetReferenceAccuracy difference (pp)95% bootstrap CI (pp)Cost difference ($/issue)95% bootstrap CI ($/issue)
LiteExpeRepair+4.67[−0.78, 10.11]−0.740[−0.880, −0.604]
LiteAdaRepair-Mem+1.89[−3.22, 7.11]−0.520[−0.638, −0.408]
VerifiedExpeRepair+8.67[4.53, 12.93]−0.560[−0.690, −0.452]
VerifiedAdaRepair-Mem+2.67[−1.20, 6.60]−0.343[−0.440, −0.256]

All four cost-difference CIs lie below zero, and the Verified accuracy CI against ExpeRepair is positive. The accuracy CIs against AdaRepair-Mem include zero. Across all three paired repeats, MARS achieves higher observed accuracy while reducing mean cost by 31.08%/22.54% on Lite/Verified, supporting a practical improvement in the accuracy–cost operating point.

Sharing the actual upstream outputs removes differences in retrieval and localization as an explanation for the within-repeat gains. Their consistent direction across repeats also shows that the result does not depend on selecting one favorable run. Higher observed accuracy together with lower expenditure supports improved use of the available search budget; the gain is not obtained by spending more on inference.

Intervals use a paired two-way bootstrap with 20,000 resamples: draw issues uniformly with replacement to preserve each dataset's size, independently draw three run identifiers with replacement, and use the same draws for every method. Mean success-rate and cost differences use the Cartesian product of the sampled issues and runs; the 2.5th and 97.5th percentiles give the endpoints. Relative savings equal 1 − mean(MARS cost)/mean(reference cost), using unrounded means. These are marginal percentile intervals from paired resampling of issues and runs; SD summarizes run-to-run variation.

The component results support distinct roles for search allocation and candidate retention. The original Table 9 evaluates the same 200-issue subset once per configuration, with CC enabled throughout.

Table R1-3. Reported AS × IC factorial, CC enabled, N = 200, one run per configuration.

ASICResolvedPass@1 (%)Cost ($/issue)Ever-correctFinal-pool correct
OffOff11859.01.39127121
OnOff12864.01.27137130
OffOn12361.51.53129127
OnOn13467.01.31140138

Enabling AS improves accuracy by +5.0/+5.5 pp and lowers cost by $0.12/$0.22 with IC off/on. Enabling IC improves accuracy by +2.5/+3.0 pp and reduces pool loss from 6/7 issues to 2 with AS off/on, at an additional $0.14/$0.04. The +0.5 pp interaction is descriptive; these single-run component results are separate from the additional paired evaluation.

The lifecycle counts clarify these different roles. AS increases the number of issues for which a correct patch is ever generated while reducing expenditure, consistent with better search allocation. IC reduces the loss of correct candidates before final selection, consistent with a more conservative replacement boundary. Both components help under either setting of the other, supporting complementary contributions.

For CC, we add a same-trajectory comparison with a fixed output allowance. On the same frozen Verified search histories, CC-prefix-3 takes the first min(3, m) distinct candidates, while Selected+recent-3 takes the active incumbent and fills the same allowance with the most recent distinct candidates, where m is the number generated. Both branches share search evidence, search cost, selector settings, validation allowance, and a branch-independent presentation order. Neither generates new patches or uses correctness labels to construct its pool.

Table R1-4. Same-trajectory commitment comparison on Verified, output allowance min(3, m).

CommitmentEver-correctFinal-pool correctResolvedPass@1 (%)All-in cost ($/issue)
CC-prefix-334233632865.61.16
Selected+recent-334232932364.61.16

CC-prefix-3 retains correct patches for seven more issues and resolves five more (+1.0 pp), with both mean costs rounding to $1.16. Shared histories remove between-branch search variation; when m ≤ 3, the candidate sets are identical. This directly supports candidate preservation under the tested k=3 allowance. It is a new sensitivity experiment, separate from the original path-dependent cap of eight and the independently executed matched control in Table 11 (+1.2 pp).

Because the generated patches and output allowance are shared, the improvement comes from which candidates remain available to the selector, rather than a larger search or candidate pool. The gain in retained correct patches translates into additional final resolutions, showing why commitment matters even after a correct patch has already been generated.

W2. Executable definitions of the two core decisions

Two core decisions of the deterministic controller are not fully operationalized, impeding reproduction and interpretation of the ablations.

The controller specification gives an explicit promotion predicate and an ordered action policy.

Promotion. The first candidate initializes the incumbent, and every challenger is registered before replacement is considered. Before strong verification begins, a distinct challenger replaces the incumbent only if both verification states are still unqueried, the incumbent has known execution and review evidence, the challenger has execution success and review score 2, and the incumbent either fails execution or has review score below 2. All other cases retain the incumbent, including ties, unknown execution, missing review, and exact duplicates. Once verification begins, the incumbent identity is fixed and the verdict is bound to that patch. Later candidates can still enter the pool for downstream selection.

Action selection. The controller applies the following gates in order and takes the first applicable action.

PriorityConditionAction
1Cost reaches $2.20 at a configured checkpointStop for budget
2No candidate is availableStop with no candidates
3The incumbent has a strong-verification passStop for verified success
4At least one candidate has review score 2; the incumbent is unverified; no verification has begunVerify the incumbent
5No candidate is review-supported, or the incumbent verdict is refine; refinement depth is below 2Refine
6No usable reproduction is available and the host has fallback expansion allowanceExpand
7No preceding rule appliesStop as exhausted

Execution affects this policy through promotion, the execution evidence supplied to the review stage, and the reproduction-availability fallback. An execution pass does not by itself trigger successful termination. The review-supported candidate triggers verification, but the target is the current incumbent. At termination, CC preserves generation order after exact deduplication; the first-eight cap applies only on capped write-out paths.

The rule checks pass 41/41 fixed cases (each replayed five times), 5,184/5,184 promotion inputs, and 15/15 state and cost checks. They cover the positive promotion predicate, patch-bound verification, and state/cost consistency. The revised Section 3.3 summarizes the policy, Appendices A.2 and A.3 give the complete rules and checks, and Algorithm 1 presents the control flow.

These definitions connect each ablation to a specific decision. Retaining the incumbent on equal support avoids replacements justified only by recency, and binding verification to its patch prevents a verdict from being transferred to a later candidate. CC separately preserves candidates for final selection, so promotion and final exposure can be examined independently.

A budget-triggered stop freezes the accumulated candidates and issues no further model calls. CC preserves the frozen candidate set for downstream selection without reopening proposal or refinement. The $2.20 threshold is a soft checkpoint because costs can exceed it between consecutive ledger checks; it is not a hard cap on final per-issue expenditure.

W3. Applicability to newer language models

The LLMs are too old. I have no idea if these methods still work on newer ones.

The observed gains extend across three proposal backbones. The main configuration uses Claude 3.5 Sonnet with o4-mini, with additional tests using DeepSeek R1 and Kimi 2.5. Table 6 reports +2.8, +1.4, and +1.0 pp over the stronger baseline within the respective settings. MARS consumes normalized execution, review, verification, and cost signals, allowing the same control interface to be used with different models.

In the additional SWE-rebench evaluation, MARS achieves higher observed success rates and lower mean costs under both model configurations. We keep ExpeRepair as the host and compare deepseek-flash and GPT configurations. The GPT configuration uses gpt-6-astra for the main model and gpt-6.1-terra for review.

Table R1-5. Additional model configurations on SWE-rebench.

Model configurationMethodPass@1 (%)Mean cost ($/issue)
deepseek-flashExpeRepair20.9$0.22/issue
deepseek-flashExpeRepair + MARS35.8$0.17/issue
gpt-6-astra / gpt-6.1-terraExpeRepair35.8$17.47/issue
gpt-6-astra / gpt-6.1-terraExpeRepair + MARS51.4$13.54/issue

Table R1-5a. Changes in success rate and cost relative to ExpeRepair.

Model configurationSuccess-rate difference (pp)95% CI for the difference (pp)Mean cost difference ($/issue)
deepseek-flash+14.9[4.9, 25.5]−$0.05/issue
gpt-6-astra / gpt-6.1-terra+15.5[−0.4, 31.5]−$3.93/issue

With deepseek-flash, MARS increases the success rate from 20.9% to 35.8%, a gain of 14.9 pp whose difference interval lies entirely above zero, while reducing mean cost from $0.22 to $0.17 per issue. These results directly demonstrate accuracy and cost benefits for the MARS system configuration on the additional benchmark.

With GPT, MARS increases the observed success rate from 35.8% to 51.4%, with a gain of 15.5 pp, while reducing mean cost from $17.47 to $13.54 per issue, approximately 22.5% based on the tabulated amounts. The success-rate gain has a positive point estimate, with an interval of [−0.4, 31.5] pp; both configurations show the same direction of higher observed success and lower mean cost.

These additional results extend the evaluation to SWE-rebench and two additional model configurations, supporting MARS's practical value beyond the original Lite/Verified settings. The deepseek-flash configuration provides an interval-supported success-rate gain, while GPT adds a positive observed result under another model configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.