acceptodds
Under review as a conference paper at ICLR 2027

plorer: Scaling Coding Agents through Reconciled Tree Search

Abstract

Test-time scaling is a promising way to improve LLM-based coding agents. However, long-horizon software repair produces extended trajectories whose partial progress must be preserved, compared, and reused; simply adding steps or branches can yield diminishing returns. We present plorer, an iterative tree-search framework built around a repeatable explore–reconcile–prune cycle. Within each iteration, path-level best-first selection extends promising trajectories; at the iteration boundary, compatible paths exchange evidence and the frontier is compressed to seed the next iteration. This bounded design lets the search use larger test-time budgets while repeatedly consolidating its most promising progress; end patch selection then compares candidates accumulated across the full search. On SWE-bench Verified with a gpt-5-mini backbone, increasing the test-time budget from 30 to 50 improves SWE-Xplorer from 54.6% to 61.4%, while the prior tree-search framework SWE-Search improves from 54.2% to 54.8%. At a matched budget of 50, SWE-Xplorer leads SWE-Search by 6.6 points while reducing average cost and runtime. It also achieves a higher resolved rate than strong single-trajectory agents—Claude Code and OpenHands—while costing less than either. Relative to its single-trajectory backbone, SWE-Xplorer improves resolved rate with all three evaluated models on both SWE-bench Verified and SWE-bench Multilingual. Together, these results show that iteration-based search can turn additional test-time compute into continued gains for long-horizon coding agents.

Primary area:
generative models
Keywords:
LLM, SWE-Agents, SWE-Bench, Tree-Search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xD8rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Xplorer improves coding agents through iterative tree search, combining recent-path scoring, periodic pruning, evidence sharing between branches with identical repository states, and final patch selection through tournament voting. It improves over mini-SWE-agent across three models and two benchmarks. With GPT-5-mini on SWE-bench Verified, it achieves 61.4% resolved rate, compared with 53.2% for mini-SWE-agent and 54.8% for SWE-Search.

Critical strengths
  1. The improvements across three backends and both Python and multilingual benchmarks suggest that the approach has practical value. The accuracy and cost improvements over the evaluated SWE-Search config are encouraging
  2. Combining evidence from branches with identical repository states is a reasonable design. The component ablations help establish that reconciliation and recent-path scoring contribute to the final system
Critical weaknesses
  1. My main concern is the limited novelty. Much of the framework combines established ideas (i.e., coding-agent tree search, iterative refinement, and tournament selection). The recent-path score is a relatively small modification, and state-compatible reconciliation appears to be the most distinctive contribution. The paper does not convincingly show how much this advances beyond closely related approaches, particularly Kim et al.'s iterative framework and trajectory-reuse methods such as SWE-Replay
  2. The eval doesn't sufficiently separate the benefits of the proposed search design from additional computation and implementation choices. Matching expansion counts does not match compute, and SWE-Xplorer costs roughly 5x as much as its single-trajectory baseline. Comparisons with independent attempts plus selection, or closely related iterative methods at matched cost, would be more informative. The lack of sensitivity studies for the path score and reward evaluator also leaves uncertainty about how robust the design is
Questions for authors

Clarifications:

C1. How many independent runs were performed per config? Table 7 provides paired confidence intervals across tasks, but what is the variability across repeated agent runs? C2. What versions, backend models, and reasoning settings were used for Claude Code, OpenHands, and OpenCode? Do reported costs include the reproduction agent and all scoring, reconciliation, and selection calls? C3. Beyond the component ablations, do you have direct quantitative evidence for C1–C3? For example, how often are promising paths stranded or is useful evidence lost during pruning?

Additional evidence that could change my assessment:

A1. How does SWE-Xplorer compare at matched cost with Kim et al.’s iterative refinement and tournament selection, or independent rollouts using the same selector? A2. How sensitive are the results to the path-score window and discount factor? How were (W=3) and (gamma=0.85) selected? A3. Does replacing Qwen-2.5-7B with a stronger reward evaluator improve the accuracy–cost tradeoff? Is the benefit of path-level scoring consistent across evaluator choices?

3 comments

Thank you for the review. Section 1 of the general response covers the matched-cost experiment and cost accounting; below we address novelty and your requests.

W1: Limited novelty relative to Kim et al. and SWE-Replay

We agree that SWE-Xplorer builds on established ideas, and the paper credits each: tree search for coding agents (SWE-Search), and iterative refinement and recursive tournament voting (Kim et al., 2026). What we add is in how the candidate pool is built, not how a patch is selected from it.

  • Kim et al. iterates over independent rollouts conditioned on the previous round's top-k solutions. With no shared tree, it cannot continue a promising trajectory from its midpoint, reuse a shared prefix, or recognize that two lines of investigation reached the same repository state.
  • SWE-Replay branches retrospectively from archived, completed trials. SWE-Xplorer branches online from live frontier nodes, and its pruning and reconciliation act on trajectories that are still running.
  • State-grounded reconciliation merges evidence from concurrent branches that reach an identical repository state; neither method has such an operation.
  • Path-level selection is a small change to the scoring rule, but removing it costs the most of any component (−6.2 points).

W2 / A1: Separating the search design from extra compute

We agree that matching expansion counts does not match compute. The revision adds independent attempts with the same tournament procedure, compared across reported monetary costs (new Figure 4, Section 4.4; Section 1 of the general response). Best-of-k is more efficient at low cost, but increasing k from two to five adds only 1.8 points. SWE-Xplorer reaches 61.4% at \0.137versusbest−of−5′s57.20.137 versus best-of-5's 57.2% at \\0.158: 4.2 points higher at 13.3% lower reported cost. Its pool coverage is also higher (67.0% vs. 65.5%), supporting improved candidate discovery, although selection errors can differ across pools.

This tests independent sampling with a shared selector, rather than Kim et al.'s full iterative method, which we did not run. Section 4.4 and Limitations acknowledge independent sampling's low-cost advantage. The comparison supports the combined system at higher reported costs, but does not isolate search design from reproduction feedback or equalize total compute under our cost exclusions.

Clarifications

C1. One run per configuration. We agree this limits how precisely small differences, such as the 1.2–3.2-point ablation effects, can be read.

C2. The main gpt-5-mini agent comparisons use nominal limits of 50 turns, steps, or expansions, as appropriate to each framework. These units do not imply equal compute. Each independent run in the new best-of-k comparison instead has a 30-step limit.

AgentVersionReasoning setting
Claude Code2.1.217Provider default (extended thinking disabled)
OpenHands0.59.0Reasoning effort high (OpenHands default)
OpenCode1.18.5Provider default
mini-SWE-agent (also SWE-Xplorer's action agent)1.17.3Provider default

Reported SWE-Xplorer costs include gpt-5-mini search and selection calls, including reconciliation, candidate summaries, and tournament judging. They exclude the reproduction agent and assign zero monetary cost to self-hosted reward-model inference (Section 1 of the general response; Section 4.1). Consequently, they do not represent total compute or full standalone deployment cost.

C3. Our quantitative evidence is primarily component-level: removing path-level selection reduces resolved rate by 6.2 points and removing reconciliation by 3.2 points in the observed runs. Appendix F illustrates two reconciliation events in a complete search tree. Pool coverage grows from 60.8% to 67.0% between B=30 and B=50, consistent with continued candidate discovery, but this does not directly measure evidence preserved by pruning or recovery from a weak action. We did not compute those event-level statistics, and the reconciliation ablation does not reallocate its extra computation to ordinary expansion.

Additional evidence

A2. W = 3 and γ = 0.85 were fixed before the reported runs to emphasize recent progress. We did not tune them or run a sensitivity grid. Ranking by a node's own reward (the "w/o path-level selection" ablation) corresponds to a window of one and reduces resolved rate by 6.2 points in the observed run.

A3. We did not test a stronger evaluator. Qwen-2.5-7B was fixed across all backbones to avoid confounding backbone comparisons, and chosen small to keep scoring cheap.

Thank you again. Your request for independent attempts plus selection at matched cost shaped the most substantial addition in the revision.

Thank you for the additional experiments and clear responses. The best-of-k comparison is a useful addition and strengthens the evidence that SWE-Xplorer can outperform independent sampling at the higher tested budgets.

However, my main concerns remain. The independent runs do not receive the same reproduction-test feedback, and the reported costs exclude reproduction-agent generation and reward-model inference. This leaves uncertainty about how much of the improvement comes from the proposed search design. Direct comparisons with Kim et al.’s iterative method and SWE-Replay are also still missing.

I appreciate the authors’ acknowledgment of the limitations around single-run ablations, hyperparameter sensitivity, and evaluator choice. These clarifications improve the paper, but the evidence does not yet sufficiently establish the distinctive contribution of its components. I am therefore maintaining my rating of 2: Weak rejection

Dear Reviewer,

We have addressed this concern already. Perhaps it is being missed. Hence, we would like to clarify here that reproduction-test outcomes adjust edit rewards for search control; they are not supplied as additional feedback to the action-generation model. Standard mini-SWE-agent has no competing frontier nodes to rank, so there is no directly equivalent use of this scoring signal in independent rollouts.

To address your underlying concern directly, we conducted one additional ablation removing the reproduction-test signal while retaining the remaining search mechanisms. SWE-Xplorer achieves 60.8%, compared with 61.4% for the full system—an observed decrease of 0.6 percentage points. Without this signal, it still exceeds best-of-5’s 57.2% by 3.6 points.

We have already added this result to Table 3 and Section 4.5. It supports that the reproduction-test signal is not the main source of the observed accuracy advantage over independent sampling.

Reviewer_LXGgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWE-Xplorer, a new technique for improving the test-time scaling capabilities of coding agents through tree search. Specifically, SWE-Xplorer allocates more computational resources to more promising search directions through path-level selection, step pruning, and cross-path reconciliation. Evaluations across different models and benchmarks show that SWE-Xplorer can outperform SWE-Search with the same or even a lower test-time budget.

Critical strengths
  • This paper explores an interesting and important direction.
  • Results clearly show that SWE-Xplorer outperforms SWE-Search in terms of final performance under the same or even lower test-time budget.
Critical weaknesses
  • The test-time budget metric used in the evaluation does not seem fair to the mini-SWE-agent baseline. One unit of this budget corresponds to one expansion for SWE-Xplorer but one agent-environment interaction step for mini-SWE-agent, so the two methods may incur very different token costs under the same nominal budget. Specifically, comparing Table 2 and Figure 3 shows that mini-SWE-agent achieves 53.2% at a cost of only about $0.30 per task (Figure 3), whereas SWE-Xplorer achieves only 47.2% - 50.8% at a similar cost (Table 2).
  • The evaluation does not directly assess the accuracy of the LLM-as-a-Judge scores assigned to individual actions in a trace. While the paper discusses LLM judge accuracy based on the quality of the final patch, it remains unclear whether the scores for individual actions are accurate.
  • Although the paper notes that SWE-Xplorer performs online tree search while SWE-Replay branches retrospectively from archived trials, this distinction does not justify excluding SWE-Replay from the experimental comparisons, as their test-time scaling performance can still be compared directly.
  • The experiments use only Qwen-2.5-7B as the evaluator, which limits the comprehensiveness of the evaluation.
Questions for authors
  • What would Figure 2 look like if the x-axis showed the actual token cost instead of the test-time budget metric?
  • Section 4.1 is missing a citation for DeepSeek-V4-Flash.
1 comment

Thank you for the review and for recognizing the direction and the gains over SWE-Search. We address each point below; the shared concern about matched compute is covered in full in our general response.

W1: The test-time budget is unfair to mini-SWE-agent

We agree that one unit of budget is one step for mini-SWE-agent but one expansion for SWE-Xplorer, so equal nominal budgets do not mean equal token cost.

One clarification on the numbers: a single mini-SWE-agent run costs $0.028 per task, not $0.30 (Figure 3 and Section 4.4). This makes your point stronger, not weaker: at B=10, SWE-Xplorer costs about the same as one mini-SWE-agent run and resolves fewer tasks.

The revision adds a comparison across reported dollar costs against best-of-k, with the final patch chosen by the same tournament procedure (new Figure 4, Section 4.4; full table in Section 1 of the general response). Independent sampling is more efficient at low cost, but SWE-Xplorer reaches 61.4% at \0.137versusbest−of−5′s57.20.137 versus best-of-5's 57.2% at \\0.158: 4.2 points higher at 13.3% lower reported cost. Section 4.4 and Limitations acknowledge the low-cost advantage of independent sampling. These costs exclude SWE-Xplorer's reproduction agent and assign zero cost to its self-hosted reward model (Section 4.1), so they do not equalize total compute or full deployment cost.

W2: Accuracy of per-action judge scores

You are right that the terminal-candidate analysis does not directly evaluate scores assigned to intermediate actions. We therefore added a new analysis of edit actions to Appendix E.1 (Table 6, "Do higher-scored edits produce better patches?"). We compare each edit's recorded score with the test-suite verdict for the code state it produces. Labels come from patch-candidate evaluations, not reproduction tests.

This yields 1,928 distinct edit-produced states across 481 tasks. Comparing passing and failing states only within the same task gives 1,292 pairs across 161 tasks. The raw evaluator score achieves an AUC of 0.586, and the shaped reward reaches 0.669. The improvement is 0.083 (95% CI: 0.031–0.139), using paired bootstrap resampling of whole tasks to account for dependence among pairs from the same task. As a stricter comparison, among 86 passing-versus-failing sibling-edit pairs across 68 tasks, both edits start with the same code and interaction history. Raw and shaped scores favor the passing edit in 60.5% and 68.6% of pairs, respectively (ties receive half credit). This directly evaluates scores assigned to intermediate edits, rather than submission scores alone. It measures whether an edit produces a resolving patch; a failing partial edit may still be useful progress. Assessing reads, searches, and test actions would require independent action-quality annotations.

W3: SWE-Replay should be compared directly

That is fair. The distinction in our related work, online tree search vs. retrospective branching from archived trials, explains how the methods differ but does not justify leaving SWE-Replay out of the experiments. We did not run it within the discussion period. The new best-of-k baseline covers the independent-rollout setting that SWE-Replay builds on, at matched cost and with the same selector. The revised Limitations (Appendix B) now names SWE-Replay as a missing comparison.

W4: Only Qwen-2.5-7B as the reward evaluator

We fixed the evaluator across all backbones to avoid confounding backbone comparisons with evaluator changes, and chose a small model to limit inference requirements. We did not test a stronger evaluator, so dependence on evaluator strength remains unresolved. In particular, the benefit of smoothing rewards over a path may change with evaluator quality; we cannot assume that a stronger evaluator would preserve the same gains or improve the accuracy–cost tradeoff.

Q1: Figure 2 with actual cost on the x-axis

Figure 4 now plots resolved rate and candidate-pool coverage against average monetary cost per task, for SWE-Xplorer across budgets 10–50 and best-of-k for k = 1–5. This addresses monetary cost; it is not a token-count plot, and it uses the accounting conventions stated in Section 4.1 and Section 1 of the general response.

Q2: Missing DeepSeek-V4-Flash citation

The missing DeepSeek-V4-Flash citation has been restored in Section 4.1.

Reviewer_m1ZxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SWE-Xplorer wraps mini-SWE-agent in a tree search for fixing GitHub issues. The budget is split into rounds. Within a round, the node with the best path score is expanded. At the end of each round, branches that reach the same codebase state are merged, and the frontier is cut to the top k. The last rounds focus on finishing one edit path. Proposed patches are kept as candidates, and an LLM tournament picks the final one. With gpt-5-mini on SWE-bench Verified, it resolves 61.4%, compared with 53.2% for mini-SWE-agent and 54.8% for SWE-Search with lower cost.

Critical strengths
  1. Clear practical motivation. Pruning only at round boundaries lets a path survive one weak step. Merging only branches with an identical codebase state is a simple, well-defined rule.

  2. Broad evaluation. The paper uses the full Verified and Multilingual benchmarks, three backbones, ablations, cost and runtime, and paired McNemar tests. It beats SWE-Search at lower cost with the same model and budget.

  3. Useful selection analysis. At budget 50, a correct patch is in the tournament pool for 67.0% of tasks, but only 61.4% are solved. The judge picks correctly in 80.8% of decisive pairs. This makes the selection bottleneck visible.

Critical weaknesses
  1. The paper does not show that tree search beats simpler uses of the same compute. "Budget" means steps for mini-SWE-agent but tree expansions for SWE-Xplorer, and each expansion samples up to two actions and calls a reward model. The paper's own numbers show what happens at equal cost. SWE-Xplorer at budget 10 costs $0.028 and solves 47.2%, while mini-SWE-agent at the same $0.028 solves 53.2%. At budget 20 ($0.046), SWE-Xplorer is still lower (50.8%). It only pulls ahead at more than 3× the cost. The scaling plot (Fig. 2) has the same problem: mini-SWE-agent uses only 13–17 actions on average (Table 8), so raising its step limit from 30 to 50 has little room to help. The usual way to scale a single agent is to run it several times and pick the best patch. The closest such methods (Kim et al. 2026) are discussed but not compared. The scaling study also covers only one model and one benchmark, with each budget run once.

  2. Part of the gain may come from an extra test signal. Before search, a separate reproduction agent runs for up to 30 steps, and its script scores every edit. This runs outside the 50-expansion budget. The paper does not say whether its cost, or the cost of the Qwen-2.5-7B reward model, is included in the $0.137. The baseline does not get this signal, and no ablation removes it.

  3. Many hand-tuned parts, and small ablation effects. The system combines path scoring (W = 3, γ = 0.85), reward shaping with many fixed constants, convergence forcing, solution augmentation, reconciliation, and tournament voting from prior work. Apart from path-level selection (−6.2 points), the ablation drops are 1.2–3.2 points. These come from single runs with no significance test. For reconciliation, the merged node also uses extra LLM calls and an extra executed action; the ablation should give that budget back to ordinary expansion. As a result, it is hard to tell which parts matter and what is new beyond SWE-Search and Kim et al. (2026).

Minor: The AUC values printed in Fig. 5 (e.g., 0.677 → 0.747) do not match those in the text of Appendix E.1 (0.683 → 0.746). The DeepSeek-V4-Flash citation is missing ("?").

Questions for authors
  1. At the same total dollar or token cost, how does SWE-Xplorer compare with N independent mini-SWE-agent runs that share its reproduction script and tournament selector?
  2. Does the $0.137 include the reproduction agent and the reward model? How much does performance drop without the reproduction test signal?
  3. If reconciliation is removed and its calls are given back to ordinary expansion, how much of the 3.2-point gain remains?
  4. How much do the main and ablation results vary across repeated runs?
3 comments

Thank you for the detailed review. Section 1 of the general response covers the matched-cost experiment; we address your questions below.

W1 / Q1: Does tree search beat simpler uses of the same compute?

You are right that our nominal budget is not compute-matched, and our own Table 2 shows it: at B=10, SWE-Xplorer costs about the same as a single mini-SWE-agent run but resolves fewer tasks.

We added independent sampling with the same tournament selector (new Figure 4; Section 1 of the general response). Best-of-k is more efficient at low cost, but increasing k from two to five adds only 1.8 points. SWE-Xplorer reaches 61.4% at \0.137versusbest−of−5′s57.20.137 versus best-of-5's 57.2% at \\0.158: 4.2 points higher at 13.3% lower reported cost. Pool coverage is also higher (67.0% vs. 65.5%), supporting improved candidate discovery, although selection errors can differ between pools. Section 4.4 and Limitations acknowledge the low-cost advantage of independent sampling.

Best-of-k shares the selector but does not receive reproduction-test feedback during generation. That feedback could also benefit independent trajectories; its contribution remains unresolved without a shared-feedback baseline or an ablation removing it.

On Figure 2: we agree that mini-SWE-agent's 13–17 actions on average leave little room for a higher step limit to help; Figure 4 instead spends the extra compute on more independent runs.

W2 / Q2: Reproduction-test signal and cost accounting

What the $0.137 covers. SWE-Xplorer's gpt-5-mini search and selection calls, including reconciliation, candidate summaries, and tournament judging. The reproduction agent is excluded and self-hosted reward-model inference is assigned zero monetary cost (Section 1 of the general response; now stated in Section 4.1). The reproduction script is reused across configurations, but its generation remains an additional cost for a standalone run. Thus, this comparison does not equalize total compute or full deployment cost.

Without the reproduction signal. We did not run this ablation, so we cannot quantify its contribution. The script shapes edit rewards (Appendix C.4), and reward and path scores also affect tournament seeding (Appendix C.7). The fallback when script generation fails does not establish the effect of removing this signal systematically.

W3 / Q3: Hand-tuned components and small ablation effects

Significance and variance. Each ablation is a single run. Path-level selection has the largest observed effect (−6.2 points), while the others range from −1.2 to −3.2 points. Without repeated runs, we cannot establish which differences exceed run-to-run variation or claim individual significance. Section 4.5 reports these as observed marginal effects within the full system; the paired tests across benchmark tasks do not measure variability across repeated agent runs.

Reconciliation and budget. A merged node uses one synthesis call, one executed action and two reward evaluations (Section 3.4), outside the 50-expansion budget. The ablation does not reallocate this work to ordinary expansion, so the 3.2-point difference does not isolate reconciliation from its extra computation. We did not run the reallocated-budget variant. The synthesis call is included in reported monetary cost, while the self-hosted reward evaluations are assigned zero cost; the best-of-k comparison does not resolve this component-level confound.

Hyperparameters. W = 3 and γ = 0.85 were fixed before the reported runs to emphasize recent progress. We did not tune them or run a sensitivity study. Table 5 lists the other search bounds, including frontier and tournament-pool size.

Novelty beyond SWE-Search and Kim et al. (2026). Tournament voting is from Kim et al., and we credit it explicitly. Our contributions concern candidate generation: path-level selection, iteration-boundary pruning with state-grounded reconciliation, and deferred termination with solution augmentation. The higher candidate-pool coverage supports the value of this combined design, but does not isolate each mechanism or substitute for a direct comparison with Kim et al.'s full iterative method.

Q4: Variance across repeated runs

Each configuration was run once. We agree this limits how precisely small differences can be read.

Minor

Both fixed (Section 2 of the general response): the AUC plot (Figure 6 in the revision) now matches Appendix E.1, and the DeepSeek-V4-Flash citation is restored.

Your Q1 motivated the new independent-sampling comparison.

The authors addressed most of my concerns. The remaining issue is that the cost calculation does not consider the reproduction agent and the reward model, but the method uses info from them. seems unfair. I will raise my score to 3, but this concern should be stated in the limitation section

Thank you for raising your score. We agree, and the revised Limitations section (Appendix B) now states that reported costs exclude the reproduction agent and the self-hosted reward model, although both guide the search and the baselines receive neither, so the cost comparisons understate SWE-Xplorer’s total compute. Our new ablation also shows that the reproduction-test signal accounts for little of the gain: without it, SWE-Xplorer still resolves 60.8% versus 61.4% with it (Section 4.5).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their careful and constructive reviews. We were glad that reviewers found the motivation practical (m1Zx), the evaluation broad, with full benchmarks, three backbones, ablations, cost/runtime and paired significance tests (m1Zx, xD8r), and the results over SWE-Search encouraging, with higher accuracy at lower cost (all reviewers).

All three reviewers raised one shared concern, which we address here. Individual replies cover the remaining points.

1. Comparing at matched cost rather than matched nominal budget

All reviewers point out that our "test-time budget" counts steps for mini-SWE-agent but tree expansions for SWE-Xplorer, so equal budgets do not mean equal compute. We agree, and we have added a cost-matched comparison (new Figure 4, Section 4.4).

Best-of-k baseline. We compare SWE-Xplorer with k independent mini-SWE-agent trajectories (gpt-5-mini), selecting the final patch with the same recursive tournament voting SWE-Xplorer uses. Each run is limited to 30 steps, matching SWE-Xplorer's per-path depth limit, and reported costs include tournament selection. Results on SWE-bench Verified:

MethodResolved (%)Pool coverage (%)Cost ($/task)Time (min/task)
Best-of-151.651.60.0262.30
Best-of-255.460.60.0635.25
Best-of-356.862.00.0947.90
Best-of-457.064.20.12610.55
Best-of-557.265.50.15813.2
SWE-Xplorer, B=1047.248.80.0283.72
SWE-Xplorer, B=2050.853.40.0465.96
SWE-Xplorer, B=3054.660.80.09512.36
SWE-Xplorer, B=4057.263.80.11715.28
SWE-Xplorer, B=5061.467.00.13718.32

What this shows.

  • At low cost, independent sampling is more efficient. Best-of-2 reaches 55.4% at $0.063, compared with 54.6% at $0.095 for SWE-Xplorer at B=30. We acknowledge this in Section 4.4 and Limitations.
  • Independent sampling shows diminishing gains over the evaluated range. Increasing k from two to five raises resolved rate by 1.8 points (55.4 → 57.2%), while SWE-Xplorer improves from 54.6% to 61.4% between B=30 and B=50. At B=50, SWE-Xplorer resolves 61.4% at $0.137 versus best-of-5's 57.2% at $0.158: 4.2 points higher at 13.3% lower reported cost. Best-of-4 provides a nearby-cost comparison, reaching 57.0% at $0.126.
  • Pool coverage supports improved candidate discovery. SWE-Xplorer's pool coverage reaches 67.0%, compared with 65.5% for best-of-5. Its coverage rises from 60.8% to 67.0% between B=30 and B=50, while best-of-k rises from 60.6% to 65.5% between k=2 and k=5. Both use the same tournament procedure, but selection errors can differ across pools; these results do not attribute the entire resolved-rate gap to discovery alone.

These results support continued gains from iterative search over the evaluated budget range, with an advantage at the higher-cost configurations rather than at every budget.

Cost accounting. SWE-Xplorer's reported cost covers its gpt-5-mini search and selection calls, including action generation, reconciliation, candidate summaries, and tournament judging. It excludes the reproduction agent, whose script is generated once per task and reused across configurations, and assigns zero monetary cost to self-hosted Qwen-2.5-7B reward-model inference. Section 4.1 now states these conventions explicitly. Reuse amortizes the reproduction cost across experiments but does not eliminate it for a standalone run; the reported comparison therefore measures monetary cost under these conventions, rather than total compute or full deployment cost.

2. Corrections already made

  • AUC figure (Figure 6 in the revision). The plotted values now match Appendix E.1 (0.683 → 0.746 for the node's own score; 0.677 → 0.727 for the path mean; 0.663 → 0.714 for the path score). (m1Zx)
  • Missing DeepSeek-V4-Flash citation. Restored. (m1Zx, LXGg)

3. Summary of changes in the revision

  • New best-of-k matched-cost baseline (Section 4.1) and Figure 4 with its analysis (Section 4.4), plotting resolved rate and pool coverage against cost.
  • Explicit statement of what reported costs include and exclude (Section 4.1).
  • New intermediate edit-score analysis with task-bootstrap confidence intervals and a sibling-edit comparison (Appendix E.1, Table 6), addressing LXGg's question about per-action scores.
  • Limitations (Appendix B) now states that best-of-k outperforms SWE-Xplorer at low budgets.
  • Corrections listed in Section 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.