- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SWE-Xplorer improves coding agents through iterative tree search, combining recent-path scoring, periodic pruning, evidence sharing between branches with identical repository states, and final patch selection through tournament voting. It improves over mini-SWE-agent across three models and two benchmarks. With GPT-5-mini on SWE-bench Verified, it achieves 61.4% resolved rate, compared with 53.2% for mini-SWE-agent and 54.8% for SWE-Search.
Critical strengths
- The improvements across three backends and both Python and multilingual benchmarks suggest that the approach has practical value. The accuracy and cost improvements over the evaluated SWE-Search config are encouraging
- Combining evidence from branches with identical repository states is a reasonable design. The component ablations help establish that reconciliation and recent-path scoring contribute to the final system
Critical weaknesses
- My main concern is the limited novelty. Much of the framework combines established ideas (i.e., coding-agent tree search, iterative refinement, and tournament selection). The recent-path score is a relatively small modification, and state-compatible reconciliation appears to be the most distinctive contribution. The paper does not convincingly show how much this advances beyond closely related approaches, particularly Kim et al.'s iterative framework and trajectory-reuse methods such as SWE-Replay
- The eval doesn't sufficiently separate the benefits of the proposed search design from additional computation and implementation choices. Matching expansion counts does not match compute, and SWE-Xplorer costs roughly 5x as much as its single-trajectory baseline. Comparisons with independent attempts plus selection, or closely related iterative methods at matched cost, would be more informative. The lack of sensitivity studies for the path score and reward evaluator also leaves uncertainty about how robust the design is
Questions for authors
Clarifications:
C1. How many independent runs were performed per config? Table 7 provides paired confidence intervals across tasks, but what is the variability across repeated agent runs? C2. What versions, backend models, and reasoning settings were used for Claude Code, OpenHands, and OpenCode? Do reported costs include the reproduction agent and all scoring, reconciliation, and selection calls? C3. Beyond the component ablations, do you have direct quantitative evidence for C1–C3? For example, how often are promising paths stranded or is useful evidence lost during pruning?
Additional evidence that could change my assessment:
A1. How does SWE-Xplorer compare at matched cost with Kim et al.’s iterative refinement and tournament selection, or independent rollouts using the same selector? A2. How sensitive are the results to the path-score window and discount factor? How were (W=3) and (gamma=0.85) selected? A3. Does replacing Qwen-2.5-7B with a stronger reward evaluator improve the accuracy–cost tradeoff? Is the benefit of path-level scoring consistent across evaluator choices?
3 comments
Thank you for the additional experiments and clear responses. The best-of-k comparison is a useful addition and strengthens the evidence that SWE-Xplorer can outperform independent sampling at the higher tested budgets.
However, my main concerns remain. The independent runs do not receive the same reproduction-test feedback, and the reported costs exclude reproduction-agent generation and reward-model inference. This leaves uncertainty about how much of the improvement comes from the proposed search design. Direct comparisons with Kim et al.’s iterative method and SWE-Replay are also still missing.
I appreciate the authors’ acknowledgment of the limitations around single-run ablations, hyperparameter sensitivity, and evaluator choice. These clarifications improve the paper, but the evidence does not yet sufficiently establish the distinctive contribution of its components. I am therefore maintaining my rating of 2: Weak rejection
Dear Reviewer,
We have addressed this concern already. Perhaps it is being missed. Hence, we would like to clarify here that reproduction-test outcomes adjust edit rewards for search control; they are not supplied as additional feedback to the action-generation model. Standard mini-SWE-agent has no competing frontier nodes to rank, so there is no directly equivalent use of this scoring signal in independent rollouts.
To address your underlying concern directly, we conducted one additional ablation removing the reproduction-test signal while retaining the remaining search mechanisms. SWE-Xplorer achieves 60.8%, compared with 61.4% for the full system—an observed decrease of 0.6 percentage points. Without this signal, it still exceeds best-of-5’s 57.2% by 3.6 points.
We have already added this result to Table 3 and Section 4.5. It supports that the reproduction-test signal is not the main source of the observed accuracy advantage over independent sampling.
Thank you for the review. Section 1 of the general response covers the matched-cost experiment and cost accounting; below we address novelty and your requests.
W1: Limited novelty relative to Kim et al. and SWE-Replay
We agree that SWE-Xplorer builds on established ideas, and the paper credits each: tree search for coding agents (SWE-Search), and iterative refinement and recursive tournament voting (Kim et al., 2026). What we add is in how the candidate pool is built, not how a patch is selected from it.
W2 / A1: Separating the search design from extra compute
We agree that matching expansion counts does not match compute. The revision adds independent attempts with the same tournament procedure, compared across reported monetary costs (new Figure 4, Section 4.4; Section 1 of the general response). Best-of-k is more efficient at low cost, but increasing k from two to five adds only 1.8 points. SWE-Xplorer reaches 61.4% at \0.137versusbest−of−5′s57.20.158: 4.2 points higher at 13.3% lower reported cost. Its pool coverage is also higher (67.0% vs. 65.5%), supporting improved candidate discovery, although selection errors can differ across pools.
This tests independent sampling with a shared selector, rather than Kim et al.'s full iterative method, which we did not run. Section 4.4 and Limitations acknowledge independent sampling's low-cost advantage. The comparison supports the combined system at higher reported costs, but does not isolate search design from reproduction feedback or equalize total compute under our cost exclusions.
Clarifications
C1. One run per configuration. We agree this limits how precisely small differences, such as the 1.2–3.2-point ablation effects, can be read.
C2. The main gpt-5-mini agent comparisons use nominal limits of 50 turns, steps, or expansions, as appropriate to each framework. These units do not imply equal compute. Each independent run in the new best-of-k comparison instead has a 30-step limit.
Reported SWE-Xplorer costs include gpt-5-mini search and selection calls, including reconciliation, candidate summaries, and tournament judging. They exclude the reproduction agent and assign zero monetary cost to self-hosted reward-model inference (Section 1 of the general response; Section 4.1). Consequently, they do not represent total compute or full standalone deployment cost.
C3. Our quantitative evidence is primarily component-level: removing path-level selection reduces resolved rate by 6.2 points and removing reconciliation by 3.2 points in the observed runs. Appendix F illustrates two reconciliation events in a complete search tree. Pool coverage grows from 60.8% to 67.0% between B=30 and B=50, consistent with continued candidate discovery, but this does not directly measure evidence preserved by pruning or recovery from a weak action. We did not compute those event-level statistics, and the reconciliation ablation does not reallocate its extra computation to ordinary expansion.
Additional evidence
A2. W = 3 and γ = 0.85 were fixed before the reported runs to emphasize recent progress. We did not tune them or run a sensitivity grid. Ranking by a node's own reward (the "w/o path-level selection" ablation) corresponds to a window of one and reduces resolved rate by 6.2 points in the observed run.
A3. We did not test a stronger evaluator. Qwen-2.5-7B was fixed across all backbones to avoid confounding backbone comparisons, and chosen small to keep scoring cheap.
Thank you again. Your request for independent attempts plus selection at matched cost shaped the most substantial addition in the revision.