- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to improve inference-time scaling for coding agents by focusing on two key factors: candidate coverage and selection quality. The authors first conduct a diagnostic study on SWE-bench Verified and show that simply increasing the number of attempts or repeatedly performing self-verification does not necessarily lead to effective scaling, due to limited exploration and unreliable verification. They propose Differential Verification to guide search using execution feedback, achieving higher resolution with substantially fewer agent steps than naive sample scaling. Based on these observations, the paper further trains these behaviors into the policy through weighted supervised fine-tuning over agent trajectories and reinforcement learning with mutation-based verification rewards and assertion-level credit assignment. The resulting policy improves both repair accuracy and verifier discrimination, with consistent gains across multiple benchmarks and model scales.
Critical strengths
-
The paper studies an important and timely problem for coding agents. Rather than treating inference-time scaling as simply increasing the number of samples, the paper provides useful insights into why additional inference compute may fail to translate into better performance, and highlights the importance of both exploration diversity and reliable verification.
-
The empirical improvements are substantial. On the held-out SWE-bench Verified split, the proposed SFT and RL stages improve pass@1 from 31.9% to 43.0% and pass@8 from 46.7% to 60.7%. The improvements also transfer to other benchmarks, suggesting that the learned behaviors are not limited to the training issues.
-
The paper provides extensive analyses and ablations. The experiments separately examine candidate coverage, selection quality, verifier discrimination, trajectory behavior, reward components, different sources of incorrect repairs, and different model scales. These analyses make it easier to understand where the improvements come from rather than relying only on end-to-end benchmark results.
Critical weaknesses
-
It remains unclear to what extent the learned policy actually reduces the need for test-time scaffolding. A central motivation of the paper is that the useful search and verification behaviors identified in the diagnostic study can be moved into the policy, such that “a policy can carry them instead of a scaffold.” However, the paper does not provide a direct, matched comparison between the trained policy and the test-time Differential Verification procedure in Section 2.1. In particular, the strongest trained-policy results still involve multi-sample candidate generation and verifier-based selection, while the Differential Verification results use a different search procedure, evaluation population, and inference budget. The strong pass@1 improvement does provide evidence that some of these behaviors are internalized, but it does not quantify how much of the test-time search and verification machinery can actually be removed after training. This makes the “policy instead of scaffold” framing somewhat stronger than what is directly established by the current experiments.
-
Regarding presentation, Figure 1 appears to be AI-generated, as some of the text on the right side, such as “self-verification,” is unclear or distorted.
Questions for authors
To what extent can the trained policy replace the test-time search and verification scaffold introduced in Section 2.1? Could the authors evaluate the trained policy and the base policy under the same test-time scaffold and matched inference budget?
Discussion
Dear Reviewers,
We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.
Thank you,
AC.