- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper solves an important problem of temporal credit assignment in RLVR for training long-horizon software-engineering agents. Typically, only the final binary test outcome is used for rewarding a trajectory. SWE-MILE develops process reward at the step level without expolicity training a reward model or leveraging an LLM judge. The solution defines and uses two potentials, a navigation potential and a verification potential. The step reward is then calculated by the changes in the potentials.
The process credit is then added to outcome advantage and optimized in a PPO style. The test results on Qwen3.5 9B and 35B-A3B models show that SWE-MILE outperforms GRPO and other variants on SWE-bench verified.
Critical strengths
Adding step-level reward/credit is crucial for improving RLVR for long-horizon tasks.
The proposed approach is cost-effective.
Consistent improvement across different benchmarks.
The paper also performed comprehensive ablations and analysis.
Critical weaknesses
-
The navigation potential seems to use gold-patch files.
-
The claim that potential differences prevent reward hacking - this is not analyzed nor guaranteed.
-
Fairness when compared with baselines. I'd love the authors to add more context on how the baselines e.g., GiGPO, are evaluated and if the evaluations are done to serve the best of these baselines.
-
The training seems limited to python-only tasks, and different tasks require training different models.
-
The authors may also want to explain the selection of the checkpoints and hyperparameters and make sure there is no leakage in the benchmark information
Questions for authors
Please refer to the weakness section.
3 comments
Reply to Reviewer xuDs (Part 2/2)
W5: Checkpoint and Hyperparameter Selection
We report the best checkpoint evaluated every 5 steps. We did so because long-horizon SWE RL is unstable, and using only the last step would penalize less stable methods. All baselines follow the same rule, so the bias is roughly symmetric. Figure 3 also shows that SWE-MILE is stable within a moderate hyperparameter range (: 0.01–0.1, : 0.1–0.4). In addition, Tables 2 and 3 show clear advantages of SWE-MILE over the baselines in response length and turns, a notable efficiency gain.
To rule out bias from checkpoint selection and benchmark information leakage, the table below reports the best checkpoint (avg@4), final checkpoint (avg@4) and last-10-step average (avg@1) for all methods. Final and Last-10 do not depend on checkpoint selection and can substitute for validation-set selection. SWE-MILE leads clearly, and by a larger margin, on every metric, with the ranking essentially unchanged. Estimating selection bias as Best − Last-10 gives 0.9–1.3 points on Verified and 0.5–1.2 on Pro across methods; SWE-MILE's is 0.9 / 0.8, not unusually large.
| Method | Verified Best / Final / Last-10 | Pro Best / Final / Last-10 |
|---|---|---|
| Base Model | 53.6 / 53.6 / 53.6 | 30.8 / 30.8 / 30.8 |
| GRPO | 58.2 / 57.9 / 57.3 | 33.9 / 33.9 / 33.4 |
| G-RA | 57.9 / 57.2 / 56.8 | 34.1 / 33.7 / 33.2 |
| SWE-TRACE | 60.5 / 59.4 / 59.2 | 36.6 / 35.9 / 35.4 |
| GiGPO | 58.4 / 58.4 / 57.3 | 34.7 / 34.2 / 33.9 |
| GraphGPO | 59.9 / 59.2 / 58.8 | 35.1 / 35.1 / 34.4 |
| SWE-MILE | 63.8 / 63.1 / 62.9 | 39.3 / 38.8 / 38.5 |
We also trained on R2E-Gym with essentially the same hyperparameters (only the batch size was changed to 32 for faster training, as R2E-Gym has more samples) and report Best and Final (avg@4) below (only some baselines due to time constraints). SWE-MILE again outperforms the baselines, indicating that its gains do not depend on hyperparameter selection on the test sets.
| Method | Verified Best / Final | Pro Best / Final |
|---|---|---|
| Base Model | 53.6 / 53.6 | 30.8 / 30.8 |
| GRPO | 57.2 / 56.7 | 33.4 / 33.0 |
| SWE-TRACE | 59.4 / 58.6 | 33.9 / 33.9 |
| GraphGPO | 59.7 / 59.3 | 36.4 / 36.2 |
| SWE-MILE | 63.1 / 62.7 | 38.6 / 38.6 |
Follow-up on Our Response
Dear Reviewer,
Thank you again for your thoughtful review and valuable feedback.
As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.
Thank you for your time and consideration!
Best regards,
The Authors
Reply to Reviewer xuDs (Part 1/2)
Thank you for your thoughtful review. Below we provide point-by-point responses to your comments and concerns. We will add the following new experimental results and analyses in the final version.
W1: Gold-patch Dependency
SWE training data mainly comes from GitHub issues and PRs, where gold patches usually exist and are easy to obtain, so the relevant-file set H is a zero-cost by-product of data construction. Mainstream SWE RL training sets (SWE-Gym, R2E-Gym, SWE-smith, SWE-rebench) all produce gold patches during construction, and some directly annotate relevant-file sets.
W2: Reward-hacking Resistance
We acknowledge that the paper does not discuss "preventing reward hacking" in detail. We will add the following analysis:
W3: Baseline Reimplementation and Tuning
Baselines use the same data, model and training parameters (batch size, learning rate, etc.) as SWE-MILE. Other training settings:
normalize_distancesearched over {True, False}, best result reported.For evaluation, as with SWE-MILE, the trained models are evaluated under the same settings, including the same inference budget, scaffold, prompts and interaction protocol.
W4: Limited Training Diversity and Cross-task Generalization
Multilingual training. SWE-MILE is language-agnostic by design: the navigation potential depends only on file-level exposure, and the verification potential only on test pass states. To verify this, we trained SWE-MILE and several baselines on 4,738 SWE-rebench V2 tasks in Python, Go, TS and JS under the same settings except batch size 32 (for faster training), and implemented test-result parsing for these four languages. SWE-MILE still outperforms all baselines:
Separate training per task. We trained separately to enable controlled comparisons with the corresponding baselines within each task family. This is an experimental design choice for research, not a limitation of the method. Whole-repository generation uses Qwen3.5-35B-A3B for a fair comparison with DeNovoSWE-Agent-35A3B of the same scale. Moreover, current LLM post-training commonly first trains task-specific expert models and then merges them into a unified model via on-policy distillation (OPD). The task-specific experts trained with SWE-MILE can be directly incorporated into this pipeline.