acceptodds
Under review as a conference paper at ICLR 2027

SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agents

Abstract

Long-horizon software engineering (SWE) agents trained with reinforcement learning with verifiable rewards (RLVR) typically receive only terminal outcome supervision, making it difficult to distinguish productive actions from redundant exploration or functional regressions. We propose SWE-MILE, an asynchronous potential-induced milestone credit assignment framework that derives fine-grained process supervision from workflow runtime, without auxiliary reward models or external evaluators. SWE-MILE quantifies task-relevant file exposure and test-state alignment as navigation and verification potentials, respectively. Differences in these potentials attribute milestone progress and regressions to individual actions, while discounted backward credit propagates supervision to preceding steps. To efficiently acquire intermediate verification states, SWE-MILE further introduces asynchronous shadow probing, which replays repository-changing actions in an isolated sandbox and runs verification in parallel with the agent’s primary interaction, largely hiding verification latency. The resulting process credit augments terminal outcome advantages and provides informative learning signals. Experiments on two representative long-horizon SWE tasks demonstrate substantial improvements in agent performance, highlighting workflow runtime signals as a practical source of process supervision for long-horizon SWE agents.

Primary area:
applications to robotics, autonomy, planning
Keywords:
software engineering agents, reinforcement learning, credit assignment, process rewards

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_xuDsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper solves an important problem of temporal credit assignment in RLVR for training long-horizon software-engineering agents. Typically, only the final binary test outcome is used for rewarding a trajectory. SWE-MILE develops process reward at the step level without expolicity training a reward model or leveraging an LLM judge. The solution defines and uses two potentials, a navigation potential and a verification potential. The step reward is then calculated by the changes in the potentials.

The process credit is then added to outcome advantage and optimized in a PPO style. The test results on Qwen3.5 9B and 35B-A3B models show that SWE-MILE outperforms GRPO and other variants on SWE-bench verified.

Critical strengths

Adding step-level reward/credit is crucial for improving RLVR for long-horizon tasks.

The proposed approach is cost-effective.

Consistent improvement across different benchmarks.

The paper also performed comprehensive ablations and analysis.

Critical weaknesses
  1. The navigation potential seems to use gold-patch files.

  2. The claim that potential differences prevent reward hacking - this is not analyzed nor guaranteed.

  3. Fairness when compared with baselines. I'd love the authors to add more context on how the baselines e.g., GiGPO, are evaluated and if the evaluations are done to serve the best of these baselines.

  4. The training seems limited to python-only tasks, and different tasks require training different models.

  5. The authors may also want to explain the selection of the checkpoints and hyperparameters and make sure there is no leakage in the benchmark information

Questions for authors

Please refer to the weakness section.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer xuDs (Part 1/2)

Thank you for your thoughtful review. Below we provide point-by-point responses to your comments and concerns. We will add the following new experimental results and analyses in the final version.

W1: Gold-patch Dependency

SWE training data mainly comes from GitHub issues and PRs, where gold patches usually exist and are easy to obtain, so the relevant-file set HH is a zero-cost by-product of data construction. Mainstream SWE RL training sets (SWE-Gym, R2E-Gym, SWE-smith, SWE-rebench) all produce gold patches during construction, and some directly annotate relevant-file sets.

W2: Reward-hacking Resistance

We acknowledge that the paper does not discuss "preventing reward hacking" in detail. We will add the following analysis:

  • No positive-reward loop can be exploited repeatedly. rpotr^{\text{pot}} is a potential difference, so the rpotr^{\text{pot}} of any cycle back to the original state always sums to zero. The navigation potential takes the historical maximum, so repeated reads earn nothing.
  • Break-then-repair is penalized at the breaking step. Process credit acts on the policy gradient per step, not as a trajectory sum. The action that breaks a P2P test immediately receives negative credit −αveri(βany+βfracbt)-\alpha_{\text{veri}}(\beta_{\text{any}}+\beta_{\text{frac}}b_t), directly lowering its probability, and more severe breakage is penalized more. Breaking comes before repair, so suppressing the breaking action suppresses the pattern. In whole-repository generation, asymmetric clipping (c−=6>c+=3c_-=6>c_+=3) further makes the penalty for regression larger than the reward for recovery.
  • Constraints at the outcome level. Repeated break-repair consumes turns and easily leads to task failure (outcome 0); hitting the turn limit also triggers the RlimR_{\text{lim}} discount.
  • Empirical evidence. In Table 2, SWE-MILE's average trajectory length and turns generally decrease relative to the base model, while the other baselines show varying increases in turns. This is the opposite of the redundant interaction that hacking typically causes.

W3: Baseline Reimplementation and Tuning

Baselines use the same data, model and training parameters (batch size, learning rate, etc.) as SWE-MILE. Other training settings:

  • GRPO: standard GRPO;
  • G-RA: no open-source code; default parameters from the paper;
  • SWE-TRACE: γ\gamma searched over {0.6, 0.75, 0.9}, best result reported;
  • GiGPO: open-source code, default parameters;
  • GraphGPO: open-source code, default parameters except normalize_distance searched over {True, False}, best result reported.

For evaluation, as with SWE-MILE, the trained models are evaluated under the same settings, including the same inference budget, scaffold, prompts and interaction protocol.

W4: Limited Training Diversity and Cross-task Generalization

Multilingual training. SWE-MILE is language-agnostic by design: the navigation potential depends only on file-level exposure, and the verification potential only on test pass states. To verify this, we trained SWE-MILE and several baselines on 4,738 SWE-rebench V2 tasks in Python, Go, TS and JS under the same settings except batch size 32 (for faster training), and implemented test-result parsing for these four languages. SWE-MILE still outperforms all baselines:

MethodVerified Best / FinalPro Best / Final
Base Model53.6 / 53.630.8 / 30.8
SWE-TRACE60.5 / 59.635.8 / 35.2
GraphGPO62.7 / 61.337.8 / 36.8
SWE-MILE65.2 / 64.740.9 / 40.3

Separate training per task. We trained separately to enable controlled comparisons with the corresponding baselines within each task family. This is an experimental design choice for research, not a limitation of the method. Whole-repository generation uses Qwen3.5-35B-A3B for a fair comparison with DeNovoSWE-Agent-35A3B of the same scale. Moreover, current LLM post-training commonly first trains task-specific expert models and then merges them into a unified model via on-policy distillation (OPD). The task-specific experts trained with SWE-MILE can be directly incorporated into this pipeline.

AuthorsOpenReview ↗

Reply to Reviewer xuDs (Part 2/2)

W5: Checkpoint and Hyperparameter Selection

We report the best checkpoint evaluated every 5 steps. We did so because long-horizon SWE RL is unstable, and using only the last step would penalize less stable methods. All baselines follow the same rule, so the bias is roughly symmetric. Figure 3 also shows that SWE-MILE is stable within a moderate hyperparameter range (αnavi\alpha_{\text{navi}}: 0.01–0.1, αveri\alpha_{\text{veri}}: 0.1–0.4). In addition, Tables 2 and 3 show clear advantages of SWE-MILE over the baselines in response length and turns, a notable efficiency gain.

To rule out bias from checkpoint selection and benchmark information leakage, the table below reports the best checkpoint (avg@4), final checkpoint (avg@4) and last-10-step average (avg@1) for all methods. Final and Last-10 do not depend on checkpoint selection and can substitute for validation-set selection. SWE-MILE leads clearly, and by a larger margin, on every metric, with the ranking essentially unchanged. Estimating selection bias as Best − Last-10 gives 0.9–1.3 points on Verified and 0.5–1.2 on Pro across methods; SWE-MILE's is 0.9 / 0.8, not unusually large.

MethodVerified Best / Final / Last-10Pro Best / Final / Last-10
Base Model53.6 / 53.6 / 53.630.8 / 30.8 / 30.8
GRPO58.2 / 57.9 / 57.333.9 / 33.9 / 33.4
G-RA57.9 / 57.2 / 56.834.1 / 33.7 / 33.2
SWE-TRACE60.5 / 59.4 / 59.236.6 / 35.9 / 35.4
GiGPO58.4 / 58.4 / 57.334.7 / 34.2 / 33.9
GraphGPO59.9 / 59.2 / 58.835.1 / 35.1 / 34.4
SWE-MILE63.8 / 63.1 / 62.939.3 / 38.8 / 38.5

We also trained on R2E-Gym with essentially the same hyperparameters (only the batch size was changed to 32 for faster training, as R2E-Gym has more samples) and report Best and Final (avg@4) below (only some baselines due to time constraints). SWE-MILE again outperforms the baselines, indicating that its gains do not depend on hyperparameter selection on the test sets.

MethodVerified Best / FinalPro Best / Final
Base Model53.6 / 53.630.8 / 30.8
GRPO57.2 / 56.733.4 / 33.0
SWE-TRACE59.4 / 58.633.9 / 33.9
GraphGPO59.7 / 59.336.4 / 36.2
SWE-MILE63.1 / 62.738.6 / 38.6
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards,

The Authors

Reviewer_WHRtOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A process reward proposal that use step wise test results and repo exploration signals to assign step level reward for software agent RL training.

The exp results look nice.

Rather than using another model to judge each action and step, the information in the repo itself is used to evaluate the process.

The async sandbox is a practical contribution.

Critical strengths

The question itself is interesting. Since terminal outputs are super sparse and after tons of actions in the process, how to assign credits to actions along the process is a great challenge as a process reward model.

The reward design is very straight forwad. Navigation is credited due to their exposing unseen files and not just "pwd" but also actual "cat".

The results look very promising over SWE-bench.

Similar to my summary, async verification sandbox is a great practical contribution.

Critical weaknesses

Path/content exposing reward looks great to me in this SWE agent, but according to Table 4 it looks like they do not contribute too much. Any exploration action level credit assignment explanation by example? Like how can I link the real contribution by each action to final success of swe task?

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer WHRt

Thank you for your thoughtful review. Below we provide point-by-point responses to your comments and concerns.

W1: Limited Contribution of Navigation Potential and Lack of Action-level Attribution Analysis

Why navigation yields smaller gains than verification. In SWE-MILE, navigation is designed as an auxiliary signal: with αnavi=0.05\alpha_{\text{navi}}=0.05, its total contribution per trajectory is at most 0.05 (vs. 0.2 for verification). This is because exposing relevant files is necessary but not sufficient for solving the task. Even so, removing navigation lowers Verified / Pro by 1.8 / 1.3 points and increases turns by 5.5 / 9.4 (Table 4), showing that its main role is helping the agent localize the issue more efficiently. Moreover, training trajectories take 25.8 steps on average before the first edit; since test states do not change before the first repository modification, navigation is the only non-zero process signal in this phase.

Example. The table below traces the potentials along one SWE-rebench V2 training trajectory (task researchobject__ro-crate-py-95, with 2 relevant files and 3 F2P tests; eventually successful). Navigation potential rises only on first exposure of a relevant file: first its path via search (+0.2), then its content via read_file (to 1.0), see steps 1–4. Verification potential rises only when edits fix F2P tests (steps 22, 29). Paginated re-reads of the same file (steps 5–6), reads of irrelevant files (steps 7–11), and script or test runs that do not change the repository (steps 12–21, 23–28) leave the total potential unchanged and thus earn no direct reward. The last column is the process credit of each step (no format errors in this trajectory). The backward term also passes a small, distance-discounted share of later rewards to earlier steps, e.g., 0.024 for steps 21 and 28.

StepActionMilestone changeΦnavi\Phi^{\text{navi}}Φveri\Phi^{\text{veri}}Φ\Phirtpotr_t^{\text{pot}}CtprocessC_t^{\text{process}}
—Initial state—000——
0search ro-crate-previewNone (no relevant file hit)00000.010
1search preview.html; preview.py in resultsFile 1 exposure 0 → 0.20.1000.0050.0050.015
2read_file preview.pyFile 1 exposure 0.2 → 1.00.5000.0250.0200.027
3search; rocrate.py in resultsFile 2 exposure 0 → 0.20.6000.0300.0050.012
4read_file rocrate.pyFile 2 exposure 0.2 → 1.01.0000.0500.0200.024
5–6Paginated reads of rocrate.pyNone (repeated exposure)1.0000.05000.004–0.005
7–11Read metadata.py, entity.py, etc.None (irrelevant files)1.0000.05000.005–0.008
12–21Run repro scripts, re-readNone (repo unchanged)1.0000.05000.009–0.024
22edit rocrate.py1/3 F2P fixed1.000.3330.1170.0670.079
23–28Run testsNone (repo unchanged)1.000.3330.11700.014–0.024
29edit rocrate.py3/3 F2P fixed1.001.0000.2500.1330.133
30–37Verify and submitNone1.001.0000.25000

Linking actions to final success. We replayed offline the file-exposure records of the 500 SWE-bench Verified evaluation trajectories from SWE-MILE's best checkpoint, taking relevant files from the gold patch and using the same scoring rules as in training. The 462 trajectories that exposed all relevant files succeed at 68.6%, versus only 5.3% for the remaining 38. Compared with w/o navigation, SWE-MILE first exposes all relevant files at turn 6.7 on average (vs. 8.5) and performs 20.1 file reads per trajectory (vs. 24.9). In addition, in a control that randomly shuffles the same credit across steps within each trajectory (Shuffle, table below), performance drops by 2.6 / 2.2 points, showing that aligning credit with the responsible actions matters.

MethodVerified Best / FinalTurnsPro Best / FinalTurns
Shuffle61.2 / 60.691.637.1 / 36.989.1
SWE-MILE63.8 / 63.181.839.3 / 38.878.4
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards,

The Authors

Reviewer_MNDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about reinforcement learning for software engineering agents on long tasks. These agents usually get only one pass or fail reward at the end, so training cannot tell which steps helped. The paper proposes SWE-MILE to give each step its own reward. A navigation potential measures how many relevant files from the reference patch the agent has found and read. A verification potential measures how many target tests pass and how many passing tests break. Each step is rewarded by how much it changes these potentials. A shadow sandbox replays code changes and runs the tests in parallel with the agent. Experiments on issue resolution and whole-repository generation show higher success rates and shorter rollouts than the baselines.

Critical strengths
  1. The step rewards are cheap and come from the environment itself. They use test results and file reads, so no reward model or LLM judge is needed. A step is rewarded only for new progress, so rereading a file earns nothing and breaking code is penalized. This keeps the reward tied to visible progress and adds no model inference cost.

  2. The method is best on all four benchmarks across two task types. It also uses shorter responses and fewer turns than the other trained methods. In issue resolution, removing any part of the method lowers performance. The shadow sandbox also hides most of the test waiting time there. This suggests the method is practical for real agent training.

Critical weaknesses
  1. The evaluation makes the gains hard to trust. The paper reports the best checkpoint on the test benchmarks, and the reward weights also seem tuned on them. The baselines are reimplemented, and their tuning is not described. The gains over the best baseline are only about two to three percentage points, so part of them may come from picking the best result.

  2. The extra test cost is large and not fully reported. In whole-repository generation, each rollout runs about 27 shadow probes of about five minutes each. Together they take more than four times as long as the agent's own interaction. The paper does not report the total training time or the extra compute over outcome-only training, so the real cost of the method is unclear.

Questions for authors
  1. What are the results of all methods at the final checkpoint, or with checkpoints chosen on a separate validation set?

  2. In issue resolution, rollouts that hit the turn or context limit but still pass get a lower reward. This reward depends on when all target tests first pass. Do the baselines also use this rule? How much of the gain and the shorter rollouts remains without it?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer MNDe

We sincerely thank the reviewer for the time and effort devoted to our work. We address each concern below and will include the full results of the following experiments in the final version.

W1 & Q1: Checkpoint and Hyperparameter Selection

We agree that reporting only the best checkpoint (every 5 steps) is incomplete. We did so because long-horizon SWE RL is unstable, and using only the last step would penalize less stable methods. All baselines follow the same rule, so the bias is roughly symmetric. Figure 3 shows SWE-MILE is stable within a moderate range (αnavi\alpha_{\text{navi}}: 0.01–0.1, αveri\alpha_{\text{veri}}: 0.1–0.4). Tables 2–3 also show SWE-MILE's clear efficiency advantage in response length and turns.

Baselines use the same data, model and training parameters (batch size, learning rate, etc.) as SWE-MILE. Other settings:

  • GRPO: standard GRPO;
  • G-RA: no open-source code; paper defaults;
  • SWE-TRACE: γ\gamma searched over {0.6, 0.75, 0.9}, best reported;
  • GiGPO: open-source code, defaults;
  • GraphGPO: open-source code, defaults except normalize_distance searched over {True, False}, best reported.

Below we report Best and Final checkpoints (avg@4) and the last-10-step average (avg@1). Last-10 does not depend on checkpoint selection and can substitute for validation-set selection. SWE-MILE leads clearly and by a larger margin on every metric; the ranking is essentially unchanged. Selection bias estimated as Best − Last-10 is 0.9–1.3 (Verified) and 0.5–1.2 (Pro) across methods; SWE-MILE's 0.9 / 0.8 is not unusually large.

MethodVerified Best / Final / Last-10Pro Best / Final / Last-10
Base Model53.6 / 53.6 / 53.630.8 / 30.8 / 30.8
GRPO58.2 / 57.9 / 57.333.9 / 33.9 / 33.4
G-RA57.9 / 57.2 / 56.834.1 / 33.7 / 33.2
SWE-TRACE60.5 / 59.4 / 59.236.6 / 35.9 / 35.4
GiGPO58.4 / 58.4 / 57.334.7 / 34.2 / 33.9
GraphGPO59.9 / 59.2 / 58.835.1 / 35.1 / 34.4
SWE-MILE63.8 / 63.1 / 62.939.3 / 38.8 / 38.5

We also trained on R2E-Gym with essentially the same hyperparameters (batch size 32 for speed, as R2E-Gym is larger; some baselines only, due to time): SWE-MILE again leads, indicating its gains do not depend on hyperparameter selection on the test sets.

MethodVerified Best / FinalPro Best / Final
Base Model53.6 / 53.630.8 / 30.8
GRPO57.2 / 56.733.4 / 33.0
SWE-TRACE59.4 / 58.633.9 / 33.9
GraphGPO59.7 / 59.336.4 / 36.2
SWE-MILE63.1 / 62.738.6 / 38.6

W2: Unreported Computational Overhead of Asynchronous Shadow Probing

Efficiency was a key design consideration:

(1) Idle CPU. SWE agent RL is dominated by GPU-bound rollout, leaving CPUs on training nodes largely idle. We use them to deploy SWE-MiniSandbox locally for the primary and shadow sandboxes, avoiding remote Docker sandbox services and raising CPU utilization.

(2) Multi-shadow parallel probing for whole-repository generation. The reviewer's "27 probes of 5 min each" sums all probe times, but probes are not serial. As in Appendix B.4, we combine:

  • Probe merge: consecutive edits with the same changed-file set run the tests only once.
  • Multi-shadow parallel probing: each rollout has 4 parallel shadow sandboxes.

Probes also overlap with other rollouts in the batch and need syncing only before the policy update. Table 8 shows 96.48% of rollouts have finished all shadow probes by the end of the batch.

(3) End-to-end overhead. Average step time (s) is below. Versus outcome-only, SWE-MILE adds only 3.2% and 4.2% in issue resolution and whole-repository generation, showing that with idle CPU and asynchronous parallel probing, shadow verification cost is largely hidden by rollout.

MethodIssue resolutionWhole-repository generation
Outcome-only (w/o milestone)4657.87029.7
w/o navigation4773.8 (+2.5%)-
w/o verification4687.9 (+0.6%)-
SWE-MILE4806.1 (+3.2%)7321.4 (+4.2%)

We will add this table and report total training time in the revision.

Q2: Terminal Reward Rule (RlimR_{\text{lim}})

Rlim=t∗/∣T∣R_{\text{lim}} = t^{\ast}/|T| requires t∗t^{\ast}, the step at which the agent first completes the task, obtainable only via shadow probing. We thus also enabled it for GiGPO and GraphGPO, which also use shadow probing; for GRPO, G-RA and SWE-TRACE, where t∗t^{\ast} is unavailable, it is uniformly set to 0.6, as in the outcome-only ablation.

RlimR_{\text{lim}} has little effect. Without it (uniformly 0.6), SWE-MILE gets 63.4 / 39.0 on Verified / Pro, close to 63.8 / 39.3, so we do not treat it as a main component. It rarely triggers: in SWE-MILE training, the average fraction of rollouts exceeding the context limit is 0 and the turn limit 0.04. It applies only to those passing verification; with mean outcome reward ~0.5, only ~2% of rollouts are discounted, too few to explain SWE-MILE's gains in performance and turns. The revision will state each method's RlimR_{\text{lim}} setting and add this ablation.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards,

The Authors

Thank you for the response, but some of my concerns still remain, so i will keep my score.

AuthorsOpenReview ↗

Request for Clarification on Remaining Concerns

Dear Reviewer MNDe,

Thank you for your reply. Could you let us know which specific concerns remain unresolved? We may not have time for further experiments, but we can still clarify the relevant claims. Knowing your remaining concerns would also help the AC reach a final decision.

Best regards, Authors

Reviewer_ZxXQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SWE-MILE, a process-credit-assignment scheme for RLVR training of long-horizon software-engineering agents. Rather than consuming the task verifier only as a terminal bit, SWE-MILE reads two scalar "potentials" off the workflow runtime: a navigation potential (Eq. 3), the mean over the relevant-file set H of a per-file cumulative exposure score q_{f,t} ∈ [0,1] that saturates at 0.2 for a path appearing in search results and 1.0 for content successfully exposed to the agent, guarded by a max over history so that re-reading a file earns nothing; and a verification potential, defined for issue resolution (Eq. 6) as the fraction of Fail-to-Pass tests repaired, p_t, minus β_any·1[b_t>0] + β_frac·b_t for the fraction of Pass-to-Pass tests regressed, and for whole-repository generation (Eq. 8) as the unnormalised pass-count delta M_t − M_0. Their weighted difference r_t^pot = Φ_t − Φ_{t−1} (Eqs. 9–10) attributes milestone progress and regression to individual actions; a discounted backward term λ Σ_u γ^{u−t} r_u^pot propagates credit to preceding steps and a format reward is added, giving C_t^process (Eq. 12), which is added to an RLOO outcome advantage (Eq. 13) and optimised with a step-level PPO-clipped loss (Eq. 15). Intermediate test states are obtained by asynchronous shadow probing (§2.6): actions that change the repository hash are replayed in an isolated shadow sandbox that runs the verifier in parallel with the primary rollout, with probe merging and multiple shadow sandboxes used for whole-repository generation.

Critical strengths

The observation that SWE RLVR discards the verifier's intrinsic structure — reducing a per-test F2P/P2P state vector to a single bit — is accurate, and recovering that structure without a reward model, an LLM judge, or additional annotation is the right trade against rubric-based PRMs (SWE-TRACE, SWE-Shepherd, rubric GRM), which add inference cost and model-estimated proxies. The trajectory-length effect is in the direction a sensible shaping scheme should produce, and Table 4's ordering (w/o verification −3.1 points on Verified, w/o navigation −1.8) is at least consistent with the stated claim that test-state changes carry more progress signal than file localisation. This matters for the recommendation because it locates the paper's value in a real and under-served part of the pipeline rather than in a benchmark delta.

Critical weaknesses

Appendix B.1 states that the authors "report the best performance among checkpoints evaluated every five training steps," and Figure 3 sweeps the two potential weights α_navi and α_veri directly on SWE-bench Verified and SWE-bench Pro — that is two independent selection passes against the reported metric, with no held-out validation split anywhere in the paper. The magnitude at stake is comparable to the ablation deltas that carry the mechanistic story (w/o navigation costs 1.8 points, w/o backward 2.4), so the ablations cannot currently be interpreted. Compounding this, the largest ablation row is not a clean comparison: the R_lim = t*/|T| shaping lives in the outcome reward and the paper states only that "in the outcome only ablation training, this reward is uniformly set to 0.6," leaving it unstated whether GRPO, G-RA, SWE-TRACE, GiGPO and GraphGPO receive the same treatment — which confounds both the 63.8-vs-59.4 comparison and the "fewer turns" headline, one of the paper's two headline outcomes. Most fundamentally, because r_t^pot is a potential difference, the forward terms telescope to Φ_T − Φ_0 over a trajectory, and the λ-backward term inflates per-trajectory credit mass by roughly 1 + λ/(1−γ) ≈ 3×; SWE-MILE is therefore close to adding a second, success-monotone bonus on top of an outcome advantage bounded in [−1,1]. No control with matched total credit mass is run, so "per-step attribution is what helps" is precisely the hypothesis the experiments are not designed to test. Two further consequences go unaddressed: C^process is never group-centred, so the leave-one-out baseline that makes RLOO what it is does not cancel and failed trajectories accumulate systematically positive shaping; and the claimed reward-hacking resistance is algebraically false in the issue-resolution setting, where breaking a P2P test (−α(β_any + β_frac·b_t)) and then repairing it (+α(β_any + β_frac·b_t)) cancel exactly, making break-then-repair a zero-sum detour rather than a penalised one. The Ng et al. (1999) framing does not license the construction either: the forward term is undiscounted, Φ^navi depends on interaction history rather than a Markov environment state, and the backward term is not a potential difference.

Questions for authors
  1. Where is validation done? Appendix B.1 reports the best checkpoint among those evaluated every five steps and Figure 3 sweeps α on SWE-bench Verified and Pro, which are also the reported benchmarks. If there is no held-out split, how many checkpoints and α configurations were evaluated on each benchmark, and what is the best estimate of the selection bias? Please report the final-checkpoint number alongside the selected one, and select checkpoints and α on a slice of SWE-rebench V2 held out from both eval sets.
  2. Is R_lim = t*/|T| applied to GRPO, G-RA, SWE-TRACE, GiGPO and GraphGPO, or only to SWE-MILE (with the outcome-only ablation fixed at 0.6)? If it is not uniform, re-report Tables 2–4 with one terminal-reward rule across all arms, since the length and turn reductions are reported as a central effect of milestone credit.
  3. Give the step-level evidence the contribution requires. What is the distribution of C^process and of per-step ΔΦ for both task families; how often is ΔΦ negative on a step that made net progress (e.g. five F2P fixed, one P2P broken, which Eq. 6 scores at −0.02); what is the correlation between per-step credit and eventual task outcome; and in a sample of trajectories, does the credit mass concentrate on the edits that solved the task or on probe-heavy steps nearby? Please also instrument the reward-hacking claims rather than arguing them: the rate of break-then-repair and of duplicate reads over training, for SWE-MILE versus outcome-only. Note that Eqs. 6/9/10 make break-then-repair exactly cancel, so the "naturally prevent reward hacking" claim currently holds only for the clipped whole-repository setting (Eq. 11).
  4. How is navigation potential to be computed in settings without gold-patch or dataset-annotation file sets, and what happens to performance when H is replaced by a noisy heuristic localisation set rather than the gold patch? Also give the precision and recall of the shell/AST search-and-read attribution against a hand-labelled sample (report both full-tool and Bash-only modes), since every navigation reward depends on it and the Bash-only result is currently attributed to reduced recall without recall ever being measured.
3 comments
AuthorsOpenReview ↗

Reply to Reviewer ZxXQ (Part 1/2)

Thank you for the thoughtful review. We address each concern below and will add all new results to the final version.

W1 & Q1: Checkpoint and Hyperparameter Selection

We agree that reporting only the best checkpoint (evaluated every 5 steps) is incomplete. We did so because long-horizon SWE RL is unstable, and using only the last step would penalize less stable methods. All baselines follow the same rule, so the bias is roughly symmetric. Figure 3 also shows SWE-MILE is stable within a moderate hyperparameter range (αnavi\alpha_{\text{navi}}: 0.01–0.1, αveri\alpha_{\text{veri}}: 0.1–0.4). In addition, Tables 2 and 3 show clear advantages of SWE-MILE over the baselines in response length and number of turns, which is a notable efficiency gain.

To show the performance advantage more directly, the table below reports the best checkpoint (avg@4), final checkpoint (avg@4), and the average of the last 10 steps (avg@1) for all methods. SWE-MILE leads clearly, and by a larger margin, under every metric, and method rankings is essentially unchanged. Estimating selection bias as Best-Last-10 gives 0.9–1.3 points on Verified and 0.5–1.2 points on Pro across methods. SWE-MILE's is 0.9 / 0.8, which is not unusually large.

MethodVerified Best / Final / Last-10Pro Best / Final / Last-10
Base Model53.6 / 53.6 / 53.630.8 / 30.8 / 30.8
GRPO58.2 / 57.9 / 57.333.9 / 33.9 / 33.4
G-RA57.9 / 57.2 / 56.834.1 / 33.7 / 33.2
SWE-TRACE60.5 / 59.4 / 59.236.6 / 35.9 / 35.4
GiGPO58.4 / 58.4 / 57.334.7 / 34.2 / 33.9
GraphGPO59.9 / 59.2 / 58.835.1 / 35.1 / 34.4
SWE-MILE63.8 / 63.1 / 62.939.3 / 38.8 / 38.5

We also trained on R2E-Gym with essentially the same hyperparameters. Since R2E-Gym has more samples, we changed only the batch size to 32 for faster training. Best and Final (avg@4) are reported below; due to time constraints, only some baselines are included. SWE-MILE again outperforms the baselines.

MethodVerified Best / FinalPro Best / Final
Base Model53.6 / 53.630.8 / 30.8
GRPO57.2 / 56.733.4 / 33.0
SWE-TRACE59.4 / 58.633.9 / 33.9
GraphGPO59.7 / 59.336.4 / 36.2
SWE-MILE63.1 / 62.738.6 / 38.6

W2 & Q2: Terminal Reward Rule

Rlim=t∗/∣T∣R_{\text{lim}}=t^{\ast}/|T| requires t∗t^{\ast}, the step at which the agent first completes the task, and t∗t^{\ast} can only be obtained via shadow probing. We therefore also enabled RlimR_{\text{lim}} for GiGPO and GraphGPO, which also use shadow probing. For GRPO, G-RA and SWE-TRACE, where t∗t^{\ast} is unavailable, this reward is uniformly set to 0.6, consistent with the outcome-only ablation.

RlimR_{\text{lim}} has little effect. Without it (uniformly 0.6), SWE-MILE reaches 63.4 / 39.0 on Verified / Pro, close to 63.8 / 39.3 for the full method, so we do not treat it as a main component. The reason is that it rarely triggers. In SWE-MILE training, the average fraction of rollouts exceeding the context limit is 0, and the fraction exceeding the turn limit is 0.04. RlimR_{\text{lim}} applies only to those that pass verification, so with an average outcome reward of about 0.5, only about 2% of rollouts are actually discounted. This fraction is too small to explain SWE-MILE's differences in performance and turns. The revision will state each method's RlimR_{\text{lim}} setting and add this ablation.

W3 & Q3: Credit Mass vs. Step-Level Attribution

Thank you for this analysis. Since rpotr^{\text{pot}} is a potential difference, the forward terms telescope to ΦT−Φ0\Phi_T-\Phi_0. The backward term amplifies each progress by 1+λγ(1−γμ−1)/(1−γ)1+\lambda \gamma (1-\gamma^{\mu-1})/(1-\gamma), which is at most 2.8 under our hyperparameters. Summed over a trajectory, SWE-MILE therefore contains a success-correlated partial-completion reward component. To isolate step-level attribution, we designed two controls that keep the total credit per trajectory unchanged:

  • Shuffle: compute CtprocessC_t^{\text{process}}, then randomly permute its allocation across steps within each trajectory, keeping the format reward at its original step. Total credit, value distribution and correlation with the final outcome are identical to SWE-MILE. The only thing removed is the link between credit and the step that caused the milestone change.
  • Partial-credit RLOO: fold the process signal into the terminal reward. For the kk-th rollout in a group, set Rk′=Rk+SkR_k'=R_k+S_k, where Sk=∑t(Ctprocess−η⋅rtfmt)S_k=\sum_t (C_t^{\text{process}}-\eta \cdot r_t^{\text{fmt}}), then apply standard RLOO without step-level credit.

All runs use the same data and hyperparameters as the main experiments:

Verified Best / FinalTurnsPro Best / FinalTurns
Shuffle61.2 / 60.691.637.1 / 36.989.1
Partial-credit RLOO60.9 / 60.3107.336.1 / 35.5112.7
SWE-MILE63.8 / 63.181.839.3 / 38.878.4

The results show that step-level attribution contributes the main gain. Aggregating process credit into the outcome reward (similar to SWE-TRACE) also helps to some extent.

AuthorsOpenReview ↗

Reply to Reviewer ZxXQ (Part 2/2)

W4: Group Centering of Process Credit

This is factually correct, but we do not see it as necessarily a flaw. SWE-MILE is designed to give positive or negative signals based on absolute progress or regression within a trajectory. For example, if a failed trajectory correctly localizes the issue or partially fixes it, credit should follow absolute progress without centering, so that those steps receive positive signals.

Moreover, step-level centering of CtprocessC_t^{\text{process}} runs into the problem that steps across trajectories have unmatched contexts or states. Forcing step-level normalization introduces length bias and contextual misalignment (as discussed in §2.5). That GiGPO/GraphGPO need compressed state definitions to match states on SWE tasks illustrates the same point.

Outcome-level centering is possible, e.g., C~k,t=Ck,tprocess−Sˉk/∣Tk∣\tilde C_{k,t} = C_{k,t}^{\text{process}}-\bar S_{k}/|T_k| with Sˉk=1K−1∑m≠k∑tCm,tprocess\bar S_{k} = \frac{1}{K-1}\sum_{m\neq k} \sum_t C^{\text{process}}_{m,t}. However, this turns absolute progress into relative progress. That conflicts with our design motivation: redundant actions in successful trajectories should receive negative signals, and correct localization or partial fixes in failed trajectories should receive positive ones.

On balance, we believe leaving the process credit uncentered is more reasonable.

W5 & Q3: Reward-Hacking Resistance

The reviewer is right that the forward potential differences of break-then-repair sum to zero over a trajectory, but this does not mean the pattern goes unpenalized. SWE-MILE's process credit acts on the policy gradient per step, not as a trajectory sum. The action that breaks a P2P test immediately receives negative credit −αveri(βany+βfracbt)-\alpha_{\text{veri}}(\beta_{\text{any}}+\beta_{\text{frac}}b_t), which directly lowers its probability, and more severe breakage is penalized more. Intuitively, in break-then-repair the breaking action comes first and the repair comes later, so suppressing the breaking action suppresses the pattern. This is exactly what our statement that such behavior "is penalized before receiving reward for the subsequent recovery" means.

In SWE-MILE, the rpotr^{\text{pot}} of any cycle back to the original state always sums to zero. The history-max constraint on the navigation potential also means repeated reads earn nothing. Hence there is no positive-reward loop to exploit repeatedly.

Repeated break-repair also consumes turns, which easily leads to task failure (outcome 0), and hitting the turn limit triggers the RlimR_{\text{lim}} discount. Hacking trajectories are therefore further suppressed at the outcome level. In whole-repository generation, asymmetric clipping (c−=6>c+=3c_-=6>c_+=3) makes the penalty for regression larger than the reward for recovery. Table 2 shows that SWE-MILE's average trajectory length and turns generally decrease relative to the base model, while the other baselines show varying increases in average turns.

Q4: Navigation Potential and Attribution Accuracy

(1) Obtaining HH. SWE training data mainly comes from GitHub issues and PRs, where gold patches usually exist and are easy to obtain, so HH is a zero-cost by-product of data construction. Mainstream SWE RL training sets (SWE-Gym, R2E-Gym, SWE-smith, SWE-rebench) all produce gold patches during construction, and some directly annotate relevant-file sets.

(2) Recall of search/read attribution. Shell/AST attribution does miss some search and read actions. Codeflow updates the navigation potential only with high-confidence evidence (Appendix B.3), so attribution errors mainly cause missed rewards rather than wrong rewards. Missed actions receive no reward, while searches and reads through dedicated tools are always attributed accurately and rewarded. As a result, the agent gradually shifts toward the dedicated tools during training: in SWE-MILE training, the share of search and read_file calls rises from 37.2% at the first step to 43.2% at the last. This also explains the slight drop in Bash-only mode. There the agent cannot switch to dedicated tools, so the navigation signal is weaker.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. We hope our responses have adequately addressed your concerns, and we would be happy to provide any further clarification if needed.

Thank you for your time and consideration!

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Rebuttal Summary

We thank all reviewers for their time and constructive feedback. Reviewers recognized SWE-MILE's environment-grounded process supervision, the practicality of asynchronous shadow probing, and its consistent gains in both performance and efficiency. During the rebuttal, we added experiments on checkpoint selection, credit-matched controls, terminal-reward fairness, training overhead, and multilingual training. Below we summarize the main concerns and how each was addressed.

Strengths

  1. SWE-MILE recovers the structure of the verifier that RLVR usually collapses into a binary pass/fail signal, and derives process supervision from workflow runtime without a reward model, an LLM judge, or extra inference cost (ZxXQ\color{red}{\text{ZxXQ}}, MNDe\color{blue}{\text{MNDe}}, WHRt\color{green}{\text{WHRt}}, xuDs\color{purple}{\text{xuDs}}).
  2. The asynchronous shadow sandbox is a practical contribution that hides most of the verification latency (MNDe\color{blue}{\text{MNDe}}, WHRt\color{green}{\text{WHRt}}).
  3. The reward design is simple and intuitive: only new progress is rewarded, so re-reading a file earns nothing and breaking code is penalized (MNDe\color{blue}{\text{MNDe}}, WHRt\color{green}{\text{WHRt}}).
  4. SWE-MILE achieves consistent improvements on all four benchmarks across two task types, with shorter responses and fewer turns, supported by comprehensive ablations (ZxXQ\color{red}{\text{ZxXQ}}, MNDe\color{blue}{\text{MNDe}}, WHRt\color{green}{\text{WHRt}}, xuDs\color{purple}{\text{xuDs}}).
  5. The paper addresses an important problem and is clearly written (ZxXQ\color{red}{\text{ZxXQ}}, WHRt\color{green}{\text{WHRt}}, xuDs\color{purple}{\text{xuDs}}).

Concerns and Author Responses

  1. Checkpoint and Hyperparameter Selection (ZxXQ\color{red}{\text{ZxXQ}}, MNDe\color{blue}{\text{MNDe}}, xuDs\color{purple}{\text{xuDs}})

Author response: We reported Best / Final / Last-10 results for all methods. SWE-MILE leads on every metric (Verified 63.8 / 63.1 / 62.9, Pro 39.3 / 38.8 / 38.5) over the strongest baseline (60.5 / 59.4 / 59.2, 36.6 / 35.9 / 35.4), with essentially unchanged rankings. Its selection bias (Best − Last-10: 0.9 / 0.8) is typical among all methods. Retraining on R2E-Gym with the same hyperparameters keeps SWE-MILE ahead (63.1 / 38.6 vs. 59.7 / 36.4). We also detailed the baseline settings, which share the same data, model, training and evaluation setup.

  1. Terminal Reward Rule (ZxXQ\color{red}{\text{ZxXQ}}, MNDe\color{blue}{\text{MNDe}})

Author response: RlimR_{\text{lim}} needs the first-completion step, which only shadow probing provides, so we also enabled it for GiGPO and GraphGPO. Removing it barely changes SWE-MILE (63.4 / 39.0 vs. 63.8 / 39.3), as only ~2% of rollouts are discounted.

  1. Credit Mass vs. Step-level Attribution (ZxXQ\color{red}{\text{ZxXQ}}, WHRt\color{green}{\text{WHRt}})

Author response: Two controls keep total credit per trajectory unchanged: Shuffle (permuting process credit across steps) and Partial-credit RLOO (folding it into the terminal reward). Both underperform SWE-MILE (61.2 / 37.1 and 60.9 / 36.1 vs. 63.8 / 39.3) and use more turns (91.6 and 107.3 vs. 81.8 on Verified), showing that step-level attribution drives the main gain. A trajectory case study and an offline replay further show that navigation credit lands on the steps exposing relevant files, and that such exposure is strongly linked to task success.

  1. Reward Hacking and Group Centering (ZxXQ\color{red}{\text{ZxXQ}}, xuDs\color{purple}{\text{xuDs}})

Author response: Any cycle back to the same state sums to zero and repeated reads earn nothing, so no positive-reward loop exists. In break-then-repair, the breaking action immediately receives negative per-step credit, and repeated cycles consume turns; empirically, SWE-MILE reduces turns. Process credit is intentionally uncentered to reward absolute progress, e.g., correct localization in failed trajectories.

  1. Verification Overhead (MNDe\color{blue}{\text{MNDe}})

Author response: Shadow probes run on idle CPUs with probe merging and 4 parallel sandboxes per rollout; 96.48% of rollouts finish all probes within the batch. Per-step time rises only 3.2% / 4.2% over outcome-only training.

  1. Gold-patch Dependency and Generality (ZxXQ\color{red}{\text{ZxXQ}}, xuDs\color{purple}{\text{xuDs}})

Author response: The relevant-file set is a zero-cost by-product of mainstream SWE RL datasets. On 4,738 tasks in Python, Go, TS and JS, SWE-MILE still outperforms all baselines (65.2 / 40.9 vs. 62.7 / 37.8).

Revisions

  1. Add Best / Final / Last-10, R2E-Gym, and multilingual results, plus baseline tuning details.
  2. Add the credit-matched controls, trajectory case study, and file-exposure replay analysis.
  3. State each method's terminal-reward setting and add the corresponding ablation.
  4. Add the training-overhead table, reward-hacking analysis, and discussion on credit centering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.