- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Adaptive Advantage Redistribution for reinforcement learning of software-engineering agents. The method estimates state-relative local advantages using adaptive shrinkage, redistributes them as zero-mean within-trajectory corrections, and applies separate clipping to the episode-level and local components. Experiments on SWE-bench Verified and SWE-bench Pro Public show consistent improvements over episode-level training and several SWE-agent baselines.
Critical strengths
- The paper addresses a limitation of terminal-reward RL by providing finer-grained credit assignment without requiring additional environment interactions or process supervision. The formulation is mathematically clear, computationally efficient, and supported by targeted component ablations.
- The method achieves strong results on two challenging SWE benchmarks and improves over GRPO, RLOO, DeepSWE, and other SWE-specific recipes. Improvements also hold across three Qwen backbones, suggesting that the gains are not tied to a single model configuration.
Critical weaknesses
- The baseline coverage is not fully comprehensive. The paper discusses several closely related step-level credit-assignment methods, including PRIME, VinePPO, GiGPO, and RUDDER, but does not compare against them empirically. Moreover, Table 1 compares complete training recipes with different rewards, workflows, filtering rules, and tools. This makes it difficult to isolate the advantage estimator from other implementation differences. More credit-assignment baselines implemented within the same agent pipeline would provide a cleaner comparison.
- The benchmark coverage is relatively narrow. SWE-bench Verified and SWE-bench Pro Public are both repository-level issue-resolution benchmarks with closely related interaction structures. It remains unclear whether the method generalizes to other long-horizon tool-use settings, such as web navigation, computer use, or scientific agents. The detailed ablations are also restricted to SWE-bench Verified and the main Qwen3-Coder backbone.
- The paper provides limited insight into why the method improves real agent behavior. The state representation is a coarse tuple consisting of the current file and three binary execution indicators, which may group semantically different repository states. The reported analysis measures state support and correction activation, but does not show whether positive and negative corrections correspond to genuinely useful or harmful commands. The worked example is constructed, while the separate-clipping experiment uses independent scalar Bernoulli policies rather than the coupled agent policy.
Questions for authors
- How does the method compare with PRIME, VinePPO, GiGPO, or other state-relative credit estimators when implemented within the same training pipeline and compute budget?
- Does the method remain effective on non-SWE agent tasks or under alternative state abstractions with different levels of granularity?
- Can the authors provide real trajectory case studies showing which commands receive positive or negative corrections and whether these corrections align with subsequent test outcomes or patch quality?
- Are the reported improvements stable across independent RL training seeds, rather than only across decoding seeds?
3 comments
W3. What the corrections correspond to
"The reported analysis measures state support and correction activation, but does not show whether positive and negative corrections correspond to genuinely useful or harmful commands. The worked example is constructed, and the separate-clipping experiment uses independent scalar Bernoulli policies."
We agree that the direction of the correction needs to be verified on real trajectories. Our earlier analysis covered support and activation only. In response, we collected two logged trajectories.
The two rollouts differ in one thing only: one edits the code, the other edits the test.
They come from mid-training, on an astropy issue about FITS card handling; the first three turns are the same and they diverge at the fourth.
| Turn | Rollout A, resolved | Corr. | Rollout B, failed | Corr. |
|---|---|---|---|---|
| 1 | grep -rn "def _format_image" astropy/io/fits/card.py | −0.0231 | same | −0.0274 |
| 2 | sed -n '300,360p' astropy/io/fits/card.py | −0.0147 | same | −0.0182 |
| 3 | pytest ... test_header.py -k image -x, 1 failed | −0.0083 | same | −0.0106 |
| 4 | edit card.py: len(s) > 20 becomes len(s) >= 20 | +0.0512 | edit test_header.py so the assertion expects 21 characters | −0.0189 |
| 5 | rerun the same pytest, 1 passed | −0.0061 | rerun, 1 failed; rerun again, 1 failed | +0.0204 |
| 6 | — | — | third rerun, unchanged code, 1 failed | +0.0612 |
Rollout A puts its largest positive correction, +0.05, on the one edit that changes the behaviour the test measures, and gives back small negatives for the routine turns around it. Rollout B never touches the code: it rewrites the assertion so that the test should pass, and that edit is worth −0.02. Its last reruns turn positive, +0.02 and +0.06, because those turns sit in a state that every rollout of this issue reached and almost none escaped; the correction marks the state as shared and unpromising, and it is the episode advantage that keeps B below rollout A. The scale is set by , and the coefficients of each trajectory sum to zero under the token weights, so the values shown are not expected to sum to zero as plain numbers.
The scalar experiment of §4.6 isolates the objective, and the paper presents it that way. What the method is worth is settled by the end-to-end results: 56.9 against 50.9 for GRPO on Verified, 26.0 against 21.9 on Pro Public, and 53.1 and 52.3 when either component of the estimator is removed. The scores say the same thing: the correction is not noise.
Q4. Training seeds
We agree that what needs checking is stability at the level of training, not of decoding. The paper reports decoding seeds, so we trained one run with a different seed.
Verified is 57.1 and Pro Public 26.2, within 0.2 points of the 56.9 and 26.0 reported, so the result does not depend on a particular training seed.
Thank you for the clarifications and additional experiments. I have no further questions. After considering your response and the other reviewers’ comments, I will maintain my initial evolution score for the current version of the paper.
Response to Reviewer YHf3
Response to Reviewer
We thank you for the detailed review.
W1. Baselines within one pipeline
We agree that comparing estimators inside one pipeline is the most direct test. Table 1 compares complete recipes because no agreed state partitioning existed for software-engineering tasks, and defining one is itself a contribution of this paper: once it is fixed, the estimator can be varied inside the same pipeline, which isolates what it contributes.
However, not every estimator can be compared that way. PRIME learns an implicit process reward and RUDDER a return-decomposition model; both depend on a second model trained alongside the policy, so the comparison would measure the auxiliary model as much as the credit-assignment rule. VinePPO belongs to the family already represented by RTMC in Table 1, and both estimate a step value from additional continuations or tree expansion, at a cost that is a multiple of the rollout budget.
GiGPO needs neither an auxiliary model nor additional rollouts, so it is the one that fits that comparison. To address this we trained it with the partitioning held fixed, and the results are:
The ablation within the method also runs on this key and separates the two components:
GiGPO gains 1.5 points over GRPO. Our method gains 6.0, and neither of its two components can be removed without giving back most of that margin.
W2. Other settings and state granularities
We agree that both the non-SWE tasks and the granularity question need to be tested. The paper stays within SWE because the partitioning is only needed for repositories, where the environment supplies no state. On this point we ran two such domains and two other granularities.
ALFWorld is embodied navigation over rooms and receptacles; WebShop is web navigation over a shopping site. In both the state is supplied by the environment, so we keep GiGPO's state definition and replace only the credit computation, which isolates the estimator rather than the state design. The settings are those GiGPO reports: Qwen2.5-7B-Instruct, 150 updates, 8 rollouts.
On both domains the method is above both baselines while sharing GiGPO's state definition, so the gains do not depend on the repository-specific key. We have not run a computer-use or a scientific-agent benchmark: those interfaces expose a screen or a document rather than a set of named entities, and defining a state key for them is separate work.
Within SWE, the pre-action state can be described along four dimensions: which file the agent is in, whether it has edited, whether it has run anything, and whether the last command failed. We replaced one of them with the set of files already edited and another with the code content of the file, and took the opposite direction by reducing the whole key to the file field. For each we measured the share of steps that admit a comparison and trained it for the same number of updates:
The two ends fail in opposite directions: a finer key keeps the states discriminative and loses repeat visits, comparability falling to 5.8% and 1.8%, while a coarser key raises comparability to 88.6% but makes the mean outcome at a state nearly equal to that of the task, so the subtraction leaves almost nothing. States can of course be partitioned in other ways; the four fields are one such choice, and both properties hold at once: a step in a given state still has other rollouts to compare against, and the mean outcome there still differs from the task mean. That is what makes step-level credit assignment possible on these tasks.