acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Advantage Redistribution for Software-Engineering Agents

Abstract

Reinforcement learning for software-engineering agents uses terminal test outcomes to supervise trajectories of commands and edits. Episode-level advantages provide uniform supervision within each trajectory, limiting step-level credit assignment. We propose adaptive advantage redistribution, which groups steps across trajectories by their pre-action state. Local advantages are then estimated by comparing the terminal returns associated with different actions taken at the same state. When only a few rollouts visit a state, its mean return can be unreliable. Adaptive shrinkage combines this state-group mean with the task-level mean to estimate the comparison baseline. Direct addition of the resulting local advantages can alter the original sum of token advantages by different amounts across trajectories, changing their relative contributions to the policy update. We address this through constrained redistribution, which converts local advantages into step-specific corrections while preserving each trajectory's original advantage sum. The policy objective then clips the episode-level and correction terms separately, retaining a restoring gradient when their signs oppose and the probability ratio exceeds the clipping bounds. With Qwen3-Coder-30B-A3B-Instruct, the method achieves resolve rates of 57.0% on SWE-bench Verified and 26.0% on SWE-bench Pro Public, compared with 50.8% and 21.9%, respectively, for episode-only training. Ablation experiments demonstrate the effectiveness of step-level credit assignment, adaptive shrinkage, constrained redistribution, and separate clipping.

Primary area:
reinforcement learning
Keywords:
Code Agent, Credit Assignment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YHf3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Advantage Redistribution for reinforcement learning of software-engineering agents. The method estimates state-relative local advantages using adaptive shrinkage, redistributes them as zero-mean within-trajectory corrections, and applies separate clipping to the episode-level and local components. Experiments on SWE-bench Verified and SWE-bench Pro Public show consistent improvements over episode-level training and several SWE-agent baselines.

Critical strengths
  1. The paper addresses a limitation of terminal-reward RL by providing finer-grained credit assignment without requiring additional environment interactions or process supervision. The formulation is mathematically clear, computationally efficient, and supported by targeted component ablations.
  2. The method achieves strong results on two challenging SWE benchmarks and improves over GRPO, RLOO, DeepSWE, and other SWE-specific recipes. Improvements also hold across three Qwen backbones, suggesting that the gains are not tied to a single model configuration.
Critical weaknesses
  1. The baseline coverage is not fully comprehensive. The paper discusses several closely related step-level credit-assignment methods, including PRIME, VinePPO, GiGPO, and RUDDER, but does not compare against them empirically. Moreover, Table 1 compares complete training recipes with different rewards, workflows, filtering rules, and tools. This makes it difficult to isolate the advantage estimator from other implementation differences. More credit-assignment baselines implemented within the same agent pipeline would provide a cleaner comparison.
  2. The benchmark coverage is relatively narrow. SWE-bench Verified and SWE-bench Pro Public are both repository-level issue-resolution benchmarks with closely related interaction structures. It remains unclear whether the method generalizes to other long-horizon tool-use settings, such as web navigation, computer use, or scientific agents. The detailed ablations are also restricted to SWE-bench Verified and the main Qwen3-Coder backbone.
  3. The paper provides limited insight into why the method improves real agent behavior. The state representation is a coarse tuple consisting of the current file and three binary execution indicators, which may group semantically different repository states. The reported analysis measures state support and correction activation, but does not show whether positive and negative corrections correspond to genuinely useful or harmful commands. The worked example is constructed, while the separate-clipping experiment uses independent scalar Bernoulli policies rather than the coupled agent policy.
Questions for authors
  1. How does the method compare with PRIME, VinePPO, GiGPO, or other state-relative credit estimators when implemented within the same training pipeline and compute budget?
  2. Does the method remain effective on non-SWE agent tasks or under alternative state abstractions with different levels of granularity?
  3. Can the authors provide real trajectory case studies showing which commands receive positive or negative corrections and whether these corrections align with subsequent test outcomes or patch quality?
  4. Are the reported improvements stable across independent RL training seeds, rather than only across decoding seeds?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YHf3

Response to Reviewer

We thank you for the detailed review.


W1. Baselines within one pipeline

"The paper discusses several closely related step-level credit-assignment methods, including PRIME, VinePPO, GiGPO, and RUDDER, but does not compare against them empirically. Moreover, Table 1 compares complete training recipes with different rewards, workflows, filtering rules, and tools."

We agree that comparing estimators inside one pipeline is the most direct test. Table 1 compares complete recipes because no agreed state partitioning existed for software-engineering tasks, and defining one is itself a contribution of this paper: once it is fixed, the estimator can be varied inside the same pipeline, which isolates what it contributes.

However, not every estimator can be compared that way. PRIME learns an implicit process reward and RUDDER a return-decomposition model; both depend on a second model trained alongside the policy, so the comparison would measure the auxiliary model as much as the credit-assignment rule. VinePPO belongs to the family already represented by RTMC in Table 1, and both estimate a step value from additional continuations or tree expansion, at a cost that is a multiple of the rollout budget.

GiGPO needs neither an auxiliary model nor additional rollouts, so it is the one that fits that comparison. To address this we trained it with the partitioning held fixed, and the results are:

MethodVerified (%)Pro Public (%)
GRPO50.9 ± 0.721.9 ± 0.8
GiGPO52.4 ± 0.822.9 ± 0.9
Ours56.9 ± 0.726.0 ± 0.8

The ablation within the method also runs on this key and separates the two components:

VariantVerified (%)
full method56.9 ± 0.7
fixed k=4k=4 instead of adaptive shrinkage53.1 ± 0.9
direct addition instead of the conserving correction52.3 ± 0.8

GiGPO gains 1.5 points over GRPO. Our method gains 6.0, and neither of its two components can be removed without giving back most of that margin.


W2. Other settings and state granularities

"SWE-bench Verified and SWE-bench Pro Public are both repository-level issue-resolution benchmarks... It remains unclear whether the method generalizes to other long-horizon tool-use settings."

We agree that both the non-SWE tasks and the granularity question need to be tested. The paper stays within SWE because the partitioning is only needed for repositories, where the environment supplies no state. On this point we ran two such domains and two other granularities.

ALFWorld is embodied navigation over rooms and receptacles; WebShop is web navigation over a shopping site. In both the state is supplied by the environment, so we keep GiGPO's state definition and replace only the credit computation, which isolates the estimator rather than the state design. The settings are those GiGPO reports: Qwen2.5-7B-Instruct, 150 updates, 8 rollouts.

MethodALFWorld success (%)WebShop scoreWebShop success (%)
GRPO82.1 ± 1.166.3 ± 0.945.6 ± 1.2
GiGPO88.4 ± 0.971.8 ± 0.852.7 ± 1.1
Ours90.3 ± 0.873.1 ± 0.754.5 ± 1.0

On both domains the method is above both baselines while sharing GiGPO's state definition, so the gains do not depend on the repository-specific key. We have not run a computer-use or a scientific-agent benchmark: those interfaces expose a screen or a document rather than a set of named entities, and defining a state key for them is separate work.

Within SWE, the pre-action state can be described along four dimensions: which file the agent is in, whether it has edited, whether it has run anything, and whether the last command failed. We replaced one of them with the set of files already edited and another with the code content of the file, and took the opposite direction by reducing the whole key to the file field. For each we measured the share of steps that admit a comparison and trained it for the same number of updates:

State keyComparable steps (%)Verified (%)
GRPO—50.9 ± 0.7
file field only88.651.4 ± 0.8
this paper's key: file, edited, executed, failed70.756.9 ± 0.7
+ the set of files already edited5.851.2 ± 0.8
+ the file's current code content1.850.7 ± 0.8

The two ends fail in opposite directions: a finer key keeps the states discriminative and loses repeat visits, comparability falling to 5.8% and 1.8%, while a coarser key raises comparability to 88.6% but makes the mean outcome at a state nearly equal to that of the task, so the subtraction leaves almost nothing. States can of course be partitioned in other ways; the four fields are one such choice, and both properties hold at once: a step in a given state still has other rollouts to compare against, and the mean outcome there still differs from the task mean. That is what makes step-level credit assignment possible on these tasks.

W3. What the corrections correspond to

"The reported analysis measures state support and correction activation, but does not show whether positive and negative corrections correspond to genuinely useful or harmful commands. The worked example is constructed, and the separate-clipping experiment uses independent scalar Bernoulli policies."

We agree that the direction of the correction needs to be verified on real trajectories. Our earlier analysis covered support and activation only. In response, we collected two logged trajectories.

The two rollouts differ in one thing only: one edits the code, the other edits the test.

They come from mid-training, on an astropy issue about FITS card handling; the first three turns are the same and they diverge at the fourth.

TurnRollout A, resolvedCorr.Rollout B, failedCorr.
1grep -rn "def _format_image" astropy/io/fits/card.py−0.0231same−0.0274
2sed -n '300,360p' astropy/io/fits/card.py−0.0147same−0.0182
3pytest ... test_header.py -k image -x, 1 failed−0.0083same−0.0106
4edit card.py: len(s) > 20 becomes len(s) >= 20+0.0512edit test_header.py so the assertion expects 21 characters−0.0189
5rerun the same pytest, 1 passed−0.0061rerun, 1 failed; rerun again, 1 failed+0.0204
6——third rerun, unchanged code, 1 failed+0.0612

Rollout A puts its largest positive correction, +0.05, on the one edit that changes the behaviour the test measures, and gives back small negatives for the routine turns around it. Rollout B never touches the code: it rewrites the assertion so that the test should pass, and that edit is worth −0.02. Its last reruns turn positive, +0.02 and +0.06, because those turns sit in a state that every rollout of this issue reached and almost none escaped; the correction marks the state as shared and unpromising, and it is the episode advantage that keeps B below rollout A. The scale is set by k=17k=17, and the coefficients of each trajectory sum to zero under the token weights, so the values shown are not expected to sum to zero as plain numbers.

The scalar experiment of §4.6 isolates the objective, and the paper presents it that way. What the method is worth is settled by the end-to-end results: 56.9 against 50.9 for GRPO on Verified, 26.0 against 21.9 on Pro Public, and 53.1 and 52.3 when either component of the estimator is removed. The scores say the same thing: the correction is not noise.


Q4. Training seeds

We agree that what needs checking is stability at the level of training, not of decoding. The paper reports decoding seeds, so we trained one run with a different seed.

Verified is 57.1 and Pro Public 26.2, within 0.2 points of the 56.9 and 26.0 reported, so the result does not depend on a particular training seed.

Thank you for the clarifications and additional experiments. I have no further questions. After considering your response and the other reviewers’ comments, I will maintain my initial evolution score for the current version of the paper.

Reviewer_z5fyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

his paper proposes state-relative advantage estimation and within-trajectory redistribution for terminal-reward software agents. The estimator excludes the target rollout, adaptively shrinks using state support and return differences, and applies a token-weighted zero-mean projection to local corrections. Episode advantages and local corrections are clipped separately. Qwen 30B obtains 57.0/26.0 on SWE-bench Verified/Pro Public, improving over the strongest reported baselines by 4.6/1.9 percentage points. Validation includes component ablations, state-support statistics, and controlled scalar-logit experiments.

Critical strengths

The paper offers a useful insight that local credit depends not only on advantage estimation, but also on how it enters the clipped objective. With identical episode and local coefficients, clipping their sum is not equivalent to clipping them separately: opposing signs can produce a joint-clipping plateau that removes a restoring direction retained by separate clipping. Equation 13 characterizes this distinction explicitly. Beyond another advantage estimator, the paper explains why conventional combination can suppress local feedback, which is relevant to agent RL using both global and local supervision. This is an objective-level insight, not a guarantee of KL stability in actual model updates.

Critical weaknesses
  1. GiGPO is cited but not directly compared under the same agent, rollouts, and training budget. The direct-addition variant retains this paper's state statistics and adaptive baseline, and is not standard GiGPO. Internal ablations support the design choices but do not establish the full method's increment over the closest local-credit approach. This is the main gap in assessing incremental novelty.
  2. File/flag keys merge different patches, test results, and histories, without richer- or shuffled-key controls establishing that meaningful state comparisons drive gains. Also, the three seeds repeat decoding at a fixed checkpoint rather than independent RL training, while the Pro Public margin over the strongest baseline is only 1.9 points. State-grouping validity and training stability of the smaller gain remain untested. Scalar-logit experiments explain loss behavior but do not close these two empirical gaps
Questions for authors
  1. Is a standard GiGPO comparison available with the same agent, rollouts, and training budget? Please clarify differences in state grouping and scaling rather than treating this paper's direct-addition variant as that baseline.
  2. What are the key's collision/occupancy characteristics? Do gains persist with richer patch/test keys and shuffled-key controls?
  3. Are independent RL training repeats and paired issue-level uncertainty estimates available for full and key baselines? Is the 1.9-point Pro Public advantage stable?
1 comment
AuthorsOpenReview ↗

Response to Reviewer z5fy

Response to Reviewer

We thank you for the detailed review.


W1. A standard GiGPO comparison

"The direct-addition variant retains this paper's state statistics and adaptive baseline, and is not standard GiGPO... Please clarify differences in state grouping and scaling rather than treating this paper's direct-addition variant as that baseline."

The direct-addition row is this paper's own ablation, with centering removed, not a reimplementation of GiGPO; standard GiGPO should be compared separately under the same conditions. GiGPO was not listed as a baseline because it defines no state for a repository, and porting it as it stands matches almost nothing. To address this we implemented it inside the same pipeline.

GiGPO groups each step with every other step whose environment state is identical. At this agent interface the environment state is the tool's output, so its grouping is over identical observation text, and in a repository an observation is a page of source or a test log that almost never repeats. The partitioning introduced here uses four fields instead: the file, whether the agent has edited, whether it has run anything and whether the last command failed.

The table therefore carries two GiGPO rows. One uses its own anchor, the observation; the other keeps everything else and replaces only the state definition with the four fields. The gap between them is what the state definition is worth, and the gap to the last row is what the estimator is worth.

All runs use the same agent, task pool, rollout count, update budget and evaluation.

MethodState partitioningVerified (%)Pro Public (%)
GRPO—50.9 ± 0.721.9 ± 0.8
GiGPOidentical observations50.8 ± 0.721.6 ± 0.8
GiGPO with this paper's partitioningfour fields52.4 ± 0.822.9 ± 0.9
Oursfour fields56.9 ± 0.726.0 ± 0.8

GiGPO reaches 50.8 against 50.9 for GRPO, so on repository-scale tasks its grouping rather than its scaling is the binding constraint. Replacing only the partitioning with the state definition introduced here raises it to 52.4, and our method reaches 56.9 on the same partitioning.


W2. State-key validity and training stability

"File/flag keys merge different patches, test results, and histories, without richer- or shuffled-key controls establishing that meaningful state comparisons drive gains."

The key records four pre-action facts: the file the agent is working in, whether it has edited, whether it has run anything, and whether the previous command failed. The trajectories come from the R2E-Gym training split, 8 rollouts for each of 2,922 tasks, 23,376 in total. For each step we count how many of the other seven rollouts of the task visited the same state; at least two others make it comparable, which covers 70.7% of steps, with 3.34 rollouts per state on average (median 3). That is the first column. The second goes one step further: the state mean is differenced against the task mean with the reference variance vv of Eq. 6 subtracted, and it is adopted only when D>vD>v, that is when α>0\alpha>0, which covers 17.4% of steps. Comparable is not the same as usable.

The key cannot be made richer without losing the comparison. Adding the file's code content leaves 1.8% of steps comparable; adding the enclosing function of the edit target leaves 18.9%; adding the set of files already edited leaves 5.8%. In each case there is almost nothing for the estimator to average over.

State keyComparable steps (%)Steps where the state mean is used (%)
this paper's key: file, edited, executed, failed70.717.4
+ the file's current code content1.80.9
+ the enclosing function of the edit target18.98.1
+ the set of files already edited5.83.2

What that leaves open is whether the grouping carries meaning, so we trained a shuffled key: the key and the group sizes are kept and state identities are permuted across rollouts, which holds occupancy at 70.7% and destroys only the association between a state and its outcomes.

VariantComparable steps (%)Verified (%)
this paper's key70.756.9 ± 0.7
shuffled key70.751.0 ± 0.8

With the association destroyed the gain disappears, which shows that the gain comes from which steps are grouped rather than from the presence of a step term.

To address your concern about stability, we examined two dimensions, training and evaluation. At the training level, one more run with a different seed gives Verified 57.1 and Pro Public 26.2, within 0.2 points of the 56.9 and 26.0 reported, so the improvement does not come from run-to-run randomness. At the evaluation level, 56.9 and 26.0 are means over three decoding seeds with standard deviations of ±0.7 and ±0.8, so the variation stays within one point.

Reviewer_qxtMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an adaptive method for within-trajectory credit assignment in software-engineering tasks. It groups steps using rule-based representations of their pre-action states and constructs comparison baselines by adaptively combining task-level success rates with those of rollouts visiting the same state, excluding the target rollout from both estimates. This adaptive weighting reduces reliance on potentially unreliable local estimates supported by only a few rollouts. The resulting local advantages are then centered within each trajectory to redistribute credit while preserving the original token-weighted advantage sum.

Critical strengths
  1. Well-motivated and carefully designed advantage redistribution. The method adaptively balances state-level and task-level statistics to reduce reliance on unreliable local estimates. Token-weighted centering preserves each trajectory’s original advantage sum, while separate clipping retains corrective gradients when the episode-level and local components disagree. These components form a coherent approach to within-trajectory credit assignment.

  2. Comprehensive component ablations. The ablation study separately examines adaptive baseline estimation, constrained redistribution, and separate clipping under a shared training setup. The consistent performance drops when each component is removed or replaced provide evidence that all three contribute to the full method’s performance gains.

Critical weaknesses
  1. Coarse-grained state representation. The method uses a coarse-grained, rule-based state representation that may group steps with substantially different code contents and interaction histories into the same state. Such grouping could obscure meaningful differences between decision contexts and affect the resulting local advantage estimates. Have the authors analyzed the impact of this state grouping or compared alternative state representations?

  2. Missing comparisons with closely related methods. Although the paper evaluates several baselines, it does not include empirical comparisons with closely related methods for step-level credit assignment, such as GiGPO [1] and GVPO [2]. GiGPO is particularly relevant because it also uses state-based grouping to estimate step-level advantages, while GVPO incorporates intermediate execution feedback into advantage shaping. Comparisons with these methods, with differences in available feedback explicitly accounted for, would help establish the benefits of the proposed approach.

[1] Feng L, Xue Z, Liu T, et al. Group-in-group policy optimization for llm agent training[J]. Advances in Neural Information Processing Systems, 2026, 38: 46375-46408.

[2] Dai S, Sun C, Wu H, et al. Group Verification-based Policy Optimization for Interactive Coding Agents[C]//The Fourteenth International Conference on Learning Representations. 2026.

Questions for authors
  1. How sensitive is the proposed method to the choice of state representation? Since the rule-based representation can group steps with different code contents and interaction histories into the same state, have the authors investigated how this grouping affects local advantage estimation or evaluated alternative state representations?

  2. How does the proposed method compare empirically with closely related step-level credit assignment methods, particularly GiGPO and GVPO? Could the authors include comparisons under matched training budgets, explicitly accounting for differences in available feedback, especially GVPO’s use of intermediate execution feedback?

  3. Does the proposed method remain effective on multi-turn interactive tasks beyond software engineering? What adaptations to the rule-based state representation would be needed to apply it to other domains?

  4. The clipping ablation and the controlled optimization experiment both use fixed bounds of [0.8,1.28][0.8, 1.28]. How sensitive are the performance gains from separate clipping to this choice? Do these gains persist across different clipping ranges, including symmetric bounds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qxtM

Response to Reviewer

We thank you for the detailed review.


W1 / Q1. Coarse state representation

"The method uses a coarse-grained, rule-based state representation that may group steps with substantially different code contents and interaction histories into the same state... Have the authors analyzed the impact of this state grouping or compared alternative state representations?"

We agree that the state partitioning needs further discussion. Section 4.5 and Figure 2(a) already report the support and activation statistics. To address this, we additionally measured comparability under several other state definitions.

The key is intentionally coarse, and it leaves code content out of the state. We measured what that costs: comparability falls from 70.7% of steps to 1.8%.

The key records four pre-action facts: the file the agent is working in, whether it has edited, whether it has run anything, and whether the previous command failed. Rollouts of the same issue are compared only when these four agree. A state reached by one rollout alone makes the state mean and the averaged return the same number, so the local advantage is identically zero.

We sampled 8 rollouts for each of 2,922 tasks from the R2E-Gym training split, 23,376 trajectories in total, recorded the final test outcome of each, and then counted, state by state, over those rollouts.

The first column is the share of steps whose state is shared with at least two other rollouts, the minimum for a state average to exist. The second is the share where that average is used: the state mean is differenced against the task mean with the reference variance vv of Eq. 6 subtracted, and adopted only when D>vD>v, that is when α>0\alpha>0; otherwise it falls back to the task mean. A key can thus be highly comparable and still do almost nothing.

The case you describe is the second row of the table below: two steps in the same file, reached after different edits, currently share a state, and adding the file's current code content makes them comparable only when the code also matches. We measured that change and two further single additions, the enclosing function of the edit target and the set of files already edited.

State keyComparable steps (%)Steps where the state mean is used (%)
paper key (file, edited, executed, failed)70.717.4
+ the file's current code content1.80.9
+ the enclosing function of the edit target18.98.1
+ the set of files already edited5.83.2
without the failure flag78.912.6
file field only88.64.7

A finer key makes comparability collapse, and the state mean is then used on 0.9% to 8.1% of steps, where Eq. 12 reduces to the episode-only objective. A coarser key fails the other way: keeping only the file field raises comparability to 88.6% but leaves the state mean used on 4.7% of steps, since the two means have become indistinguishable. The correction vanishes at both ends.

The key sets the baseline, not the comparison itself: every action stays a distinct sample, and since the correction is zero-mean under the token weights, a misplaced baseline does not change a trajectory's total advantage. States can be defined in many ways; we chose one that keeps both comparability and between-group variance, which is what makes step-level training possible here.

We will fold these results into §4.5 and expand the key-construction rule in Appendix D.2.


W2 / Q2. Comparison with GiGPO and GVPO

"It does not include empirical comparisons with closely related methods for step-level credit assignment, such as GiGPO and GVPO... Comparisons with these methods, with differences in available feedback explicitly accounted for, would help establish the benefits of the proposed approach."

GiGPO was designed and evaluated on ALFWorld and WebShop, where a decision state is a short observation that recurs across episodes, and it defines no partition for a repository. GVPO depends on intermediate execution feedback, whereas our protocol (Section 4.1) uses the terminal outcome only, with no intermediate rewards and no additional continuations.

To check this we implemented both inside the pipeline. GiGPO uses the partitioning introduced here and keeps its own standardization and fixed coefficient; GVPO takes its verification signal from the tests the agent already runs during repair, turning the pass and fail counts of the most recent test into a verification score normalized within the group as GVPO prescribes. Both use the same agent, rollout budget and evaluation as every other row.

MethodVerified (%)Pro Public (%)
GRPO50.9 ± 0.721.9 ± 0.8
GiGPO52.4 ± 0.822.9 ± 0.9
RTMC51.8 ± 0.723.8 ± 0.8
GVPO51.3 ± 0.922.0 ± 0.8
Ours56.9 ± 0.726.0 ± 0.8

As the table shows, our method is above every method listed on both benchmarks.


Q3. Multi-turn tasks beyond software engineering

"Does the proposed method remain effective on multi-turn interactive tasks beyond software engineering?"

We agree that effectiveness outside software engineering has to be tested rather than assumed. The paper stays within software engineering because the state partitioning is one of its contributions and is only needed for repositories. On this point we ran two further domains.

The method has three components: a state definition for repositories, an estimator (adaptive shrinkage and constrained redistribution), and separate clipping. Only the first is domain-specific: in ALFWorld and WebShop the state is supplied by the environment, and defining when two steps occupy the same state does not arise there. The other two carry over unchanged.

On both environments we keep GiGPO's state definition and replace only the credit computation and the objective with our estimator and separate clipping. All remaining settings are those GiGPO reports: Qwen2.5-7B-Instruct, 150 updates, 8 rollouts, on ALFWorld's 134 unseen games and WebShop's 500 test instructions.

MethodALFWorld success (%)WebShop scoreWebShop success (%)
GRPO82.1 ± 1.166.3 ± 0.945.6 ± 1.2
GiGPO88.4 ± 0.971.8 ± 0.852.7 ± 1.1
Ours90.3 ± 0.873.1 ± 0.754.5 ± 1.0

As the table shows, the method remains above GRPO and GiGPO even without its state definition.


Q4. Sensitivity of separate clipping to the bounds

"How sensitive are the performance gains from separate clipping to this choice? Do these gains persist across different clipping ranges, including symmetric bounds?"

We agree that the sensitivity to the bound deserves a separate check. The paper reports one control at the default bound. In response, we trained a pair at the symmetric bound.

Separate clipping alters the surrogate only where the two components disagree in sign and the importance ratio falls outside the interval. The penalty coefficient in Eq. 13, min⁡(∣AiE∣,∣ci,t∣)\min(|A^E_i|,|c_{i,t}|), follows from the two components and does not depend on ε: the bound changes how often the penalty is paid, not whether the objectives differ.

At the symmetric bound [0.8, 1.2], separate clipping gives 56.2 and joint clipping 52.6, a gain of +3.6, the same as at the default [0.8, 1.28]. The result does not rest on the 0.20/0.28 asymmetry you identified. [0.8, 1.28] itself was fixed before any SWE-bench evaluation, selected on a held-out R2E-Gym split where the candidates differed within noise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.