acceptodds
Under review as a conference paper at ICLR 2027

AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning

10 views

Abstract

Long-horizon LLM agents are typically trained with sparse outcome rewards, making trajectory-level objectives too coarse to distinguish the contribution of individual decisions. Step-level credit assignment provides finer-grained supervision, but its estimates can be unreliable because observed returns also depend on subsequent actions, environment transitions, and trajectory length. We propose **AdaStep**, an **Ada**ptive **Step**-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal. We formulate this weighting as a mean-squared-error estimation problem for the latent step advantage and, under an explicit conditional sampling assumption, derive an optimal per-state shrinkage coefficient. The coefficient admits a **signal-to-total-variance** interpretation: it preserves local credit when return variation is attributable to the selected action and suppresses it when variation is dominated by downstream randomness. AdaStep requires only lightweight scalar computation, with no critic, additional rollouts, or extra model inference. Experiments with three model backbones on ALFWorld, WebShop, and ScienceWorld show **consistent improvements** over baselines at low computational cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_twYwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AdaStep, which adapts the strength of GiGPO's step-level advantage correction using an estimated ratio of action-dependent return variance to total return variance. From limited theoretical lens, author gives some intuitive explanation on calibration effect on the advantage estimation. Empirical studies across three model backbones and three agent benchmarks report consistent improvements with minimal additional credit-computation cost.

Critical strengths

1, This method addresses a clear limitation of fixed-weight group-based credit assignment methods like GiGPO and it is simple to integrate into existing training pipelines.
2, The theoretical explanation on weighted factor derivation is intuitive coming with some insightful explanation.
3, Empirical study is comprehensive with multiple size model backbones, different RL algos, and some ablation studies.

Critical weaknesses

1, In line 248, it would be great to clearly define what is the ground-truth step-level advantage function by moving Eq 7 ahead.
2, It would be great to see AdaStep performance and detailed analysis in long horizon agentic tasks and coding/math reasoning tasks where rewards are more sparse and verifiable.
3, The theoretical induction assumes independent action-return pairs conditioned on the same state. However, groups can include repeated visits from the same trajectory, while policy prompts contain history and step-count info that may differ across identical anchor observations.
4, It would be better to provide empirical justification in terms of less computation needed for the AdaStep comparing with other Group-based baselines other than GiGPO.

Questions for authors

1, What is the nature of noise \eta defined in Equation 3? Or what is the underlying intuition/theory to construct Eq 3 here?
2, What exactly constitutes an achor-state key, and how do you handle differing histories, remaining horizons, and repeated visits within one trajectory?
3, How sensitive is the coefficient to group size and action counts? Could a bias-corrected or cross-fitted estimator improve its reliability?

8 comments
AuthorsOpenReview ↗

Author Response (Part 1)


W1 & Q1: Move Eq. 7 forward to clearly define the ground-truth step-level advantage function; clarify the nature of the noise \eta in Eq. 3 and the intuition/theory behind its construction.

Thank you for the comment. We have clarified the following three points.

  • Definition of the true advantage. At the beginning of Section 4.2, we now explicitly define
=Q(\boldsymbol{s},\boldsymbol{a})-V(\boldsymbol{s}),$$ where $Q(\boldsymbol{s},\boldsymbol{a})=\mathbb{E}_{\pi}[R\mid\boldsymbol{s},\boldsymbol{a}] , V(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\sim\pi(\cdot\mid\boldsymbol{s})}[Q(\boldsymbol{s},\boldsymbol{a})]$. - **Nature of the noise.** GiGPO estimates the step-level advantage using sampled returns and a group-mean baseline: $$\begin{aligned} A_S(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) &=R(\boldsymbol{s},\boldsymbol{a}_t^{(j)})-\hat V_{N_s}(\boldsymbol{s}),\quad \hat V_{N_s}(\boldsymbol{s}) =\frac{1}{N_s}\sum_{(j',t')\in\mathcal{G}(\boldsymbol{s})} R(\boldsymbol{s},\boldsymbol{a}_{t'}^{(j')}). \end{aligned}$$ The noise is the difference between this estimate and the true advantage: $$\begin{aligned} \eta_t^{(j)} &=A_S(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) -A_S^{*}(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) =\bigl(R(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) -Q(\boldsymbol{s},\boldsymbol{a}_t^{(j)})\bigr) -\bigl(\hat V_{N_s}(\boldsymbol{s})-V(\boldsymbol{s})\bigr). \end{aligned}$$ It therefore captures estimation errors in both the sampled return and the baseline. - **Intuition and theoretical rationale.** We observe that GiGPO may assign negative advantages to correct state–action pairs and positive advantages to incorrect ones. We illustrate this issue in Figure 1, provide two concrete examples as motivation in Section 4.1, and present detailed rollout trajectories in Appendix L. Based on the theoretical derivation above, we attribute this issue to estimation error, which can be viewed as noise. Appendix L further demonstrates that our coefficient can correct the resulting advantages: for example, a correct action receiving a negative advantage under GiGPO receives a positive final advantage after applying our method. *** **`W2: It would be great to see AdaStep performance and detailed analysis in long horizon agentic tasks and coding/math reasoning tasks where rewards are more sparse and verifiable.`** To evaluate AdaStep on more demanding long-horizon tasks, we extended our ScienceWorld experiments to include tasks with expert trajectories of at most 70 steps and allowed up to 100 environment interactions per episode during training. With Qwen2.5-14B-Instruct, AdaStep achieves **52.07 ± 1.8%** success compared with **49.73 ± 1.7%** for GiGPO, a gain of **2.34 percentage points**. It also reduces the average validation episode length from **44.54 ± 1.8** to **35.72 ± 2.2** environment steps, a reduction of **8.82 steps**. These results are reported in **Appendix G.2, Table 11**, and the training curves are shown in **Figure 7(b)**. | Method | Success (%) | Average episode length (environment steps) | | --- | --- | --- | | GiGPO | 49.73 ± 1.7 | 44.54 ± 1.8 | | **AdaStep (Ours)** | **52.07 ± 1.8** | **35.72 ± 2.2** | | **Δ vs. GiGPO** | **+2.34** | **−8.82** | The success rate and episode length are measured at the training step with the highest seed-averaged validation success rate, consistent with **Appendix G.2, Table 11**. AdaStep focuses on agentic reinforcement learning, where an LLM interacts with an environment. Like GiGPO, it uses recurring environment observations as anchor states to group actions and their returns across rollouts of the same task. Extending the current formulation to standalone coding or mathematical reasoning would require a way to define comparable anchor states across rollouts.
AuthorsOpenReview ↗

Author Response (Part 2)


W3: Groups can include repeated visits from the same trajectory, while policy prompts contain history and step-count info that may differ across identical anchor observations.

In the revised manuscript, we add two complementary ablations: the first tests AdaStep in a practical setting satisfying the sampling assumptions; the second compares weighting strategies in practice.

1. First-visit grouping. Both methods retain only the first occurrence of each anchor state per trajectory. Under a fixed task and policy, independently sampled trajectories yield i.i.d. first-visit action–return pairs conditional on reaching that state. On WebShop with Qwen2.5-7B-Instruct, AdaStep improves score by 2.23 points and success by 3.90 percentage points over GiGPO under this protocol (Section 5.4, Table 4).

MethodScoreSuccess (%)
GiGPO86.57 ± 0.675.50 ± 1.9
AdaStep88.80 ± 1.479.40 ± 1.9

2. Revisit-gate ablation. Under the original grouping, we set the coefficient to one for groups with within-trajectory revisits, reverting them to GiGPO’s fixed weighting. AdaStep achieves higher mean success than this gated variant. The higher mean success supports retaining adaptive weighting in groups containing revisits (Section 5.4, Table 5; Appendix B).

ModelMethodALFWorld In-Suc. (%)ALFWorld Out-Suc. (%)WebShop ScoreWebShop Suc. (%)
Qwen3-4BAdaStep93.01 ± 0.986.58 ± 1.288.76 ± 0.580.27 ± 0.4
Qwen3-4BAdaStep + gate86.93 ± 1.584.63 ± 1.291.13 ± 1.378.93 ± 2.2
Qwen2.5-7B-InstructAdaStep97.66 ± 0.893.30 ± 1.390.46 ± 0.482.42 ± 1.2
Qwen2.5-7B-InstructAdaStep + gate93.23 ± 1.888.77 ± 1.890.13 ± 1.378.67 ± 2.0

3. History-conditioned prompts. The anchor-conditioned action distribution is obtained by marginalizing over the histories and step counts induced by the rollout and grouping protocol; corresponding action and state values are conditional return expectations under that distribution. This permits prompt contexts to differ across identical anchor observations. As in GiGPO and HGPO, state-history context includes only the current and immediately two preceding states.


W4: It would be better to provide empirical justification in terms of less computation needed for the AdaStep comparing with other Group-based baselines other than GiGPO.

Table 6 reports empirical measurements of the time required to compute step-level advantages. AdaStep increases GiGPO’s computation time by only 1.0% while improving success by 4.85 percentage points; it is also faster than HGPO with higher success.

MethodTime (s)Success (%)
GiGPO0.4968 ± 0.032477.57±4.1
HGPO0.6340 ± 0.071178.51 ± 1.4
AdaStep0.5018 ± 0.011482.42 ± 1.2

Appendix H analyzes the time and space complexity for GiGPO, HGPO, and AdaStep, where MM is the total number of sampled steps and KK is HGPO’s history depth. AdaStep matches GiGPO in both time and peak additional space complexity.

Method / implementationExpected timePeak additional space
GiGPOO(M)O(M)O(M)O(M)
AdaStepO(M)O(M)O(M)O(M)
HGPO (all layers stored)O(M(K+1))O(M(K+1))O(M(K+1))O(M(K+1))
HGPO (layer by layer)O(M(K+1))O(M(K+1))O(M)O(M)
AuthorsOpenReview ↗

Author Response (Part 3)


Q2: What exactly constitutes an achor-state key, and how do you handle differing histories, remaining horizons, and repeated visits within one trajectory?

For exact matching, the anchor-state key consists solely of the current environment state string at the agent's decision step; histories and remaining horizons are excluded. Appendix L illustrates repeated visits to the same interface, such as "product page 1." When the state string is unchanged, these visits are matched exactly through hash-based lookup.

For history, as in GiGPO and HGPO, the state-history context contains only the current and immediately two preceding states, not the full interaction history.

For remaining horizons, their effect is captured by AdaStep's coefficient rather than the grouping key. Under sparse terminal rewards, R(s,at(j))=γTj−trTj(j)R(\boldsymbol{s},\boldsymbol{a}_t^{(j)})=\gamma^{T_j-t}r_{T_j}^{(j)}, so different horizons yield different returns. By Eq. (9), horizon differences caused by the action change Q(s,a)Q(\boldsymbol{s},\boldsymbol{a}) (signal), whereas those arising after the same action enter the within-action variance (noise) and lower w^(s)\widehat w(\boldsymbol{s}). In Example 1 (Section 4.1), the same correct action has 10 or 25 remaining steps: GiGPO assigns 3.33 and -1.51, whereas AdaStep assigns 1.07 and 0.35.

For repeated visits, as in W3, AdaStep outperforms GiGPO under first-visit grouping, which satisfies the theoretical sampling assumptions. In practice, retaining repeated visits with our derived coefficients yields better mean success than reverting affected groups to GiGPO's fixed weighting.


Q3: How sensitive is the coefficient to group size and action counts? Could a bias-corrected or cross-fitted estimator improve its reliability?

  • group size. We ablate the number of rollouts per task on ALFWorld using Qwen2.5-7B-Instruct, setting NN to 4, 8, and 16. The results are shown below (Appendix F.2, Table 9).
Rollouts per task NNMethodIn-domain success (%)Out-of-domain success (%)
4GiGPO85.43 ± 1.281.73 ± 0.5
4AdaStep (Ours)90.36 ± 2.087.76 ± 0.9
8GiGPO92.71 ± 0.989.06 ± 1.4
8AdaStep (Ours)97.66 ± 0.893.30 ± 1.3
16GiGPO94.53 ± 2.188.28 ± 0.8
16AdaStep (Ours)98.15 ± 0.494.97 ± 1.5

AdaStep consistently outperforms GiGPO across all three rollout counts, improving in-domain success by 3.62–4.95 percentage points and out-of-domain success by 4.24–6.69 percentage points.

  • bias-corrected

Under the i.i.d. sampling assumption, we use the sample count as the denominator when estimating the variances in our coefficient, which yields biased estimates of the individual population variances. Replacing NsN_s and ns,an_{s,\boldsymbol{a}} with Ns−1N_s-1 and ns,a−1n_{s,\boldsymbol{a}}-1 gives the standard unbiased variance estimators for those individual variance terms; this does not imply that the resulting variance ratio or coefficient is unbiased. Proposition C.1 shows that the bias-corrected coefficient can become negative, and Appendix K (Figure 11) empirically confirms the occurrence of negative values. We further evaluate the bias-corrected variant using Qwen2.5-7B-Instruct on ALFWorld and WebShop. As shown in Appendix C.4 (Table 7), it performs worse than the default formulation across all four metrics.

MethodALFWorld In-SuccessALFWorld Out-SuccessWebShop Test ScoreWebShop Success
GiGPO92.71±0.989.06±1.489.13±1.577.57±4.1
AdaStep (default)97.66±0.893.30±1.390.46±0.482.42±1.2
AdaStep (bias-corrected)93.50±1.684.93±0.585.23±2.775.40±1.2
  • cross-fitted estimator

We split the eight rollouts sampled for each task into two groups of four. For anchor states appearing in both groups, we perform cross-fitting: coefficients estimated from group A are applied to group B, and vice versa. We evaluate this setup on ALFWorld using Qwen2.5-7B, with the results shown below.

MethodALFWorld In-domain Success (%)ALFWorld Out-of-domain Success (%)
GiGPO (N=4)(N=4)85.43 ± 1.281.73 ± 0.5
Cross-fit89.83 ± 2.185.97 ± 1.3
AdaStep (N=4)(N=4)90.36 ± 2.087.76 ± 0.9
Reviewer_jWpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper replaces the fixed step advantage coefficient ω in GiGPO with a weight calculated per group, w(s) = 1 − (within-group variance) / (total variance), and derives this weight from the perspective of MSE. Experiments were conducted in three environments: ALFWorld, WebShop, and ScienceWorld, and the reported results are superior to GiGPO and HGPO.

Critical strengths
  1. The method is simple and has low additional overhead;

  2. The theoretical derivation is correct under the assumptions given in the paper.

Critical weaknesses

The relationship to ECPO needs to be clarified (my main concern)

The relationship between this paper and ECPO (Li et al., 2026, arXiv:2606.05885) needs to be clarified. This is my main concern.

The core formulas are in fact equivalent. ECPO's Variance-Gated Credit Weighting is defined as

ρVG(s)=gs⋅BsBs+Ws+ε,\rho_{\mathrm{VG}}(s) = g_s \cdot \frac{B_s}{B_s + W_s + \varepsilon},

where BsB_s is the between-action variance and WsW_s is the within-action variance, both using nn-denominators (ECPO Eqs. 14–18). Since Bs+WsB_s + W_s equals the total group variance v^s\hat{v}_s, we have

BsBs+Ws=1−Wsv^s,\frac{B_s}{B_s + W_s} = 1 - \frac{W_s}{\hat{v}_s},

which is exactly Eq. (17) of this paper. The only difference is ECPO's extra factor gs=tanh⁡(ns/τ)g_s = \tanh(n_s/\tau). On the groups where this paper estimates the coefficient (at least 4 samples), gs≥0.96g_s \ge 0.96, so this factor matters little.

The weight is used in the same way and under nearly the same conditions. ECPO's final advantage is AGRPO+ω⋅ρVG⋅AstepA^{\mathrm{GRPO}} + \omega \cdot \rho_{\mathrm{VG}} \cdot A_{\mathrm{step}}, which has the same structure as AE+w^⋅ASA_E + \hat{w} \cdot A_S here. ECPO's StrictValid condition requires at least two distinct actions and Bs+Ws>εB_s + W_s > \varepsilon; Section 5.3 of this paper requires at least two distinct actions and positive total variance.

ECPO already evaluates this configuration. ECPO's ablation includes a "VarGate-only" variant, which applies this weight to raw GiGPO step advantages without ECA shrinkage. Algorithmically, this is essentially the proposed method. ECPO reports that it improves ALFWorld success from 87.5 to 89.6.

ECPO is described incompletely. The paper cites ECPO only as "heuristically motivated calibrated action-level advantages", which covers the ECA component alone. It does not mention VarGate, and ECPO is not included as a baseline.

The "optimality" assumptions do not hold in the implementation

The derivation assumes that the samples within a group are i.i.d. conditional on ss. However, the same trajectory can enter a group multiple times. Back-solving the numbers in Example 2 implies a group of roughly 18 or more samples, yet each task has only 8 trajectories. In addition, the prompts include interaction history, so the policy π(⋅∣s)\pi(\cdot \mid s) assumed by the theory is not well defined in practice.

The fallback rule contradicts the paper's own theory

For groups with a single action but varying returns, Eq. (17) gives w=0w = 0, yet the paper sets w=1w = 1. This value appears to have been chosen based on test results (Table 3b). The paper also does not report what fraction of groups meet the estimation criteria.

The experimental comparison may be unfair

  • Several 7B baseline numbers are identical to those in the GiGPO paper, including their standard deviations. GiGPO used a 15-step WebShop limit, while this paper says all methods were run with 30 steps.
  • The normalization setting chosen for each benchmark is the one that favors AdaStep.
  • Many of the gains fall within one standard deviation.
Questions for authors
  1. Is Eq. (17) equivalent to ECPO's VarGate weight when gs=1g_s = 1 and ε=0\varepsilon = 0?

  2. Can you add comparisons with ECPO and with VarGate-only?

  3. What fraction of groups meet the estimation criteria during training?

  4. Were the 7B baselines rerun under your setting, or taken from prior work?

9 comments
AuthorsOpenReview ↗

Author Response (Part 1)


W1: The relationship to ECPO needs to be clarified (Part 1/2) .

  • AdaStep and ECPO are, in fact, not equivalent in their complete advantage formulations or weighting rules. AdaStep computes the combined advantage as

AAdaStep(s,at(j))=AE(τj)+wAdaStep(s)AS(s,at(j))A_{\mathrm{AdaStep}}(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) = A_E(\tau_j) + w_{\mathrm{AdaStep}}(\boldsymbol{s}) A_S(\boldsymbol{s},\boldsymbol{a}_t^{(j)})

where wAdaStep(s)w_{\mathrm{AdaStep}}(\boldsymbol{s}) estimates the theoretically derived coefficient:

wAdaStep(s)=Var⁡a∼π(⋅∣s)[Q(s,a)]Var⁡a∼π(⋅∣s)[Q(s,a)]+Ea∼π(⋅∣s)[Var⁡(R(s,a)∣s,a)]=1−Ea∼π(⋅∣s)[Var⁡(R(s,a)∣s,a)]Var⁡(R(s,at(j))∣s).w_{\mathrm{AdaStep}}(\boldsymbol{s}) = \frac{\operatorname{Var}_{\boldsymbol{a}\sim\pi(\cdot\mid\boldsymbol{s})}\bigl[Q(\boldsymbol{s},\boldsymbol{a})\bigr]}{\operatorname{Var}_{\boldsymbol{a}\sim\pi(\cdot\mid\boldsymbol{s})}\bigl[Q(\boldsymbol{s},\boldsymbol{a})\bigr] + \mathbb{E}_{\boldsymbol{a}\sim\pi(\cdot\mid\boldsymbol{s})}\bigl[\operatorname{Var}(R(\boldsymbol{s},\boldsymbol{a})\mid\boldsymbol{s},\boldsymbol{a})\bigr]} = 1 - \frac{\mathbb{E}_{\boldsymbol{a}\sim\pi(\cdot\mid\boldsymbol{s})}\bigl[\operatorname{Var}(R(\boldsymbol{s},\boldsymbol{a})\mid\boldsymbol{s},\boldsymbol{a})\bigr]}{\operatorname{Var}(R(\boldsymbol{s},\boldsymbol{a}_t^{(j)})\mid\boldsymbol{s})}.

ECPO comprises two components: Evidence-Calibrated Action Advantage (ECA) and Variance-Gated Credit Weighting (VarGate). Disabling ECA yields the VarGate-only ablation, rather than the full ECPO algorithm. This variant applies VarGate directly to the original occurrence-level step advantages, giving

AVarGate(s,at(j))=AE(τj)+wVarGate(s)AS(s,at(j))A_{\mathrm{VarGate}}(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) = A_E(\tau_j) + w_{\mathrm{VarGate}}(\boldsymbol{s}) A_S(\boldsymbol{s},\boldsymbol{a}_t^{(j)})

where

w_{\mathrm{VarGate}}(\boldsymbol{s}) = \omega\\,\mathbf{1}_{\mathrm{StrictValid}(\boldsymbol{s})}\max\\!\left\\{\tanh\\!\left(\frac{N_s}{\tau}\right)\frac{B_s}{B_s+W_s+\varepsilon},\\ \rho_{\min}\right\\}

and

1StrictValid(s)=1[depth⁡(s)≥dmin⁡,  Ns≥2,  ∣As∣≥2,  Bs+Ws>ε]\mathbf{1}_{\mathrm{StrictValid}(\boldsymbol{s})}=\mathbf{1}\left[\operatorname{depth}(\boldsymbol{s})\ge d_{\min},\; N_s\ge2,\; |\mathcal A_s|\ge2,\; B_s + W_s>\varepsilon\right]

Here, depth⁡(s)\operatorname{depth}(\boldsymbol{s}) denotes the trajectory depth of state s\boldsymbol{s}; NsN_s is the state-group size. The coefficient additionally depends on the hyperparameters ω,τ,ε,ρmin⁡\omega,\tau,\varepsilon,\rho_{\min}, and dmin⁡d_{\min}.

The algebraic correspondence identified by the reviewer is limited to a single expression within VarGate that coincides with our estimator of wAdaStep(s)w_{\mathrm{AdaStep}}(\boldsymbol{s}), namely, w^AdaStep(s)=BsBs+Ws.\widehat w_{\mathrm{AdaStep}}(\boldsymbol{s})=\frac{B_s}{B_s+W_s}. However, sharing this intermediate expression does not make the complete coefficients equivalent: VarGate further transforms it through sample-size modulation, validity gating, a reliability floor, and global scaling.

  • AdaStep's coefficient is principled and carries an explicit estimation-theoretic meaning: it is derived as the optimal scalar correction for noisy step-level advantage estimation under the stated assumptions. We begin by analyzing how sampled returns and the shared group baseline introduce error into the estimated advantage, and determine the coefficient by minimizing its mean-squared deviation from the true advantage. The resulting weight quantifies how much of the observed return variability reflects differences in expected action values, relative to downstream randomness. It therefore specifies how strongly the sampled step-level credit should contribute to approximating the true advantage. AdaStep gives the coefficient a precise role as an optimal estimator correction, grounded in an explicit noise analysis and optimization objective. This principled derivation and its optimality characterization constitute an independent theoretical contribution, distinct from ECPO.
AuthorsOpenReview ↗

Author Response (Part 2)


W1: The relationship to ECPO needs to be clarified (Part 2/2) .

  • ECPO does not isolate the variance-ratio coefficient itself as a standalone contribution. Although ECPO reports a VarGate-only ablation, this variant removes ECA while retaining the additional mechanisms associated with VarGate, including sample-size modulation, depth-based gating, and a reliability floor. Consequently, the reported improvement establishes the effectiveness of that configured module, but does not isolate the contribution of the variance ratio alone. ECPO’s broader design combines action-level calibration with reliability gating; its ablations do not separately establish how the additional mechanisms affect the effectiveness of the unmodified coefficient. AdaStep makes the coefficient itself the central object of study: we derive it from an explicit advantage-estimation objective, characterize its optimality under stated assumptions, and examine its direct use for scaling occurrence-level step credit. The presence of the same computational expression in ECPO therefore does not, by itself, establish that its standalone role and estimation-theoretic significance have already been fully characterized.

  • We have clarified in the citation that ECPO consists of two components: action-dependent ECA and state-dependent VarGate.


W2: The "optimality" assumptions do not hold in the implementation.

Thank you for raising this concern. We clarify the scope of the optimality result and provide two complementary ablations examining the effect of within-trajectory revisits.

  • “Optimal” refers to the minimizer of the stated estimation objective under the theorem’s assumptions. The variance-ratio coefficient is theoretically optimal under conditional i.i.d. sampling and the other stated conditions; this is not a claim of exact optimality for arbitrary dependent rollout groups.

  • We verify AdaStep’s performance gains in a setting that satisfies the theoretical sampling assumptions. Both GiGPO and AdaStep retain only the first occurrence of each anchor state per trajectory. Under a fixed task and rollout policy, independently sampled trajectories yield i.i.d. first-visit action–return pairs conditional on reaching the anchor state. On WebShop with Qwen2.5-7B-Instruct, AdaStep improves score by 2.23 points and success by 3.90 percentage points (see Section 5.4 and Table 4).

MethodScoreSuccess (%)
GiGPO86.57 ± 0.675.50 ± 1.9
AdaStep88.80 ± 1.479.40 ± 1.9
  • Our practical weighting scheme remains effective even when conditional independence is not guaranteed. Without independence, the optimal coefficient involves unknown action-value terms and cannot be computed directly from the available return statistics. We therefore introduce a gate that sets the coefficient to one for groups containing within-trajectory revisits, reverting those groups to GiGPO’s fixed weighting. AdaStep achieves higher mean success rates than this gated variant across all evaluated model–benchmark splits, demonstrating the practical benefit of retaining our adaptive coefficients in these groups (Section 5.4, Table 5).
ModelMethodALFWorld In-Suc. (%)ALFWorld Out-Suc. (%)WebShop ScoreWebShop Suc. (%)
Qwen3-4BAdaStep93.01 ± 0.986.58 ± 1.288.76 ± 0.580.27 ± 0.4
Qwen3-4BAdaStep + gate86.93 ± 1.584.63 ± 1.291.13 ± 1.378.93 ± 2.2
Qwen2.5-7B-InstructAdaStep97.66 ± 0.893.30 ± 1.390.46 ± 0.482.42 ± 1.2
Qwen2.5-7B-InstructAdaStep + gate93.23 ± 1.888.77 ± 1.890.13 ± 1.378.67 ± 2.0
  • History-conditioned prompts do not make the anchor-conditioned action distribution ill-defined. In our analysis, this distribution is defined by marginalizing over histories induced by the specified rollout and grouping protocol. The corresponding action and state values are conditional return expectations under that same distribution. In practice, we use the same context configuration as our GiGPO and HGPO baselines: the state-history context is limited to the current state and the immediately two preceding state, rather than the full interaction history.
AuthorsOpenReview ↗

Author Response (Part 3)


W3: The fallback rule contradicts the paper's own theory.

  • Under our theoretical formulation, insufficient action coverage makes the coefficient unavailable for estimation, rather than theoretically equal to zero. With only one observed action, the group provides no comparison across actions. Interpreting the zero obtained by direct empirical substitution as the population coefficient would incorrectly equate missing action diversity with the absence of action-related signal. We therefore apply the prescribed fallback instead.

  • Choosing a unit fallback is consistent with our premise that GiGPO’s step-level advantages may contain useful information despite their noise. When the available statistics are insufficient to determine an appropriate correction, setting the weight to one preserves that potentially informative signal, whereas setting it to zero discards it entirely as if it were pure noise. Table 3(b) supports this theoretically motivated design rationale by showing that the unit fallback performs best among the evaluated choices; it does not define the rationale retrospectively.

  • We further emphasize a distinction between AdaStep and ECPO: even when all auxiliary mechanisms are removed, an identical algebraic expression does not imply identical rules for estimating the coefficient. AdaStep treats the coefficient as unavailable for estimation when only one action is observed. ECPO’s empirical variance ratio instead evaluates to zero when total return variance is positive, because the empirical between-action variance is zero.

Thank you for the reminder. Appendix K (Figure 8) now reports, over training, the fraction of state groups with nonzero step advantages in which AdaStep's coefficient actively adjusts the local correction (please see our response to Q3).


W4 & Q4: The experimental comparison may be unfair. Were the 7B baselines rerun under your setting, or taken from prior work?

  • Baseline source and reruns. The Qwen2.5-7B-Instruct baseline results of WebShop in Table 1 were originally taken from HGPO, whose setup uses a 30-step episode limit. To ensure a fair comparison, we have rerun these WebShop baselines under the same 30-step setting used for AdaStep and updated the results in Table 1 of the manuscript.

  • Normalization choices follow GiGPO. Table 1 adopts the better-performing settings reported in the original GiGPO paper, with std normalization on ALFWorld and without it on WebShop. We apply these settings to GiGPO, HGPO, and AdaStep alike. These settings were not selected to favor AdaStep. Table 2 tests both normalization settings with the 4B and 7B models, and AdaStep achieves higher mean performance in every ALFWorld and WebShop comparison. We have also added the complete results under both settings for all three backbones on all three benchmarks, including ScienceWorld. (Please see Table 8 in Appendix F.1: for some benchmarks and models, alternative normalization settings yield larger gains than those used in Table 1.)

  • The gain and standard deviations. All RL results report mean ± std over three independent seeds, and comparisons use these means for all methods. AdaStep achieves higher means than both GiGPO and HGPO in all 15 model–metric comparisons in Table 1. Against GiGPO, among the 12 success-rate comparisons, 7/12 gains exceed GiGPO's std, and 6/12 exceed each method's std individually. Gains over HGPO are particularly pronounced on WebShop across all three model sizes and on ScienceWorld with Qwen3-1.7B. In these success-rate comparisons, the gains exceed the sum of both standard deviations, yielding nonoverlapping mean ± one-standard-deviation ranges. These results support improvements beyond the scale of the reported run-to-run variability.


Q1: Is Eq. (17) equivalent to ECPO's VarGate weight when g_s = 1 and \varepsilon = 0?

Eq. (17) in the original submission corresponds to Eq. (20) in the revised manuscript.

The complete weighting rules are not equivalent. Beyond setting gs=1g_s=1 and ε=0\varepsilon=0, recovering AdaStep's calculation would require:

  • removing the reliability floor (ρmin⁡=0\rho_{\min}=0)
  • disabling the depth restriction and StrictValid gating
  • setting the global scale ω=1\omega=1
  • adopting AdaStep's estimation-eligibility and fallback rules

These changes would establish computational equivalence. AdaStep’s distinct theoretical contribution is the principled derivation of this coefficient as the optimal scalar correction under an explicit advantage-estimation objective and the stated sampling assumptions.

Reviewer_DP7HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a new method of calculating advantages for adapting LLMs to agentic tasks. Typically, the advantage of a state-action pair is equivalent to the advantage of the entire trajectory since the reward is sparse: zero at every timestep until the end. This feature makes credit assignment difficult in long-horizon rollouts from the LLM since it is unclear which tokens were most responsible for the “success of the trajectory”. Prior work (GiGPO) showed it is possible to augment the standard trajectory-level advantage by introducing an advantage that enables credit assignment to individual steps of the trajectory. This augmentation is weighted by a constant parameter ω\omega. The authors of this paper propose a method to automatically choose ω\omega as a state-dependent quantity.

Critical strengths

The authors derive an optimal coefficient for weighting step-level advantages. This represents a principled approach to an important problem in LLM post-training. There are a good number of baselines that show that the authors’ method improves performance.

Critical weaknesses

Novelty: The paper’s main contribution is quite narrow. Essentially, the entire paper amounts to automating the choice of a scalar coefficient in a training objective proposed in another paper. While the authors do show this marginally improves performance over the original GiGPO fixed coefficient scenario it feels like a thin contribution for ICLR.

Theory: All theorems and lemmas should have proofs or at least a proof sketch in the main text.

Formatting Issues: The notation can be confusing at times: In the prose before Equation 1 it states that there is a set of nn trajectories but then in the equation itself it uses the symbol NN.

For example there are the symbols: RtiR_t^i, RS(si,t)R_S(s_{i,t}), RE(τi)R_E(\tau_i). Despite appearing with the same capital RR they take different arguments so it's unclear what they depend on. I found this to be difficult to parse as a reader. Particularly when switching what arguments are passed to that function.

Another example is the difference between s~k\tilde{s}_k and sts_t where the subscript in the former refers to one of KK anchor states whereas in the latter it refers to an index in time. I suggest using subscripts for time only.

Questions for authors

How do you match similar states together? In my interpretation, a state is either a token or a passage of text. If this is the case, how can you get an exact match between the states via the hash map approach mentioned in the paper?

What does the term ω(s)\omega(s) look like when plotted over different states? Is there any interpretation that can be gained from visualizing / inspecting it?

6 comments
AuthorsOpenReview ↗

Author Response (Part 1)


W1: The paper’s main contribution is quite narrow.

Thank you for this concern. AdaStep contributes: an estimation objective for step-credit reliability, a theoretically derived, state-dependent weighting rule, and consistent empirical gains at minimal additional cost, extending beyond automatically choosing GiGPO's global coefficient.

  • AdaStep addresses a significant technical challenge in long-horizon agent RL: determining how much to trust estimated step-level credit. Sparse outcome rewards make trajectory-level supervision too coarse to distinguish individual decisions, motivating finer-grained credit assignment, including GiGPO[1], HGPO[2], GraphGPO[3] and BEACON[4]. HGPO extends GiGPO’s state-based grouping with historical context; GraphGPO replaces GiGPO’s trajectory-derived step returns with graph-distance-based returns; and BEACON replaces GiGPO’s repeated-state comparisons with milestone-aligned segment comparisons and subgoal rewards. While these methods refine how local credit is constructed, AdaStep addresses the complementary limitation of GiGPO’s fixed weighting. Observed returns mix action information with downstream randomness from subsequent decisions and reward timing. AdaStep therefore adapts the influence of local credit on policy updates to its estimated reliability.
  • The contribution is an explicit estimation formulation and a theoretically motivated, state-dependent weighting rule. The target advantage is Aπ(s,a)=Qπ(s,a)−Vπ(s)A^\pi(\boldsymbol{s},\boldsymbol{a})=Q^\pi(\boldsymbol{s},\boldsymbol{a})-V^\pi(\boldsymbol{s}), whereas the sampled estimator R−RˉR-\bar R contains errors from both the individual return and the estimated baseline. We study these errors and formulate local weighting as minimizing the mean-squared error relative to the target advantage. Under the stated conditional-sampling assumptions, we derive an optimal scalar shrinkage coefficient with a signal-to-total-variance interpretation. This criterion preserves action-related information while attenuating downstream noise. Different state groups receive different coefficients, changing their relative contributions to policy updates in ways a global constant cannot reproduce.
  • Empirical evidence shows meaningful task improvements and benefits beyond tuning GiGPO’s fixed coefficient. Across three backbones and three benchmarks, AdaStep achieves higher mean performance than GiGPO in every model–benchmark combination (Table 1). On ScienceWorld with Qwen3-1.7B, success increases from 20.57% to 29.93%, a gain of 9.36 points. With Qwen2.5-7B, ALFWorld in-domain success increases from 92.71% to 97.66%, corresponding to a 67.9% relative reduction in failure rate; and WebShop success improves by 4.85 points. Sections 5.5 and 5.6 show that how the coefficients are determined matters: adapting weights to state-specific reliability yields better performance than automating the selection of a globally fixed coefficient.
  • AdaStep’s simplicity is a practical strength (not a weakness) because it delivers these gains with minimal integration and computational costs. The method uses existing rollout statistics and introduces no additional critic, rollouts, model forward passes, or tunable shrinkage hyperparameters. On WebShop, credit-computation time increases from 0.4968 to 0.5018 seconds per iteration—about 5 milliseconds, or 1% of credit-computation time (Table 6). It also uses 20.9% less credit-computation time than HGPO while achieving higher success. This makes AdaStep a lightweight module for training pipelines that provide comparable state groups and action-return statistics. It improves the learning signal without adding model or sampling complexity.
  • Principled adaptation of scalar weights has established value in machine learning when supported by clear analysis and empirical evidence. For example, SAC’s automatic temperature adjustment [5] derives an adaptive coefficient governing the reward-entropy trade-off; GradNorm [6] dynamically balances scalar weights in an existing multitask objective; and Min-SNR Weighting [7] improves diffusion training through timestep-dependent loss weights. These precedents show that a compact weighting rule can embody a substantive optimization principle. We ask that AdaStep be assessed through its estimation formulation, theoretical characterization, diagnostic evidence, and demonstrated performance–cost benefits, alongside the simplicity of its implementation.

[1] Feng et al. GiGPO. NeurIPS, 2025.

[2] He et al. HGPO. ICLR, 2026.

[3] Cheng et al. GraphGPO. ICML, 2026.

[4] Wang et al. BEACON. ICML, 2026.

[5] Haarnoja et al. SAC. arXiv:1812.05905, 2018.

[6] Chen et al. GradNorm. ICML, 2018.

[7] Hang et al. Min-SNR Weighting. ICCV, 2023.

AuthorsOpenReview ↗

Author Response (Part 2)


W2: Theory: All theorems and lemmas should have proofs or at least a proof sketch in the main text.

Thank you for careful reading. We have added a proof sketch after each theorem and lemma in the main text, outlining the key ideas and techniques.


W3: Formatting Issues: The notation can be confusing at times.

Thank you again for your careful reading. We have revised and standardized the notation throughout the paper. Specifically, R(τj)R(\tau_j) denotes the return of trajectory τj\tau_j, which is either 0 or 1, while R(s,at(j))R(\boldsymbol{s},\boldsymbol{a}_{t}^{(j)}) denotes the realized discounted return from time step tt onward in trajectory τj\tau_j, where action at(j)\boldsymbol{a}_{t}^{(j)} is taken in state s\boldsymbol{s}.

We have also removed the potentially confusing notation s~k\tilde{\boldsymbol{s}}_k. Instead, for a fixed state s\boldsymbol{s}, we define

G(s)={(j,t)∣st(j)=s, 1≤j≤N, 1≤t≤Tj}.\mathcal{G}(\boldsymbol{s})=\{(j,t)\mid \boldsymbol{s}_t^{(j)}=\boldsymbol{s},\ 1\leq j\leq N,\ 1\leq t\leq T_j\}.

Here, NN is the number of sampled trajectories, and TjT_j is the length of trajectory τj\tau_j. Thus, G(s)\mathcal{G}(\boldsymbol{s}) contains all trajectory–time-step index pairs (j,t)(j,t) at which the state equals s\boldsymbol{s}, with jj indexing the trajectory and tt indexing the time step.


Q1: How do you match similar states together?

We use the same state-grouping procedure as GiGPO, which supports two matching modes. Here, the state used for grouping is the environment state at an agent decision step, typically represented as a passage of text. Grouping is performed within rollouts of the same task.

Hash-based exact matching. The environment renders its predefined state configurations as text, so revisiting the same state configuration can yield exactly the same state string. Thus, different rollouts of the same task can encounter identical textual states. Each observation is converted into a hashable key and placed in a hash map. Two states are grouped together only when their state strings are exactly equal. This shared state is referred to as an anchor state in the manuscript, and its associated actions and returns form a step-level group. We use this to match identical states.

Text similarity matching. The optional similarity mode accommodates small textual variations using character-level sequence similarity. Each state is compared with the representative state of each existing group and joins the first group whose similarity meets a specified threshold, otherwise it creates a new group. The representative is the first state assigned to that group and remains fixed. This mode relaxes exact anchor-state matching to approximate grouping.


Q2: What does the term $w(\boldsymbol{s})$ look like when plotted over different states? Is there any interpretation that can be gained from visualizing / inspecting it?

Thank you for this question. Appendix K of the revised manuscript presents (1) the fraction of state groups with nonzero step advantages in which w(s)w(\boldsymbol{s}) is active, i.e., w(s)<1w(\boldsymbol{s})<1 (Figure 8), and (2) the distribution of these active weights over training (Figure 9). The active weights cover a broad range within [0,1][0,1] on both ALFWorld and WebShop, so different state groups receive substantially different corrections rather than a single global coefficient. On WebShop, they shift toward larger values later in training, indicating that return differences increasingly reflect action quality.

To illustrate how this weighting affects credit assignment, Appendix L presents three concrete trajectory examples (Figures 12–15). More specifically, one example shows that GiGPO assigns a negative combined advantage to a correct action, whereas AdaStep automatically attenuates the misleading local correction through w(s)w(\boldsymbol{s}), resulting in a positive combined advantage.

AuthorsOpenReview ↗

Author Response (Part 3)


Summary Response

We thank you for recognizing the importance of the problem, the principled derivation, and the performance improvements over multiple baselines. We clarify that AdaStep formulates step-credit weighting as minimizing estimation error relative to the target advantage and derives a state-dependent rule under the stated assumptions, rather than merely automating the choice of a global coefficient (W1). Empirically, AdaStep achieves higher mean performance than GiGPO in every evaluated model–benchmark combination across three backbones and three benchmarks, including a 9.36-percentage-point gain on ScienceWorld with Qwen3-1.7B, at minimal additional computational cost. Further analyses show the benefit of state-specific reliability weighting over automated selection of a global coefficient (W1). We have added proof sketches in the main text and standardized the notation (W2–3). We also clarify what constitutes a state and how exact matching and optional similarity-based grouping work (Q1). Weight-distribution analyses and concrete examples illustrate how state-dependent weighting attenuates misleading local credit, including a case where AdaStep restores a positive combined advantage for a correct action (Q2). We would appreciate your reconsideration in light of these clarifications and additional evidence.

Comments on the paper

2 comments

Dear Reviewers,

We thank all reviewers for their constructive feedback. Below, we summarize the revisions to the manuscript and the additional experiments and analyses.

Additional Experiments

  • Section 5.1, Table 1: Rerun the Qwen2.5-7B-Instruct WebShop baselines under the same 30-step episode limit used for AdaStep.
  • Section 5.4, Tables 4–5: First-visit grouping and revisit-gate ablations to examine the conditional i.i.d. assumption and the practical effect of within-trajectory revisits.
  • Appendices F.1–F.2, Tables 8–9: Complete normalization ablations across all three backbones and benchmarks, and rollout group-size ablations with 4, 8, and 16 trajectories per task.
  • Appendices G.1–G.2, Tables 10–11 and Figure 7: Experiments with Qwen2.5-14B-Instruct on ScienceWorld, including longer-horizon tasks allowing up to 100 environment interactions per episode.
  • Appendix C.4, Table 7: Comparison with a variant using bias-corrected variance estimates.
  • Appendix K, Figures 8–11: Analyses of active-coefficient fractions and coefficient distributions throughout training, including the effects of revisit gating and alternative variance denominators.
  • Reviewer responses: Additional comparisons with ECPO and VarGate-only, and an evaluation of cross-fitted coefficient estimation.

Revisions

  • Standardized the notation and clarified anchor-state grouping, the target step-level advantage, and the sources of estimation error (Sections 3.2 and 4.2–4.3).
  • Added proof sketches in the main text, expanded the conditional-error analysis, and derived the general optimal coefficient when conditional independence is relaxed (Appendices A–B).
  • Updated the discussion of ECPO to describe both its ECA and VarGate components.
  • Added a finite-sample analysis explaining the variance-denominator choice and the resulting coefficient range (Appendix C.2, Proposition C.1).
  • Added detailed time and space complexity comparisons with GiGPO and HGPO (Appendix H).
  • Expanded the explanations accompanying trajectory examples to illustrate how AdaStep attenuates misleading local credit (Appendix L).

In response to the reviewers’ feedback, we have clarified the scope of our theoretical claims: the variance-ratio coefficient is optimal for the stated estimation objective under the specified sampling assumptions. The new first-visit experiment supports AdaStep under this sampling protocol, improving WebShop success over GiGPO by 3.90 percentage points. The revisit-gate ablation further supports retaining adaptive weighting in practice. We also clarify the shared variance-ratio expression and the differences between AdaStep’s complete weighting rule and ECPO’s. Additional experiments support AdaStep across normalization choices, rollout group sizes, and a larger backbone. On long-horizon ScienceWorld, AdaStep improves success by 2.34 percentage points while reducing average validation episode length by 8.82 steps.

Additions and changes to the manuscript are highlighted in blue in the revised PDF.

We sincerely appreciate the reviewers’ thoughtful comments, which have helped us improve the clarity, rigor, and empirical support of this work.

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

Thank you for overseeing the review of our paper. During rebuttal, we addressed the reviewers’ concerns through the following additional experiments and clarifications:

  • Contribution and distinction from ECPO (DP7H, jWpT). We clarified AdaStep’s contribution: formulating step-credit weighting as an estimation problem and deriving a state-dependent correction with a signal-to-total-variance interpretation. We explained the differences in the complete weighting rules and added comparisons with ECPO and VarGate-only. The performance results and coefficient-distribution analyses demonstrate the practical impact of these differences.

  • Theoretical assumptions and practical validity (jWpT, twYw). We added first-visit grouping experiments to evaluate AdaStep under the stated sampling assumptions, alongside revisit-gate ablations to examine its use with dependent samples. Both support the effectiveness of adaptive weighting. We also clarified history conditioning and derived the general optimal coefficient without conditional independence.

  • Experimental fairness and estimator robustness (jWpT, twYw). We reran the questioned baselines under matched settings and completed normalization ablations. Additional experiments examine rollout group sizes, bias-corrected variance estimates, and cross-fitted coefficient estimation. We further explained the fallback rule and reported how frequently adaptive weighting is active during training.

  • Long-horizon effectiveness and efficiency (twYw). We added more demanding long-horizon experiments, showing improved success and shorter average episodes. Detailed trajectory analyses illustrate how AdaStep attenuates misleading local credit. Runtime measurements and complexity analyses establish its minimal additional cost over GiGPO and lower credit-computation time than HGPO.

  • Presentation and interpretation (DP7H, twYw). We standardized notation, defined the target advantage and estimation error more clearly, added proof sketches, and expanded explanations of state grouping and coefficient behavior.

The reviewers recognized AdaStep’s principled derivation, simplicity, empirical improvements, and low overhead. Following the revisions, DP7H raised their rating to weak accept, and twYw confirmed that the additional empirical results addressed their clarity concerns. We subsequently provided further analyses addressing twYw’s follow-up questions on long trajectories and computational cost.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.