- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AdaStep, which adapts the strength of GiGPO's step-level advantage correction using an estimated ratio of action-dependent return variance to total return variance. From limited theoretical lens, author gives some intuitive explanation on calibration effect on the advantage estimation. Empirical studies across three model backbones and three agent benchmarks report consistent improvements with minimal additional credit-computation cost.
Critical strengths
1, This method addresses a clear limitation of fixed-weight group-based credit assignment methods like GiGPO and it is simple to integrate into existing training pipelines.
2, The theoretical explanation on weighted factor derivation is intuitive coming with some insightful explanation.
3, Empirical study is comprehensive with multiple size model backbones, different RL algos, and some ablation studies.
Critical weaknesses
1, In line 248, it would be great to clearly define what is the ground-truth step-level advantage function by moving Eq 7 ahead.
2, It would be great to see AdaStep performance and detailed analysis in long horizon agentic tasks and coding/math reasoning tasks where rewards are more sparse and verifiable.
3, The theoretical induction assumes independent action-return pairs conditioned on the same state. However, groups can include repeated visits from the same trajectory, while policy prompts contain history and step-count info that may differ across identical anchor observations.
4, It would be better to provide empirical justification in terms of less computation needed for the AdaStep comparing with other Group-based baselines other than GiGPO.
Questions for authors
1, What is the nature of noise \eta defined in Equation 3? Or what is the underlying intuition/theory to construct Eq 3 here?
2, What exactly constitutes an achor-state key, and how do you handle differing histories, remaining horizons, and repeated visits within one trajectory?
3, How sensitive is the coefficient to group size and action counts? Could a bias-corrected or cross-fitted estimator improve its reliability?
8 comments
Author Response (Part 2)
W3: Groups can include repeated visits from the same trajectory, while policy prompts contain history and step-count info that may differ across identical anchor observations.
In the revised manuscript, we add two complementary ablations: the first tests AdaStep in a practical setting satisfying the sampling assumptions; the second compares weighting strategies in practice.
1. First-visit grouping. Both methods retain only the first occurrence of each anchor state per trajectory. Under a fixed task and policy, independently sampled trajectories yield i.i.d. first-visit action–return pairs conditional on reaching that state. On WebShop with Qwen2.5-7B-Instruct, AdaStep improves score by 2.23 points and success by 3.90 percentage points over GiGPO under this protocol (Section 5.4, Table 4).
| Method | Score | Success (%) |
|---|---|---|
| GiGPO | 86.57 ± 0.6 | 75.50 ± 1.9 |
| AdaStep | 88.80 ± 1.4 | 79.40 ± 1.9 |
2. Revisit-gate ablation. Under the original grouping, we set the coefficient to one for groups with within-trajectory revisits, reverting them to GiGPO’s fixed weighting. AdaStep achieves higher mean success than this gated variant. The higher mean success supports retaining adaptive weighting in groups containing revisits (Section 5.4, Table 5; Appendix B).
| Model | Method | ALFWorld In-Suc. (%) | ALFWorld Out-Suc. (%) | WebShop Score | WebShop Suc. (%) |
|---|---|---|---|---|---|
| Qwen3-4B | AdaStep | 93.01 ± 0.9 | 86.58 ± 1.2 | 88.76 ± 0.5 | 80.27 ± 0.4 |
| Qwen3-4B | AdaStep + gate | 86.93 ± 1.5 | 84.63 ± 1.2 | 91.13 ± 1.3 | 78.93 ± 2.2 |
| Qwen2.5-7B-Instruct | AdaStep | 97.66 ± 0.8 | 93.30 ± 1.3 | 90.46 ± 0.4 | 82.42 ± 1.2 |
| Qwen2.5-7B-Instruct | AdaStep + gate | 93.23 ± 1.8 | 88.77 ± 1.8 | 90.13 ± 1.3 | 78.67 ± 2.0 |
3. History-conditioned prompts. The anchor-conditioned action distribution is obtained by marginalizing over the histories and step counts induced by the rollout and grouping protocol; corresponding action and state values are conditional return expectations under that distribution. This permits prompt contexts to differ across identical anchor observations. As in GiGPO and HGPO, state-history context includes only the current and immediately two preceding states.
W4: It would be better to provide empirical justification in terms of less computation needed for the AdaStep comparing with other Group-based baselines other than GiGPO.
Table 6 reports empirical measurements of the time required to compute step-level advantages. AdaStep increases GiGPO’s computation time by only 1.0% while improving success by 4.85 percentage points; it is also faster than HGPO with higher success.
| Method | Time (s) | Success (%) |
|---|---|---|
| GiGPO | 0.4968 ± 0.0324 | 77.57±4.1 |
| HGPO | 0.6340 ± 0.0711 | 78.51 ± 1.4 |
| AdaStep | 0.5018 ± 0.0114 | 82.42 ± 1.2 |
Appendix H analyzes the time and space complexity for GiGPO, HGPO, and AdaStep, where is the total number of sampled steps and is HGPO’s history depth. AdaStep matches GiGPO in both time and peak additional space complexity.
| Method / implementation | Expected time | Peak additional space |
|---|---|---|
| GiGPO | ||
| AdaStep | ||
| HGPO (all layers stored) | ||
| HGPO (layer by layer) |
Author Response (Part 3)
Q2: What exactly constitutes an achor-state key, and how do you handle differing histories, remaining horizons, and repeated visits within one trajectory?
For exact matching, the anchor-state key consists solely of the current environment state string at the agent's decision step; histories and remaining horizons are excluded. Appendix L illustrates repeated visits to the same interface, such as "product page 1." When the state string is unchanged, these visits are matched exactly through hash-based lookup.
For history, as in GiGPO and HGPO, the state-history context contains only the current and immediately two preceding states, not the full interaction history.
For remaining horizons, their effect is captured by AdaStep's coefficient rather than the grouping key. Under sparse terminal rewards, , so different horizons yield different returns. By Eq. (9), horizon differences caused by the action change (signal), whereas those arising after the same action enter the within-action variance (noise) and lower . In Example 1 (Section 4.1), the same correct action has 10 or 25 remaining steps: GiGPO assigns 3.33 and -1.51, whereas AdaStep assigns 1.07 and 0.35.
For repeated visits, as in W3, AdaStep outperforms GiGPO under first-visit grouping, which satisfies the theoretical sampling assumptions. In practice, retaining repeated visits with our derived coefficients yields better mean success than reverting affected groups to GiGPO's fixed weighting.
Q3: How sensitive is the coefficient to group size and action counts? Could a bias-corrected or cross-fitted estimator improve its reliability?
- group size. We ablate the number of rollouts per task on ALFWorld using Qwen2.5-7B-Instruct, setting to 4, 8, and 16. The results are shown below (Appendix F.2, Table 9).
| Rollouts per task | Method | In-domain success (%) | Out-of-domain success (%) |
|---|---|---|---|
| 4 | GiGPO | 85.43 ± 1.2 | 81.73 ± 0.5 |
| 4 | AdaStep (Ours) | 90.36 ± 2.0 | 87.76 ± 0.9 |
| 8 | GiGPO | 92.71 ± 0.9 | 89.06 ± 1.4 |
| 8 | AdaStep (Ours) | 97.66 ± 0.8 | 93.30 ± 1.3 |
| 16 | GiGPO | 94.53 ± 2.1 | 88.28 ± 0.8 |
| 16 | AdaStep (Ours) | 98.15 ± 0.4 | 94.97 ± 1.5 |
AdaStep consistently outperforms GiGPO across all three rollout counts, improving in-domain success by 3.62–4.95 percentage points and out-of-domain success by 4.24–6.69 percentage points.
- bias-corrected
Under the i.i.d. sampling assumption, we use the sample count as the denominator when estimating the variances in our coefficient, which yields biased estimates of the individual population variances. Replacing and with and gives the standard unbiased variance estimators for those individual variance terms; this does not imply that the resulting variance ratio or coefficient is unbiased. Proposition C.1 shows that the bias-corrected coefficient can become negative, and Appendix K (Figure 11) empirically confirms the occurrence of negative values. We further evaluate the bias-corrected variant using Qwen2.5-7B-Instruct on ALFWorld and WebShop. As shown in Appendix C.4 (Table 7), it performs worse than the default formulation across all four metrics.
| Method | ALFWorld In-Success | ALFWorld Out-Success | WebShop Test Score | WebShop Success |
|---|---|---|---|---|
| GiGPO | 92.71±0.9 | 89.06±1.4 | 89.13±1.5 | 77.57±4.1 |
| AdaStep (default) | 97.66±0.8 | 93.30±1.3 | 90.46±0.4 | 82.42±1.2 |
| AdaStep (bias-corrected) | 93.50±1.6 | 84.93±0.5 | 85.23±2.7 | 75.40±1.2 |
- cross-fitted estimator
We split the eight rollouts sampled for each task into two groups of four. For anchor states appearing in both groups, we perform cross-fitting: coefficients estimated from group A are applied to group B, and vice versa. We evaluate this setup on ALFWorld using Qwen2.5-7B, with the results shown below.
| Method | ALFWorld In-domain Success (%) | ALFWorld Out-of-domain Success (%) |
|---|---|---|
| GiGPO | 85.43 ± 1.2 | 81.73 ± 0.5 |
| Cross-fit | 89.83 ± 2.1 | 85.97 ± 1.3 |
| AdaStep | 90.36 ± 2.0 | 87.76 ± 0.9 |
Author Response (Part 1)
W1 & Q1: Move Eq. 7 forward to clearly define the ground-truth step-level advantage function; clarify the nature of the noise \eta in Eq. 3 and the intuition/theory behind its construction.Thank you for the comment. We have clarified the following three points.
- Definition of the true advantage. At the beginning of Section 4.2, we now explicitly define
=Q(\boldsymbol{s},\boldsymbol{a})-V(\boldsymbol{s}),$$ where $Q(\boldsymbol{s},\boldsymbol{a})=\mathbb{E}_{\pi}[R\mid\boldsymbol{s},\boldsymbol{a}] , V(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\sim\pi(\cdot\mid\boldsymbol{s})}[Q(\boldsymbol{s},\boldsymbol{a})]$. - **Nature of the noise.** GiGPO estimates the step-level advantage using sampled returns and a group-mean baseline: $$\begin{aligned} A_S(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) &=R(\boldsymbol{s},\boldsymbol{a}_t^{(j)})-\hat V_{N_s}(\boldsymbol{s}),\quad \hat V_{N_s}(\boldsymbol{s}) =\frac{1}{N_s}\sum_{(j',t')\in\mathcal{G}(\boldsymbol{s})} R(\boldsymbol{s},\boldsymbol{a}_{t'}^{(j')}). \end{aligned}$$ The noise is the difference between this estimate and the true advantage: $$\begin{aligned} \eta_t^{(j)} &=A_S(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) -A_S^{*}(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) =\bigl(R(\boldsymbol{s},\boldsymbol{a}_t^{(j)}) -Q(\boldsymbol{s},\boldsymbol{a}_t^{(j)})\bigr) -\bigl(\hat V_{N_s}(\boldsymbol{s})-V(\boldsymbol{s})\bigr). \end{aligned}$$ It therefore captures estimation errors in both the sampled return and the baseline. - **Intuition and theoretical rationale.** We observe that GiGPO may assign negative advantages to correct state–action pairs and positive advantages to incorrect ones. We illustrate this issue in Figure 1, provide two concrete examples as motivation in Section 4.1, and present detailed rollout trajectories in Appendix L. Based on the theoretical derivation above, we attribute this issue to estimation error, which can be viewed as noise. Appendix L further demonstrates that our coefficient can correct the resulting advantages: for example, a correct action receiving a negative advantage under GiGPO receives a positive final advantage after applying our method. *** **`W2: It would be great to see AdaStep performance and detailed analysis in long horizon agentic tasks and coding/math reasoning tasks where rewards are more sparse and verifiable.`** To evaluate AdaStep on more demanding long-horizon tasks, we extended our ScienceWorld experiments to include tasks with expert trajectories of at most 70 steps and allowed up to 100 environment interactions per episode during training. With Qwen2.5-14B-Instruct, AdaStep achieves **52.07 ± 1.8%** success compared with **49.73 ± 1.7%** for GiGPO, a gain of **2.34 percentage points**. It also reduces the average validation episode length from **44.54 ± 1.8** to **35.72 ± 2.2** environment steps, a reduction of **8.82 steps**. These results are reported in **Appendix G.2, Table 11**, and the training curves are shown in **Figure 7(b)**. | Method | Success (%) | Average episode length (environment steps) | | --- | --- | --- | | GiGPO | 49.73 ± 1.7 | 44.54 ± 1.8 | | **AdaStep (Ours)** | **52.07 ± 1.8** | **35.72 ± 2.2** | | **Δ vs. GiGPO** | **+2.34** | **−8.82** | The success rate and episode length are measured at the training step with the highest seed-averaged validation success rate, consistent with **Appendix G.2, Table 11**. AdaStep focuses on agentic reinforcement learning, where an LLM interacts with an environment. Like GiGPO, it uses recurring environment observations as anchor states to group actions and their returns across rollouts of the same task. Extending the current formulation to standalone coding or mathematical reasoning would require a way to define comparable anchor states across rollouts.