- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses state-matching for GRPO training of CLI agents, where varying terminal transcripts make exact-state matches rare. It groups steps by pre-action feedback type and interaction depth, then compares returns combining final task rewards with capped scores accumulated from each step onward. A theorem is proposed to quantify the trade-off between larger comparison groups and greater within-group value heterogeneity. Experiments across 4B/7B/8B on OpenThoughts Terminal-Bench-format tasks and SETA shows improved performance compared to exact-state step-grouping baseline and greater zero-shot transfer to Terminal-Bench 2.0.
Critical strengths
The abstract comparison trade-off between comparison-group size, within-group value differences, and baseline estimation error provides insights on the state-grouping mechanism for RL training. It also explains well why abstract-state grouping can provide more learning signals for step-by-step progression, and the extent to which such grouping is appropriate.
Critical weaknesses
- The feedback mode and execution-feedback scores (process rewards) are defined heuristically, particularly for CLI agents. Terminal-Bench transfer is useful evidence within CLI domain, but broader applicability to web, GUI, or other tool environments remains unknown. So I have concerns about the generality of the proposed method.
- Insufficient comparison with the closest prior work.
- No comparison with full GiGPO or its similarity-based grouping variant.
- HGPO[1], ProxMO[2], and RTMC[3] are discussed as alternatives that relax raw-state identity but receive no quantitative comparison. BiPACE is evaluated only at 4B using the authors’ reimplementation, with different training/evaluation settings and checkpoint coverage. The externally reported PPO result also uses a different setup. These results provide limited evidence that ASGPO improves over competitive existing credit-assignment methods.
Suggestions for the presentation: Better to add benchmark and baseline details in Section 6.1. Also, Figure 3 is overlapping with table 1-4 and Figure 4 is overlapping with table 3. They can be deleted or removed to the appendix.
References:
- He, Shuo, et al. "Hierarchy-of-groups policy optimization for long-horizon agentic tasks." International Conference on Learning Representations. 2026.
- Fang, Yangyi, et al. "Proximity-based multi-turn optimization: Practical credit assignment for llm agent training." ACL 2026.
- Wang, Tao, Suhang Zheng, and Xiaoxiao Xu. "RTMC: Step-Level Credit Assignment via Rollout Trees." arXiv preprint arXiv:2604.11037 (2026).
Questions for authors
- What is the value of in Eq.6? Is it assigned to each step in a short failed episode or a failed episode whose prompts exceed the context budget?
- Since the premise of Theorem 1 is leave-in centering, does it still apply to the leave-one out setting, such as the RLOO algorithm?
- Are the comparable samples within a group drawn from the entire training batch, or only from trajectories that originated from the same initial state?
- What do the cells in Table 1 with no values signify, such as 7B-ASGPO sing., and BiPACE using 7B/8B agent?
5 comments
Thanks for the authors' responses. However, without experiment results on environments besides CLI domains and mentioned baselines (e.g., similarity-based GiGPO & HGPO, etc.), my concerns still remained. So I maintain the original score.
Dear Reviewer a1EW,
Thank you again for your careful review and for following up on transfer beyond CLI and the additional baselines.
Our earlier response reported the completed 7B CLI comparison against full GiGPO. It used GiGPO’s default exact matching and both episode-level and step-level advantages.
We also ran an initial zero-shot test on 100 synthetic WebShop tasks using the CLI-trained 7B policies. Two-level ASGPO succeeded on 23 tasks, and standard GiGPO succeeded on 22. Training ASGPO on WebShop requires defining comparison groups and progress-aware returns from shopping feedback.
The 7B HGPO runs are still training, and the CLI run may take longer than the remaining discussion period. If the held-out evaluation finishes in time, we will include the HGPO results in a final update.
As the discussion period draws to a close, we would welcome any further questions about the transfer setup or the baseline comparisons. Thank you again for your time and consideration.
Best regards, The Authors
Thank you for identifying the questions about the comparison protocol and the implementation. We address them below. The revised appendix highlights the corresponding clarifications.
W1. Scope of the estimator. ASGPO centers progress-aware step returns in groups defined by pre-action feedback and interaction depth. The same estimator applies when an environment exposes feedback for selecting peers and observable events for constructing progress-aware returns. In a web or GUI environment, a form validation error could define a feedback label, and a verified application-state change could contribute to the step return. Our transfer evidence covers CLI tasks. ASGPO reaches 12.7% on 79 scored Terminal-Bench 2.0 tasks (Appendix B.6).
W2. Full GiGPO and other baselines. The original Exact baseline tests how concrete-state matching supplies peers for step credit. Given the limited rebuttal window, we prioritized completing the full GiGPO comparison at 7B, the middle scale in Table 1. We used default exact matching and separate episode and step advantages. Both runs use the same SFT initialization, OpenThoughts RL split, and evaluation tasks, with four rollouts per task. The original 7B ASGPO leads standard GiGPO in both shared-checkpoint summaries. The original Table 1 entry is 34.59%, and standard GiGPO's checkpoint mean is 28.15% (Appendix B.2).
When all sampled trajectories for a hard task receive zero verifier reward, episode centering gives zero throughout the group. The episode term also leaves exact step groups unchanged. Abstract grouping supplies peers across transcripts, and bounded progress in the step return distinguishes execution outcomes within those groups. This mechanism applies at any model size under these reward and grouping conditions. Table 1 shows gains over Exact at 4B, 7B, and 8B across two CLI task suites.
Appendix B.4 compares the other estimators. GiGPO's similarity variant uses a text threshold. HGPO's nested histories refine current-state groups and therefore cannot add peers to a singleton at the coarsest level. ProxMO's soft TF–IDF baseline can be dominated by the target's own return. RTMC estimates values for state–action signatures, and we could not verify a complete public release of its original training implementation. These comparisons concern the estimators. Table 1 reports our paper-based 4B BiPACE implementation with thinking on during training and off during evaluation. Its checkpoint coverage differs from the 4B ASGPO run. Table 2 reports the published PPO score under the Endless Terminals setup (Appendices A and B.7).
Q1. Return adjustment in Eq. (6). In the verified original 4B configuration, each retained step receives the same Γt. A failed trajectory incurs −0.01 per recorded action short of three. The adjustment is −0.02 for a one-action failure and −0.01 for a two-action failure. A successful short rollout has no short-episode penalty. A recorded PROMPT_TRUNCATION termination adds −0.10. The terms add when both conditions hold and the guard is zero when neither applies (Appendix B.7).
Q2. Leave-one-out. For n>1, let Rˉ be the group mean and Rˉ−i the mean of the other n−1 returns. Then
Ri−Rˉ=(1−1/n)(Ri−Rˉ−i).
Leave-one-out removes the self-inclusion factor. Under the conditional action-law and independent-peer assumptions, the peer baseline has zero expected product with the target score. The peer-count and value-aliasing trade-off still applies with n−1 independent peers. A singleton has no leave-one-out peer. Appendix B.5 distinguishes strict centering from the archived implementation's singleton fallback.
Q3. Comparison pool. The original ASGPO pools steps sharing a label across the current update batch, including across tasks. Both branches of the same-size shuffle in Appendix B.3 group within each task. Task conditioning removes the between-task component of value heterogeneity. Batch pooling can supply additional peers (Appendix B.6).
Q4. Missing entries. The 7B ASGPO dash denotes an unreported singleton diagnostic. The 7B and 8B BiPACE dashes denote unreported scores (Appendix B.7).
Presentation. Section 6.1 now consolidates the benchmark and baseline setup. We removed the original Figure 3 and moved the original Figure 4 to Appendix D, where it appears as Figure 3. Appendix B.7 records these changes.