- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Skill-Probed Policy Optimization (SPPO) to alleviate gradient starvation in group-relative reinforcement learning for long-horizon language agents. It introduces current and lagged skills to shape rollout groups and employs asymmetric credit attribution to internalize skill-guided successful behaviors into a skill-free actor. Experiments on ALFWorld show improved task success rates over conventional agent RL baselines.
Critical strengths
-
Meaningful motivation and a coherent technical framework
The paper addresses a practical challenge in agent reinforcement learning: sparse rewards and unsuccessful rollouts can limit exploration and effective policy optimization. Using natural-language skills to guide exploration and subsequently internalizing successful behaviors into a standalone policy is a reasonable approach. The reported improvements over GRPO and GiGPO are encouraging.
Critical weaknesses
-
Limited novelty and insufficient differentiation from closely related work
The core ideas of skill-guided exploration, group-level rollout shaping, and skill internalization overlap substantially with existing research. In particular, GOLF [1] already uses natural-language feedback to introduce successful trajectories into low-reward groups and improve exploration. SKILL0 [2] and Skill0.5 [3] also investigate skill internalization in agentic RL. Although SPPO introduces temporal skill contrast and asymmetric credit attribution, its distinct methodological contributions over existing guided exploration and internalization approaches remain insufficiently established.
-
Missing strong baselines and inadequate component comparisons
The main experiments compare SPPO primarily against conventional RL methods, while omitting closely related skill-based approaches such as SKILL0 and Skill0.5, as well as a GOLF-style guided exploration baseline. Moreover, it omits essential controlled baselines, including a frozen skill generator, successful-trajectory SFT/distillation, and standard guided GRPO with matched rollout data. Without these comparisons, it is difficult to determine whether the improvements arise from the proposed optimization objective or simpler forms of skill-guided training.
-
Limited evaluation benchmarks and insufficient generalization evidence
All main experiments are conducted on a single benchmark, ALFWorld, using only 128 games from its in-distribution valid_seen split. The available valid_unseen split is not evaluated, and no experiments are reported on other agent environments such as WebShop or ScienceWorld. Given the paper's broader motivation concerning long-horizon language agents, the current evaluation is insufficient to establish whether SPPO generalizes across unseen environments, task structures, and interaction patterns. Furthermore, using a single training seed limits the reliability of the reported improvements.
-
Unclear performance attribution due to additional expert supervision and computational overhead
The skill generator is initialized using GPT-5.5-generated skills distilled from reference walkthroughs. This introduces privileged demonstration-derived information that is not explicitly matched in the main RL baselines. Additionally, SPPO requires an auxiliary trainable generator and additional skill-generation and optimization overhead. Without comparisons under matched supervision and total training compute, the reported improvements cannot be confidently attributed to the proposed algorithmic mechanism.
-
Insufficient validation of the proposed optimization mechanism
The existing ablations demonstrate the importance of lagged skills and double-scoring within the proposed framework, but do not establish their necessity over simpler alternatives. In particular, the skill-stripped objective updates the unguided policy using trajectories sampled under skill-conditioned distributions, introducing a distribution mismatch that deserves further justification. The paper also lacks comparisons with conventional off-policy distillation and simpler auxiliary skill-conditioned losses. These limitations make it difficult to isolate the benefits of asymmetric credit attribution from those of guided exploration and successful-trajectory reuse.
[1] Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
[2] SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
[3] Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning
Questions for authors
See weaknesses
2 comments
Dear Reviewer,
Thank you for your constructive feedback. We have submitted a revised version of the paper in line with your comments. Every requested change is included except the 3-run experiments, which we did not have time to complete. We will run the main experiments with three seeds and add the mean and standard deviation in the next revision.
Best regards, The Authors
Thank you for the careful and constructive review. Below we respond to each point in turn.
W1 (novelty). We will add a detailed comparison to the paper. The main differences are:
The SKILL0 paper reports 87.9% (3B) and 89.8% (7B) on ALFWorld for SKILL0 and 82.4% and 89.9% for SkillRL, which at 7B do not exceed GiGPO or SPPO. GOLF and SKILL0 also use different settings (math, code and open tasks for GOLF, Qwen2.5-VL for SKILL0). A direct comparison needs reruns under a matched protocol, which we could not finish within the rebuttal period.
W2 (baselines and component comparisons). For the skill-based baselines, see W1. For the controlled baselines (Qwen2.5-1.5B, test set valid_seen with 128 games, versus 90.6% for SPPO):
W3 (benchmarks). Following GiGPO, we evaluate on valid_seen, since the baselines are only evaluated there. The 3-run experiments and the additional WebShop experiments are still running, and we will report them as soon as they finish.
W4 (attribution of extra supervision and compute). The GPT-5.5/walkthrough-derived initialization of the skill generator is part of our method. Other methods also carry components that we do not use, such as extra credit assignment and reward terms, and these contribute to their results. To separate the effect of the extra supervision and compute, we ran the controls listed in W2 (also in our reply to Reviewer hUh3, W1 and W2).
W5 (validation of the optimization mechanism).
Off-policy argument. SPPO reshapes the rollout group with skill guidance, which changes the sampling distribution. We argue that the resulting update is still an ascent direction. Let p=πθ(⋅∣x,H) be the skill-free policy and q=πθ(⋅∣x,H,s) the skill-conditioned behavior policy.
We will add this analysis and a discussion of related work on RL from off-policy guidance and hints to the paper.