- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Skill-Augmented Policy Optimization (SAPO), an online reinforcement learning framework designed to curate and utilize reusable procedural knowledge (skills) for complex agentic tasks. Addressing the critical flaw in prior methods where unvalidated, mixed-quality skills are blindly injected into the skill bank, SAPO introduces a "validate-before-store" paradigm. It splits the rollout budget to estimate the marginal utility of candidate skills via reward gaps. Furthermore, SAPO repurposes the agent policy as both a skill generator and a scorer, utilizing the generation likelihood to maintain the long-term skill bank and rerank skills during retrieval. Extensive experiments on ALFWorld, WebShop, and Search-QA demonstrate that SAPO achieves competitive performance while avoiding costly proprietary LLM API calls.
Critical strengths
-
Insightful Motivation and Pragmatic Design: The paper correctly identifies a significant bottleneck in prior skill-augmented RL methods: the direct injection of unvalidated skills (even those generated by frontier LLMs) can mislead policy learning. The proposed "validate-before-store" mechanism using marginal utility estimation is intuitive, theoretically sound, and practically elegant, as it requires no additional rollout budget.
-
Comprehensive Empirical Validation: The authors provide abundant and rigorous experimental results across diverse benchmarks (interactive decision-making and search-augmented QA). The comparison against various baselines, including memory-augmented RL and proprietary LLMs, convincingly demonstrates that SAPO achieves strong performance while significantly reducing API costs. The training dynamics and skill utility analyses further substantiate the claims.
Critical weaknesses
-
Superficial Treatment of Skill Iteration and Maintenance: While the paper details the skill induction and validation processes, the mechanisms for long-term skill iteration, deletion, and update (Section 4.3) feel relatively weak and under-explored. Relying solely on the policy's generation likelihood as a proxy for skill utility to prune outdated skills is somewhat heuristic. The paper lacks a deep discussion on how skills genuinely evolve or interact over time. Furthermore, it is unclear whether this likelihood-based pruning might inadvertently lead to a "rich-get-richer" effect, stifling the exploration of novel but initially low-likelihood skills.
-
Limitations in the Policy Optimization Objective: The policy optimization in SAPO relies on GRPO (a REINFORCE-style method without a value function baseline). In complex, long-horizon agentic tasks, the absence of value guidance (e.g., a critic network as in PPO) can lead to high variance and suboptimal convergence. This limitation likely explains why SAPO's performance gains are not universally dominant across all tasks (e.g., certain subtasks in WebShop or Search-QA), as the policy struggles to accurately attribute long-term rewards to specific skill-guided actions without a value baseline.
Questions for authors
-
Regarding the Choice of RL Objective: Why did you opt for a value-free objective like GRPO instead of a value-guided objective (e.g., PPO)? Given the high variance inherent in skill-augmented agentic rollouts, wouldn't a critic network provide more stable advantage estimation, potentially leading to better skill generation and overall performance? Could you provide an ablation or theoretical justification for this choice?
-
Regarding Performance Bottlenecks: Although SAPO is novel, its performance improvement over baselines like SkillRL or D2Skill is not overwhelmingly significant in all tasks (e.g., WebShop's "Pick2" or certain out-of-domain Search-QA datasets). Could you provide a detailed analysis of why the performance plateaus or struggles in these specific scenarios? Is it directly related to the lack of value guidance mentioned above, or are there other bottlenecks in the skill retrieval/maintenance process?
-
Regarding Skill Maintenance Dynamics: Since you use the policy's generation likelihood to score and prune old skills, how do you ensure that the skill bank doesn't suffer from mode collapse (i.e., the policy only retains skills it already knows how to generate well, ignoring diverse but potentially useful skills that have low initial likelihood)? Have you monitored the diversity of the long-term skill bank over training steps?