acceptodds
Withdrawn from ICLR 2027

SAPO: Skill-Augmented Policy Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Skill-augmented agents leverage reusable procedural knowledge to improve performance on complex agentic tasks. Despite promising progress, existing methods often rely on strong language models to analyze past experience, generate skills, and directly insert them into a retrievable skill bank. However, we find that even skills generated by proprietary frontier LLMs exhibit highly mixed utility, with many having little or even negative effect on performance. Although prior work tracks skill utility from later rollouts where the skill is retrieved, this feedback is delayed and entangled: once inserted, low-quality skills mislead the agent and slow down policy learning before their negative effects are recognized; meanwhile, because multiple skills are usually retrieved together, the observed performance change reflects the aggregate effect of the skill set rather than the marginal contribution of any individual skill. To address these limitations, we propose Skill-Augmented Policy Optimization (SAPO), a novel online reinforcement learning framework that enforces a validate-before-store paradigm for skill curation. Given a task, SAPO splits the standard rollout budget into base and skill-augmented halves: it generates base rollouts with the currently retrieved skills, induces a candidate skill from these trajectories, and evaluates it through skill-augmented rollouts under the same retrieval context. The performance gap between the two halves estimates the candidate skill's marginal contribution beyond existing skills, enabling validation before long-term storage without additional rollout overhead. This utility signal further trains the policy itself as a stronger skill generator, whose skill-generation likelihood serves as a skill score for retrieval-time reranking and outdated-skill pruning. Extensive experiments show that SAPO outperforms prior skill-augmented RL methods and avoids costly API calls to proprietary models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5bCYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Skill-Augmented Policy Optimization (SAPO), an online reinforcement learning framework designed to curate and utilize reusable procedural knowledge (skills) for complex agentic tasks. Addressing the critical flaw in prior methods where unvalidated, mixed-quality skills are blindly injected into the skill bank, SAPO introduces a "validate-before-store" paradigm. It splits the rollout budget to estimate the marginal utility of candidate skills via reward gaps. Furthermore, SAPO repurposes the agent policy as both a skill generator and a scorer, utilizing the generation likelihood to maintain the long-term skill bank and rerank skills during retrieval. Extensive experiments on ALFWorld, WebShop, and Search-QA demonstrate that SAPO achieves competitive performance while avoiding costly proprietary LLM API calls.

Critical strengths
  1. Insightful Motivation and Pragmatic Design: The paper correctly identifies a significant bottleneck in prior skill-augmented RL methods: the direct injection of unvalidated skills (even those generated by frontier LLMs) can mislead policy learning. The proposed "validate-before-store" mechanism using marginal utility estimation is intuitive, theoretically sound, and practically elegant, as it requires no additional rollout budget.

  2. Comprehensive Empirical Validation: The authors provide abundant and rigorous experimental results across diverse benchmarks (interactive decision-making and search-augmented QA). The comparison against various baselines, including memory-augmented RL and proprietary LLMs, convincingly demonstrates that SAPO achieves strong performance while significantly reducing API costs. The training dynamics and skill utility analyses further substantiate the claims.

Critical weaknesses
  1. Superficial Treatment of Skill Iteration and Maintenance: While the paper details the skill induction and validation processes, the mechanisms for long-term skill iteration, deletion, and update (Section 4.3) feel relatively weak and under-explored. Relying solely on the policy's generation likelihood as a proxy for skill utility to prune outdated skills is somewhat heuristic. The paper lacks a deep discussion on how skills genuinely evolve or interact over time. Furthermore, it is unclear whether this likelihood-based pruning might inadvertently lead to a "rich-get-richer" effect, stifling the exploration of novel but initially low-likelihood skills.

  2. Limitations in the Policy Optimization Objective: The policy optimization in SAPO relies on GRPO (a REINFORCE-style method without a value function baseline). In complex, long-horizon agentic tasks, the absence of value guidance (e.g., a critic network as in PPO) can lead to high variance and suboptimal convergence. This limitation likely explains why SAPO's performance gains are not universally dominant across all tasks (e.g., certain subtasks in WebShop or Search-QA), as the policy struggles to accurately attribute long-term rewards to specific skill-guided actions without a value baseline.

Questions for authors
  1. Regarding the Choice of RL Objective: Why did you opt for a value-free objective like GRPO instead of a value-guided objective (e.g., PPO)? Given the high variance inherent in skill-augmented agentic rollouts, wouldn't a critic network provide more stable advantage estimation, potentially leading to better skill generation and overall performance? Could you provide an ablation or theoretical justification for this choice?

  2. Regarding Performance Bottlenecks: Although SAPO is novel, its performance improvement over baselines like SkillRL or D2Skill is not overwhelmingly significant in all tasks (e.g., WebShop's "Pick2" or certain out-of-domain Search-QA datasets). Could you provide a detailed analysis of why the performance plateaus or struggles in these specific scenarios? Is it directly related to the lack of value guidance mentioned above, or are there other bottlenecks in the skill retrieval/maintenance process?

  3. Regarding Skill Maintenance Dynamics: Since you use the policy's generation likelihood to score and prune old skills, how do you ensure that the skill bank doesn't suffer from mode collapse (i.e., the policy only retains skills it already knows how to generate well, ignoring diverse but potentially useful skills that have low initial likelihood)? Have you monitored the diversity of the long-term skill bank over training steps?

Reviewer_3f1MOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the evaluation of newly generated skills and on the maintenance of a useful skill bank. The proposed solution, SAPO, divides an existing trajectory budget into two parts: base rollouts and skill-augmented one. The policy generates this candidate skill from the base trajectories. The reward difference between the two parts estimates the marginal utility of the candidate under the current retrieval context. The proposed solution uses this signal for two purposes: 1) selects the skills to promote, and trains the policy as a skill generator, and 2) uses the likelihood of the generator to rerank and prune retrieved skills.

Critical strengths

No critical things, but the candidate-specific utility comparison is a reasonable design choice.

Critical weaknesses

The reviewer thinks that this paper does not provide independent validation or a clear component analysis of the central utility-based mechanisms. In particular, Figure 2 uses the same utility estimates to select the promoted skills and to show their higher utility, so this gap is not independent evidence. The paper also does not show that the generation likelihood of the trained generator correctly ranks candidate skills by their current marginal utility.

Questions for authors
  1. What does the preliminary analysis show? Figure 1 supports the observation that skill evolution gives little additional benefit in the evaluated setting. However, it does not show that harmful skill admission causes this limit. In addition, the SAPO curve is close to the other curves.

  2. How does the utility analysis relate to the baseline algorithms? Honestly, the reviewer is not familiar with this domain, so there is no background in SkillRL (the prior method). In other words, does the implemented SkillRL training procedure use the utility of Section 3? Or is this utility only a post-hoc diagnostic? At least, the authors have to provide how SkillRL generates, stores, and retrieves skills, and how it learns with them. Without this information, it is difficult to evaluate the connection between the preliminary observations and the three stated limitations.

  3. How does SAPO use intra-prompt and cross-prompt utilities when K>1K>1?

  4. Which incremental contributions make SAPO different from existing utility-based skill maintenance?

  5. Does SAPO exclude temporary candidates with non-positive utility from retrieval? If so, please specify this rule. If no, how does the algorithm support the claim that SAPO discards harmful skills before they affect later retrieval and policy learning?

  6. As far as the reviewer understands, for each candidate, SAPO needs three dependent generation phases: base rollouts, skill generation, and skill-augmented rollouts. And they never run in parallel. Please let us know the computational trade-off, e.g., end-to-end training time, GPU-hours, token generation and scoring costs, and retrieval latency.

Reviewer_QnwnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Skill-Aware Policy Optimization, to tackle the issue in skill bank maintenance of measuring the utility of a newly created skill. They show with prior self-evolving skill bank baselines that newly created skills have a neutral or negative impact on agentic performance. For SAPO, they divide the number of trajectories in half, where the first half they only used retrieved existing skills, then create a skill with the outputs, and then the second half uses the newly created skill. The difference in rewards helps measure the utility of the new skill. This utility score both helps maintain the bank and also provides a training signal to the skill generator. Across different benchmarks, SAPO achieves competitive performance against baselines.

Critical strengths

SAPO method is consistently the top-2 performing methods across benchmarks

I believe for the most part the formalization and method described is solid. To estimate the utility of a skill in relation to the existing skill bank, performing trajectories with and without the skill is a intuitive and simple solution and gives the added benefit of also acting as a training signal for the skill generator

Critical weaknesses

The findings in Section 3 is only based off of Qwen 2.5 7B Instruct as the task agent which seems outdated for a 2026 conference submission. If I understand correctly, Figure 1 has the Qwen model be both the agent and skill generator (except for SkillRL/S) and Figure 2 has GPT-5.4 as the skill generator (with results using Opus-4.6 as the generator in the Appendix)? Moreover, in their main experiments they use GPT-4o, Gemini-2.5-Pro, and Qwen 2.7-7B. I do not agree with the authors on lines 410-411 that they “represent state-of-the-art general-purpose reasoning models”. Furthermore, Sections 3 and 5 could be strengthened with qualitative examples of the failure cases and how SAPO mitigates them. Right now, the paper leaves a lot left to analyze to fully understand the problem and motivation for SAPO.

The authors mention multiple times as motivation that prior work like SkillRL rely on closed-source LLMs. While those works certainly use them for their papers, I wouldn’t say they “rely on closed-source”. Maybe I’m missing something, but anything that SkillRL uses GPT for, an open-source model could do.

Questions for authors

How is K determined?

Temporary horizon should be defined before being discussed in Lines 279-286. Also if the skill that is validated is too similar (I’m assuming semantic similarity like cosine) to skills in B_{long} and ends up being discarded by the end of the horizon, shouldn’t that be checked first before anything else?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.