SAPO: Skill-Augmented Policy Optimization

Skill-augmented agents leverage reusable procedural knowledge to improve performance on complex agentic tasks. Despite promising progress, existing methods often rely on strong language models to analyze past experience, generate skills, and directly insert them into a retrievable skill bank. However, we find that even skills generated by proprietary frontier LLMs exhibit highly mixed utility, with many having little or even negative effect on performance. Although prior work tracks skill utility from later rollouts where the skill is retrieved, this feedback is delayed and entangled: once inserted, low-quality skills mislead the agent and slow down policy learning before their negative effects are recognized; meanwhile, because multiple skills are usually retrieved together, the observed performance change reflects the aggregate effect of the skill set rather than the marginal contribution of any individual skill. To address these limitations, we propose Skill-Augmented Policy Optimization (SAPO), a novel online reinforcement learning framework that enforces a validate-before-store paradigm for skill curation. Given a task, SAPO splits the standard rollout budget into base and skill-augmented halves: it generates base rollouts with the currently retrieved skills, induces a candidate skill from these trajectories, and evaluates it through skill-augmented rollouts under the same retrieval context. The performance gap between the two halves estimates the candidate skill's marginal contribution beyond existing skills, enabling validation before long-term storage without additional rollout overhead. This utility signal further trains the policy itself as a stronger skill generator, whose skill-generation likelihood serves as a skill score for retrieval-time reranking and outdated-skill pruning. Extensive experiments show that SAPO outperforms prior skill-augmented RL methods and avoids costly API calls to proprietary models.