- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Skill-Conditioned Policy Optimization (SCPO), a closed-loop framework for training LLM agents on long-horizon tasks with a reusable skill bank. The core observation is that existing skill-augmented agents (e.g., SkillRL, D2Skill) provide retrieved skills only as contextual information, leaving the agent's choice of which skill to follow implicit and therefore unevaluable. SCPO makes three additions to a GRPO backbone: (1) explicit skill declaration — the agent must output a skill identifier before its reasoning and action, making skill selection an observable policy output; (2) step-level retrospective attribution — an LLM judge (Gemini-3-Flash) labels each declared skill invocation as appropriate/inappropriate and correct/incorrectly executed, producing a dense binary step-level advantage that is combined with the trajectory-level advantage (weight α); and (3) dual-track credit-guided skill-bank evolution — soft updates to retrieval weights based on cumulative credit, plus hard updates (LLM rewriting of low-credit skills and success-driven generation of new skills).
Validation is on ALFWorld and WebShop with Qwen2.5-7B/3B. SCPO reports state-of-the-art results (92.7% ALFWorld, 79.7% WebShop at 7B), outperforming SkillRL by 5.7%/4.2%. Ablations cover random judge, removal of each evolution track, phase-level vs. skill-level declaration, and judge robustness (GPT-4o, QwQ-32B). Human validation of the skill judge is reported (89.7% accuracy vs. 94.3% human-human agreement on ALFWorld), along with training dynamics, per-category analyses, and training-cost measurements.
Critical strengths
1. The explicit declaration mechanism is a clean, well-motivated answer to a real diagnostic problem, and it is theoretically and empirically validated. The paper's motivating analysis (Appendix D) is genuinely informative: among SkillRL trajectories, those with observable skill declarations succeed at 86.8% vs. 78.7% without, and among declared trajectories, correct declarations succeed at 95.5% vs. 48.3% for incorrect ones. This demonstrates that skill selection is a high-variance, previously hidden decision — precisely the kind of variable that dense credit assignment should target. Theorem 1 (Appendix A) then makes the mechanism precise: under autoregressive factorization, the declared skill identifier receives its own policy-gradient term, which is absent under implicit injection because the retriever q(m|z) carries no gradient through θ. The honest caveat — that the theorem establishes an optimization path, not a performance guarantee — strengthens rather than weakens the analysis. This combination of diagnostic, theory, and controlled ablation (skill-level vs. phase-level declaration, +2.1%/+1.5%) is exactly the kind of evidence that distinguishes a contribution from a formatting trick.
2. The evaluation is unusually thorough for an agent-training paper. Beyond headline numbers, the paper includes: matched SkillRL-based ablations with identical skill banks and teachers; judge robustness across three LLM judges; human validation of the attribution labels with disagreement analysis (72 cases categorized into three interpretable failure patterns); sensitivity to the attribution weight α; per-category training dynamics; and a training-cost breakdown showing only ~11% overhead over SkillRL. The judge-validation appendix in particular (89.7% accuracy, macro-F1 0.895, with error analysis) is above the norm for papers using LLM judges as training signal sources, and it directly addresses the most obvious reliability concern a reviewer would raise.
Critical weaknesses
1. The objective, learning signal, and skill bank are mutually dependent, and this circularity is neither analyzed nor controlled for. As the paper itself frames it, SCPO is a closed loop: the policy's gradients depend on the judge's attribution labels; the judge's labels depend on the current skill bank's contents (it must decide whether a declaration matches the state and whether behavior follows the declared skill's principle); and the skill bank is updated based on the same credit signals derived from those labels. This creates a potential feedback loop — a poorly worded or wrongly retrieved skill can induce "incorrect execution" labels that then trigger revision of that skill, while a dominant or overlapping skill can crowd out others through the retrieval-weight updates. The paper does not analyze whether this loop converges, oscillates, or drifts; there is no measurement of skill-bank content change over training (e.g., how many skills are revised/added, whether revisions help or hurt the revised skills' subsequent credit), and no control experiment with a frozen skill bank (hard updates disabled but soft updates enabled is not the same thing — both are removed only jointly in "w/o Dual Track"). Since the claimed advantage over SkillRL is precisely this co-evolution, the absence of evidence that the loop behaves stably — and ablations isolating which track drives the gains — leaves the central mechanism under-substantiated.
2. Gains over SkillRL are real but modest relative to the machinery added, and the attribution of these gains across components is incomplete. SCPO improves ALFWorld success from 87.0% (SkillRL) to 92.7%, but the ablations show that the "w/o Dual Track" variant (declaration + step-level attribution, fixed skill bank) already reaches 88.0%, i.e., the entire dual-track evolution contributes only ~4.7 points, while the two tracks individually contribute only ~2 points each (90.6/90.1 vs. 88.0 baseline — and the soft/hard-update ablations are not additive, suggesting high variance in where credit lands). Meanwhile, the comparison that would isolate the paper's core claim — declaration + attribution without any of the skill-bank machinery, against SkillRL with the same training budget and teacher access — is only partially provided (the matched SkillRL variants are mentioned in Section 4.3 but not reported with full numbers). Relatedly, the dependence on external closed-source components is heavier than acknowledged: the judge (Gemini-3-Flash) and the skill-revision/generation teacher are both external LLMs whose quality the framework inherits, yet the robustness study swaps the judge but not the evolution teacher, and the WebShop judge's agreement with humans (86.3%) is noticeably lower than ALFWorld's. Whether the conclusions hold when the teacher is weaker — the realistic open-source setting — is untested.
Questions for authors
The answers to the following questions could materially affect my assessment.
- Circularity of the closed loop. The judge labels depend on the skill bank's contents; the bank is updated from those labels. Did you observe any instability or drift in skill-bank composition during training (e.g., skills oscillating between revision rounds, retrieval weights concentrating on a few modes)? Can you report the number of skills revised/added per run and their post-revision credit trajectories? This is a clarification request about the loop's dynamics, not a suggestion for future work.
- Frozen-bank control. Does an ablation exist with the skill bank entirely frozen (no soft or hard updates) but declaration + step-level attribution retained? If so, what are the numbers? If not, this is the key missing control for attributing gains to co-evolution rather than to declaration/attribution alone.
- Dependence on the initial skill bank. The initial banks are taken from SkillRL's released resources (45/55 entries). How sensitive are results to bank initialization — e.g., a smaller bank, randomly initialized principles, or banks constructed without access to the training distribution? Given that your objective is defined relative to the retrieved candidate set S_t, the choice of bank effectively defines the action space for declaration; quantifying this dependence would clarify what the method actually requires to work.
- Attribution-label quality as training bottleneck. The judge disagrees with humans on ~10–14% of steps, and your own error analysis shows the largest disagreement class (43%) is stale declarations after phase transitions — exactly the kind of label that is plausibly systematically wrong rather than random. Since the Random Judge ablation shows that uninformative labels hurt, did you test biased labels (e.g., flipping 10–15% of labels in a structured, non-random way)? The robustness of the training signal to realistic judge bias is materially different from robustness to random noise.
- Efficiency of the 13-candidate retrieval set. Each step retrieves 6 task-specific + 6 general + 1 fallback skills, and the policy must read all of them. How does performance vary with the number of retrieved candidates (K)? If a smaller retrieved set performs similarly, the declaration mechanism's cost-benefit changes substantially.