acceptodds
Withdrawn from ICLR 2027

Skill in the Loop: Explicit Skill Declaration and Attribution for Agent Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agents increasingly rely on reusable skills distilled from historical experience to solve long-horizon decision-making tasks. However, existing methods primarily provide skills as contextual information, leaving the agent's skill selection implicit and offering limited mechanisms to assess and improve how skills are used. We propose Skill-Conditioned Policy Optimization (SCPO), a closed-loop framework for explicit skill use and continual skill evolution. Before each action, the agent explicitly declares the skill guiding its decision. A retrospective mechanism then evaluates the effectiveness of skill usage over multi-step interactions and provides fine-grained feedback, which jointly optimizes the policy via GRPO and updates the skill bank based on observed utility. Experiments on ALFWorld and WebShop show that SCPO achieves state-of-the-art performance and consistently improves both skill utilization and skill-bank quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agents, Skill Selection,Skill Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_StLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Skill-Conditioned Policy Optimization (SCPO), a closed-loop framework for training LLM agents on long-horizon tasks with a reusable skill bank. The core observation is that existing skill-augmented agents (e.g., SkillRL, D2Skill) provide retrieved skills only as contextual information, leaving the agent's choice of which skill to follow implicit and therefore unevaluable. SCPO makes three additions to a GRPO backbone: (1) explicit skill declaration — the agent must output a skill identifier before its reasoning and action, making skill selection an observable policy output; (2) step-level retrospective attribution — an LLM judge (Gemini-3-Flash) labels each declared skill invocation as appropriate/inappropriate and correct/incorrectly executed, producing a dense binary step-level advantage that is combined with the trajectory-level advantage (weight α); and (3) dual-track credit-guided skill-bank evolution — soft updates to retrieval weights based on cumulative credit, plus hard updates (LLM rewriting of low-credit skills and success-driven generation of new skills).

Validation is on ALFWorld and WebShop with Qwen2.5-7B/3B. SCPO reports state-of-the-art results (92.7% ALFWorld, 79.7% WebShop at 7B), outperforming SkillRL by 5.7%/4.2%. Ablations cover random judge, removal of each evolution track, phase-level vs. skill-level declaration, and judge robustness (GPT-4o, QwQ-32B). Human validation of the skill judge is reported (89.7% accuracy vs. 94.3% human-human agreement on ALFWorld), along with training dynamics, per-category analyses, and training-cost measurements.

Critical strengths

1. The explicit declaration mechanism is a clean, well-motivated answer to a real diagnostic problem, and it is theoretically and empirically validated. The paper's motivating analysis (Appendix D) is genuinely informative: among SkillRL trajectories, those with observable skill declarations succeed at 86.8% vs. 78.7% without, and among declared trajectories, correct declarations succeed at 95.5% vs. 48.3% for incorrect ones. This demonstrates that skill selection is a high-variance, previously hidden decision — precisely the kind of variable that dense credit assignment should target. Theorem 1 (Appendix A) then makes the mechanism precise: under autoregressive factorization, the declared skill identifier receives its own policy-gradient term, which is absent under implicit injection because the retriever q(m|z) carries no gradient through θ. The honest caveat — that the theorem establishes an optimization path, not a performance guarantee — strengthens rather than weakens the analysis. This combination of diagnostic, theory, and controlled ablation (skill-level vs. phase-level declaration, +2.1%/+1.5%) is exactly the kind of evidence that distinguishes a contribution from a formatting trick.

2. The evaluation is unusually thorough for an agent-training paper. Beyond headline numbers, the paper includes: matched SkillRL-based ablations with identical skill banks and teachers; judge robustness across three LLM judges; human validation of the attribution labels with disagreement analysis (72 cases categorized into three interpretable failure patterns); sensitivity to the attribution weight α; per-category training dynamics; and a training-cost breakdown showing only ~11% overhead over SkillRL. The judge-validation appendix in particular (89.7% accuracy, macro-F1 0.895, with error analysis) is above the norm for papers using LLM judges as training signal sources, and it directly addresses the most obvious reliability concern a reviewer would raise.

Critical weaknesses

1. The objective, learning signal, and skill bank are mutually dependent, and this circularity is neither analyzed nor controlled for. As the paper itself frames it, SCPO is a closed loop: the policy's gradients depend on the judge's attribution labels; the judge's labels depend on the current skill bank's contents (it must decide whether a declaration matches the state and whether behavior follows the declared skill's principle); and the skill bank is updated based on the same credit signals derived from those labels. This creates a potential feedback loop — a poorly worded or wrongly retrieved skill can induce "incorrect execution" labels that then trigger revision of that skill, while a dominant or overlapping skill can crowd out others through the retrieval-weight updates. The paper does not analyze whether this loop converges, oscillates, or drifts; there is no measurement of skill-bank content change over training (e.g., how many skills are revised/added, whether revisions help or hurt the revised skills' subsequent credit), and no control experiment with a frozen skill bank (hard updates disabled but soft updates enabled is not the same thing — both are removed only jointly in "w/o Dual Track"). Since the claimed advantage over SkillRL is precisely this co-evolution, the absence of evidence that the loop behaves stably — and ablations isolating which track drives the gains — leaves the central mechanism under-substantiated.

2. Gains over SkillRL are real but modest relative to the machinery added, and the attribution of these gains across components is incomplete. SCPO improves ALFWorld success from 87.0% (SkillRL) to 92.7%, but the ablations show that the "w/o Dual Track" variant (declaration + step-level attribution, fixed skill bank) already reaches 88.0%, i.e., the entire dual-track evolution contributes only ~4.7 points, while the two tracks individually contribute only ~2 points each (90.6/90.1 vs. 88.0 baseline — and the soft/hard-update ablations are not additive, suggesting high variance in where credit lands). Meanwhile, the comparison that would isolate the paper's core claim — declaration + attribution without any of the skill-bank machinery, against SkillRL with the same training budget and teacher access — is only partially provided (the matched SkillRL variants are mentioned in Section 4.3 but not reported with full numbers). Relatedly, the dependence on external closed-source components is heavier than acknowledged: the judge (Gemini-3-Flash) and the skill-revision/generation teacher are both external LLMs whose quality the framework inherits, yet the robustness study swaps the judge but not the evolution teacher, and the WebShop judge's agreement with humans (86.3%) is noticeably lower than ALFWorld's. Whether the conclusions hold when the teacher is weaker — the realistic open-source setting — is untested.

Questions for authors

The answers to the following questions could materially affect my assessment.

  1. Circularity of the closed loop. The judge labels depend on the skill bank's contents; the bank is updated from those labels. Did you observe any instability or drift in skill-bank composition during training (e.g., skills oscillating between revision rounds, retrieval weights concentrating on a few modes)? Can you report the number of skills revised/added per run and their post-revision credit trajectories? This is a clarification request about the loop's dynamics, not a suggestion for future work.
  2. Frozen-bank control. Does an ablation exist with the skill bank entirely frozen (no soft or hard updates) but declaration + step-level attribution retained? If so, what are the numbers? If not, this is the key missing control for attributing gains to co-evolution rather than to declaration/attribution alone.
  3. Dependence on the initial skill bank. The initial banks are taken from SkillRL's released resources (45/55 entries). How sensitive are results to bank initialization — e.g., a smaller bank, randomly initialized principles, or banks constructed without access to the training distribution? Given that your objective is defined relative to the retrieved candidate set S_t, the choice of bank effectively defines the action space for declaration; quantifying this dependence would clarify what the method actually requires to work.
  4. Attribution-label quality as training bottleneck. The judge disagrees with humans on ~10–14% of steps, and your own error analysis shows the largest disagreement class (43%) is stale declarations after phase transitions — exactly the kind of label that is plausibly systematically wrong rather than random. Since the Random Judge ablation shows that uninformative labels hurt, did you test biased labels (e.g., flipping 10–15% of labels in a structured, non-random way)? The robustness of the training signal to realistic judge bias is materially different from robustness to random noise.
  5. Efficiency of the 13-candidate retrieval set. Each step retrieves 6 task-specific + 6 general + 1 fallback skills, and the policy must read all of them. How does performance vary with the number of retrieved candidates (K)? If a smaller retrieved set performs similarly, the declaration mechanism's cost-benefit changes substantially.
Reviewer_EJa7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCPO trains an LLM based agent with a set of skills. After the episode, a judge model labels each step which is then aggregated over the whole trajectory. The skill bank is updated via 2 updates: a soft update for the retrieval weights and a hard update rewriting skills and distilling new skills. Experiments use Qwen on ALFWorld and WebShop and comparisons include recent baselines.

Critical strengths

Explicit skill declaration makes the skill choice observable for training and diagnosis. The paper compares judge labels with human annotations, checks disagreements, and tests phase declarations as a control study. The last directly addresses whether the benefit comes merely from adding a structured output field.

Critical weaknesses

The experiments do not separate the skill optimization method from stronger supervision. SCPO receives dense step-level labels from a frontier model that already performs well , the baselines primarily learn from terminal rewards (but adding teacher-generated skills). The random-judge ablation shows that much of the advantage depends on informative judge feedback. The missing control is needs to use the same judge and supervision budget. A close previous method might be missing. Skill1, cited, already renders skill selection policy output. It uses the same Qwen2.5 backbone and benchmarks, requires no external judge, and reports high scores. Smaller gains are difficult to assess. Many ablation differences are small and missing CIs. Both benchmarks are close to saturation which limits what these differences show. Comparisons with other step-level credit methods would further help clarify the gains.

Questions for authors

What does explicit declaration add? How does SCPO compare with Skill1? How does SCPO compare with other RL methods receiving process rewards from the same judge? Please report mean and stdev across seeds for the ablations In the future, less saturated agent benchmarks would provide a stronger ground for claims.

Reviewer_1WR2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Existing approaches use skills to solve long-horizon decision-making tasks, but typically provide them as contextual information without making skill use explicit, making it difficult to evaluate and improve how skills are used. To address this limitation, this paper proposes Skill-Conditioned Policy Optimization (SCPO), which requires the agent to declare a skill before each action. Starting from an initial skill bank, the framework combines step-level retrospective attribution to evaluate skill use, GRPO-based policy learning that incorporates this feedback into the advantage, and mechanisms for selecting skills and updating the skill bank. Experiments on ALFWorld and WebShop demonstrate performance improvements. Component ablations and a comparison with phase-level declarations examine the contributions of the proposed design, while human validation and experiments with alternative judges assess judgment reliability and robustness to evaluator choice.

Critical strengths

S1. Explicit and evaluable skill use
The paper identifies the limitations of using skills as implicit contextual guidance and proposes SCPO to make skill use explicit through per-action declarations. Its retrospective mechanism evaluates both the appropriateness of the selected skill and the consistency of subsequent behavior with that skill. This feedback is converted into advantages and incorporated into a GRPO backbone, providing a straightforward way to improve skill selection and execution through policy learning.

S2. Evaluation-guided skill refinement and strong empirical performance
The paper proposes evaluation-guided skill selection and mechanisms for refining the skill bank through retrieval-weight adjustments and skill-content updates. SCPO achieves strong overall performance on ALFWorld and WebShop across two model scales. Component ablations and additional analyses of skill use, together with human validation of the attribution judgments, provide supporting evidence for the proposed framework.

Critical weaknesses

W1. Clarifying the methodological contribution beyond prior work.
Several components build on ideas explored in prior work: explicit policy-level skill selection in Skill1 [1], retrospective and terminal feedback for GRPO in CriticSearch [2], and joint policy and skill-bank evolution in SkillRL [3]. SCPO integrates these ideas through per-action declarations and skill-use feedback for policy learning and bank updates. Further clarifying the methodological advances enabled by this integration would help strengthen the paper’s contribution.

W2. Further validation of skill-specific attribution and skill-bank evolution
The ablations and additional analyses provide useful evidence for the framework’s effectiveness. Comparing skill-specific evaluation with generic action evaluation under matched conditions would further clarify the benefit of the proposed attribution mechanism. Examples of skills before and after revision, together with newly generated skills and their measured effects, would also help illustrate how the skill bank evolves and improves.

[1] Yaorui Shi et al. (2026). Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning. Sections 3.1-3.3.

[2] Yaocheng Zhang et al. (2026). CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic. Findings of ACL 2026, Sections 3.2-3.3.

[3] Peng Xia et al. (2026). SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. Sections 3.1-3.3.

Questions for authors

Q1. Clarification of error attribution: Wrong skill selection and correct selection followed by incorrect execution both receive negative raw attribution, and the resulting signals inform both policy and bank updates. Although the revision prompt allows a “no-change” response, could you clarify how the framework handles cases where execution failures reduce the retrieval weight of an otherwise appropriate skill?

Q2. Rationale for attribution normalization: When the batch standard deviation is nonzero, uniformly GOOD and uniformly BAD trajectories both yield zero trajectory-centered attribution. Could you explain the rationale for using this relative signal for skill-bank updates? Clarifying the numerical handling of zero or very small batch standard deviations would also be helpful.

Q3. Broader evaluation and baseline comparisons: Evaluation on additional benchmarks such as WebArena or AppWorld would help assess SCPO’s applicability across more diverse interaction settings. A comparison with Skill1, which is cited in the paper but not included in the experiments, under matched base models and training budgets would also be valuable. This comparison would further clarify SCPO’s benefits over existing approaches that learn explicit skill selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.