- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to maintain useful skill-based supervision as an LLM agent improves: fixed skills may become redundant or even harmful, while the agent may benefit from new or refined skills. To address this challenge, the paper proposes SkillCycle, a framework that alternates between (1) training an agent to internalize skills through on-policy distillation (OPD), using the same policy with access to a skill as the teacher, and (2) updating the skill bank while holding the policy fixed. Experiments on ALFWorld and WebShop provide evidence that adapting the skill bank can help sustain useful supervision as the learner evolves.
Critical strengths
-
The problem studied in this paper is well-motivated. Skill-based guidance can become unnecessary or counterproductive as a policy improves. Jointly evolving the policy and its skill bank is a reasonable approach to this changing need for supervision and goes beyond training with a fixed set of skills.
-
WebShop results are promising. The improvements over a controlled baseline that trains the student with a static skill bank suggest that updating the skills can provide value beyond policy training alone.
Critical weaknesses
-
There is limited evidence for the value of repeated co-evolution. In Table 2, the gains over a single skill-bank refresh are only 1.26 percentage points on ALFWorld and 1.82 percentage points on WebShop. Without confidence intervals or results across multiple seeds, it is difficult to determine whether these gains reflect a reliable benefit from iterative co-evolution or experimental noise.
-
The advantage over relevant baselines and prior work is unclear. SkillCycle evolution depends on a proposer, yet the outcome-only RL baseline, like GiGPO, outperforms SkillCycle on some tasks without access to such stronger model. This makes it difficult to assess whether SkillCycle’s additional resources and complexity are justified. Moreover, although the paper discusses OPID, which provides hierarchical skills and token-level supervision, and UCOB, which connects self-distillation with skill-memory evolution, the experiments do not clearly establish SkillCycle’s advantages over these closely related approaches.
Questions for authors
-
Can you provide qualitative examples showing both the emergence of useful skills absent from the initial skill bank and evidence that the policy subsequently internalizes those skills?
-
How does SkillCycle prevent the student from learning unsupported references to the skill bank? For example, a teacher with skill-bank access may generate phrases such as “referring to the skill bank,” which distillation could encourage the student to reproduce even when it lacks that access. Prior work has identified this type of hallucination; how does SkillCycle explicitly address it?
-
What evidence distinguishes the benefit of policy training from that of improved inference-time context management? In particular, how does SkillCycle compare with approaches that evolve the agent harness or manage skill context without updating the policy?
2 comments
Continued Official Comment to Reviewer fyQn
Q6: Separating policy learning from context management
SkillCycle is motivated by the changing relationship between policy capability and useful supervision. As the policy learns, a fixed bank may become redundant, misleading, or incomplete.
Our matched controls distinguish these contributions:
- Frozen policy, revised bank: measures the effect of bank revision and skill-context management.
- Trained student, fixed bank: measures policy internalization under unchanged supervision.
- Full SkillCycle: measures the interaction between policy learning and bank revision.
- No-OPD/proposer-only: removes distillation diagnostics while preserving the candidate-edit budget.
The no-skill condition provides a shared reference. Preliminary matched Full/no-OPD runs separately record the final bank effect and the diagnostic contribution.
Importantly, the final student is evaluated without skill input. This allows the policy-learning contribution to be assessed independently of skill retrieval at inference.
We are carefully revising the manuscript and will share additional results as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your consideration of our responses.
Official Comment to Reviewer fyQn
We sincerely thank the reviewer for the careful assessment and constructive questions. We appreciate your recognition of the changing supervision needs of a learning policy and the promising comparison with a static bank. We address the reliability, mechanism, and practical value of SkillCycle below.
Q1: Reliability of the gains over a single bank update
We will add independent training seeds and paired bootstrap confidence intervals to assess the reliability of the 1.26- and 1.82-point improvements over a single update.
Existing fixed three-seed WebShop evaluations show mean success-rate gains over the no-bank condition of 5.7292 points for Cycle 1 and 5.4688 points for Cycle 2. Cycles 2 and 3 are identical for all three seeds because Cycle 3 retains the accepted bank.
ALFWorld probes also show that later bank versions may perform worse. We will clarify this observed behavior and avoid implying that every additional cycle necessarily improves performance.
Q2: Value relative to the additional proposer and system cost
SkillCycle uses the proposer within a diagnostic and validation procedure. OPD feedback guides local revisions, while environmental acceptance determines whether those revisions enter the bank.
Under the matched candidate-edit budget, the Phase 2 candidate outperforms the no-OPD control by 7.111 percentage points, with a 95% confidence interval of [2.667, 11.778]. This supports the additional value of diagnostic screening. The overall improvement over B0 is smaller and its confidence interval includes zero.
We will report token consumption, environment rollouts, and API costs so that the benefits can be assessed against the additional resources.
Q3: Distinction from OPID and UCOB
OPID provides hierarchical skills and token-level supervision. UCOB combines skill-conditioned and skill-free on-policy trajectories with return comparisons and credit assignment to learn skill utilization, utility, and a reflective skill writer.
SkillCycle explicitly treats the persistent bank as an object of intervention. With the policy frozen, rules undergo keep/rewrite/drop/internalize decisions, and candidate edits must pass both paired-trajectory checks and bank-level environmental validation before the next policy-internalization stage.
Existing comparisons support the contribution of OPD diagnostics and bank revision. We will add direct OPID and UCOB comparisons with matched skill inputs, initialization, and budgets.
Q4: A qualitative example of skill emergence and internalization
A WebShop example concerns product-variant selection. The initial bank contains general search, item-selection, and purchase rules, but lacks an explicit check of whether an item supports the requested
fit_typeorshape_option. The student may find a relevant item and then attempt to select an unavailable variant or proceed to purchase.Token-level OPD discrepancies concentrate around variant selection and the subsequent purchase action. The proposer introduces a rule: first check whether the requested variant is supported; if it is available, select it; otherwise, return to the candidate list and continue searching.
The candidate undergoes paired-trajectory and bank-level validation, including checks for regressions and appropriate routing, before entering the accepted Cycle 1/Phase 2 bank. The analysis also distinguishes unsupported options from over-budget items.
To demonstrate subsequent internalization, we will provide before-and-after trajectories and success-rate comparisons for the student under the original no-skill prompt. The key behavior is whether the student checks variant availability and returns to search when necessary without receiving bank text or rule names at inference.
Q5: Unsupported references to an unavailable skill bank
The student uses the original no-skill prompt during internalization and inference. It does not receive skill-bank text, rule names, or router-provided candidate actions. Skill knowledge enters the parameter updates through teacher action and token-level signals.
We will additionally audit student outputs for unsupported references to a bank or rule, such as “refer to the skill bank,” and report their frequency under skill-free inference.