acceptodds
Withdrawn from ICLR 2027

SkillCycle: Co-Evolving Agent Policies and Skill Banks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Internalizing external skills changes a language agent's capabilities and, with them, the value of its remaining guidance: rules can become redundant, misleading, or insufficient for newly encountered decisions. This creates a coupled problem of learning from skills and adapting the skills that supervise further learning. We introduce SkillCycle, a framework for co-evolving agent policies and skill banks through a feedback loop between skill internalization and rule revision. Our central contribution is to give distillation feedback a second role: token-level contextual differences help locate rules for inspection, while interaction outcomes guide edits to their content and applicability. SkillCycle alternates between two phases: policy learning with a fixed skill bank and router, and rule revision with a frozen policy. Candidate edits undergo rule-level and whole-bank environment comparisons before they guide the next learning cycle. On WebShop, SkillCycle with a 3B model achieves a success rate of 74.74% and a score of 88.37 without inference-time skill inputs, representing relative improvements of 0.73% and 3.96% over the state-of-the-art (SOTA) model, respectively. In Cycle 3 ablations on ALFWorld and WebShop, SkillCycle's no-skill success rates improve by 10.18% and 18.11% relative to a static skill bank, and by 2.41% and 2.50% relative to a single bank update, respectively. These results show that continually revising skill guidance as the agent's capabilities change helps transform external skills into policy capabilities that require no skill inputs at inference. We will release code, configurations, skill banks, and evaluation protocols.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Language Agents, Skill Internalization, Skill Bank Evolution, On-Policy Distillation, Agentic Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_fyQnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to maintain useful skill-based supervision as an LLM agent improves: fixed skills may become redundant or even harmful, while the agent may benefit from new or refined skills. To address this challenge, the paper proposes SkillCycle, a framework that alternates between (1) training an agent to internalize skills through on-policy distillation (OPD), using the same policy with access to a skill as the teacher, and (2) updating the skill bank while holding the policy fixed. Experiments on ALFWorld and WebShop provide evidence that adapting the skill bank can help sustain useful supervision as the learner evolves.

Critical strengths
  1. The problem studied in this paper is well-motivated. Skill-based guidance can become unnecessary or counterproductive as a policy improves. Jointly evolving the policy and its skill bank is a reasonable approach to this changing need for supervision and goes beyond training with a fixed set of skills.

  2. WebShop results are promising. The improvements over a controlled baseline that trains the student with a static skill bank suggest that updating the skills can provide value beyond policy training alone.

Critical weaknesses
  1. There is limited evidence for the value of repeated co-evolution. In Table 2, the gains over a single skill-bank refresh are only 1.26 percentage points on ALFWorld and 1.82 percentage points on WebShop. Without confidence intervals or results across multiple seeds, it is difficult to determine whether these gains reflect a reliable benefit from iterative co-evolution or experimental noise.

  2. The advantage over relevant baselines and prior work is unclear. SkillCycle evolution depends on a proposer, yet the outcome-only RL baseline, like GiGPO, outperforms SkillCycle on some tasks without access to such stronger model. This makes it difficult to assess whether SkillCycle’s additional resources and complexity are justified. Moreover, although the paper discusses OPID, which provides hierarchical skills and token-level supervision, and UCOB, which connects self-distillation with skill-memory evolution, the experiments do not clearly establish SkillCycle’s advantages over these closely related approaches.

Questions for authors
  1. Can you provide qualitative examples showing both the emergence of useful skills absent from the initial skill bank and evidence that the policy subsequently internalizes those skills?

  2. How does SkillCycle prevent the student from learning unsupported references to the skill bank? For example, a teacher with skill-bank access may generate phrases such as “referring to the skill bank,” which distillation could encourage the student to reproduce even when it lacks that access. Prior work has identified this type of hallucination; how does SkillCycle explicitly address it?

  3. What evidence distinguishes the benefit of policy training from that of improved inference-time context management? In particular, how does SkillCycle compare with approaches that evolve the agent harness or manage skill context without updating the policy?

2 comments
AuthorsOpenReview ↗

Official Comment to Reviewer fyQn

We sincerely thank the reviewer for the careful assessment and constructive questions. We appreciate your recognition of the changing supervision needs of a learning policy and the promising comparison with a static bank. We address the reliability, mechanism, and practical value of SkillCycle below.

Q1: Reliability of the gains over a single bank update

We will add independent training seeds and paired bootstrap confidence intervals to assess the reliability of the 1.26- and 1.82-point improvements over a single update.

Existing fixed three-seed WebShop evaluations show mean success-rate gains over the no-bank condition of 5.7292 points for Cycle 1 and 5.4688 points for Cycle 2. Cycles 2 and 3 are identical for all three seeds because Cycle 3 retains the accepted bank.

ALFWorld probes also show that later bank versions may perform worse. We will clarify this observed behavior and avoid implying that every additional cycle necessarily improves performance.

Q2: Value relative to the additional proposer and system cost

SkillCycle uses the proposer within a diagnostic and validation procedure. OPD feedback guides local revisions, while environmental acceptance determines whether those revisions enter the bank.

Under the matched candidate-edit budget, the Phase 2 candidate outperforms the no-OPD control by 7.111 percentage points, with a 95% confidence interval of [2.667, 11.778]. This supports the additional value of diagnostic screening. The overall improvement over B0 is smaller and its confidence interval includes zero.

We will report token consumption, environment rollouts, and API costs so that the benefits can be assessed against the additional resources.

Q3: Distinction from OPID and UCOB

OPID provides hierarchical skills and token-level supervision. UCOB combines skill-conditioned and skill-free on-policy trajectories with return comparisons and credit assignment to learn skill utilization, utility, and a reflective skill writer.

SkillCycle explicitly treats the persistent bank as an object of intervention. With the policy frozen, rules undergo keep/rewrite/drop/internalize decisions, and candidate edits must pass both paired-trajectory checks and bank-level environmental validation before the next policy-internalization stage.

Existing comparisons support the contribution of OPD diagnostics and bank revision. We will add direct OPID and UCOB comparisons with matched skill inputs, initialization, and budgets.

Q4: A qualitative example of skill emergence and internalization

A WebShop example concerns product-variant selection. The initial bank contains general search, item-selection, and purchase rules, but lacks an explicit check of whether an item supports the requested fit_type or shape_option. The student may find a relevant item and then attempt to select an unavailable variant or proceed to purchase.

Token-level OPD discrepancies concentrate around variant selection and the subsequent purchase action. The proposer introduces a rule: first check whether the requested variant is supported; if it is available, select it; otherwise, return to the candidate list and continue searching.

The candidate undergoes paired-trajectory and bank-level validation, including checks for regressions and appropriate routing, before entering the accepted Cycle 1/Phase 2 bank. The analysis also distinguishes unsupported options from over-budget items.

To demonstrate subsequent internalization, we will provide before-and-after trajectories and success-rate comparisons for the student under the original no-skill prompt. The key behavior is whether the student checks variant availability and returns to search when necessary without receiving bank text or rule names at inference.

Q5: Unsupported references to an unavailable skill bank

The student uses the original no-skill prompt during internalization and inference. It does not receive skill-bank text, rule names, or router-provided candidate actions. Skill knowledge enters the parameter updates through teacher action and token-level signals.

We will additionally audit student outputs for unsupported references to a bank or rule, such as “refer to the skill bank,” and report their frequency under skill-free inference.

AuthorsOpenReview ↗

Continued Official Comment to Reviewer fyQn

Q6: Separating policy learning from context management

SkillCycle is motivated by the changing relationship between policy capability and useful supervision. As the policy learns, a fixed bank may become redundant, misleading, or incomplete.

Our matched controls distinguish these contributions:

  • Frozen policy, revised bank: measures the effect of bank revision and skill-context management.
  • Trained student, fixed bank: measures policy internalization under unchanged supervision.
  • Full SkillCycle: measures the interaction between policy learning and bank revision.
  • No-OPD/proposer-only: removes distillation diagnostics while preserving the candidate-edit budget.

The no-skill condition provides a shared reference. Preliminary matched Full/no-OPD runs separately record the final bank effect and the diagnostic contribution.

Importantly, the final student is evaluated without skill input. This allows the policy-learning contribution to be assessed independently of skill retrieval at inference.

We are carefully revising the manuscript and will share additional results as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your consideration of our responses. We would be grateful if you would consider raising your score.\texttt{We would be grateful if you would consider raising your score.}

Reviewer_FB6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents SkillCycle, a framework in which a language agent's policy and its external skill bank improve together through an alternating loop. In the internalization phase, the bank and a state-aware router stay fixed. SkillCycle combines OPD and outcome reward for skill internalization. The goal is to move the value of skill guidance into the parameters, so the agent needs no skill input at inference time. In the revision phase, the policy is frozen. The token-level gaps point to which rules shaped unsuccessful decisions, trajectories supply outcome labels, and an offline proposer issues KEEP, REWRITE, DROP, or ADD edits. Each candidate edit is checked with paired rollouts and a whole-bank gate before it is committed. Experiments are conducted on WebShop and ALFWorld.

Critical strengths

The central idea is well motivated. The authors identify a real gap. Internalization methods remove the agent's dependence on skills, but nobody asks how the skill bank itself should change once the policy has learned from it. A rule that helped at cycle one can be redundant or misleading at cycle three. The design follows naturally from that observation. Keeping the bank fixed during learning and the policy fixed during revision is a clean way to attribute effects, and it lets every candidate edit be tested before it influences further training. It's a good design to leverage the distillation feedback for skill revision.

Critical weaknesses
  • The writing is hard to follow. For example, the abstract's phrase "In Cycle 3 ablations on ALFWorld and WebShop" appears without context (what is Cycle 3?). I can find AI slops here and there.

  • The authors have visibly compressed the official template. Line spacing is tight, the text is crowded, and figures, tables, and equations compete for space. The structure is problematic; even with this compression, the related work section is defered to appendix.

  • The experimental benchmarks (ALFWorld and WebShop) are toy.

  • The advantage over recent skill-based baselines is unclear to me. For example, in your abstract, you claim improvements of 0.73% and 3.96% over the state-of-the-art (SOTA) model. However, I am not sure what do you mean by SOTA models. Is it SOTA skill-based baseline methods or frontier models like Claude opus 5.5?

Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Official Comments to Reviewer FB6B

We sincerely thank the reviewer for the thoughtful review and encouraging recognition of our motivation and validation design. We also appreciate the specific suggestions on presentation and experimental scope, which will help us communicate the work more clearly.

Q1: Meaning of Cycle 3 and writing clarity

“Cycle 3” refers to the SkillCycle iteration index. Each cycle involves policy internalization and a subsequent bank-revision stage, with an existing bank retained when no candidate is accepted.

We will define the cycle terminology at its first appearance, explain the corresponding bank and router states, and replace the context-free phrase “Cycle 3 ablation” in the abstract. We will also revise the manuscript for consistent terminology, concrete explanations, and clearer prose.

Q2: Template spacing and paper organization

We agree that the current presentation is too compressed. We will restore readable spacing consistent with the official template, improve separation around figures and tables, and move related work into the main text.

The revised PDF will undergo a page-by-page visual check.

Q3: Evaluation beyond ALFWorld and WebShop

ALFWorld and WebShop allow controlled comparisons and detailed inspection of skill routing and interaction trajectories. Their task coverage remains limited, and broader environments are important for assessing applicability.

We will explicitly discuss this limitation and extend evaluation to tool-using, multi-tool, and cross-environment tasks, including tau-bench or a comparable public benchmark. Additional results will be reported as soon as they are available.

Q4: Meaning of the reported improvements and the comparison baseline

The reported 0.73% and 3.96% improvements are both relative to OPID, the skill-based baseline used in our comparison. They do not refer to comparisons with frontier commercial models.

We will name OPID explicitly in the abstract and provide the corresponding metrics and absolute values, replacing the ambiguous phrase “state-of-the-art model.”

We are carefully revising the manuscript and will share additional results as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your consideration of our responses. We would be grateful if you would consider raising your score.

Reviewer_uPsMOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This submission points out that as an agent internalizes external skills, the remaining skills can become unnecessary, even harmful, or unable to handle new situations. To address this, the authors propose SkillCycle, which jointly optimizes the policy and the skill bank. SkillCycle alternates between two stages. In the policy-training stage, the skill bank and a rule-based router are kept fixed, and the policy is trained with PPO using a reward that combines the task reward with the token-wise probability difference from a teacher that receives the skills as context. In the skill-revision stage, the policy is kept fixed and the skill bank is revised. The authors claim that SkillCycle outperforms OPID, the previous state of the art, on Qwen3 and Qwen3.5, and they report results supporting the usefulness of both the internalization and the skill-bank revision components.

Critical strengths
  • Jointly optimizing the skill bank and the policy is a good idea. It appears to be a useful lever for internalizing existing skills while also achieving better performance.
  • The computation required by the framework appears efficient compared with the other methods.
  • The authors run experiments not only on open-weight models but also on API models. These experiments are limited, but they broaden the coverage of the paper.
Critical weaknesses
  • The most serious concern is that the comparison does not appear to be fair. Only the proposed method is initialized from weights trained with GiGPO or OPID, while the remaining methods in Table 1 are taken directly from the numbers reported in the baseline paper (OPID), so the same initialization is not applied across methods. This prevents an independent evaluation of the improvement that SkillCycle itself provides, and it makes a fair comparison with the baselines difficult. I believe Table 1 should report the performance of GiGPO (i.e., the base weights used to train SkillCycle), as well as the performance of OPID trained under the same computation budget as SkillCycle. At present, GiGPO is not included as a baseline in Table 1, and the performance of the initial weights is not reported for the 3B model. In addition, the paper should explain when OPID is used as the initial weights and when GiGPO is used, and what criteria govern this choice. Without this, it is hard to be confident that the initialization is done in a principled way rather than an arbitrary one.
  • SkillRL and Skill1 are cited, but their performance is not presented. Results for methods that also use skills are needed to judge whether the teacher provides a meaningful advantage over such baselines, and they would also help justify the proposed alternating algorithm.
  • Looking at the Student performance of SkillCycle in Table 1, apart from the WebShop Score, it is either lower than OPID and the other methods, or the gains do not appear to be statistically significant. If they are significant, I would appreciate it if the authors could provide supporting evidence.
  • In the ablation study, why does SkillCycle perform worse on ALFWorld than simply continuing to train with GiGPO from the same initialization?
  • Internalizing skills presumably interacts with the capabilities the agent or LLM already has, yet there is no study on whether those capabilities are preserved. The Limitations section mentions transfer across tasks and retention in lifelong learning as future work. However, I think preserving existing capabilities is essential for practicality and is too important to be deferred to future work.
  • The router appears to be rule-based, which relies heavily on human-designed rules for each environment and limits the achievable upper bound. In practice, the conditions under which a skill applies are often hard to parse clearly, but SkillCycle is validated only on ALFWorld and WebShop, where parsing is relatively easy. This issue does not seem to be sufficiently considered.
  • Alternating algorithms like this can be unstable, but there is little empirical study of stability or convergence across different training seeds or over a larger number of cycles.
  • In the OPID-init part of Table 7 in the Appendix, the WebShop entries are still marked as TBD.
Questions for authors
  • Why are there no results for cycle 2 with Qwen2.5-3B-Instruct? Does instability arise as the cycles repeat? The missing WebShop result at cycle 4 further raises this concern.
  • Overall, too much vertical space seems to have been cut, and this seriously hurts readability (e.g., the space above the Table 2 caption and the space below Figure 3).
  • Beyond these points, I would appreciate it if the authors could respond to, and run experiments addressing, the weaknesses raised above.
3 comments
AuthorsOpenReview ↗

Official Comments to Reviewer uPsM

We sincerely thank the reviewer for the detailed and thoughtful assessment. We appreciate your recognition of the joint optimization idea and computational efficiency. Your concerns about fairness, reproducibility, and capability retention are valuable, and we address them individually below.

Q1: Fairness of initialization and training comparisons

We agree that the sources and initialization of the reported results should be clearer. The revised tables will explicitly distinguish results re-evaluated from shared checkpoints, results reported by prior work, and our matched-budget comparisons.

To isolate SkillCycle’s contribution, we will present controls using the same initialization, training steps, and evaluation task list. Existing no-OPD, B0, no-skill, and fixed-reference controls will be included in the main comparison, while literature results will be marked as “reported by prior work.”

Q2: GiGPO, initial-checkpoint, and matched-budget OPID baselines

We will complete the GiGPO and OPID comparisons using matched checkpoints, evaluation tasks, seeds, and budgets.

For reference, the existing WebShop TCOD-3B evaluation on 128 fixed seen tasks across three evaluation seeds gives:

Model or stageScoreSuccess rate (%)
Cycle 1 student85.498269.5313
Parent teacher86.641873.4375
Phase 2 teacher87.020373.9583

The Phase 2 teacher improves over the parent teacher by 0.3785 Score and 0.5208 percentage points in success rate. We will identify these as training-side evaluations and report the requested baseline rows separately.

Q3: Principles for selecting initialization

GiGPO initialization is used for comparisons with outcome-reward policy training. OPID initialization provides a shared starting point for comparisons involving skill-conditioned distillation.

When only one corresponding public checkpoint is available for an environment, we use that checkpoint and explicitly identify it in the table. We will document the initialization of every setting and use a common starting checkpoint within each controlled comparison.

Q4: Comparisons with SkillRL and Skill1

We agree that these are important comparisons. We will add SkillRL and Skill1 evaluations under shared initialization, training budgets, and evaluation protocols.

These comparisons will help assess the value of the skill-conditioned teacher and the alternating optimization procedure relative to other skill-based approaches.

Q5: Statistical significance of the gains

We will report paired bootstrap confidence intervals and McNemar tests for binary success comparisons.

For bank revision, the recorded Phase 2 candidate improves over the no-OPD control by 7.111 percentage points, with a 95% confidence interval of [2.667, 11.778]. Its improvement over B0 is 1.778 points, with an interval of [-2.000, 5.778]. This supports the diagnostic-selection benefit relative to the no-OPD control.

We will separately report uncertainty for student performance and provide the three-seed WebShop evaluations as descriptive stability evidence.

Q6: Lower ALFWorld performance than continued GiGPO training

Continuing GiGPO directly updates the policy distribution. During SkillCycle’s revision stage, the policy is frozen, so improvement depends on rule granularity, routing accuracy, and interactions among candidate edits.

Our ALFWorld records include rejected candidate banks, and fixed-policy probes found revised bank versions that underperformed the no-skill condition. These observations show why bank revision requires environmental acceptance checks.

We will report unchanged-bank, no-OPD, and no-skill controls and make this behavior explicit in the discussion.

Q7: Retention of existing capabilities

We agree that capability retention is important for practical use. Existing evaluations compare no-skill, bank-conditioned, and fixed-reference settings on the same task list, but do not yet provide a complete assessment of general capabilities or cross-task transfer.

We will add at least one seen/unseen task-family retention evaluation and report the results alongside task performance.

Q8: Dependence on an environment-specific rule-based router

The current router uses observable state predicates, allow/block conditions, priorities, and abstention. This makes individual skill injections inspectable, while introducing dependence on environment-specific conditions.

For environments where applicability is difficult to express explicitly, learned routing, retrieval confidence, and abstention provide natural extensions. We will clearly describe the current router’s scope and evaluate broader routing settings in future experiments.

AuthorsOpenReview ↗

Continued Official Comments to Reviewer uPsM

Q9: Stability across seeds and cycles

Under the existing fixed three-seed WebShop evaluation, mean success rates are 62.5000% without a bank, 68.2292% for Cycle 1, and 67.9688% for Cycle 2. Cycle 3 exactly matches Cycle 2 because the previously accepted bank is retained.

ALFWorld probes also show that successive bank versions do not automatically improve performance. We will report these results as evidence of the observed stability and non-monotonic behavior, and supplement them with additional training seeds and cycles.

Q10: Missing Cycle 2 and Cycle 4 results

These entries correspond to runs that remain incomplete or have not yet passed the experimental validation checks. They do not represent documented training divergence.

Our reporting pipeline requires checks of saved artifacts, trajectory pairing, prompts, and runtime settings before results enter the formal tables. We will state the status and reason for each missing entry and add the validated results when available.

Q11: The TBD entry in Appendix Table 7

We will complete the OPID-initialized WebShop evaluation before the deadline using the same evaluation task list, budget, and seeds, and replace the TBD entry with the validated result.

Q12: Readability and excessive spacing compression

Thank you for identifying these issues. We will restore the official template’s standard spacing, improve the space around figures and tables, and move related work into the main text.

We will recompile and inspect the PDF, including the spacing above the Table 2 caption and below Figure 3.

We are carefully revising the manuscript and will share additional results as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your consideration of our responses. We would be grateful if you would consider raising your score.\texttt{We would be grateful if you would consider raising your score.}

Reviewer_uPsMOpenReview ↗

Comments to Authors about the discussion period

The reviewer sincerely thank the authors for their efforts in preparing the rebuttal. I would like to share how I approached this discussion phase, along with my updated assessment.

I read the rebuttal as soon as it was posted, and I then waited for the authors to complete the promised experiments. As of almost November 15 (AoE), however, none of those experiments has been reported. About 9-10 days have passed since the reviews were released. In my view, the fact that the promised experiments are still unavailable suggests that the submission needs a substantial amount of further experimentation and revision. I believe that work goes beyond what is expected at ICLR. I see the rebuttal period as a chance to strengthen a paper that is already reasonably complete. It is not meant to finish a paper that is still in progress, and that places a heavy demand on the authors', the reviewers' and even AC's time.

For these reasons, I am lowering my recommendation to 1: Clear Reject.

The responses to my questions consist of commitments without supporting results, so I cannot evaluate them at this stage. One minor point on formatting: the numbering in the rebuttal does not follow the numbering of the Weaknesses and Questions in my review, which made it somewhat difficult to match each response to the concern it addresses. Following the reviewer's original structure would make future responses easier to follow. I hope the authors find these comments useful as they continue to develop this work.

Reviewer_LyJ3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SkillCycle. The framework switches between skill internalization into model parameters through contextual self distillation and RL, and feedback driven skill bank revision using token level diagnostics, LLM based rule editing, and environment based verification. Experiments on ALFWorld and WebShop show improvements over static and single-refresh skill banks

Critical strengths

The framework targets a practical setting - The framework aims to internalize external skills into model parameters so that the agent can run without skill retrieval at inference time.

Critical weaknesses

Their proposed frameowrk just combines existing techniques. The methodological contribution beyond integrating these components is limited. The experiments are also restricted to only just two environments which makes it difficult to assess how well the this method generalizes to other settings. Also, The skill bank is revised using trajectories from the same environments and task families used during training. So it remains unclear whather the learned skills generalize to unseen task or different environments Also there is very marginal improvements over baselines as we can see from table 2.

Questions for authors

See weakness Also What is the additional computational cost introduced by this framework.

1 comment
AuthorsOpenReview ↗

Official Comment to Reviewer LyJ3

We sincerely thank the reviewer for the time and care devoted to evaluating our work. We appreciate your recognition of the practical value of skill-free inference. Your comments help us clarify the contribution, experimental scope, and computational requirements.

Q1: Motivation and methodological contribution

SkillCycle studies how a supervision source should evolve as the policy learns from it. A useful rule may later become redundant, misleading, or insufficient for newly exposed errors. This motivates explicit management of the skill bank’s lifecycle.

The method connects four designs:

  1. Frozen-stage alternation: the bank and router remain fixed during policy internalization, and the policy remains fixed during bank revision.
  2. Token-level OPD diagnosis: distillation feedback connects unsuccessful decisions to rule-level evidence.
  3. Environmental acceptance: candidate edits undergo paired-trajectory checks and bank-level validation before adoption.
  4. Explicit lifecycle decisions: rules are assigned keep/rewrite/drop/internalize/insufficient-evidence outcomes.

Together, these designs make skill revision responsive to the learner’s current needs and allow proposed changes to be tested before they affect subsequent training.

Q2: Benchmark coverage and generalization

ALFWorld and WebShop support controlled comparisons and detailed inspection of routing and interaction trajectories. We agree that evaluation in these two environments does not establish transfer to different environments or unseen task families.

We will add seen/unseen task-family evaluations and extend testing to tau-bench or comparable interactive benchmarks. The revision will clearly distinguish within-environment evaluation from broader transfer.

Q3: Interpretation of the gains in Table 2

Table 2 is an ablation of skill-bank update strategies. The 1.26- and 1.82-point gains over a single update measure the additional benefit of continued adaptation. The static-bank comparison addresses the broader question of whether revising the supervision source adds value beyond training with unchanged skills.

We will make this distinction clearer and report multiple training seeds and confidence intervals so that the reliability of the incremental gains can be assessed directly.

Q4: Additional computational cost

The additional training cost comes from skill-conditioned teacher evaluations, candidate proposals, paired rollouts, and bank-level validation. We will report training steps, teacher and proposer tokens, environment rollouts, and API costs under matched budgets.

At inference, the student receives only the original task input and does not require skill retrieval, bank text, or proposer calls. Thus, the additional revision machinery is used during training.

We are carefully revising the manuscript and will share additional results as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your consideration of our responses. We would be grateful if you would consider raising your score.\texttt{We would be grateful if you would consider raising your score.}

Reviewer_xL7XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SkillCycle changes between learning from a skill bank and revising that bank as the policy improves. It uses distillation feedback to identify rules for inspection, then verifies proposed edits through environment interactions. Experiments on ALFWorld and WebShop evaluate performance without skill inputs at inference.

Critical strengths
  1. We find the motivation sensible: as an agent learns, its existing guidance may become less useful and need updating.

  2. We agree with the matched-budget comparisons with static and single-refresh banks. Checking edits at both the rule and whole-bank levels is also a thoughtful design choice.

Critical weaknesses
  1. We find the conceptual novelty somewhat limited. As Appendix A discusses, prior methods already explore policy–skill co-evolution and skill internalization. The main addition appears to be using distillation feedback to guide rule inspection, followed by environment-based verification. This is a reasonable extension, but we are not yet convinced it offers a substantial new insight beyond combining these existing ideas. It is still in the cycle of SKill with anything.

  2. The experiments do not clearly isolate the value of this central addition. The bank-update comparisons show that revising guidance helps, but do not establish how much distillation-based diagnosis adds over simpler outcome-based revision.

Questions for authors
  1. Could you explain more directly what SkillCycle adds conceptually beyond skillrl and ucbo?

  2. Can you have a matched-budget comparison between distillation-guided revision and revision based only on interaction outcomes?

  3. Are the gains over a single refresh consistent across independent training runs?

  4. I understand alfworld and webshop such text world are popular in skill-style tasks due to the limited resource. But the task type of them is single and easy to "hack". It's optional for you to test on some terminal-style datasets like tau-bench, just a suggestion.

1 comment
AuthorsOpenReview ↗

Official Comment to Reviewer xL7X

Comment:

We sincerely thank the reviewer for the careful reading and constructive feedback. We especially appreciate your recognition of our motivation, budget-matched comparisons, and validation design. We address your concerns below.

Q1: Conceptual contribution beyond SkillRL and UCOB

SkillRL focuses on recursive skill-augmented policy optimization and skill-bank evolution. UCOB combines skill utilization and evolution through credit-aware, on-policy bidirectional self-distillation, using feedback to update skill utility and a reflective skill writer.

SkillCycle explicitly addresses how a persistent skill bank should change after the policy has internalized some of its knowledge. It alternates between two stages: during internalization, the bank and router remain fixed while the policy learns through OPD and outcome rewards; during revision, the policy remains fixed while token-level distillation diagnostics identify rules for keep/rewrite/drop/internalize decisions.

Candidate edits must then pass paired-trajectory checks and bank-level environmental validation. Our contribution is this explicit connection between the policy’s changing supervision needs, rule-level diagnosis, and validated skill lifecycle management.

Q2: Budget-matched comparison with outcome-only revision

We agree that this comparison is essential for isolating the diagnostic mechanism. We have implemented a matched-budget no-OPD/proposer-only control that preserves the candidate-edit budget, skill-document budget, and evaluation protocol while removing OPD-based diagnostic screening.

In the recorded Phase 2 comparison, the candidate outperforms the no-OPD control by 7.111 percentage points, with a 95% cluster confidence interval of [2.667, 11.778]. The improvement over the reference baseline B0 is 1.778 points, with an interval of [-2.000, 5.778]. These results support the contribution of OPD diagnostics to candidate selection.

We will report the matched training steps, candidate-edit budgets, and token consumption, together with convergence speed and computational cost.

Q3: Consistency across runs and interpretation of repeated updates

We appreciate the opportunity to clarify the cycle terminology. A later cycle does not necessarily introduce another bank update: the existing bank is retained when no candidate is accepted.

Under the fixed three-seed evaluation protocol, WebShop success rates average 68.2292%, 67.9688%, and 67.9688% for Cycles 1, 2, and 3. Cycles 2 and 3 are identical for every seed because Cycle 3 retains the bank accepted in Cycle 2. The ALFWorld M0 probe likewise produces no new bank in Cycle 3.

We will clarify these protocol states and distinguish evaluation-seed results from independent training runs. Additional training seeds will assess run-to-run consistency.

Q4: Evaluation beyond ALFWorld and WebShop

Thank you for this helpful suggestion. ALFWorld and WebShop enable controlled inspection of skill routing, paired trajectories, and environmental acceptance decisions. We agree that broader environments are needed to evaluate generalization.

We will extend the evaluation to tau-bench or comparable tool-using benchmarks and explicitly discuss the current benchmark coverage in the limitations.

We are carefully revising the manuscript and will share additional results as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your consideration of our responses. We would be grateful if you would consider raising your score.\texttt{We would be grateful if you would consider raising your score.}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.