acceptodds
Under review as a conference paper at ICLR 2027

Beyond Initial Skills: Contrast-Guided Skill Evolution for Robust Learning in LLM Agents

Abstract

Large language model (LLM) agents can learn continually by accumulating and updating external procedural skills while keeping their model parameters frozen. Existing approaches typically evolve these skills by revising prior skills based on interaction experiences collected from new tasks. However, we find that their learning gains vary substantially with initial skills, even under the same sequence of subsequent training tasks, resulting in unstable or even negative skill evolution. Further analysis shows that initial skills influence both the experiences collected through agent interactions and the subsequent skill updates, causing their effects to persist throughout the evolution process. To enable robust skill evolution beyond initial skills, we propose Skill-CoE, a Contrast-guided skill Evolution framework that explores alternative skill update directions and leverages their complementary strengths for effective updates. Specifically, given the same interaction experience, Skill-CoE constructs two evolution candidates by revising the current skill and reconstructing a new skill, respectively. It then compares their executions on the same tasks using a pairwise preference objective, with the current skill serving as a fixed reference to identify complementary strengths. Guided by these comparisons, Skill-CoE recombines the candidates to incorporate useful new procedures while preserving effective existing experience. Experiments on ALFWorld, AppWorld, and ScienceWorld show that Skill-CoE consistently achieves stable learning gains across all evaluated initial skills and outperforms existing methods in final test performance. Code is available at https://anonymous.4open.science/r/Skill-CoE-2737.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
large language model agents, continual learning, procedural skills, skill evolution, initialization dependence, learning from experience

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ozbcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates skill evolution for LLM agents. This paper proposes Skill-CoE, a contrast-guided skill evolution framework, which consists of (1) contrastive sampling, (2) within-task trajectory preferences, (3) contrast-guided skill recombination. Extensive experiments on ALFWorld, ScienceWorld, and AppWorld demonstrate the effectiveness of the proposed framework.

Critical strengths
  1. The idea of contrast-guided skill evolution with non-parametric DPO is novel.
  2. The robustness of the initial skill is an interesting problem for skill evolution.
Critical weaknesses
  1. Although the investigated problem is interesting, the proposed solution introduces substantial complexity whose necessity is insufficiently justified. Section 3.3 combines several heuristic choices, while the ablations mainly establish the usefulness of broad components rather than the distinctive recombination mechanism. A comparison against a streamlined recombination procedure, under matched computation and validation conditions, would help determine whether the additional machinery is warranted. The main body should also explain all essential update and acceptance steps. Without this evidence, the paper’s algorithmic insight remains less convincing than its reported performance gains.

  2. This paper uses ALFWorld, ScienceWorld, and AppWorld as the benchmark; however, these three benchmarks cannot serve as a valid agent benchmark for modern LLMs. Please follow SkillOpt to introduce more benchmarks, or use more recent agent benchmarks with enough complexity.

  3. Lack of repeated runs for the experiments.

  4. Lack of token cost analysis. It seems that Skill-CoE uses much more tokens to produce enough rollouts due to the contrast nature.

Questions for authors

I think that Weakness #1 is my biggest concern, which cannot be solved in the rebuttal phase.

3 comments
AuthorsOpenReview ↗

Response to reviewer ozbc

Thank you for your careful review and constructive feedback. We address your main concerns below.

W1. Necessity of the recombination mechanism and clarity of the update procedure

1. Comparison with streamlined recombination

To test whether the added complexity is justified, we compare Skill-CoE with simplified recombination (SR), which fixes the base and donor roles, samples cases uniformly, and directly generates local edits. SR uses the same validation protocol and comparable learning tokens: 6.3% fewer on ALFWorld and 1.5% more on AppWorld (Appendix B.5). RR retains staged generation but randomizes selection; Single-stage retains our selection but generates edits directly. Results are in Table 4.

VariantALFWorld SR (%)AppWorld TGC (%)AppWorld SGC (%)
Skill-CoE93.03 ± 1.8885.71 ± 2.0673.81 ± 2.73
Simplified recombination (SR)85.57 ± 2.6278.37 ± 2.8167.86 ± 3.09
Random recombination (RR)86.07 ± 3.1180.16 ± 3.0565.48 ± 4.49
Single-stage generation87.81 ± 2.1582.74 ± 2.5964.29 ± 3.57

Skill-CoE exceeds SR by 7.46 points in ALFWorld SR and 7.34 points in AppWorld TGC, showing the benefit of the full design over this simpler alternative at comparable token cost. The RR and Single-stage results further support guided selection and staged generation, respectively.

We also isolate the effect of case sampling while holding the other recombination inputs and generation settings fixed. Preference-guided sampling yields larger mean execution gains than uniform sampling on the validation sets: an additional 2.19 points in ALFWorld SR and 1.87 in AppWorld TGC. Every generated candidate is evaluated, without acceptance filtering (Appendix B.4).

2. Update and acceptance steps

Sections 3.3.1–3.3.3 now explain skill and case selection, generation, and updates. The first generated skill with lower loss than the base is accepted; if none succeeds within J attempts, the base is used. Validation selects the final skill for testing and does not reject learning updates (Section 4.1). Further details are in Appendices A.2–A.4.

W2. Evaluation on an additional benchmark

Following your suggestion, we added SpreadsheetBench [1]. We use its official 400-task Verified subset, holding out 280 tasks for testing and splitting the remainder between training and validation (Appendix B.2). Agents manipulate real workbooks through Python/Bash, and the evaluator checks output cell values. Results are shown in Table 1:

MethodSpreadsheetBench SR (%)
ExpeL67.74 ± 1.09
ACE68.10 ± 3.30
SkillOpt74.40 ± 3.85
Skill-Pro73.21 ± 2.23
Skill-CoE81.07 ± 1.86

Skill-CoE exceeds the strongest baseline, SkillOpt, by 6.67 percentage points in mean SR, extending the observed advantage to spreadsheet manipulation.

W3. Repeated runs

The original manuscript reported the best of three runs for each method. We apologize for not reporting the full statistics. Tables 1–4 now report means over three independent full learning runs, with standard deviations in Table 1 and Appendix B.6. Each run starts from the same initialization, collects fresh experience, updates its knowledge, and selects its final checkpoint using validation before testing with knowledge frozen.

Skill-CoE retains the highest mean performance across all four benchmarks.

W4. Token cost of contrastive learning

We match executed training rollouts: Skill-CoE runs for one epoch and learning baselines for three on the same tasks, counting executions under the current skill and both candidates. Local editing and scoring reuse recorded trajectories without additional training rollouts. Token counts cover training interactions and knowledge updates, including generation, scoring, failed calls, and retries; validation and test evaluation are excluded (Section 4.7; Appendix B.8). Appendix B.8 also reports LLM calls under the same scope.

Table 6 reports three-run mean learning tokens (input / output, in millions).

MethodALFWorldAppWorldScienceWorldSpreadsheetBench
Executed training rollouts180135180240
ExpeL12.64 / 0.14910.92 / 0.30512.28 / 0.29519.84 / 0.977
ACE15.95 / 0.37217.70 / 0.39716.03 / 0.35620.41 / 1.504
SkillOpt15.89 / 0.22919.63 / 0.49816.09 / 0.26319.26 / 0.968
Skill-Pro18.44 / 0.19729.98 / 0.31816.60 / 0.26323.97 / 0.937
Skill-CoE21.87 / 0.28328.85 / 0.40918.64 / 0.32525.94 / 1.234

Skill-CoE generally uses more learning tokens but achieves higher mean performance. Figure 6 shows the cost–performance trade-off relative to ExpeL (Appendix B.8).

[1] Ma Z, Zhang B, Zhang J, et al. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

Thanks for the response and the newly added experiment results. For Weakness #1, I still feel that the proposed skill recombination method is overly engineered (with a complex decision logic as presented in the appendix) that we cannot identify which part takes the central role and which contribute only marginally. Thus, I would maintain my score as a weak reject.

AuthorsOpenReview ↗

Response to reviewer ozbc

Thank you for the follow-up. Let us explain why we use this design and which parts contribute most. We also hope the additional explanations below will be helpful in your reassessment.

1. Why the recombination design has this structure

The design centers on local complementarity: a skill with higher aggregate preference loss can still better support preferred behavior on individual cases. We therefore use the same preference objective at two levels. Aggregate loss selects the base and evaluates proposed edits; pairwise loss differences identify a complementary donor and evidence for what to change or preserve. Analysis, alignment, and planning then translate this evidence into concrete skill edits (Section 3.3; Appendices A.2–A.3). This connects the main decisions around a common goal: incorporating useful new procedures while preserving effective existing behavior.

The simplified recombination variant (SR) directly tests a less elaborate alternative: fixed base/donor roles, uniform case sampling, and direct editing. It uses the same validation protocol and comparable learning tokens. Skill-CoE still exceeds SR by 7.46 points in ALFWorld SR and 7.34 in AppWorld TGC (Appendix B.5). Thus, the additional structure brings measurable benefits over this simpler procedure at comparable learning cost.

2. Central mechanisms and supplementary contributions

The ablations show that the main benefit of recombination comes from incorporating complementary experience from different skills into an improved skill. Loss-guided selection identifies what to incorporate and preserve, while staged generation turns this evidence into concrete skill edits. These two mechanisms jointly support this process. Successful replay provides additional preservation support by supplying successful behaviors that preference pairs can miss.

Tables 3–4 report the following performance drops relative to Skill-CoE (percentage points); variant details are in Appendix B.5.

AblationALFWorld SRAppWorld TGCAppWorld SGC
NR: select a skill without recombination9.458.1311.90
RR: randomize skill and case selection6.975.568.33
SG: replace staged generation with direct editing5.222.989.52
SR: simplify both selection and generation7.467.345.95
WR: remove successful replay2.744.173.57

NR has the largest drops across all three metrics, supporting the central role of recombination: selecting a single skill leaves useful procedures from the other skills unused.

The substantial drops in RR and SG show that neither mechanism alone sustains the full method's performance. Without loss guidance, generation lacks targeted evidence; without staged generation, useful evidence is harder to turn into effective edits in a single step. Loss guidance finds experience worth learning from, while staged analysis, alignment, and planning turn it into concrete changes. Together, they support the effective use of complementary information.

WR's smaller drops than NR and RR support a supplementary role for replay. It supplies successful trajectories that preference pairs can miss when returns tie, helping preserve effective behavior during updates.

Reviewer_yoiWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies continual learning in LLM agents that update external procedural skills while keeping model parameters frozen. The authors first show that existing skill evolution methods are sensitive to the initial skill: under an identical sequence of subsequent training tasks, their learning gains vary substantially across initializations and can even turn negative. The authors attribute this to the practice of repeatedly revising the current skill, which lets the influence of the initial skill propagate through later interactions and updates. To address this, the paper proposes Skill-CoE, a contrast guided skill evolution framework that generates two candidates from the same interaction experience by revising the current skill and by independently reconstructing a skill from raw experience, executes both on shared tasks, forms within task trajectory preferences, and adapts a pairwise preference objective with the current skill as a fixed reference to guide the recombination of complementary strengths. Experiments on three interactive benchmarks show stable positive learning gains across different initial skills and stronger final performance than existing methods.

Critical strengths
  1. The paper identifies and systematically studies a real and previously underexamined problem: the sensitivity of external skill evolution to initialization. The observation that several strong existing methods can yield unstable or even negative learning gains under the same training sequence is a compelling and well supported motivation, and the follow up analysis of why initial skill influence persists gives the subsequent method design a clear logical basis.

  2. The method is thoughtfully designed and its components are individually validated. Exploring two update directions that isolate revision from reconstruction, constructing within task preferences under a fixed reference, and selecting a base and a donor for guided recombination each address one of the stated challenges, and the ablation study confirms that removing any of these components degrades performance.

  3. The empirical evaluation is unusually comprehensive for this area, covering three benchmarks with distinct action spaces, three families of baselines, several initialization construction schemes, multiple backbone models, and process level analyses of success coverage and repair versus regression that help explain why the proposed mechanisms work rather than only reporting aggregate scores.

Critical weaknesses
  1. Computational cost is a serious concern that the paper never quantifies. Each skill update executes two candidate skills on shared tasks, runs an iterative search with repeated generation and scoring calls, and validates the search output before deployment, which adds substantial overhead relative to the baselines being compared against. Without measurements of overhead in terms of LLM calls and executed trajectories, both the practicality of the method and the fairness of the performance comparison remain unclear.

  2. The preference objective is a heuristic surrogate whose connection to actual behavioral improvement is loose. The authors themselves acknowledge that scoring recorded actions under fixed histories is neither a trajectory probability nor a guarantee of higher return, and the pipeline depends on a long chain of LLM mediated stages with many hand crafted rules for failure handling and tie breaking. The link between lower surrogate loss and real improvement is established only empirically, and the method depends on sufficient capability of the underlying model, with results on the smallest backbone remaining low in absolute terms, raising questions about the robustness of the whole procedure.

Questions for authors
  1. Are the reported results averaged over multiple runs with different seeds, and do the observed gaps over baselines remain stable under such repetition?
  2. How does the method behave over longer learning horizons, and when the initial skill contains actively harmful or misleading rules rather than merely suboptimal ones?
3 comments
AuthorsOpenReview ↗

Response to reviewer yoiW - Part 1

Thank you for your careful review and constructive feedback. We address W1 and W2 below; our responses to Q1 and Q2 follow in Part 2.

W1. Computational overhead and comparison fairness

We have added measurements of learning tokens and LLM calls under matched training rollout budgets: Skill-CoE runs for one epoch and the learning baselines for three on the same tasks, counting current-skill and both candidate executions. Local editing and scoring reuse recorded trajectories without additional training rollouts (Section 4.7; Appendix B.8).

Table 6 reports three-run mean learning tokens (input / output, in millions). These counts cover training interactions and skill updates, including probability scoring and retries, but exclude validation and test evaluation.

MethodALFWorldAppWorldScienceWorldSpreadsheetBench
Executed training rollouts180135180240
ExpeL12.64 / 0.14910.92 / 0.30512.28 / 0.29519.84 / 0.977
ACE15.95 / 0.37217.70 / 0.39716.03 / 0.35620.41 / 1.504
SkillOpt15.89 / 0.22919.63 / 0.49816.09 / 0.26319.26 / 0.968
Skill-Pro18.44 / 0.19729.98 / 0.31816.60 / 0.26323.97 / 0.937
Skill-CoE21.87 / 0.28328.85 / 0.40918.64 / 0.32525.94 / 1.234

Skill-CoE achieves higher final performance with approximately 1.93–2.66× the LLM calls of ExpeL and 1.05–1.11× those of Skill-Pro (Appendix B.8). Its normalized gain per additional learning token relative to ExpeL is comparable to SkillOpt's on ALFWorld and AppWorld and higher than Skill-Pro's on all four benchmarks (Figure 6; normalization in Appendix B.8).

W2. Behavioral meaning of the objective and robustness of recombination

1. Connection between the preference objective and behavioral improvement

We agree that a lower surrogate loss does not guarantee a higher execution return. A lower pairwise loss means a larger preferred–dispreferred action-score margin under fixed recorded histories, relative to the current skill (Appendix A.1).

To test its behavioral usefulness, we compared loss-weighted and uniform sampling within the same case categories, holding the other recombination inputs and generation settings fixed. Compared with uniform sampling, loss-weighted sampling increased mean execution gains over the base by 2.19 points in ALFWorld SR and 1.87 in AppWorld TGC on the validation tasks. All generated edits were evaluated, without acceptance filtering or best-candidate selection (Table 5, lower block; Appendix B.4). This supports using preference differences to select useful editing evidence.

The full learning ablations are consistent with this finding: replacing paired scoring and behavioral feedback with a single-trajectory objective and individual outcome feedback (UP), or randomizing skill and case selection (RR), lowers final test performance on both benchmarks (Tables 3–4; variant definitions in Appendix B.5 and run statistics in Appendix B.6).

We have clarified the original implementation: validation selects the final checkpoint and does not reject intermediate updates. Adding a validation-based acceptance gate did not improve mean test performance on either benchmark (Appendix B.7).

2. Contribution of the multi-stage generation chain

The single-stage variant retains our skill and case selection procedure but passes the selected materials directly to the editor. Its mean scores are lower than those of staged generation (Table 4; mean ± SD over three full learning runs):

VariantALFWorld SR (%)AppWorld TGC (%)AppWorld SGC (%)
Skill-CoE93.03 ± 1.8885.71 ± 2.0673.81 ± 2.73
Single-stage generation87.81 ± 2.1582.74 ± 2.5964.29 ± 3.57

This suggests that analysis, alignment, and planning help turn execution feedback into effective edits. The simplified recombination variant (SR) also performs worse on both benchmarks, even with more attempts and comparable measured learning token usage (Appendix B.5). Failure handling and tie-breaking rules are specified in Appendices A.2–A.3.

3. Backbone dependence across skill-evolution methods

Skill-CoE improves over both baselines across all three tested backbones, with a particularly large advantage on Qwen3.5-9B (Figure 5). However, absolute performance on Ministral-3-8B-Instruct remains low. Skill evolution still requires sufficient execution and editing capabilities and cannot fully compensate for a weak backbone (Section 4.6).

AuthorsOpenReview ↗

Response to reviewer yoiW - Part 2

This second part addresses Q1 and Q2, continuing our response in Part 1.

Q1. Repeated runs and stability of the performance gaps

We apologize for not reporting the run statistics clearly. The original manuscript showed the best result from three runs for each method. The revised Tables 1–4 report means across three independent full learning runs using master seeds 42, 43, and 44, with standard deviations in Table 1 and Appendix B.6. Each run starts from the same initialization, collects fresh experience, updates its knowledge, and selects its own checkpoint using validation before frozen test evaluation.

The mean advantages over the strongest baseline remain on all four benchmarks:

Benchmark and metricStrongest baselineBaseline mean ± SDSkill-CoE mean ± SDMean gain (points)
ALFWorld SR (%)SkillOpt86.07 ± 2.6293.03 ± 1.88+6.96
AppWorld TGC (%)ACE82.54 ± 3.4985.71 ± 2.06+3.17
ScienceWorld ASACE74.60 ± 2.2977.38 ± 2.95+2.78
SpreadsheetBench SR (%)SkillOpt74.40 ± 3.8581.07 ± 1.86+6.67

The positive mean gaps therefore persist when best-of-three results are replaced with three-run averages. Appendix B.6 reports additional run statistics for the initialization and ablation experiments.

Q2. Longer learning horizons and misleading initial skills

You raise two settings beyond the original initialization study. We have added experiments for both.

Longer learning. We extend learning from empty initialization on ALFWorld to 2× and 3× each method's original epoch budget, using the same training tasks and carrying learned knowledge across epochs. The 1× setting is the main comparison. Appendix B.10 reports test SR for the best-validation checkpoint at each budget (mean ± SD):

Method1×2×3×
SkillOpt86.07 ± 2.6286.57 ± 2.2484.08 ± 3.02
Skill-Pro85.82 ± 3.4283.08 ± 3.8386.32 ± 3.11
Skill-CoE93.03 ± 1.8893.53 ± 1.7292.79 ± 2.15

Skill-CoE has the highest mean SR at all three budgets, varying by only 0.74 points. Its advantage therefore persists for the selected checkpoints over the tested horizons.

Misleading initial skills. We replaced part of the Easy initial libraries with explicitly incorrect rules, such as swapping heating and cooling procedures in ALFWorld or ignoring pagination and contact/time constraints in AppWorld. Appendix B.9 describes the modifications and reports the results below (mean ± SD); gains are relative to each method's own performance before learning with the misleading library.

MethodALFWorld SR (%)Gain (points)AppWorld TGC (%)Gain (points)
ExpeL85.82 ± 1.29−1.4968.25 ± 2.41−6.75
ACE86.32 ± 3.02+6.4777.18 ± 3.44+4.56
SkillOpt82.34 ± 2.62+1.7473.61 ± 1.91+0.40
Skill-Pro72.39 ± 3.25−9.7077.58 ± 3.28+1.98
Skill-CoE91.04 ± 1.97+8.9682.54 ± 2.09+8.13

Skill-CoE achieves the highest final performance and largest mean learning gain on both benchmarks, showing that it can learn despite actively misleading initial guidance in these settings.

AuthorsOpenReview ↗

Follow-up on Our Responses

We have submitted our point-by-point responses and hope they address your concerns. We would greatly appreciate any further feedback when you have time.

Reviewer_TibcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper "Beyond Initial Skills: Contrast-Guided Skill Evolution for Robust Learning in LLM Agents" addresses a critical flaw in current LLM agents that learn by updating external procedural skills: their evolution process is highly sensitive to initial skills, often leading to unstable updates or negative learning gains. To achieve robust skill evolution independent of initialization, the authors introduce Skill-CoE (Contrast-guided skill Evolution). Given an interaction experience, Skill-CoE generates two evolution candidates via revision (refining the current skill) and reconstruction (building a new skill from scratch). It then compares their task executions using a fixed-reference, DPO-inspired pairwise preference objective to identify behavioral strengths. Guided by these preferences, Skill-CoE recombines the candidates to incorporate effective new procedures while preserving valid existing strategies. Benchmark evaluations across ALFWorld, AppWorld, and ScienceWorld show that Skill-CoE consistently delivers positive learning gains regardless of initial skills and outperforms strong baseline methods.

Critical strengths
  1. Overcomes Sensitivity to Initial Skills: Unlike existing methods whose performance heavily depends on the quality or difficulty of starting skills, Skill-CoE demonstrates consistent positive learning gains across diverse initialization setups (e.g., Easy, Medium, and Hard source task groups) and diverse backbone models.

  2. Effective Contrastive Exploration and Recombination: By combining independent skill reconstruction with traditional revision, Skill-CoE escapes local optima imposed by early assumptions. Using a DPO-style pairwise preference mechanism to evaluate execution trajectories ensures that helpful new procedures are absorbed while existing effective behaviors are protected against regression.

Critical weaknesses
  1. High Computational and Execution Overhead during Skill Evolution. Skill-CoE requires generating two separate candidate skills per batch (revision and reconstruction), executing both candidates across shared tasks alongside the existing skill to construct preference pairs, and then running an iterative local editing/scoring search loop for up to JJ attempts.

  2. Reliance on Validation-Based Update Acceptance as a Safety Net. After constructing preference pairs, scoring candidate actions, and performing targeted recombination, the framework does not deploy the newly recombined skill directly. Instead, it falls back on an empirical validation step: executing both the proposed skill and the pre-update skill on a separate validation set V\mathcal{V}, accepting the update only if the mean validation return strictly improves. This introduces explicit validation feedback directly into the deployment decision loop. It obscures how much of the robust performance is genuinely driven by the preference-based recombination loss versus simple rejection sampling via validation rollouts. Without an explicit ablation evaluating Skill-CoE without validation filtering (i.e., directly deploying ωout\omega^{out}) it remains unclear whether the contrastive recombination objective is sufficient on its own or if it frequently produces sub-optimal skills that are merely shielded by the validation guardrail.

  3. Incomplete Ablation Study on Key Architectural Components. While the paper provides high-level ablations for contrastive sampling, preference pairs, and recombination (Table 3), it fails to isolate several critical design choices:

  • Validation Acceptance Filter (Eq. A7): The framework relies on validation rollouts to reject bad updates before deployment. No ablation is provided for directly deploying the recombined skill ωout\omega^{out} without validation filtering, making it unclear how much performance gain is protected by brute-force rollout validation versus the preference optimization algorithm itself.
  • Replay Buffer (Sk\mathcal{S}_k): Successful historical trajectories are explicitly replayed during recombination to mitigate regression. The authors do not evaluate a variant without replay cases, obscuring its individual contribution to the reported low regression rates.
  • Multi-stage LLM Chain: The recombination stage employs a heavily engineered 4-step LLM prompt pipeline (Behavioral Analysis →\rightarrow Rule Alignment →\rightarrow Edit Planning →\rightarrow Local Editing). It is unproven whether this complex pipeline is strictly necessary compared to simpler direct synthesis.
Questions for authors

see Weaknesses

2 comments
AuthorsOpenReview ↗

Response to reviewer Tibc

Thank you for your careful review and constructive feedback. We address your main concerns below.

W1. Computational and execution overhead

We match the number of executed training rollouts by running Skill-CoE for one epoch and the learning baselines for three on the same tasks. Skill-CoE's rollout count includes executions under the current skill and both candidates. Local editing and scoring reuse recorded trajectories without further training rollouts.

Section 4.7 and Table 6 report three-run mean learning tokens (input / output, in millions). These include training execution, skill updates, and candidate scoring; validation and test evaluation are excluded. Appendix B.8 provides budget and accounting details, along with LLM call counts under the same scope.

MethodALFWorldAppWorldScienceWorldSpreadsheetBench
Executed training rollouts per run180135180240
ExpeL12.64 / 0.14910.92 / 0.30512.28 / 0.29519.84 / 0.977
ACE15.95 / 0.37217.70 / 0.39716.03 / 0.35620.41 / 1.504
SkillOpt15.89 / 0.22919.63 / 0.49816.09 / 0.26319.26 / 0.968
Skill-Pro18.44 / 0.19729.98 / 0.31816.60 / 0.26323.97 / 0.937
Skill-CoE21.87 / 0.28328.85 / 0.40918.64 / 0.32525.94 / 1.234

Skill-CoE generally uses more learning tokens, so we also examine the gains from this additional computation. Figure 6 relates the fraction of ExpeL's remaining score gap closed to the relative increase in total learning tokens. Skill-CoE's cost–gain coefficients are close to SkillOpt's on ALFWorld and AppWorld and exceed Skill-Pro's on all four benchmarks, while reaching higher final performance.

W2. Validation-based update acceptance

We apologize for the ambiguity in our original description. Eq. A7 described an optional extension and was not used in any experiment reported in the original submission. All methods continued learning from their updated knowledge, with validation used only to select the final checkpoint.

We added a direct comparison with the A7 gate enabled. Both versions select the best-validation checkpoint for final testing; only the gated version can reject intermediate updates based on validation return. Results are mean ± SD (Appendix B.7).

Update ruleALFWorld SR (%)AppWorld TGC (%)AppWorld SGC (%)
Skill-CoE, without the A7 gate93.03 ± 1.8885.71 ± 2.0673.81 ± 2.73
Skill-CoE, with the A7 gate92.54 ± 1.4984.72 ± 1.8270.83 ± 2.73

Enabling the gate did not improve mean performance on any of the three metrics. Our reported gains do not rely on validation-based rejection of intermediate updates. A skill with lower aggregate validation return may still contain useful procedures for later recombination, which a strict gate may discard. We have clarified the update and validation procedures in Sections 3.3.3 and 4.1 and Appendix A.4.

W3. Successful replay and the multi-stage generation chain

The two ablations in Table 4 retain Skill-CoE's candidate scoring, loss-based search acceptance, generation-attempt limit, and validation protocol. Details are in Appendices B.5 and B.6.

VariantALFWorld SR (%)AppWorld TGC (%)AppWorld SGC (%)
Skill-CoE93.03 ± 1.8885.71 ± 2.0673.81 ± 2.73
Without successful replay90.30 ± 2.6981.55 ± 2.3870.24 ± 3.72
Single-stage generation87.81 ± 2.1582.74 ± 2.5964.29 ± 3.57

Successful replay. Preference pairs do not capture every successful behavior worth preserving. When all three skills obtain the same return on a task, no pair is formed, even if their trajectories contain useful procedures. Replay supplies successful execution records as additional preservation evidence during recombination. Removing only these records lowers performance on both benchmarks despite retaining improvement and preservation preference pairs, supporting the value of this additional evidence.

Figure 4 further shows that removing replay increases regression on fixed validation tasks in both benchmarks, consistent with its role in preserving effective behaviors. Evaluation settings and metric definitions are provided in Appendix B.3.

Multi-stage generation. The single-stage variant retains the original base, donor, and case selection, including replay, but replaces behavioral analysis, rule alignment, and edit planning with direct local editing. Its lower performance on both benchmarks supports the contribution of these intermediate stages under the same search-attempt limit. The steps are described in Section 3.3.3 and Appendix A.3, with prompts in Appendix C.

Simplified recombination (SR) simplifies both selection and generation. Even with more attempts and similar learning-token costs, it performs worse than Skill-CoE on both benchmarks (Table 4; Appendix B.5).

AuthorsOpenReview ↗

Follow-up on Our Responses

We have submitted our point-by-point responses and hope they address your concerns. We would greatly appreciate any further feedback when you have time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for your careful reviews and the constructive discussion. We have uploaded a revised manuscript incorporating the additional experiments and clarifications presented in our responses. The main updates are:

  • The method description now clarifies the preference objective, recombination steps, and the distinction between learning updates and validation-based checkpoint selection (Sections 3.3 and 4.1; Appendix A).
  • Additional ablations examine simplified recombination at comparable learning-token cost, staged generation, successful replay, and validation-based update acceptance. A controlled diagnostic also tests preference-guided case sampling (Tables 3–5; Appendices B.4–B.5 and B.7).
  • We added evaluation on SpreadsheetBench and robustness experiments with misleading initial skills and longer learning horizons (Table 1; Appendices B.2 and B.9–B.10).
  • Tables 1–4 now report means over three independent full learning runs, with standard deviations in Table 1 and Appendix B.6.
  • We added learning-stage token and LLM-call measurements under matched training rollout budgets, together with an analysis of the cost–performance trade-off (Section 4.7; Appendix B.8).

Our replies to each reviewer provide further details on these updates. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.