- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates skill evolution for LLM agents. This paper proposes Skill-CoE, a contrast-guided skill evolution framework, which consists of (1) contrastive sampling, (2) within-task trajectory preferences, (3) contrast-guided skill recombination. Extensive experiments on ALFWorld, ScienceWorld, and AppWorld demonstrate the effectiveness of the proposed framework.
Critical strengths
- The idea of contrast-guided skill evolution with non-parametric DPO is novel.
- The robustness of the initial skill is an interesting problem for skill evolution.
Critical weaknesses
-
Although the investigated problem is interesting, the proposed solution introduces substantial complexity whose necessity is insufficiently justified. Section 3.3 combines several heuristic choices, while the ablations mainly establish the usefulness of broad components rather than the distinctive recombination mechanism. A comparison against a streamlined recombination procedure, under matched computation and validation conditions, would help determine whether the additional machinery is warranted. The main body should also explain all essential update and acceptance steps. Without this evidence, the paper’s algorithmic insight remains less convincing than its reported performance gains.
-
This paper uses ALFWorld, ScienceWorld, and AppWorld as the benchmark; however, these three benchmarks cannot serve as a valid agent benchmark for modern LLMs. Please follow SkillOpt to introduce more benchmarks, or use more recent agent benchmarks with enough complexity.
-
Lack of repeated runs for the experiments.
-
Lack of token cost analysis. It seems that Skill-CoE uses much more tokens to produce enough rollouts due to the contrast nature.
Questions for authors
I think that Weakness #1 is my biggest concern, which cannot be solved in the rebuttal phase.
3 comments
Thanks for the response and the newly added experiment results. For Weakness #1, I still feel that the proposed skill recombination method is overly engineered (with a complex decision logic as presented in the appendix) that we cannot identify which part takes the central role and which contribute only marginally. Thus, I would maintain my score as a weak reject.
Response to reviewer ozbc
Thank you for the follow-up. Let us explain why we use this design and which parts contribute most. We also hope the additional explanations below will be helpful in your reassessment.
1. Why the recombination design has this structure
The design centers on local complementarity: a skill with higher aggregate preference loss can still better support preferred behavior on individual cases. We therefore use the same preference objective at two levels. Aggregate loss selects the base and evaluates proposed edits; pairwise loss differences identify a complementary donor and evidence for what to change or preserve. Analysis, alignment, and planning then translate this evidence into concrete skill edits (Section 3.3; Appendices A.2–A.3). This connects the main decisions around a common goal: incorporating useful new procedures while preserving effective existing behavior.
The simplified recombination variant (SR) directly tests a less elaborate alternative: fixed base/donor roles, uniform case sampling, and direct editing. It uses the same validation protocol and comparable learning tokens. Skill-CoE still exceeds SR by 7.46 points in ALFWorld SR and 7.34 in AppWorld TGC (Appendix B.5). Thus, the additional structure brings measurable benefits over this simpler procedure at comparable learning cost.
2. Central mechanisms and supplementary contributions
The ablations show that the main benefit of recombination comes from incorporating complementary experience from different skills into an improved skill. Loss-guided selection identifies what to incorporate and preserve, while staged generation turns this evidence into concrete skill edits. These two mechanisms jointly support this process. Successful replay provides additional preservation support by supplying successful behaviors that preference pairs can miss.
Tables 3–4 report the following performance drops relative to Skill-CoE (percentage points); variant details are in Appendix B.5.
| Ablation | ALFWorld SR | AppWorld TGC | AppWorld SGC |
|---|---|---|---|
| NR: select a skill without recombination | 9.45 | 8.13 | 11.90 |
| RR: randomize skill and case selection | 6.97 | 5.56 | 8.33 |
| SG: replace staged generation with direct editing | 5.22 | 2.98 | 9.52 |
| SR: simplify both selection and generation | 7.46 | 7.34 | 5.95 |
| WR: remove successful replay | 2.74 | 4.17 | 3.57 |
NR has the largest drops across all three metrics, supporting the central role of recombination: selecting a single skill leaves useful procedures from the other skills unused.
The substantial drops in RR and SG show that neither mechanism alone sustains the full method's performance. Without loss guidance, generation lacks targeted evidence; without staged generation, useful evidence is harder to turn into effective edits in a single step. Loss guidance finds experience worth learning from, while staged analysis, alignment, and planning turn it into concrete changes. Together, they support the effective use of complementary information.
WR's smaller drops than NR and RR support a supplementary role for replay. It supplies successful trajectories that preference pairs can miss when returns tie, helping preserve effective behavior during updates.
Response to reviewer ozbc
Thank you for your careful review and constructive feedback. We address your main concerns below.
W1. Necessity of the recombination mechanism and clarity of the update procedure
1. Comparison with streamlined recombination
To test whether the added complexity is justified, we compare Skill-CoE with simplified recombination (SR), which fixes the base and donor roles, samples cases uniformly, and directly generates local edits. SR uses the same validation protocol and comparable learning tokens: 6.3% fewer on ALFWorld and 1.5% more on AppWorld (Appendix B.5). RR retains staged generation but randomizes selection; Single-stage retains our selection but generates edits directly. Results are in Table 4.
Skill-CoE exceeds SR by 7.46 points in ALFWorld SR and 7.34 points in AppWorld TGC, showing the benefit of the full design over this simpler alternative at comparable token cost. The RR and Single-stage results further support guided selection and staged generation, respectively.
We also isolate the effect of case sampling while holding the other recombination inputs and generation settings fixed. Preference-guided sampling yields larger mean execution gains than uniform sampling on the validation sets: an additional 2.19 points in ALFWorld SR and 1.87 in AppWorld TGC. Every generated candidate is evaluated, without acceptance filtering (Appendix B.4).
2. Update and acceptance steps
Sections 3.3.1–3.3.3 now explain skill and case selection, generation, and updates. The first generated skill with lower loss than the base is accepted; if none succeeds within J attempts, the base is used. Validation selects the final skill for testing and does not reject learning updates (Section 4.1). Further details are in Appendices A.2–A.4.
W2. Evaluation on an additional benchmark
Following your suggestion, we added SpreadsheetBench [1]. We use its official 400-task Verified subset, holding out 280 tasks for testing and splitting the remainder between training and validation (Appendix B.2). Agents manipulate real workbooks through Python/Bash, and the evaluator checks output cell values. Results are shown in Table 1:
Skill-CoE exceeds the strongest baseline, SkillOpt, by 6.67 percentage points in mean SR, extending the observed advantage to spreadsheet manipulation.
W3. Repeated runs
The original manuscript reported the best of three runs for each method. We apologize for not reporting the full statistics. Tables 1–4 now report means over three independent full learning runs, with standard deviations in Table 1 and Appendix B.6. Each run starts from the same initialization, collects fresh experience, updates its knowledge, and selects its final checkpoint using validation before testing with knowledge frozen.
Skill-CoE retains the highest mean performance across all four benchmarks.
W4. Token cost of contrastive learning
We match executed training rollouts: Skill-CoE runs for one epoch and learning baselines for three on the same tasks, counting executions under the current skill and both candidates. Local editing and scoring reuse recorded trajectories without additional training rollouts. Token counts cover training interactions and knowledge updates, including generation, scoring, failed calls, and retries; validation and test evaluation are excluded (Section 4.7; Appendix B.8). Appendix B.8 also reports LLM calls under the same scope.
Table 6 reports three-run mean learning tokens (input / output, in millions).
Skill-CoE generally uses more learning tokens but achieves higher mean performance. Figure 6 shows the cost–performance trade-off relative to ExpeL (Appendix B.8).
[1] Ma Z, Zhang B, Zhang J, et al. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.