- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates an interesting research problem, i.e., whether skills learned by agents can generalize to held-out tasks? This paper provides a controlled comparison among several skill evolution methods on six benchmarks with consistent backbone LLM. Then, this paper proposes Generalizable Skill Optimization (GSO), which learns a persistent guide for generating a fresh skill for each task. GSO achieves the highest reported test score across all six columns.
Critical strengths
- The research problem is interesting.
Critical weaknesses
- While the investigated research problem is interesting, the experimental setups are not convincing enough for me. More specifically, (1) this paper only reports results with Qwen3-Coder-480B-A35B-Instruct-FP8, which is not a common choice for the community. Thus, the SOTA performance of GSO built on this model is hard for the community to evaluate. (2) This paper only reports results with one single pass under the greedy decoding setting. However, greedy decoding is not a common practice for most LLM agents since modern LLMs typically exhibit performance deterioration with that. Thus, it is unknown whether the SOTA performance just comes from noises. (3) All the investigated baselines are more like context engineering or prompt engineering, rather than genuine skills that are widely recognized. Thus, the conclusions drawn on top of these methods make the investigated research problem less practical.
- The empirical analyses seem plain and naïve. Skill overfitting is a well-known issue for skill evolution methods. The memorization of specific instance in the skill is also investigated by many prior works. However, this paper does not provide novel insights beyond that phenomenon.
- Please provide the complete pseudo-code-alike algorithm for GSO. It is hard for me to follow the algorithm details.
- The use of “Definition” in Section 3.1 is confusing. This paper does not provide formal definitions for self-evolving skills or generalization.
- I disagree that the overfit should be defined as G_train > 0 and retention < 0. Consider a case (A=No Skill, B=With Skill), A_train = 20%, B_train = 60% (G_train=40%), A_test = 90%, B_test = 100% (G_test=10%, retention=-30%). Clearly this is not an overfit.
Questions for authors
See weakness.
3 comments
W3. Pseudo-code
We added Algorithm 1, shown next to Figure 4. One learning round of GSO works as follows:
- Collect failed training tasks. In the first round, these are the failures of the agent with no skill; later, the failures of the current metaskill.
- Write one edit. From these failures, write one edit to the metaskill that does not name any specific task.
- Compare. Run the old and the new metaskill three times each on the validation tasks.
- Decide. Keep the edit only if it turns more than twice as many failed task attempts into passes as the reverse, and a paired statistical test agrees.
After five rounds the metaskill is frozen. For each test task, the model writes a skill from the metaskill, uses it once, and throws it away.
W4. Formal definitions
Section 3.1 now gives two definitions.
Definition 1 (self-evolving skill method).
- The method starts from a first skill.
- It runs the agent with this skill on training tasks.
- It edits the skill based on the feedback, and keeps an edit only if it passes the method's own check.
- The final skill is frozen before testing; for GSO, the final metaskill is frozen. Test tasks are never used to edit or choose it.
Definition 2 (generalization).
- Same kind of task: training, validation, and test tasks are separate samples of the same kind of task, such as editing spreadsheets.
- When a skill generalizes: if its gain over no skill on test tasks is at least as large as its gain on training tasks. This compares gains in points, as in the submission.
- Refinement from your W5 example: in your example, no skill already scores 90% on test, so a skill can gain at most 10 points there. Points alone cannot show that such a skill took all the room it had. We therefore refined the definition: next to the gain in points, we now report how much of the possible gain the skill takes (Appendix A.4; see W5).
W5. For our definition of generalization
Our definition says a skill generalizes if its gain on test tasks is at least as large as its gain on training tasks. In the reviewer's point is judging by points alone is unfair to such a skill.
Our change. The reviewer's point is a good supplement to our definition, and we have refined the definition to cover this case (Definition 2, Appendix A.4). Besides the gain in points, we now also report how much of the possible gain the skill takes, that is, its gain divided by the room left between no skill and 100%:
- Training tasks: 40 of a possible 80 points, so half of the possible gain.
- Test tasks: 10 of a possible 10 points, so all of it.
I thank the authors for adding these new results and analyses. However, my concerns are fundamental ones for this manuscript. For the current version, I think it is still clearly below the bar of ICLR.
Thank you for the review and the helpful example in W5. To answer the review, we ran a new study (summary in Section 5.6, full results in Appendix C). It runs Qwen3.5-35B-A3B on SpreadsheetBench, a public benchmark of spreadsheet-editing tasks. There are three random splits of 100 training, 80 validation, and 100 test tasks, and three runs per method. We compare with two methods, both run from their official code: Trace2Skill merges lessons from many runs into one skill, and SkillOpt edits one skill document.
GSO learns a metaskill, a short guide that tells the model how to write a skill for each new task. It learns in rounds, and keeps each proposed edit only if it passes a check on the validation tasks. Below, the starting metaskill is the hand-written guide before learning, and the learned metaskill is the guide after learning.
W1 (1). Do the results depend on an uncommon model?
Section 5 uses Qwen3-Coder-480B-A35B-Instruct, and those results stay there. To check that the findings do not depend on that model, the new study uses Qwen3.5-35B-A3B, an open model. On SpreadsheetBench, GSO's learned metaskill scores 48.8%, against 42.8% with no skill (+6.0 points; 95% confidence interval: +1.4 to +10.4).
We also gave the metaskill learned with Qwen3.5, unchanged, to two other public models. Each model then writes and uses its own skills; nothing is learned again (Appendix C.4, Table 10):
On both models, the learned metaskill beats the starting metaskill, but it does not beat their own no-skill scores; neither the −1.3 nor the −0.7 difference is significant. On both models, the starting metaskill alone scores below no skill. So the learning gain carries over, but on these models the learned metaskill stays slightly below no skill.
W1 (2). Are the results reliable with a single greedy run?
The Section 5 results use temperature zero. To test reliability, all new runs sample at temperature 0.6, with three runs per method on each of three random splits, and paired 95% confidence intervals (Appendix C.1):
The learned metaskill scores 6.0 points above no skill (95% confidence interval: +1.4 to +10.4). The gain is positive on each split: +8.7, +5.3, and +4.0 points. GSO also scores above both baselines, but these two differences are not statistically significant.
W1 (3). Are the baselines prompt engineering rather than skills?
By a skill, we mean a written procedure that an agent loads before a task and that a method improves from feedback. Both baselines fit this definition: they produce
SKILL.mdfiles, the format that agent frameworks load before a task, and we run their official code.W2. Is there a new insight beyond "skills can overfit"?
Yes. The new analyses (Appendix C.5) show which learned rules in the metaskill carry over, and why:
On this benchmark, then, the useful rules are about how to deliver the answer, and a typical rejected edit is within run-to-run noise. This supports GSO's rule of accepting an edit only after a paired test over repeated runs.