acceptodds
Under review as a conference paper at ICLR 2027

Do Self-Evolving Skills Generalize to Held-Out Tasks?

Abstract

AI agents can externalize what they learn from past tasks into reusable skills, such as procedures, checklists, code, or other executable artifacts, that can be retrieved and reused when solving new tasks. Self-evolving skill methods keep rewriting these skills after each round of practice on training tasks, and the skill is then used on new tasks of the same kind. We ask a question: does the improvement a skill shows on its training tasks carry over to new test tasks? We test five self-evolving methods and a one-shot skill on six benchmarks, with the same model, the same agent, and the same train/test split for every method. Of the 21 skills that improve on their training tasks, 5 keep all of that improvement on the test tasks, 13 keep part of it, and 3 keep none of it. No existing method is best everywhere. When we read the skills, the ones that carry over badly often copy details of specific training tasks, such as column names, or turn a fix for one failure into a rule for every task. An LLM judge that reads the skill content can often see this: it ranks finished skills the same way the test results do in 86% of pairs. But it predicts the effect of a single edit poorly, so edits still have to be tested by running them. Based on these findings, we describe Generalizable Skill Optimization (GSO), which keeps only a guide for writing skills and writes a new skill for each task; it scores highest on all six benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agentic systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yqADOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates an interesting research problem, i.e., whether skills learned by agents can generalize to held-out tasks? This paper provides a controlled comparison among several skill evolution methods on six benchmarks with consistent backbone LLM. Then, this paper proposes Generalizable Skill Optimization (GSO), which learns a persistent guide for generating a fresh skill for each task. GSO achieves the highest reported test score across all six columns.

Critical strengths
  • The research problem is interesting.
Critical weaknesses
  • While the investigated research problem is interesting, the experimental setups are not convincing enough for me. More specifically, (1) this paper only reports results with Qwen3-Coder-480B-A35B-Instruct-FP8, which is not a common choice for the community. Thus, the SOTA performance of GSO built on this model is hard for the community to evaluate. (2) This paper only reports results with one single pass under the greedy decoding setting. However, greedy decoding is not a common practice for most LLM agents since modern LLMs typically exhibit performance deterioration with that. Thus, it is unknown whether the SOTA performance just comes from noises. (3) All the investigated baselines are more like context engineering or prompt engineering, rather than genuine skills that are widely recognized. Thus, the conclusions drawn on top of these methods make the investigated research problem less practical.
  • The empirical analyses seem plain and naïve. Skill overfitting is a well-known issue for skill evolution methods. The memorization of specific instance in the skill is also investigated by many prior works. However, this paper does not provide novel insights beyond that phenomenon.
  • Please provide the complete pseudo-code-alike algorithm for GSO. It is hard for me to follow the algorithm details.
  • The use of “Definition” in Section 3.1 is confusing. This paper does not provide formal definitions for self-evolving skills or generalization.
  • I disagree that the overfit should be defined as G_train > 0 and retention < 0. Consider a case (A=No Skill, B=With Skill), A_train = 20%, B_train = 60% (G_train=40%), A_test = 90%, B_test = 100% (G_test=10%, retention=-30%). Clearly this is not an overfit.
Questions for authors

See weakness.

3 comments

Thank you for the review and the helpful example in W5. To answer the review, we ran a new study (summary in Section 5.6, full results in Appendix C). It runs Qwen3.5-35B-A3B on SpreadsheetBench, a public benchmark of spreadsheet-editing tasks. There are three random splits of 100 training, 80 validation, and 100 test tasks, and three runs per method. We compare with two methods, both run from their official code: Trace2Skill merges lessons from many runs into one skill, and SkillOpt edits one skill document.

GSO learns a metaskill, a short guide that tells the model how to write a skill for each new task. It learns in rounds, and keeps each proposed edit only if it passes a check on the validation tasks. Below, the starting metaskill is the hand-written guide before learning, and the learned metaskill is the guide after learning.

W1 (1). Do the results depend on an uncommon model?

Section 5 uses Qwen3-Coder-480B-A35B-Instruct, and those results stay there. To check that the findings do not depend on that model, the new study uses Qwen3.5-35B-A3B, an open model. On SpreadsheetBench, GSO's learned metaskill scores 48.8%, against 42.8% with no skill (+6.0 points; 95% confidence interval: +1.4 to +10.4).

We also gave the metaskill learned with Qwen3.5, unchanged, to two other public models. Each model then writes and uses its own skills; nothing is learned again (Appendix C.4, Table 10):

Model that uses the metaskillLearned minus starting metaskill (points)Learned metaskill minus no skill (points)
gpt-oss-120b+3.8−1.3
gpt-oss-20b+4.4−0.7

On both models, the learned metaskill beats the starting metaskill, but it does not beat their own no-skill scores; neither the −1.3 nor the −0.7 difference is significant. On both models, the starting metaskill alone scores below no skill. So the learning gain carries over, but on these models the learned metaskill stays slightly below no skill.

W1 (2). Are the results reliable with a single greedy run?

The Section 5 results use temperature zero. To test reliability, all new runs sample at temperature 0.6, with three runs per method on each of three random splits, and paired 95% confidence intervals (Appendix C.1):

MethodTest pass rate (%)
No skill42.8
Trace2Skill45.1
SkillOpt45.7
GSO, learned metaskill48.8

The learned metaskill scores 6.0 points above no skill (95% confidence interval: +1.4 to +10.4). The gain is positive on each split: +8.7, +5.3, and +4.0 points. GSO also scores above both baselines, but these two differences are not statistically significant.

W1 (3). Are the baselines prompt engineering rather than skills?

By a skill, we mean a written procedure that an agent loads before a task and that a method improves from feedback. Both baselines fit this definition: they produce SKILL.md files, the format that agent frameworks load before a task, and we run their official code.

W2. Is there a new insight beyond "skills can overfit"?

Yes. The new analyses (Appendix C.5) show which learned rules in the metaskill carry over, and why:

  1. What carries over is knowledge about how to deliver the answer. Learning on three separate splits found the same main kinds of rules, such as "write computed values, not formulas", "save the output file", and "keep all sheets of the input workbook". No rule names a specific cell, column, or task. They share one principle: the answer is checked only through the output file, so that file must hold the finished result and keep what the task provided. Every task asks for different content, but all tasks share this way of delivering the answer. GSO writes a new skill for each task to handle the content, so the metaskill learns the shared part, which also holds for new tasks.
  2. On the first split, one such rule gives most of the learning gain. Learning raises the test pass rate by 21.3 points over the starting metaskill (first split only: no skill 40.3%, starting metaskill 27.7%, learned metaskill 49.0%). Part of this gain makes up for the starting metaskill's deficit; the learned metaskill is 8.7 points above no skill on this split. Adding only the rule "write computed values, not formulas" to the starting metaskill raises its test pass rate by 15.7 of these 21.3 points.
  3. Most proposed edits were rejected (11 of 15). A typical rejected edit changes validation results about as much as two runs of the same metaskill differ from each other.

On this benchmark, then, the useful rules are about how to deliver the answer, and a typical rejected edit is within run-to-run noise. This supports GSO's rule of accepting an edit only after a paired test over repeated runs.

W3. Pseudo-code

We added Algorithm 1, shown next to Figure 4. One learning round of GSO works as follows:

  1. Collect failed training tasks. In the first round, these are the failures of the agent with no skill; later, the failures of the current metaskill.
  2. Write one edit. From these failures, write one edit to the metaskill that does not name any specific task.
  3. Compare. Run the old and the new metaskill three times each on the validation tasks.
  4. Decide. Keep the edit only if it turns more than twice as many failed task attempts into passes as the reverse, and a paired statistical test agrees.

After five rounds the metaskill is frozen. For each test task, the model writes a skill from the metaskill, uses it once, and throws it away.

W4. Formal definitions

Section 3.1 now gives two definitions.

Definition 1 (self-evolving skill method).

  1. The method starts from a first skill.
  2. It runs the agent with this skill on training tasks.
  3. It edits the skill based on the feedback, and keeps an edit only if it passes the method's own check.
  4. The final skill is frozen before testing; for GSO, the final metaskill is frozen. Test tasks are never used to edit or choose it.

Definition 2 (generalization).

  • Same kind of task: training, validation, and test tasks are separate samples of the same kind of task, such as editing spreadsheets.
  • When a skill generalizes: if its gain over no skill on test tasks is at least as large as its gain on training tasks. This compares gains in points, as in the submission.
  • Refinement from your W5 example: in your example, no skill already scores 90% on test, so a skill can gain at most 10 points there. Points alone cannot show that such a skill took all the room it had. We therefore refined the definition: next to the gain in points, we now report how much of the possible gain the skill takes (Appendix A.4; see W5).

W5. For our definition of generalization

Our definition says a skill generalizes if its gain on test tasks is at least as large as its gain on training tasks. In the reviewer's point is judging by points alone is unfair to such a skill.

Our change. The reviewer's point is a good supplement to our definition, and we have refined the definition to cover this case (Definition 2, Appendix A.4). Besides the gain in points, we now also report how much of the possible gain the skill takes, that is, its gain divided by the room left between no skill and 100%:

  • Training tasks: 40 of a possible 80 points, so half of the possible gain.
  • Test tasks: 10 of a possible 10 points, so all of it.

I thank the authors for adding these new results and analyses. However, my concerns are fundamental ones for this manuscript. For the current version, I think it is still clearly below the bar of ICLR.

Reviewer_uv3mOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether self-evolving agent skills generalize from training tasks to unseen instances. Through a controlled evaluation of six existing methods across six benchmarks, it finds that training improvements often fail to transfer, partly because task-specific fixes become overly general rules. To address this, the authors propose GSO, which learns a reusable metaskill to generate task-specific skills and achieves the highest reported test performance across all six benchmarks.

Critical strengths
  • The systematic comparison across multiple methods and benchmarks provides valuable empirical evidence of the generalization limitations of self-evolving skills.

  • The retention metric and concrete examples of harmful skill edits offer an intuitive and interpretable analysis of skill overfitting.

  • GSO introduces a simple and practical alternative to persistent skill refinement, demonstrating promising improvements across diverse task domains.

Critical weaknesses
  • Section 5.1 explicitly states that GSO generates a new skill for every test task, validates its output, and may retry once, whereas competing methods lack these additional steps. Consequently, the reported gains may reflect greater test-time computation and adaptation rather than improved skill generalization.
  • Table 2 evaluates only 17 and 23 test tasks on the two RBioBench tracks and 20 on SciVisAgentBench, without uncertainty estimates or repeated splits. Notably, GSO underperforms No Skill on both training and validation for SciVisAgentBench but substantially outperforms it on test, raising concerns about split sensitivity.
Questions for authors

see wekaness.

2 comments

Thank you for the review. To answer it, we ran a new study (summary in Section 5.6, full results in Appendix C). It runs Qwen3.5-35B-A3B on SpreadsheetBench, a public benchmark of spreadsheet-editing tasks. There are three random splits of 100 training, 80 validation, and 100 test tasks, and three runs per method. We compare with two methods, both run from their official code: Trace2Skill merges lessons from many runs into one skill, and SkillOpt edits one skill document.

GSO learns a metaskill, a short guide that tells the model how to write a skill for each new task. It learns in rounds, and keeps each proposed edit only if it passes a check on the validation tasks. Below, the starting metaskill is the hand-written guide before learning, and the learned metaskill is the guide after learning.

W1. Is GSO's gain simply due to more computation at test time?

No. At test time, GSO writes a skill for each task. The agent may also check its own output and retry once within the same attempt at the task; we call this the check-and-retry step. We tested these extra steps in three ways:

ConditionTest pass rate (%)Tokens per task
No skill42.8151k
GSO, starting metaskill (same test-time steps, no learning)33.8190k
GSO, learned metaskill48.8209k
GSO, learned metaskill, check-and-retry step removed51.2151k
SkillOpt45.7212k
SkillOpt with GSO's check-and-retry step added38.0265k
  1. Same extra steps, no learning: with the hand-written starting metaskill, GSO scores 33.8%, below no skill.
  2. Step removed: without the check-and-retry step, the learned metaskill scores 51.2%, slightly higher; the difference is not significant on any split. We still report GSO with the step, because the step was fixed as part of GSO before any test run; switching after seeing test results would mean using the test set to choose a method. Without the step, GSO uses 151k tokens per task, the same as no skill, even though it makes fewer model calls per task (13.1 against 18.0 for no skill, including the call that writes the skill).
  3. Step added to another method: adding the same step to SkillOpt lowers its score from 45.7% to 38.0%.

So the gain does not come from extra test-time computation.

W2. Are the results reliable, given small test sets and no repeated splits?

We added the new study to answer this:

  • Test tasks: 100 per split.
  • Splits: three random splits.
  • Runs: three runs of each method on each split.
  • Uncertainty: we report paired 95% confidence intervals over tasks.
MethodTest pass rate (%)
No skill42.8
Trace2Skill45.1
SkillOpt45.7
GSO, learned metaskill48.8

The learned metaskill scores 6.0 points above no skill (95% confidence interval: +1.4 to +10.4), and the gain is positive on each split:

Split 1Split 2Split 3
GSO minus no skill (points)+8.7+5.3+4.0

GSO also scores above both baselines, but these two differences are not statistically significant.

The original results on six benchmarks stay in Section 5. The new study adds repeated random splits, with the test tasks hidden until every method is final (Appendix C.1).

Thank you to the authors for their detailed response, which has addressed some of my concerns. Taking into account the other reviewers' comments and the authors' rebuttal, I have decided to maintain my original score.

Reviewer_ck5iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks a good question: when a self-evolving skill improves on training tasks, how much of that improvement survives on held-out tasks of the same type? The authors run six existing methods plus No Skill on six benchmarks with a shared model, agent, and train/validation/test split, and report that of 21 skills with positive training gain, 5 keep the full gain, 13 keep part of it, and 3 keep none; 10 of 36 test scores fall below No Skill. They attribute the gap to skills that hard-code task details or turn a single fix into a global rule, and report that an LLM judge ranks finished skills consistently with test results (108/126 pairs) but cannot predict the effect of a single edit. They then propose GSO, which learns a metaskill that writes a fresh per-task skill, and report the best test score on all six benchmarks.

Critical strengths
  • The question matters and is underexamined. Most skill papers report gains on the tasks they optimized on. Measuring what reaches held-out tasks is a real contribution.

  • The judge result is useful. "Content predicts the quality of a finished skill but not the effect of an edit" is a concrete warning for anyone using an LLM judge as an accept/reject signal.

Critical weaknesses
  1. The paper does not deliver the analysis its title promises. The title asks an empirical question, and the abstract answers it with a set of counts. But the "why" is supported by three skill excerpts (Figure 3a) and two GSO examples (Figure 3b), and the authors themselves call these "examples, not a count." The paper then spends more than half its length on a new method. The material for a real analysis is already there — 36 final skills, 45 tracked edits, per-round checkpoints, task-level outcomes — but it is not used. In particular, Table 3 lists each method's update and retention rule; crossing those design choices with retention would directly answer "which designs transfer and why," and this is never done.

  2. GSO's gains are not attributed. GSO differs from the baselines in at least four ways: a human-written one-page metaskill m(0), a per-task skill-writing call at test time, one bounded retry, and a different acceptance gate. None is ablated. The most likely outcome of a frozen-m(0) baseline is that most of the gain remains, which would make the contribution "a good generic guide plus per-task instantiation," not "learning how to write skills." Note also that m(0) contains runtime policies (m4 "check the result", m6 "repair in one retry"), so it is an agent scaffold, not only a guide for writing skills.

  3. The retry makes the comparison unfair and contaminates the core metric. Baselines get no retry; GSO does. This is not just extra compute: retention is defined as G_test − G_train, and GSO's G_test includes a retry while the baselines' does not. The paper does not say whether the retry is an in-episode repair or a second full attempt. Either way, the comparison is between a method and a method-plus-scaffold.

  4. The retention metric is under-specified, and GSO's own numbers break it. Section 3.1 says "retention of zero or more means the whole gain carried over." It never explains what retention > 0 means, yet GSO has positive retention on five of six benchmarks (roughly +12 to +20 points; I computed these from Table 2), and the paper only reports the one negative case (HealthBench, −5.2). Under the paper's own framework, GSO carries over more than the whole training gain, which needs an explanation. The "Train" column is also undefined: which checkpoint, whether it includes the retry, and whether it uses the same harness as the test runs. If the train and test columns are not measured the same way, the retention comparison is not meaningful.

  5. There is no uncertainty quantification. Test sets have 17–40 tasks, so one task is worth 2.5–5.9 points. Four of the six "best" margins amount to at most four tasks (RBio Clinical ≈2/17, SciVis ≈1.7/20, HealthBench ≈1.7/40, Omics ≈4/23). No seeds, confidence intervals, or paired tests are reported, yet the abstract claims the highest score on all six benchmarks. The 5/13/3 taxonomy is also over-precise: several of the "partial" cells (e.g., −0.8, −0.9, −2.0) are indistinguishable from zero. Per-task paired bootstrap and McNemar tests are feasible with the accounting the authors already keep.

  6. RBioBench is author-built and carries the largest gains. GSO leads by +11.8 and +17.4 on Clinical and Omics, the two benchmarks built by the authors. On Omics, no existing method beats No Skill, which is suspicious and may indicate a distribution shift or ambiguous prompts. The AI-use statement says generative AI was used to generate and clean candidate RBioBench tasks, and the verifier is exact-match on reference outputs for 356 of 405 tasks. If prompts are ambiguous, exact matching penalizes methods that guess and rewards the one whose metaskill says "abstain when evidence is insufficient." This needs a human audit, an ambiguity estimate, and ideally one public benchmark as a check.

  7. Baselines and cost are under-documented. The five baselines are reimplementations, but the appendix gives only a one-line update rule for each — no prompts, no comparison with the numbers reported in the original papers. Two results look like harness artifacts: SkillOpt collapses to 15.0 train / 25.0 validation on RBio Clinical (No Skill: 42.5), and GEPA falls below No Skill on SpreadsheetBench and Omics. No cost table is reported either, although GSO evaluates two metaskills on the validation set each round and makes an extra call per test task.

  8. The judge analysis is circular and statistically loose. The rubric's "Generalizability" dimension asks exactly what GSO optimizes, and the GSO metaskill is scored with a rubric written for skills (the authors call this a "loose comparison") yet is still counted in the 126 pairs. A leave-one-out analysis excluding GSO is needed; I suspect the 85.7% figure rests largely on GSO being the outlier. The 126 pairs are not independent, and the treatment of ties is inconsistent between the text and the Figure 8 caption. Finally, for single edits the judge sees only the text — not the failed tasks or traces — so the negative result is about a deliberately blind judge, not about judges in general.

  9. The writing is hard to follow. The style is very terse and telegraphic, with few connectives and heavy use of "this/it" as subjects. Some passages are genuinely ambiguous: the retention definition above is one example; another is "whether the distance between training and test tasks explains it" in Section 3.2, where "it" has no clear referent. The metaskill is described with three different numbering schemes (m1–m6, four learning steps, six stages in Figure 4), and the reader has to map between them. The abstract is a wall of numbers without a topic sentence. Terminology drifts between "carry over," "keep," "retain," "transfer," and "generalize."

Questions for authors
  1. What exactly does the "Train" column measure for GSO — which checkpoint, and does it include the retry? Why is GSO's test score higher than its train score on five of six benchmarks?
  2. What is the result for GSO with m(0) frozen (no metaskill updates)?
  3. What is the result for GSO without the retry? And for SkillOpt with the same per-task instantiation and retry?
  4. Did the reimplemented baselines reproduce the numbers reported in their original papers? What explains SkillOpt's collapse on RBio Clinical?
  5. What fraction of RBioBench tasks were AI-generated, how were they audited, and how ambiguous are the prompts? Why does no method beat No Skill on Omics?
  6. What is the judge agreement after excluding GSO? Does a judge that also sees the failed tasks predict edit effects better?
  7. Were the per-round test curves in Figure 2 computed after freezing, and was test ever visible to any stopping or selection decision?
5 comments

Thank you for the detailed review. To answer it, we ran a new study (summary in Section 5.6, full results in Appendix C). It runs Qwen3.5-35B-A3B on SpreadsheetBench, a public benchmark of spreadsheet-editing tasks. There are three random splits of 100 training, 80 validation, and 100 test tasks, and three runs per method. We compare with two methods, both run from their official code: Trace2Skill merges lessons from many runs into one skill, and SkillOpt edits one skill document.

GSO learns a metaskill, a short guide that tells the model how to write a skill for each new task. It learns in rounds, and keeps each proposed edit only if it passes a check on the validation tasks, which the paper calls the gate. Below, the starting metaskill is the handwritten guide before learning, and the learned metaskill is the guide after learning.

W1. Which learned rules carry over to new tasks, and why?

Our title asks whether self-evolving skills carry over to new tasks. We added three analyses that answer this:

  1. What does learning produce? Rules about how to deliver the answer, not about how to solve any one task. They share one principle, explained below (Appendix C.5).
  2. Which of these rules gives the gain? Mostly one, which tells the agent to write computed values into cells instead of formulas (Appendix C.5).
  3. Why are most proposed edits rejected? Most of them are no larger than run-to-run noise, so GSO's gate rejects them (Appendix C.5).

So what carries over is knowledge about how to deliver the answer, and GSO's gate keeps out edits that are only noise.

1. What does learning produce? On each of the three splits, the first learning round added a block of four rules, written only from that split's own training tasks. We grouped all 12 rules by what they ask the agent to do:

What the rule asks the agent to doRules (of 12)
Write computed values into cells, not formulas3
Save the output file explicitly3
Keep all sheets of the input workbook3
Keep the example values already in the answer cells2
Check that every answer cell is filled1

The three most common kinds were learned on every split, and no rule names a specific cell, column, file, or task. Together, the rules express one principle: the answer is checked only through the output file, so that file must hold the finished result and must keep what the task provided. The rules apply this principle in three ways:

  • Deliver the finished result: write computed values, not formulas that still have to be calculated.
  • Do not damage the input: keep all sheets, and keep the example values already in the answer cells.
  • Make sure the answer is delivered: save the file, and check that every answer cell is filled.

This explains why these rules carry over. On SpreadsheetBench, every task asks for different content, but all tasks share this way of delivering the answer. GSO writes a new skill for each task to handle that task's content, so what the metaskill learns from training tasks is the shared part, and the shared part also holds for new tasks. The next analysis tests which of these rules matters most.

2. Which learned rule gives the gain? On the first split, learning raised the test pass rate by 21.3 points over the starting metaskill (27.7% to 49.0%). Averaged over all three splits, the gain is 15.0 points (see W2). We used the learned metaskill of split 1. Besides rules of the kinds above, it contains two checks that GSO also added during learning: a "done" checklist and a check of row counts and empty cells. We removed one rule at a time and measured the change (three runs each):

Rule removed from the learned metaskill (learned metaskill: 49.0%)Change in test pass rate (points)
Write computed values, not formulas−11.3
Keep all sheets of the input workbook−6.0
Keep the example values already in the answer cells−2.3
End each task's skill with a "done" checklist−1.0
Check row counts and empty cells+1.3

The table gives each rule a short name. The rule "write computed values, not formulas" carries most of the gain. Its full text in the learned metaskill, exactly as GSO wrote it, is:

### Cache Formula Results
When: Writing calculation logic to cells.
- Do not rely on the environment to recalculate formulas
- Compute results using Python libraries immediately
- Write the numeric or string result directly into the cell instead of the formula string
- If formulas are required, ensure values are pre-calculated

The rule is short, but it targets one frequent failure: the agent writes a formula into a cell and relies on the spreadsheet program to calculate it, instead of computing the value itself. As a further check, we added only this rule to the starting metaskill: the pass rate rose by 15.7 points, and the 95% confidence interval excludes zero.

3. Why are most proposed edits rejected? GSO keeps an edit only if, on the validation tasks, it turns more than twice as many failed task attempts into passes as the reverse, and a paired statistical test agrees. To see how much change is just noise, we ran the same metaskill twice on the validation tasks. Out of 240 task attempts (80 tasks × 3 runs), that alone flips about 19 from fail to pass and about 19 from pass to fail. Of the 15 edits proposed on the three splits, 4 were kept:

Compared with the metaskill before the editAttempts fixedAttempts broken
Kept edits (average of 4)30.88.8
Rejected edits (average of 11)18.319.9
Same metaskill, run twice19.218.9

Counts are out of 240 validation task attempts (80 tasks × 3 runs). For the last row, the second run is compared with the first. A typical rejected edit fixes and breaks about as many attempts as running the same metaskill twice, which is why the gate rejects it. The kept edits fix far more task attempts than they break.

W2. Is GSO's gain due to a hand-tuned starting metaskill rather than to learning?

No. We also tested the starting metaskill, and here are the results:

ConditionTest pass rate (%)
No skill42.8
A skill written for each task, with no metaskill32.7
GSO, starting metaskill (no learning)33.8
GSO, learned metaskill48.8

The starting metaskill alone scores 9 points below no skill. Learning lifts it by 15.0 points, to 6.0 points above no skill, and it is above no skill on every split. The gain comes from what is learned, not from the hand-written starting guide. The next reply tests GSO's check-and-retry step directly.

W3. Does GSO's check-and-retry step make the comparison unfair?

At test time, GSO's agent may check its own output and retry once, within the same attempt at the task. It does not start a second, independent attempt. We tested this step in both directions:

ConditionTest pass rate (%)
GSO, learned metaskill48.8
GSO, learned metaskill, check-and-retry step removed51.2
SkillOpt45.7
SkillOpt with GSO's check-and-retry step added38.0
  • Step removed from GSO: it scores 51.2%, slightly higher; the difference is not significant on any split. We still report GSO with the step, because the step was fixed as part of GSO before any test run; switching after seeing test results would mean using the test set to choose a method.
  • Step added to SkillOpt: its score drops by 7.7 points. So the step is not where GSO's gain comes from.

W4. How exactly are training and test gains measured?

We now give these definitions in Section 3.1.

  • Training score and test score: the share of training or test tasks the method passes, with the same agent and the same scoring rule. For GSO, this means the skills written from the learned metaskill.
  • Training gain and test gain: the training or test score minus that of no skill, in points.
  • Retention: the test gain minus the training gain. Near zero means the skill keeps its training gain on new tasks; negative means it loses part of it; positive means its test gain is larger.

Following another reviewer's suggestion, we also measure each gain as a share of the room left above no skill. This matters when no skill is already near the top. For example, if no skill scores 90% on test, a skill can add at most 10 points there. In the new study, the learned metaskill's test gain is about as large as its training gain: retention is +2.5 points, and its 95% confidence interval includes zero (Appendix C.2).

W5. Are the results reliable, given the test set size and run-to-run variation?

We added the new study to answer this:

  • Test tasks: 100 per split.
  • Splits: three random splits.
  • Runs: three runs of each method on each split.
  • Uncertainty: we report paired 95% confidence intervals over tasks.
MethodTest pass rate (%)
No skill42.8
Trace2Skill45.1
SkillOpt45.7
GSO, learned metaskill48.8

The learned metaskill scores 6.0 points above no skill (95% confidence interval: +1.4 to +10.4). The gain is positive on each split: +8.7, +5.3, and +4.0 points.

GSO also scores above both baselines, but these two differences are not statistically significant (Appendix C.2).

W6. How was RBioBench built and checked?

RBioBench is our benchmark of R programming tasks for clinical and bioinformatics analysis. Appendix D now describes how we built and checked it. Every task follows one written standard:

  • What a task contains:
    • a prompt that names the input and output files;
    • the input files;
    • a reference R solution;
    • the expected outputs, made by running that solution.
  • Review checklist: the code runs, the biological method is correct, the prompt is clear, the test data are reasonable, and the difficulty level fits.
  • Use of AI: as stated in our AI Use Statement, we used generative AI to draft and clean candidate tasks. The authors reviewed them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading. Our paper asks whether skills that an agent learns for itself still help on new tasks. GSO takes a meta-learning perspective: rather than optimizing individual skills, it evolves a metaskill that guides agents in generating task-specific skills. The metaskill is iteratively refined based on agent execution feedback using the generated skills, enabling better generalization and reducing overfitting to individual tasks. In this response, we expanded our experiments on SpreadsheetBench to include three additional models—Qwen3.5-35B-A3B, gpt-oss-120b, and gpt-oss-20b—and conducted evaluations across three random data splits to quantify the variability of the results (Section 5.6; full results in Appendix C).

Below are short answers to four questions that several reviewers asked. Each reviewer reply gives the details.

1. Do the results hold with more test tasks and repeated runs? Yes. GSO scores 48.8%, against 42.8% with no skill: a gain of 6.0 points, positive on all three splits.

2. Does GSO gain only because it does more work at test time? No. At test time, GSO's agent may check its own output and retry once, within the same attempt at the task. Without this check-and-retry step, GSO scores 51.2%, slightly higher; the difference is not significant on any split. Adding the same step to SkillOpt, an existing skill-learning method, lowers its score from 45.7% to 38.0%.

3. Do the results depend on one model? We gave the metaskill learned with Qwen3.5, unchanged, to two other models. It improves them over the hand-written starting metaskill (the guide before learning) by 3.8 points (gpt-oss-120b) and 4.4 points (gpt-oss-20b). On these two models it does not beat their own no-skill scores; the differences are not significant.

4. Which learned rules carry over, and why? Rules about how to deliver the answer, not about any one task. Three separate learning runs found the same kinds of rules. On the first split, learning adds 21.3 points over the starting metaskill (which scores below no skill: 27.7% vs 40.3%). Adding only the rule "write computed values, not formulas" to the starting metaskill gains 15.7 of them; removing it from the learned metaskill costs 11.3.

We also added formal definitions (Section 3.1) and pseudo-code (Algorithm 1). New text in the paper is shown in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.