acceptodds
Under review as a conference paper at ICLR 2027

Understanding Agent Skills: How Well Do They Transfer to Smaller Models?

Abstract

Skill evolution gives agents a way to improve over time by turning task-execution experience into reusable procedural guidance. Expressed in text, these lessons can be shared across models without transferring weights. We investigate whether skills developed by frontier models can help much smaller agents generalize to unseen problems while reducing reliance on expensive models. Though text makes this transfer more convenient, smaller models may interpret and follow the same instructions differently from their authors. Across distinct skill-generation frameworks and model families, we find that strong-model skills do not consistently improve smaller executors, and feedback-driven evolution does not consistently outperform direct generation. Analyses of instruction content, model computation, and reasoning trajectories show that responding to a skill does not ensure a better solution. Experiments that introduce guidance at different stages further show that its usefulness can change as a solution develops. Finally, because smaller models are comparatively inexpensive to fine-tune, we compare external skill transfer with internalizing experience through reinforcement learning and on-policy distillation. Training improves performance even without external guidance. After skill-guided training, retaining the document at inference provides further gains, although adding it to a model trained without that guidance can sometimes hurt. Together, these findings show both the promise and the difficulty of transferring experience through skills.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Skills, Agents, LLM Reasoning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rCkyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether agent skills, i.e. reusable procedural guidance in text form produced by frontier models through skill-evolution frameworks (SkillOpt, EvoSkill, TextGrad), help much smaller executor models. The evaluation uses Qwen3.5 (4B/9B/27B) and Gemma-4 (12B/26B-A4B) on HotpotQA, DAPO-Math, BigCodeBench, and ALFWorld. The authors find that strong-model skills do not consistently help smaller executors: the same skill can help one executor and hurt another. Feedback-driven evolution also does not reliably outperform skills the strong model writes directly. To explain why, the paper analyzes skill content through instruction-category ablations, reasoning trajectories, and continuation experiments that inject the same guidance at different solution stages. These analyses show that executors follow the skills but may apply advice where it does not fit, and that a skill's usefulness depends on when it enters the solution. Finally, the paper compares external skill transfer with internalization through reinforcement learning (GiGPO) and on-policy distillation. Training improves performance even without skills. After skill-guided training, keeping the skill at inference brings further gains, but adding it to a model trained without the skill can hurt. The authors conclude that producing a useful procedure and enabling another model to apply it are distinct parts of skill transfer.

Critical strengths
  1. Timely and practically relevant research question. Frontier models are stronger at summarizing experience and executing tasks, so skills that transfer to small models would let a strong model build a procedure once and cheaper executors reuse it without weight updates. The paper tests this assumption directly, which existing skill-evolution work leaves under-examined.
  2. Broad and controlled comparison. The study covers three skill-evolution frameworks, SkillOpt (Yang et al., 2026 [1]), EvoSkill (Alzubi et al., 2026 [2]), and TextGrad (Yuksekgonul et al., 2025 [3]), along with several generation models and five executors from two model families (dense and MoE). Within each comparison, baseline and skill conditions share the same task instructions and execution interface, and evolved and directly generated skills from the same generation model are compared side by side. This makes the main conclusion less likely to be an artifact of a single framework or executor.

[1] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.

[2] Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.

[3] Yuksekgonul et al. Optimizing Generative AI by Backpropagating Language Model Feedback. Nature, 639:609–616, 2025.

Critical weaknesses
  1. Dated, relatively easy benchmarks with contamination and ceiling risks. HotpotQA (2018) and ALFWorld (2021) predate the evaluated executors (Qwen3.5, Gemma-4) by many years, and BigCodeBench (2024) also predates them. Their test data has likely been exposed during pre-training, which can inflate no-skill baselines and mask how much a skill actually contributes (Sainz et al., 2023 [4]; Golchin & Surdeanu, 2024 [5]). Several settings are also near saturation. HotpotQA no-skill accuracy is already 81–88% (Table 1) and reaches 93.8–96.3% in the continuation experiments (Table 4), which leaves little headroom to separate helpful skills from harmful ones (Ott et al., 2022 [6]; Kiela et al., 2021 [7]). These tasks also have fairly short procedures compared with the multi-step, tool-heavy settings where agent skills are typically deployed. It is therefore unclear whether the conclusion that skills transfer unreliably carries over to the use cases that motivate the paper.
  2. Small effect sizes relative to evaluation noise. Many differences in Table 1 are within ±1–3 points. The paper's own repeated runs (Table 7) show standard deviations of 0.40–1.21 points, and these runs cover only one skill per task, two tasks, two executors, and three repetitions. BigCodeBench has only 114 test problems (Table 9), so a single problem changes accuracy by about 0.9 points. No confidence intervals or significance tests are reported. Accuracy variance across independent skill-generation runs is also never measured: Table 6 reports only the semantic similarity of the generated documents. As a result, many individual "helps" or "hurts" cells may not be distinguishable from noise. The weak cross-executor agreement (Appendix B.4) may also be partly a consequence of noise, since measurement noise attenuates Pearson and CCC estimates. The claim would be better supported by statistical tests across seeds and skill-generation runs, or by restricting it to the larger effects (e.g., the category ablations and the early/late continuation gaps).
  3. Unaddressed conflict with the transfer results of the original frameworks. SkillOpt reports that all of its cross-model transfers are positive. Its GPT-5.4 skills improve GPT-5.4-mini and GPT-5.4-nano by +3.0 to +9.4 points, and "no row falls below the target's no-skill baseline" (Yang et al., 2026 [1], Table 4a). EvoSkill reports positive zero-shot transfer from SealQA to BrowseComp (+5.3) (Alzubi et al., 2026 [2]). The present paper finds transfer that is frequently negative, but it does not discuss this discrepancy.

[1] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.

[2] Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.

[3] Yuksekgonul et al. Optimizing Generative AI by Backpropagating Language Model Feedback. Nature, 639:609–616, 2025.

[4] Sainz et al. NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. Findings of EMNLP, 2023.

[5] Golchin & Surdeanu. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. ICLR, 2024.

[6] Ott et al. Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence. Nature Communications, 13, 2022.

[7] Kiela et al. Dynabench: Rethinking Benchmarking in NLP. NAACL, 2021.

Questions for authors
  1. On the timing experiments (Section 4.3). The early and late branch points are defined by task events (Appendix B.6), so the Early and Late rows may cover different subsets of questions. This would explain why the no-skill scores differ between stages, e.g., BigCodeBench 68.24 vs. 72.60 for Qwen3.5-4B and 73.03 vs. 68.92 for Qwen3.5-9B (Table 4). Could the authors report the number of questions in each stage, and confirm whether the 9.15–11.97 point gap is computed only on questions that reach both branch points? In addition, the prefix is generated without a skill and the remaining budget is fixed. When a skill is injected early, does the model spend budget on reading and re-planning according to the skill (Figure 2 shows that it first reads skill.md)? Measuring this would help separate the effect of advice timing from the effect of budget consumption. Finally, does the finding lead to a deployable strategy? For example, running from scratch and exposing the skill only after the first public-test result, compared with exposing it at the start.
Reviewer_x56DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of strong-to-weak skill transfer. It presents a study on whether (and why) the skills generated by strong models transfer to weak models. The paper presents empirical studies on four different benchmarks, two (small) model families spanning five models, and several skill generation techniques. The results show that the the skills generated by large models result in mixed performance change for the small models that execute the skills. The paper then presents an analysis on what category these skills belong to, and how small models use those skills. The paper then discusses when the skills should be incorporated during task execution. The paper also presents results on training agentic models with skills.

Critical strengths
  1. The problem that the paper studies is timely and important.
Critical weaknesses
  1. While the authors present results related to several research questions, the paper does not appear to have key takeaway(s) that readers can learn from. The results in the main Table, Table 1, can be regarded as negative results, since the skills that strong models generate do not yield consistently good results on small model executors. However, the analysis the paper later presents is mostly descriptive and does not address either of the following two questions: 1. why this happens 2. how to mitigate this/improve the performance of smaller model executors with skills.

  2. The paper does not analyze whether or not the generated skills are themselves good, at least when large/strong models that generate these skills are the executor.

  3. Originally I wrote this in the 'Question' section, but I believe it is worth to mention in critical weakness, exceeding the two critical weaknesses budget. I followed the suggestion in Appendix B.8 to open the anonymous link to get 'more results' , but I only got very few results that do not match the claim in Table 1. After another review of the code repository, I don't believe it has enough details to actually reproduce the results claimed in the paper. For example, I think there is almost no code related to the skill-evolving methods mentioned (SkillOpt, TextGrad, EvoSkill). While releasing code and other artifacts in the review phase is optional, this mismatch between claimed open-source and actual released artifacts is concerning.

Questions for authors

Please address the concerns in weakness section. Moreover,

  1. Considering the very minor numerical difference for comparable entries in Table 1, it is highly recommended to include the mean and standard deviation for the tested models.

  2. For the training with skills part, it is suggested to compare the approach with existing papers that explicitly considers skill in agentic training, e.g. [1]

[1] Co-Evolving Skill Generation and Policy Optimization https://arxiv.org/abs/2606.08755

Reviewer_CCZLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the cross-model transferability of textual "skills"—reusable procedural instructions generated by frontier models (GPT-5.4, Claude Opus 4.8, Claude Sonnet 4.6, and Qwen3.8-Max)—when deployed on smaller executor models (the Qwen3.5 and Gemma-4 families). The authors compare evolved skills, iteratively refined via execution feedback using frameworks such as SkillOpt, EvoSkill, and TextGrad, against zero-shot directly generated skills and no-skill baselines. They find that frontier-evolved skills do not consistently improve smaller models, nor does feedback-driven evolution reliably outperform direct skill generation. To analyze this failure mode, the authors examine instruction taxonomy, instruction following capability, mid-trajectory instruction injection, and skill internalization via RL. They conclude that while RL and distillation improve baseline performance, external skills only reliably provide additive benefits at inference time if the small model was explicitly exposed to skill-conditioned supervision during training.

Critical strengths
  • Practical and Timely Research Question: Decoupling expensive frontier models (as skill curators/authors) from cheap, low-latency small models (as routine executors) is a highly practical deployment paradigm. Rigorously questioning whether text-based procedural memory actually transfers across capability scales addresses an important gap in the agentic prompt/skill optimization literature.

  • Interesting Synergy Finding in Internalization: The comparison between GiGPO and OPD under skill-conditioned vs. skill-free training offers a clean insight: adding a task-relevant skill at inference time can hurt a model trained without skills, whereas training with skill guidance makes the student consistently receptive to retaining the skill document at test time.

Critical weaknesses
  1. Confounded Evolution Baseline (Missing "Strong Curator + Small Executor" Loop): Throughout the main experiments, the evolved skills are generated using the frontier model as both the skill curator and the rollout executor during the evolutionary loop (see Footnote 1: "a single model name means that generation and rollout use the same model"). Consequently, the skill evolves to fix the frontier model's failure modes rather than the smaller executor's failure modes. For example, in Figure 3 (Section 4.1), evolution on HotpotQA reduces evidence-gathering instructions from 59.2% to 16.7% because the frontier model already retrieves evidence well; however, a 4B or 9B executor still struggles with basic multi-hop evidence retrieval. Evaluating a "Strong Curator + Small Executor" evolution baseline—where the cheap small model generates validation rollouts and the frontier model curates the skill—is essential to separate whether text skills inherently fail to guide small models, or if the skills in this paper simply overfit to the frontier model's error distribution.

  2. Qualitative Over-Reliance in Trajectory Analysis: The current analysis in section 4.2 relies on qualitative than quantitative study. A more fine-grained study which evaluates step-by-step trajectory prefixes with a Process Reward Model (PRM) or using the token-level log-probability advantages from the 27B teacher (Section 5) would provide a much more rigorous, quantitative measure of when following a skill rule helps vs. derails a smaller model.

  3. Lack of Statistical Significance: All experiments use stochastic decoding, and the test sets for BigCodeBench (N=114N=114) and ALFWorld (N=134N=134) are small (1 problem ≈0.88\approx 0.88 pp on BigCodeBench). Yet Tables 1, 2, 4, and 5 omit standard errors or confidence intervals. Many highlighted effects in Figure 1 and Table 1 (±0.4\pm 0.4 to ±1.3\pm 1.3 pp) represent 0.5 to 1.5 problems and fall well within the ±0.40\pm 0.40 to ±1.21\pm 1.21 pp sample standard deviations reported in Table 7. Where paired 95% confidence intervals are mentioned in the prose of Appendix B.5.1 for the Table 2 evidence/state ablation ([-11.40, 4.39], [-7.02, 5.26], [-7.02, 0.88]), every interval crosses zero, rendering the claimed performance drops statistically insignificant.

  4. Absence of Frontier Model Upper Bounds: The paper does not report the baseline or skill-augmented performance of the frontier generator models (GPT-5.4, Opus 4.8, Sonnet 4.6, Qwen3.8-Max) on the evaluated benchmarks. Thus, it is unclear how much each evolved skill actually improved the authoring model itself before transfer was attempted.

Questions for authors
  1. Footnote 1 states that "Appendix tables retain explicit generation–rollout pairs where different models are used". Where are these results located, and how well do skills transfer when the small executor itself is used to generate the rollouts during SkillOpt/EvoSkill/TextGrad evolution while the frontier model acts as the curator?

  2. Table 1 vs. Table 2 discrepancy: Why do the BigCodeBench scores for SkillOpt Opus in Table 2 (71.05%, 81.58%, 85.09%) differ by up to 3.5 percentage points from the SkillOpt Opus scores in Table 1 (69.3%, 78.1%, 82.0%)?

  3. Ablation design: In section 4.1, the authors define six instruction categories (Figure 3) but only ablate two categories on two tasks (Table 2 and Table 11). Why are these two categories selected but others are omitted? Moreover, why is it testing leave-one-out deletions, not testing single-category ("keep-only") instructions?

  4. In Section 4.2, the paper reports that HotpotQA trajectories across skills and executors contain 1.79–2.90 searches and 0.44–1.32 document reads per question, concluding that "evidence gathering recurs during solving". Could you provide the same stats for no-skill baseline to backup this claim?

  5. Sample Filtering: In Section 4.3 (Table 4), the authors argue that introducing a skill Late in a trajectory outperforms introducing it Early. However, the Early and Late checkpoints seems to be defined by task-specific milestones (e.g., Late on HotpotQA requires having already retrieved evidence supporting an answer candidate). Can you report the exact number of questions evaluated in each row of Table 4, explain the filtering criteria that caused No-Skill accuracy on HotpotQA (4B) to rise from 83.1% (Table 1) to 93.82% / 96.31% (Table 4), and recompute the Early vs. Late comparison restricted strictly to the identical intersection of questions that reached both checkpoints? If Early and Late are not evaluated on the same set of questions, it is a confounding factor for reaching a meaningful conclusion.

  6. Given the ±1.16\pm 1.16 pp baseline standard deviation on BigCodeBench (Table 7) and the wide paired bootstrap intervals in Appendix B.5.1, which specific skill gains/losses in Table 1, Table 2, and Table 5 are statistically significant at p<0.05p < 0.05 under paired bootstrap testing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.