- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether agent skills, i.e. reusable procedural guidance in text form produced by frontier models through skill-evolution frameworks (SkillOpt, EvoSkill, TextGrad), help much smaller executor models. The evaluation uses Qwen3.5 (4B/9B/27B) and Gemma-4 (12B/26B-A4B) on HotpotQA, DAPO-Math, BigCodeBench, and ALFWorld. The authors find that strong-model skills do not consistently help smaller executors: the same skill can help one executor and hurt another. Feedback-driven evolution also does not reliably outperform skills the strong model writes directly. To explain why, the paper analyzes skill content through instruction-category ablations, reasoning trajectories, and continuation experiments that inject the same guidance at different solution stages. These analyses show that executors follow the skills but may apply advice where it does not fit, and that a skill's usefulness depends on when it enters the solution. Finally, the paper compares external skill transfer with internalization through reinforcement learning (GiGPO) and on-policy distillation. Training improves performance even without skills. After skill-guided training, keeping the skill at inference brings further gains, but adding it to a model trained without the skill can hurt. The authors conclude that producing a useful procedure and enabling another model to apply it are distinct parts of skill transfer.
Critical strengths
- Timely and practically relevant research question. Frontier models are stronger at summarizing experience and executing tasks, so skills that transfer to small models would let a strong model build a procedure once and cheaper executors reuse it without weight updates. The paper tests this assumption directly, which existing skill-evolution work leaves under-examined.
- Broad and controlled comparison. The study covers three skill-evolution frameworks, SkillOpt (Yang et al., 2026 [1]), EvoSkill (Alzubi et al., 2026 [2]), and TextGrad (Yuksekgonul et al., 2025 [3]), along with several generation models and five executors from two model families (dense and MoE). Within each comparison, baseline and skill conditions share the same task instructions and execution interface, and evolved and directly generated skills from the same generation model are compared side by side. This makes the main conclusion less likely to be an artifact of a single framework or executor.
[1] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.
[2] Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.
[3] Yuksekgonul et al. Optimizing Generative AI by Backpropagating Language Model Feedback. Nature, 639:609–616, 2025.
Critical weaknesses
- Dated, relatively easy benchmarks with contamination and ceiling risks. HotpotQA (2018) and ALFWorld (2021) predate the evaluated executors (Qwen3.5, Gemma-4) by many years, and BigCodeBench (2024) also predates them. Their test data has likely been exposed during pre-training, which can inflate no-skill baselines and mask how much a skill actually contributes (Sainz et al., 2023 [4]; Golchin & Surdeanu, 2024 [5]). Several settings are also near saturation. HotpotQA no-skill accuracy is already 81–88% (Table 1) and reaches 93.8–96.3% in the continuation experiments (Table 4), which leaves little headroom to separate helpful skills from harmful ones (Ott et al., 2022 [6]; Kiela et al., 2021 [7]). These tasks also have fairly short procedures compared with the multi-step, tool-heavy settings where agent skills are typically deployed. It is therefore unclear whether the conclusion that skills transfer unreliably carries over to the use cases that motivate the paper.
- Small effect sizes relative to evaluation noise. Many differences in Table 1 are within ±1–3 points. The paper's own repeated runs (Table 7) show standard deviations of 0.40–1.21 points, and these runs cover only one skill per task, two tasks, two executors, and three repetitions. BigCodeBench has only 114 test problems (Table 9), so a single problem changes accuracy by about 0.9 points. No confidence intervals or significance tests are reported. Accuracy variance across independent skill-generation runs is also never measured: Table 6 reports only the semantic similarity of the generated documents. As a result, many individual "helps" or "hurts" cells may not be distinguishable from noise. The weak cross-executor agreement (Appendix B.4) may also be partly a consequence of noise, since measurement noise attenuates Pearson and CCC estimates. The claim would be better supported by statistical tests across seeds and skill-generation runs, or by restricting it to the larger effects (e.g., the category ablations and the early/late continuation gaps).
- Unaddressed conflict with the transfer results of the original frameworks. SkillOpt reports that all of its cross-model transfers are positive. Its GPT-5.4 skills improve GPT-5.4-mini and GPT-5.4-nano by +3.0 to +9.4 points, and "no row falls below the target's no-skill baseline" (Yang et al., 2026 [1], Table 4a). EvoSkill reports positive zero-shot transfer from SealQA to BrowseComp (+5.3) (Alzubi et al., 2026 [2]). The present paper finds transfer that is frequently negative, but it does not discuss this discrepancy.
[1] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.
[2] Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.
[3] Yuksekgonul et al. Optimizing Generative AI by Backpropagating Language Model Feedback. Nature, 639:609–616, 2025.
[4] Sainz et al. NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. Findings of EMNLP, 2023.
[5] Golchin & Surdeanu. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. ICLR, 2024.
[6] Ott et al. Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence. Nature Communications, 13, 2022.
[7] Kiela et al. Dynabench: Rethinking Benchmarking in NLP. NAACL, 2021.
Questions for authors
- On the timing experiments (Section 4.3). The early and late branch points are defined by task events (Appendix B.6), so the Early and Late rows may cover different subsets of questions. This would explain why the no-skill scores differ between stages, e.g., BigCodeBench 68.24 vs. 72.60 for Qwen3.5-4B and 73.03 vs. 68.92 for Qwen3.5-9B (Table 4). Could the authors report the number of questions in each stage, and confirm whether the 9.15–11.97 point gap is computed only on questions that reach both branch points? In addition, the prefix is generated without a skill and the remaining budget is fixed. When a skill is injected early, does the model spend budget on reading and re-planning according to the skill (Figure 2 shows that it first reads skill.md)? Measuring this would help separate the effect of advice timing from the effect of budget consumption. Finally, does the finding lead to a deployable strategy? For example, running from scratch and exposing the skill only after the first public-test result, compared with exposing it at the start.