acceptodds
Under review as a conference paper at ICLR 2027

Evolving to Match Capabilities: Improving Harness Self-Evolution via Skill Adaptation

Abstract

Harness self-evolution offers a practical route for LLM agents to improve from execution experience without updating model parameters. Its effectiveness depends on whether models can successfully act on the guidance provided by their harnesses. However, we find that harness benefits are not uniform across model capabilities: skills that improve stronger models can provide little benefit or even degrade performance for weaker ones. Across seven models and three benchmarks, we identify 103 task–skill pairs where skills improve GPT-5.6-luna by 31.1 points but degrade Gemma-4-E4B by 8.8 points. In 86% of failed weak-model runs, the violated requirement is already stated in the skill. This motivates skill adaptation: reformulating existing skills to match the executor’s capabilities and make their guidance easier to execute. To achieve this, we diagnose failures in tool execution, context management, instruction following, and verification, and distill these diagnoses into a fixed meta-skill that guides adaptation during harness self-evolution. Guided by the meta-skill, a capable external rewriter, Kimi-K3, adapts skills without execution evidence, improving the three weak executors by 5.1–14.1 points over the original skills. We then incorporate this adaptation into a five-round self-evolution loop, where each model revises its own skill from execution trajectories and verifier feedback. Across seven executors, from Gemma-4-E4B to Kimi-K2.5, the meta-skill consistently improves final scores by 2.0–5.1 points over the same loop without the meta-skill. Our findings provide practical design guidance for developing more effective harness self-evolution methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Agents, Skill Adaptation, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EyXZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why weaker models struggle to use skills that benefit larger models. It identifies four types of failure and designs a meta-skill that guides an external rewriter to rewrite the skill. Experimental results show that after rewriting, the performance of all weaker executors has improved.

Critical strengths
  1. The paper identifies four types of failures that the weak models violate requirements already present in the skill and designs a meta-skill to rewrite them. This is a practical approach because it explains how available guidance can be made easier for an agent to follow.

  2. Experimental results show that after rewriting by the rewriter using the meta-skill, all the weak models gain improvements. It also performs better under the same five-round revision budget, suggesting that the guidance itself helps, beyond using a stronger model or more attempts

Critical weaknesses
  1. The paper uses the same 103 pairs for failure analysis, meta-skill construction, and evaluation, without a reported held-out test set. It is unclear whether the meta-skill also works well on tasks that were not used to develop it.

  2. The method relies on a manually designed, fixed meta-skill. It cannot be automatically updated or modified to adapt to new tasks and failure patterns.

  3. There are already several methods that study the problem of skill adaptation, such as MASA[1] and SkillRevise[2], but neither is included as an experimental baseline. The paper only makes comparisons with original skills and generic rewriting prompts.

[1] Yu, J., Zhu, J., Lin, B., Cui, Q., Ding, Z., & Li, X. (2026). Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents. arXiv preprint arXiv:2605.30723.

[2] Liu, Y., et al. (2026). SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision. arXiv preprint arXiv:2606.01139.

Questions for authors

Please see the weaknesses above.

Reviewer_mzJaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why weak models fail to benefit from skills. It selects 103 task–skill pairs on which skills help strong models but not weak ones. It groups weak-model failures into four modes and turns them into a meta-skill that guides skill rewriting. The meta-skill is tested in one-shot rewriting by strong models and in five-round self-evolution by weak models. Results show that skills rewritten by the strong model gains more.

Critical strengths

None.

Critical weaknesses
  • This paper selects 103 skill-task pairs precisely and all subsequent experiments rely on these pairs. It may introduce significant bias, making it unclear how general the findings are. Moreover, it remains unclear whether the meta-skill can benefit more general tasks.
  • All experiments are conducted in single run, which makes the results fragile. To my knowledge, agent performance can vary substantially across runs, even under identical settings. In single-run setting, it is difficult to distinguish gains brought by the skills from run-to-run variance.
Questions for authors

See weaknesses.

Reviewer_Q42SOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why skills that help strong models can fail with weaker ones, and proposes a fixed meta-skill to guide skill rewriting and self-evolution.

Critical strengths

The paper is well written and easy to follow. The layout and figures are polished, and the examples explain the method clearly. The problem also has practical value.

Critical weaknesses

My biggest concern is that there are so many similar papers in skill evolution, which makes me concerned about the novelty for this paper. The deatils are here:

  • MASA (arXiv:2605.30723) already studies how skill effectiveness depends on the model and proposes model-aware adaptation. The paper cites it, but describing this issue as “unexamined” seems too strong. The authors should explain what their capability-matching approach adds beyond a simpler adaptation recipe.

  • SkillRevise (arXiv:2606.01139) already uses execution diagnosis and repair principles to guide skill revision. I know you already cited SkillRevise, but the difference from the proposed meta-skill is not clear. Starting from human-written rather than LLM-written skills changes the setting, but does not by itself establish a new revision mechanism.

  • Two other relevant works appear to be missing: SkillAdaptor (arXiv:2606.01311), which locates the first actionable failure and makes targeted skill updates, and SkillAligner (arXiv:2608.06880, August 2026), which adapts skills to the task and execution interface before execution. These overlap with the paper’s earliest-failure diagnosis and one-shot rewriting settings, respectively. A discussion would help explain which parts are genuinely capability-specific rather than general improvements to skill instructions.

  • The current baselines do not resolve this novelty concern. Comparing against a generic rewriting instruction or removing the meta-skill shows that detailed guidance helps, but not whether this particular diagnosis framework is necessary. A stronger rewriting baseline with task constraints, error recovery, and verification would be more informative. Besides, the same 103 tasks are used to develop and evaluate the meta-skill, so its usefulness on independent tasks remains unclear.

Questions for authors
  • Please explain the differences in what gets adapted, how failures are diagnosed, and how skills are revised. Focusing on weak models alone does not make the methodological difference clear.
  • Could you add a comparison with one representative skill-revision method under comparable models, feedback access, and execution budgets?
  • Could you add a held-out evaluation with the meta-skill fixed beforehand, along with repeated runs or confidence intervals. This would make both the generalization claim and the relatively small self-evolution gains more convincing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.