acceptodds
Withdrawn from ICLR 2027

The Contract–Injection Mismatch: An Execution-Conditioned Audit of Skill Self-Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-evolving skill frameworks write a reusable skill document from an agent's own task experience, and are accepted by a single average success rate: measured under one benchmark, one base model, and one deployment configuration, the evolution counts as effective once the number goes up. Whether this score represents the skill—whether it holds outside the configuration that produced it, and what it conceals within—has never been examined, yet self-evolved skills are being certified, compared, and deployed on it. We therefore conduct an execution-conditioned audit of SkillOpt, a representative self-evolution system: we compare skill and no-skill conditions on the same task pool, repeat every task five times, and read every rule evolution wrote. The audit finds that the score is a property of the execution condition: +13.4 points on SearchQA become −6.2 on OfficeQA, flipping or splitting again when only the base model or the deployment harness changes; even within the original configuration, the score is the residual of 315 tasks improved against 116 harmed. Separating the two sides: the gains are stabilization—turning intermittently correct tasks into always-correct ones—while the costs land on tasks that never failed, where injection introduces error types that did not exist before. The text explains both sides: evolution writes operating rules with preconditions, not new knowledge, yet injection places every rule on every task. Because a rule's preconditions can only reveal themselves in execution, the audit's rule of use is: do not inject by default; retry once with the skill on failure—outperforming both no-skill and full injection on all three benchmarks. In simpler terms: rules written for some executions are deployed into all of them, and whether they help is only knowable after execution.

Primary area:
interpretability and explainable AI
Keywords:
skill self-evolution, language agents, execution-conditioned audit, skill evaluation, contract–injection mismatch, repeated paired execution, LLM reliability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GHxfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits SkillOpt, a skill self-evolution system, by running skill and no-skill conditions on the same task pools with five repetitions per task, and by categorizing the rules that evolution adds to the skill. The authors find that the paired gain changes sign across benchmarks, base models, and harnesses, e.g., +13.4 points on SearchQA but −6.2 on OfficeQA. They further show that the gains mainly come from tasks that were only sometimes solved, while the harm falls on tasks the no-skill executor always solves, and that the added rules are operational conventions rather than knowledge. Based on this, they propose to keep the skill off by default and retry once with the skill on failure, which reaches 59.1% on SearchQA versus 37.0% without the skill and 50.4% with full injection.

Critical strengths
  1. The evaluation protocol is careful and reports what happens at the task level. Every comparison pairs the same tasks under skill and no-skill conditions with five repetitions, reports how many tasks improved and how many were harmed, and uses McNemar tests and task-cluster bootstraps. For example, the +14.2 net gain of one SearchQA skill comes from 315 improved tasks against 116 harmed ones. This is useful because it shows that a single average score can hide large changes in both directions, which is a good practice for how self-evolved skills should be evaluated.
  2. The paper also checks what evolution actually writes into the skill. The 189 rules added by evolution are categorized with released, deterministic patterns rather than LLM judges, and the paper reports that only about one in ten proposed revisions is accepted (10.7%). This is useful because it gives a concrete and checkable picture of the evolved skill, instead of only reporting its final score.
Critical weaknesses
  1. The major concern is that the technical contribution of this work seems limited. The proposed rule, i.e., keep the skill off by default and retry once with the skill after a failure, is quite intuitive and more like a simple engineering trick than a new method. It is also not surprising that a second attempt helps, since almost any retry strategy would. Besides, the main finding that skills mostly stabilize tasks that were already sometimes solved, rather than adding new capability, has already been reported by prior work, e.g., Jiang et al. (2026) find that a given skill mostly stabilizes existing behavior. The paper argues that its difference is studying the evolution process rather than given skills, but the analysis itself, i.e., paired skill vs. no-skill runs and categorizing the added rules, is largely a standard evaluation, and it only covers one system, SkillOpt. So it is hard to tell what new insight or technique the paper brings beyond a careful audit of one system.

  2. Another concern is that the analysis behind the main findings may be misleading. The paper groups tasks into easy, mixed, and hard bands based on how the no-skill runs perform, and then compares them with new runs using the skill. The problem is that the grouping itself already decides the result: a task picked because it was correct in all five no-skill runs can only go down in new runs, and a task that was never correct can only go up. This would happen even if the skill does nothing. In fact, the paper's own human seed, which is just a short placeholder without any rules, shows the same pattern, i.e., −23.33 on the easy band and +6.18 on the hard band. So the claim that the skill helps hard tasks and hurts easy tasks is not well supported. In addition, the retry rule is not compared fairly. It decides whether to retry using the official evaluator, which needs the gold answer, and it gets two attempts while the baselines only get one. Since more than half of the SearchQA tasks are already sometimes solved without the skill, simply retrying without the skill may also give a large gain. The authors are encouraged to address these issues.

Questions for authors
  1. What is new in the findings compared with prior skill audits, e.g., Jiang et al. (2026)?
  2. Have the authors compared the retry rule with simply retrying once without the skill?
Reviewer_L6MfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits SkillOpt, a representative self-evolving skill framework, asking whether the single average success rate that certifies such skills actually represents them. The protocol is careful: same task pool, same executor, same evaluation protocol, skill vs. no-skill paired task-by-task, five repetitions per task to separate lucky from reproducible successes, and the text of every evolved skill read and categorized deterministically. Findings: (1) the score does not travel — the same evolved skill is worth +13.4 points on SearchQA and −6.2 on OfficeQA, and flips or splits again across base models and deployment harnesses; (2) the net score conceals 315 improved against 116 harmed tasks even within its own configuration; (3) the mechanism is that evolution writes operational contracts with preconditions (answer formats, evidence checks) which are then injected wholesale into every execution — where preconditions hold, intermittently-correct tasks become always-correct; where they fail, previously-correct tasks acquire error types that did not exist without injection; (4) no pre-execution signal predicts which case applies, but the pass/fail outcome of a first execution does — yielding a simple deployable policy, "default off, retry once with the skill on failure," which beats both extremes on all three benchmarks.

Critical strengths
  1. The protocol is exemplary and the paper is unusually scrutable. Paired same-task/same-seed executions, five repetitions, frozen protocol before consulting official outcomes, exact McNemar tests, task-cluster bootstrap with Holm correction within declared families, and — most commendably — fully deterministic audit scripts with archived inputs/outputs behind every content-level table (Table 8). The provenance map (Table 7) explicitly ties each table to its pool and statistic. This is close to a model of what empirical auditing in this area should look like, and it directly addresses the post-hoc-selection and LLM-judge-confound problems that plague skill evaluations.

  2. The human-seed control is a genuinely clever experimental design move. Showing that on OfficeQA the evolved product's harm nearly equals its human seed's (−6.2 vs. −6.1) reveals that part of what would otherwise be booked as "evolution helped" is actually the cost of injection itself being booked as a gain of evolution. This single row changes how one reads the whole literature, and the recommendation to keep an empty/human control in the loop is cheap and immediately actionable.

  3. The two-sided ledger is the right object. Prior audits study given skills at single-execution granularity; this paper studies what one evolution run changed relative to its seed, and it records both sides — stabilization on the mixed band (+14.6) and new errors on the easy band (−9.3, with value/evidence mismatch appearing 0% → 9.0%). The verbatim harm attribution (Table 3) — wrong page numbers, wrong statistical basis, formulas never materialized — grounds the mechanism in concrete, checkable events rather than vibes.

  4. The deployment policy is honest and validated against its own alternatives. "Default off, retry on failure" is not the only strategy the authors tried; Table 22 registers the ones that lost to always-injecting, and the oracle analysis (Table 21) is reported with both discounts (52.6% capability-limited failures; best-skill instability across test seeds 45.7% of the time). Reporting the roads that do not work, pre-registered-style, materially raises my confidence in the one that does.

  5. The GPT-evolved checkpoint vs. self-evolved comparison (Table 13) is a nice, clean demonstration that constraints travel with the evolver — the stronger optimizer's more elaborate document leaves the small target model worse at 5.5× the cost per correct outcome, with the missing delivery clause as the concrete mechanism.

Critical weaknesses
  1. One system is audited, but the framing repeatedly generalizes beyond it. The abstract and conclusions speak of "self-evolving skill frameworks" and state the mismatch as if it were a property of the class; the evidence is SkillOpt plus one official checkpoint and one SkillCreator document. Whether Voyager (code skills, cumulative library), Trace2Skill (trajectory-local lessons), or ExpeL (verbal reflections) exhibit the same mismatch is unknown — code skills, for instance, are gated by execution success rather than prompt injection, which plausibly changes the harm profile. The claim "we verified whole-document injection in 105/105 examined cells" establishes deployment practice only for this system. Either audit a second system (even one cell of one) or consistently scope the claims to whole-document prompt-injection systems.

  2. The models are small and the headline phenomena may be scale-dependent. Much of the drama comes from instability that a 2B model exhibits (54.2% mixed band on SearchQA; 3.34 distinct answers per unstable task). "The gains are stabilization" and "the skill records constraints written for the target model" are most consequential exactly for weak, high-variance executors. Whether a frontier base model — which has both fewer intermittently-correct tasks and different failure modes — shows a contract–injection mismatch of comparable size is untested, and it matters for how the findings should be read by practitioners. At minimum, discuss explicitly why the qualitative mechanism (preconditioned rules injected unconditionally) should be scale-independent even if the quantitative effect sizes are not.

  3. The central causal claim stops one ablation short of proof. The paper is careful — "log cues are treated as evidence of where the failure appears, not as rule-level causation" — but the mismatch story would be directly testable by injecting ablated documents (drop the output-format rules, keep evidence rules, etc.) and re-running the harmed easy band. The correlation between the two most-written rule categories (formatting 43, evidence 35) and the two dominant harm modes (wrong-value 54.5%, non-materialized 33.5%) is suggestive but confounded by benchmark delivery contracts. This ablation is feasible within the existing infrastructure and would convert a plausible narrative into a demonstrated mechanism.

  4. The deployment policy's cost is not analyzed. Retry-on-failure means up to 2× inference on every failed task (and failure detection itself requires a full first execution plus an evaluator signal — which in deployment is often the expensive part). The paper reports skill-call savings (37% on SearchQA) but never states the aggregate budget multiplier of the policy, nor compares against cheaper alternatives in the same cost class (e.g., inject only the evidence-verification rules; inject with a "if this does not apply, ignore" wrapper — admittedly half-heartedly testable). For a paper whose punchline is a deployment recommendation, the missing cost column in Table 4 is a real gap.

  5. The "no pre-execution information suffices" claim is over-general from a handful of probes. Table 17 tests a surface-similarity router, structured features, and same-task history. A learned router trained on paired outcomes — rather than nearest-neighbor over question text — is the natural baseline the field would try, and its absence leaves "the decision cannot be made before execution" as an assertion tested mainly at information-poor points (which Table 22 usefully shows lose). Soften the claim or add the learned-router baseline.

  6. Statistical robustness is uneven across cells. OfficeQA pools are small (172 tasks in Table 1, 374 in the stability audit) and effects flip sign per seed; the harness axis (Table 10) rests on seed means of ±15–20 point swings with per-seed CIs that mostly include zero, and two cells are flagged single-seed/directional-only. The strong pooled claims are fine; the per-harness sign claims are weaker than the prose suggests. ALFWorld (160 tasks, seed skill only, point estimates without intervals) is the weakest leg of Table 4.

  7. Overlap with concurrent work needs sharper differentiation. Demystifying Agent Skills already finds that "a given skill mostly stabilizes existing behavior" at scale. The genuinely new elements here are: auditing self-produced text, the two-sided paired ledger with a human-seed control, and the deployment policy. That is enough, but the introduction should say plainly which of their findings this paper confirms vs. extends, rather than differentiating on the given-skills/self-evolved axis alone.

Questions for authors
  1. Can you run the rule-ablation injection experiment (even on the SearchQA easy band) to directly test the contract–injection mechanism?
  2. What is the aggregate cost multiplier of retry-on-failure, and does the policy still dominate under a budget constraint that binds at 1.5× base cost?
  3. Why do pooled base rates differ across tables (e.g., OfficeQA 13.1% vs. 32.89%)? A single consolidated pool table in the main text would preempt reviewer confusion.
  4. For the harness axis: can you pair executions over tasks (not just seeds) to get task-cluster CIs for Claude Code / Codex CLI rows, given the seed-level swings?
  5. Does the retry policy survive when skill injection changes the conversation context rather than the system prompt (retrieval-style), where a retry is not a fresh independent draw?
  6. Given hosted-version drift and missing serving manifests for older runs, which headline numbers would you expect to be most fragile to re-fingerprinting?
Reviewer_NGXROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper's main claim is that execution is needed for skill selection, as evolved skills may not generalize sufficiently and may only perform well on the benchmarks on which they were evolved. Execution is therefore needed to determine whether a skill is useful for a particular task. This is not a particularly valuable research question, as execution is already a very common component of skill validation.

Critical strengths

The cases in the appendix are very detailed, and the cross-references between the appendix and the main text are clear.

Critical weaknesses
  1. The writing is very unclear. Although I cannot be certain that it is AI-generated, it has a very strong AI-generated feel.
  2. The authors frequently introduce unnecessary numerical results in the introduction. These results should primarily appear in the experiments section rather than the introduction.
  3. In the introduction, why does the paper outline start with Appendix A and then move to Section 2? Is this a normal order for presenting the structure of a paper?
  4. Section 3.3 claims to establish the direction of causality based on the increase in errors from 0% in the easy band. However, the easy band itself was selected to include only tasks answered correctly in all five runs under the no-skill condition.
Questions for authors

Why does the paper show so many signs of AI Vibe research? Even if you used AI, shouldn’t you have reviewed the manuscript yourself before submitting it? Why are the numerical results in the LaTeX tables written in Markdown syntax?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.