acceptodds
Under review as a conference paper at ICLR 2027

Skill Lock-In: Silent Retirement Failures in Self-Evolving LLM Skill Libraries

Abstract

Self-evolving LLM agents can retain harmful advice in growing skill libraries despite continued evaluation. Judge bias affects skill removal and admission differently. When feedback weakens measured effects without changing their sign, zero remains a meaningful decision boundary, while a permissive admission rule can allow much greater true harm. We identify sparse skill use, optimistic feedback and routing changes as interacting causes of this skill lock-in. We introduce PAWL (Paired Anytime-valid Watchdog for Libraries), a lifecycle that estimates a skill's contribution on the tasks where it is used. It randomly withholds skills to maintain task-matched no-skill references and uses confidence bounds valid under repeated inspection to govern removal and admission. Experiments across four solver families and two live LLM judges show stronger harmful-skill detection and more selective admission. In one matched optimistic-feedback setting, PAWL retires 16 of 18 planted harmful skills, versus two for the success-count baseline. Separate replays show that normalizing success counts by skill use remains insufficient. On screened, loop-synthesized coding skills, PAWL admits 15 of 18 helpful candidates and none of 18 harmful candidates under strong simulated optimistic feedback within the tested budget. These findings establish measurement, threshold choice and sustained exposure as central design considerations for self-evolving skill libraries.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, self-evolving agents, skill lifecycle management, LLM-as-a-judge, noisy feedback, confidence sequences, adaptive evidence allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DQgzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper shows that by comparing the performance with and without certain text-based skills, one can select the set of skills that are most helpful to get higher performance on simple benchmarks, and exclude skills that hurt performance on these benchmarks.

Critical strengths

The high-level idea makes sense, is clearly explained, and looks like the sort of approach that is not very common while looking like it ought to work (given the right context, which I am not convinced the paper is studying).

Critical weaknesses

The evaluations are very toy and simple. These tasks do not capture the sort of relatively big and open ended tasks that skills are mostly meant to help with. These are likely tasks that are already saturated by frontier models, and I find it suspicious that some skills are measured to be net-helpful. The paper's Figures do not clearly show the magnitude of the uplift nor what kind of skills result in any uplift on these likely saturated tasks.

The paper has relatively quantitative and concrete qualitative results showing what the approach would look like in practice. It also does not present comparisons with simple baselines (e.g. like asking an LLM if a skill looks helpful or not).

Questions for authors
  • Do the helpful skills actually provide an uplift? What do they look like? Why are these strong LLMs failing simple tasks?
  • What do the harmful skills look like? Is it trivial to notice these are harmful?
  • How would the method perform on more realistic evaluations with either harder tasks or simply more realistic and messy real world settings? I am especially interested in settings where harmful skills are likely to emerge naturally.

Minor

  • the lower threshold should likely be <0, since many skills likely reduce performance a tiny amount on most tasks but are extremely useful on some (potentially in ways which outcome based measurements don't capture). Do you agree?
  • Could you avoid calling models "Opus"? It would be better for the reader to be able to parse the Figure and get the actual version of the model without having to read the section detailing what version of the model is studied.
  • I suggest you rename "harmful" to "performance-reducing", or a similar expression that conveys there is nothing malign about those skills.
Reviewer_sPpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why harmful natural-language skills can persist in the libraries of self-evolving LLM agents. It identifies three interacting causes: exposure dilution in retirement scores, distorted judge feedback, and routing changes that deprive unresolved skills of further observations. It proposes PAWL, which uses randomized skill withholding to maintain task-matched no-skill references and applies anytime-valid confidence sequences to residual feedback. These bounds govern admission and retirement, alongside probation and evidence-allocation policies.

The analysis explains when judge noise preserves a zero retirement threshold and how it weakens a permissive admission threshold. Validation includes coding and instruction-following experiments, four solver families, two live judges, simulated feedback corruption, recorded-sequence comparisons, and held-out utility evaluations. The strongest results demonstrate improved retirement of planted harmful skills and useful admission selectivity within the tested budgets. The formal certificate covers a weighted residual mean; interpreting it as true skill harm additionally requires control of reference error and suitable assumptions about the judge.

Critical strengths
  1. The paper provides a clear and mathematically explicit account of several distinct retirement failures. The identity Gs=aszˉsG_s=a_s\bar z_s explains why a fixed negative cutoff can become unreachable when a skill is rarely used. The judge-channel analysis explains why normalizing by use does not resolve the measurement problem. Connecting these observations to task-matched comparisons and sequential inference produces a coherent lifecycle design. The distinction between zero-threshold retirement and permissive admission is particularly informative. The authors also specify the certificate's target and the additional assumptions needed to interpret it as true harm. This careful separation of claims strengthens the contribution.

  2. The controlled experiments provide substantial evidence that PAWL addresses the studied failure modes. At false-pass rate 0.450.45, PAWL retires 16/1816/18 harmful plants compared with 2/182/18 for the count rule, with a paired seed-cluster analysis supporting the difference. Additional comparisons examine normalized counting, alternative confidence bounds, reference estimators, other solvers, and live judges. Executable audit intervals support negative effects for 21 of 23 certified Opus-judge planted-harm retirements. In the expanded coding admission cohort under strong simulated false-pass feedback, PAWL admits 15/1815/18 screened helpful candidates and none of 18 screened harmful candidates within the tested budget. This demonstrates a useful finite-budget operating point, although harmful candidates are predominantly excluded through probation expiry rather than certified rejection.

Critical weaknesses
  1. The connection between the operational certificate and actual harm remains insufficiently validated for naturally generated skills. Section 4 correctly states that E[di′∣Fi−1]=ci′+p0′(xi)−p^0(xi)\mathbb{E}[d_i' \mid \mathcal{F}_{i-1}] = c_i' + p_0'(x_i)-\widehat p_0(x_i). The certificate therefore covers a residual that includes reference-estimation error, while the deployed interval does not account for uncertainty in that reference. Reference reuse is important to the reported power: Table 6 shows 66/9866/98 harmful retirements with shared references, 24/9824/98 with audit-only references, and 1/981/98 with the tested randomized AIPW alternative. These differences do not establish bias, but they make validation at actual decision crossings consequential.

Live-judge distortion also has a relevant direction. Appendix C.6 reports judged-minus-executable residual differences from −0.10-0.10 to −0.18-0.18 on planted harms, whereas arm-blind optimism predicts positive differences from +0.24+0.24 to +0.47+0.47. This indicates a relative penalty on skill outputs in those experiments. It does not establish the same distortion on near-neutral skills, but it raises a question about specificity. The separate long-horizon study retires 12/1812/18 screened near-neutral plants, with eleven corresponding executable audit intervals including zero. Among 29 retired synthesized skills, four task-bootstrap intervals lie below zero, 17 include zero, and eight single-task cases have no interval. These audits use a different target from the certificate and do not establish false-retirement rates. They leave the reliability of natural-skill harm decisions unresolved, which limits the practical interpretation of the method.

  1. The admission result depends on the evaluation budget, while the broader comparison does not yet establish the practical advantage of the complete lifecycle. Under false-pass rate 0.700.70, Table 8 reports zero harmful admissions, but 17 of the 18 harmful candidates expire and one remains open; none receives certified rejection. Under Proposition 2's matched-reference, constant arm-blind assumptions, κ=0.3\kappa=0.3 and b=−0.15b=-0.15 imply only μˉt≥b/κ=−0.5\bar{\mu}_t \geq b/\kappa=-0.5 at admission. The screened harmful effects range from −0.69-0.69 to −0.16-0.16, overlapping that permissive range. Screening and routed effects can differ, and retirement takes precedence over admission, so longer-budget behavior remains an empirical question. Helpful admissions also increase from 3/183/18 in the default cohort to 15/1815/18 in the expanded cohort, where capacity, insertion timing, and duration change together.

The adapted gate leaves 17 of 18 helpful candidates unread under clean MBPP+ feedback, evaluates skills regardless of applicability, and uses additional solves. These disclosed choices limit conclusions about PAWL's advantage over a comparably allocated paired validation procedure. The natural-skill population is also selected and small: six coding designs per class, screened on one to thirteen source tasks, with seven source tasks shared across the helpful designs. Finally, the held-out PAWL-versus-no-skill intervals in Table 9 all include zero. A broad accuracy gain is not necessary for this diagnostic contribution, but the evidence leaves the tradeoff between reliable removal, useful retention, evaluation cost, and downstream value insufficiently resolved.

Questions for authors
  1. Clarification and reanalysis: Can the existing retirement audits separate the contributions of reference-estimation error and judge distortion, using weights aligned with the certificate's target? Please report planted harms, synthesized skills, and near-neutral controls separately. This would help determine whether disagreements reflect different evaluation targets or a failure of the residual to track executable contribution. An executable interval containing zero should remain an inconclusive result.

  2. Budget sensitivity: Do the existing longer-budget studies show what happens to harmful candidates that expire in Table 8 when probation is extended while the admission bar and other allocation settings are held fixed? Do they reach certified retirement, admission, or continued uncertainty? Proposition 2 explains the permissiveness of the threshold, but does not determine which decision occurs first.

  3. Candidate-selection clarification: Appendix B.4 selects some natural candidates using lower bounds above +0.10+0.10 or upper bounds below −0.10-0.10 from earlier runs. Were these PAWL's own paired confidence bounds, using the same construction evaluated here? Please report how many candidates were excluded at each selection stage and whether existing results cover candidates not selected through those bounds, including the intermediate screened-effect range. The fresh executable screen provides independent outcome evidence, but selection by the evaluated method's statistic could still limit the population to which the admission result applies.

  4. Targeted comparative validation: Is there an existing comparison against a paired validation gate with applicability-aware tasks, comparable access to candidate evaluations, and a matched total solver-and-judge budget? A focused comparison on an existing cohort would help distinguish the value of sequential inference from the effects of scheduling and measurement choices.

  5. Judge hacking: The threshold analysis assumes arm blind feedback, and the live judges in Appendix C.6 penalize skill outputs (judged minus executable residuals of −0.10-0.10 to −0.18-0.18), which favors retirement. The reverse case seems more concerning for self evolving libraries: a skill that makes outputs more persuasive to the judge without making them more correct, for example by adding self verification commentary, asserted test cases, or confident justifications. This produces a positive skill specific offset Δeff>0\Delta_{\mathrm{eff}} > 0. Following Lemma 1 and the argument of Propositions 1 and 2, retirement remains sound, but a harmful skill is never retired once Δeff≥κ∣μˉt∣\Delta_{\mathrm{eff}} \ge \kappa|\bar\mu_t|, and admission at Lt≥bL_t \ge b certifies only μˉt≥(b−Δeff)/κ\bar\mu_t \ge (b - \Delta_{\mathrm{eff}})/\kappa, so even the zero and positive bars could admit harmful skills. Because admission selects on judged contribution, repeated synthesis may also favor judge pleasing skills over time, and in the Opus setting the same model synthesizes skills, solves tasks, and judges outputs, which raises the self preference concern the paper cites (Panickssery et al., 2024). Did any synthesized or planted skill show a positive offset? Could you plant judge targeted skills with screened executable effect at most zero and report their retirement and admission outcomes under both live judges?

Reviewer_4L1fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies harmful-skill persistence in self-evolving LLM skill libraries under sparse routing and imperfect feedback. It proposes PAWL, which collects routed no-skill references and uses anytime-valid confidence sequences for skill retirement and admission. Experiments on coding and instruction-following tasks evaluate planted harms, synthesized skills, simulated label corruption, and live LLM judges.

Critical strengths

1.The paper clearly explains why harmful skills can persist despite continued evaluation. It distinguishes between infrequent use, distorted feedback, and routing changes that cut off further evaluation. This is useful because these problems require different remedies; collecting more success counts alone cannot address them all. The threshold analysis further shows how weaker feedback can make a fixed admission rule more permissive toward harmful skills.

2.The paper provides a broad empirical evaluation across multiple benchmarks (MBPP+, IFBench, and BIRD-SQLite, with transfer tests on HumanEval+ and BigCodeBench-Hard) and solver models (Opus 4.7, Kimi K2.5, GLM-5, and GPT-5.5), using both simulated and live-judge feedback. This breadth helps assess whether the findings extend beyond a single task, model, or feedback setting.

Critical weaknesses

NA

Questions for authors

Does a library curated by PAWL achieve higher accuracy on new tasks than an append-only library that keeps all accumulated skills?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.