acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Final Skill: Revaluing Skill Evolution History After Executor Replacement

Abstract

Skill evolution produces a sequence of reusable Skill versions, but deployment usually keeps only the version selected by the source model. When that model is replaced, an earlier version may work better with the new executor. Existing transfer approaches typically reuse the source-selected Skill or optimize a new target-specific Skill, leaving this historical choice unexplored. We propose HEIRS (History-preserving Executor-aware Inheritance through Revaluation and Selection), which preserves the Skill history, keeps a compact set of distinct candidates, and selects one unchanged archived Skill using separate target-side screening and confirmation sets. Across Qwen3.5 migrations and a DeepSeek-to-GLM cross-family transfer, HEIRS improves selected migrations: mean Qwen ALFWorld success rises by 4.5 percentage points over endpoint transfer, and DeepSeek-to-GLM Spreadsheet success rises by 11.5 points. Across Qwen Spreadsheet and ALFWorld migrations, HEIRS uses 52.4% fewer logical target evaluations than fresh SkillOpt while avoiding target-side Skill optimization. The code implementation is available at https://anonymous.4open.science/r/HEIRSanonymous.4open.science/r/HEIRS.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, skill evolution, cross-model transfer, lifelong learning, model migration, agent skills

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_pbBiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HEIRS (History-preserving Executor-aware Inheritance through Revaluation and Selection), a four-stage framework that selects and directly transfers the best-performing version from the historical Skill archive under a limited target-side evaluation budget, without modifying the selected Skill.

Critical strengths
  1. The paper is the first to identify and validate the “historical option value” of historical Skill versions after model migration, showing that Skills rejected by the source model may perform better on the target model.
  2. It proposes HEIRS, a framework that combines source-side history coverage with two-stage target-side screening and confirmation to select the most suitable historical Skill under a fixed evaluation budget.
  3. It systematically evaluates the benefits, selection errors, and cost boundaries of HEIRS, clarifying when historical re-evaluation succeeds and where it may fail.
Critical weaknesses

The framework proposed in this paper is heuristic in nature and relies on strong prior assumptions. Furthermore, the lack of detailed descriptions makes it difficult to gain a thorough understanding or conduct a nuanced analysis of the approach. For instance, what are the specific details and the formal definition of "Task-Cover"? Similarly, further elaboration on "History-Cover" is also required.

Questions for authors
  1. If the executor condition involves changes to the runtime/harness, how can one demonstrate that an observed ranking reversal is caused by the model replacement rather than by changes in the execution environment?
  2. After performing multiple rollouts for the same skill task, are the historical rankings and the final selection stable? Does this have any implications for our observations and methodology?
Reviewer_AYxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to reuse Skills when the executor LLM of a self-evolving agent is replaced. The authors argue that a Skill version rejected during source evolution may suit the new executor better than the final selected version. They propose HEIRS, which keeps the whole Skill history, picks a small set of textually different candidates, screens them on a few target tasks, confirms the best ones on a separate set, and transfers one archived Skill unchanged. Experiments on SpreadsheetBench-Verified and ALFWorld across six migrations between Qwen3.5 models, one from DeepSeek to GLM, and one from Qwen9 to Llama8 show gains over directly transferring the endpoint on ALFWorld (+4.5 points on average) and on SpreadsheetBench when migrating from DeepSeek to GLM (+11.5 points), while the average Spreadsheet gain within Qwen is negligible.

Critical strengths
  1. The paper raises a real and well-motivated problem. In seven of the 11 complete cases studied, the final Skill from source evolution is not the best one for the new executor, and in each of them the best Skill had actually been rejected during source evolution. Simple source-side signals, such as the validation score, do not reliably find these Skills either. This is useful because it shows that simply keeping the latest Skill is not enough when the executor changes, which is a practical issue for self-evolving agents.
  2. The paper reports its failures and uncertainty openly. It shows a clear failure case on Qwen9→Llama8, reports intervals that cross zero, and states that the analysis on the expanded task sets is inconclusive. This is useful because it makes the evidence easy to check, and readers can see which results are solid and which are not.
Critical weaknesses
  1. The major concern is that the gain over simply transferring the final Skill is small and not consistent. On SpreadsheetBench, the average over the six Qwen migrations is 33.7% for HEIRS vs. 33.6% for direct transfer, and HEIRS is actually worse in three Qwen directions and on Qwen9→Llama8. The gains on ALFWorld and DeepSeek→GLM come from only a few extra tasks, e.g., 6 of 52 tasks on DeepSeek→GLM. In addition, it is unclear how many runs each result is based on, and the significance tests are done for an earlier version of the selector rather than the final HEIRS. Given the randomness of LLMs, these small gains may be within run-to-run variance. The authors are encouraged to address these issues.

  2. Another concern is that the technical novelty of HEIRS seems somewhat limited. HEIRS is essentially picking the best archived Skill on a few target tasks and then confirming it on a separate set, which is a fairly standard selection procedure. In fact, simply screening the whole history gets a higher Spreadsheet score than HEIRS (35.0% vs. 33.7%), and on ALFWorld HEIRS even costs more evaluations than screening the whole archive. Since each archive only has 4 to 13 Skills, screening all of them is already cheap. Besides, a rewrite baseline in the appendix gets higher averages than HEIRS on both benchmarks, but it is left out of the main table. So it is hard to tell what HEIRS adds beyond simple selection.

Questions for authors
  1. How many runs is each result based on? Have the authors conducted significance tests for the final HEIRS against direct transfer?
  2. Have the authors compared HEIRS with a simple baseline that tests all past Skill versions on the target tasks and picks the best one, using the same number of evaluations?
Reviewer_VUQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which archived agent skill should be deployed after replacing its underlying language model. HEIRS retains accepted and rejected skill versions, selects candidates through text-based coverage, screens them on shared target tasks, and chooses an unchanged version using a separate confirmation set. Experiments on SpreadsheetBench and ALFWorld show that some source-rejected versions transfer better than the source-selected endpoint and examine the cost of selecting from an existing archive.

Critical strengths
  1. The paper provides concrete evidence that source-rejected skill versions can become useful after executor replacement. The task-level analysis, including the workbook output interventions, helps explain how an archived version can behave differently across models.

  2. Screening, confirmation, and final testing use separate task sets. The results include useful gains: average Qwen ALFWorld success increases from 54.5% to 59.0%, and DeepSeek-to-GLM Spreadsheet transfer gains six successes out of 52 tasks. The evaluation also reports migrations where reselection provides no benefit or reduces performance.

Critical weaknesses
  1. The contribution beyond existing skill-transfer and candidate-selection work is modest. MASA(https://arxiv.org/abs/2605.30723) and WikiSkill(https://arxiv.org/abs/2608.27454) already show that skill utility depends on the executing model. Selecting from a fixed candidate pool under an evaluation budget is also well studied, including in TRIPLE(https://proceedings.neurips.cc/paper_files/paper/2024/hash/b46bc1449205888e1883f692aff1a252-Abstract-Conference.html). HEIRS adds the specific idea of reconsidering complete versions from the same evolution history, including rejected proposals, after executor replacement. This is a useful extension, but the method combines established clustering, screening, and validation procedures. Its methodological advance is limited, leaving the empirical findings to carry most of the contribution.

  2. The experiments establish the usefulness of archive re-evaluation more clearly than the added value of the particular HEIRS selector. Appendix E.1 includes matched-candidate controls, but History-Cover, Source Top-K, and archive-order uniform admission all achieve zero candidate regret on the two development matrices. Figure 6 shows a retention advantage over Source Top-K on broader diagnostic panels, but does not establish the final performance advantage under matched screening and confirmation. The main Full History reference uses one-stage selection. Table 1 also shows almost no average improvement on Qwen Spreadsheet, from 33.6% to 33.7%. An end-to-end comparison with simpler archive-selection procedures is needed to demonstrate that the proposed selection design makes a consequential difference.

Questions for authors
  1. How does the final two-stage HEIRS procedure compare with Source Top-K and archive-order uniform admission when candidate counts, total target-evaluation budgets, and confirmation rules are matched? This would help distinguish the value of retaining the archive from the value of the proposed selector.
Reviewer_59auOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the behavior of evolved natural-language skills when the underlying executor LLM is replaced. The authors demonstrate that executor replacement can reorder skill effectiveness: on a 16-task spreadsheet panel, rejected intermediate checkpoints outperform the source-selected endpoint in 7 out of 11 views, while source-side scores and checkpoint order correlate poorly with target utility. Mechanistic analysis via task-level decomposition and an uncached-formula intervention highlights how executor-dependent skill interpretations drive these discrepancies.

To address this, the authors propose HEIRS, a four-stage transfer framework: (1) preserving the full optimization trajectory; (2) admitting a diverse subset of checkpoints via embedding distance (History-Cover), anchoring on the endpoint; (3) screening candidates on 16 target tasks; and (4) confirming a shortlist on a disjoint task set before deploying the selected skill unchanged. HEIRS is evaluated across six intra-family migrations (Qwen3.5 4B/9B/27B) and two cross-family pairs (DeepSeek→GLM, Qwen9→Llama8) on SpreadsheetBench-Verified and ALFWorld. Headline claims include a +4.5pp mean gain over Direct Endpoint on ALFWorld, a +11.5pp gain on DeepSeek→GLM Spreadsheet, and a 52.4% reduction in target evaluations relative to optimizing skills from scratch.

Critical strengths
  1. Executor replacement is an increasingly common practical problem as model APIs evolve. Section 3 presents a clear empirical case that an optimization endpoint on a source executor is frequently suboptimal on a target model, and that source-side metrics fail to predict target performance. The task-level disruption/recovery decomposition (Eq. 3) and the targeted output-step intervention (Fig. 4d) substantiate this claim with concrete operational mechanisms rather than just aggregate metrics.
  2. The evaluation framework is conscientiously constructed: screening, confirmation, and test sets are strictly disjoint; hyperparameters are chosen on development data; and the selected-gain formulation (Eq. 4) systematically isolates admission error from screening/selection error. The authors also candidly report regressions (e.g., Qwen 27B→9B) and null directions (Qwen9→Llama8).
Critical weaknesses

My primary concern is that the marginal and inconsistent empirical gains lack statistical validation. For example, on Qwen Spreadsheet migrations, HEIRS averages 33.7% versus 33.6% for the zero-cost Direct Endpoint baseline. It actually degrades performance in 3 of 6 directions (most notably 27B→9B, which drops from 28.9% to 24.2%, or roughly 6 out of 128 tasks) and achieves a clear advantage only on 4B→27B.

Furthermore, the reported percentage gains translate to very few discrete tasks—for instance, +6.2pp on ALFWorld corresponds to approximately 3 of 48 tasks, and the headline +11.5pp on DeepSeek→GLM corresponds to roughly 6 of 52 tasks.

Finally, the paper relies on single evaluation runs without reporting variance across repeated executions. More critically, the paired significance analysis in Table 10 evaluates only the single-stage B16 baseline (where only 1 of 10 rows is statistically significant), omitting paired significance tests (e.g., McNemar’s test or bootstrap CIs) for the complete two-stage HEIRS framework. Without this evidence, it remains unproven whether the reported gains reflect genuine algorithmic transfer or stochastic evaluation noise.

Questions for authors
  1. Can you provide paired significance tests (e.g., McNemar's test and bootstrap confidence intervals) specifically comparing the final two-stage HEIRS against Direct Endpoint across all evaluated migrations, rather than only reporting this for the one-stage B16 baseline?

  2. Given that the observed gains often correspond to 3–6 individual tasks, what is the empirical variance when evaluating the selected Skill across multiple seeds or non-zero temperatures on the target executor?

  3. Is the reordering of candidate checkpoints consistent across independent evolution runs (multiple source-side trajectories), or was this observation derived from a single optimization history?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.