- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HEIRS (History-preserving Executor-aware Inheritance through Revaluation and Selection), a four-stage framework that selects and directly transfers the best-performing version from the historical Skill archive under a limited target-side evaluation budget, without modifying the selected Skill.
Critical strengths
- The paper is the first to identify and validate the “historical option value” of historical Skill versions after model migration, showing that Skills rejected by the source model may perform better on the target model.
- It proposes HEIRS, a framework that combines source-side history coverage with two-stage target-side screening and confirmation to select the most suitable historical Skill under a fixed evaluation budget.
- It systematically evaluates the benefits, selection errors, and cost boundaries of HEIRS, clarifying when historical re-evaluation succeeds and where it may fail.
Critical weaknesses
The framework proposed in this paper is heuristic in nature and relies on strong prior assumptions. Furthermore, the lack of detailed descriptions makes it difficult to gain a thorough understanding or conduct a nuanced analysis of the approach. For instance, what are the specific details and the formal definition of "Task-Cover"? Similarly, further elaboration on "History-Cover" is also required.
Questions for authors
- If the executor condition involves changes to the runtime/harness, how can one demonstrate that an observed ranking reversal is caused by the model replacement rather than by changes in the execution environment?
- After performing multiple rollouts for the same skill task, are the historical rankings and the final selection stable? Does this have any implications for our observations and methodology?