acceptodds
Under review as a conference paper at ICLR 2027

Where Should a Skill Edit Live? Scoped Root Evolution in Hierarchical Agent Skill Libraries

Abstract

Self-evolving agents turn execution experience into edits to a skill library, but most edit locally, so what they learn stays where it was written. In a hierarchical library only the shared root skills can carry an edit to tasks that did not produce it; for the same reason a root edit's scope is the whole subtree, so it can repair the tasks it was written for while silently breaking a sibling, which admission rules that score only the proposing tasks or the subtree mean cannot see. We recast rootlevel evolution as an edit-placement problem with a single rule: an edit belongs in the root only if no evaluated descendant regresses under paired execution, and otherwise in the leaves of the descendants it helps. ScopeEvo instantiates the rule with root-aware revision, a subtree gate that promotes, localizes, or discards each candidate, and release recursion, and treats the leaf editor as a pluggable singleskill optimizer. On SkillsBench it further improves a library already evolved by a strong baseline, and on SearchQA and DocVQA it matches the strongest baseline with fewer editor calls and a more compact library. Its promoted roots repair tasks the loop never saw, and its refused root edits are recovered in the leaves rather than lost. We further show why a root edit needs subtree-wide validation while a localized edit does not, and that what a gate can guarantee from a few executions is bounded by the coverage of its validation set, which more runs cannot extend.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agent skill, llm agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dFwhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ScopeEvo, studying how the skill should be modified when a new line of experience is summarized from real tasks. It proposes that the skill should be structured as a tree; its root is responsible for the general skills and knowledge, while its leaves are responsible for specific tasks. The two levels of skills combine to finish the task. The authors points out that locating the skill modifications to either the root or a certain leaf is the challenging part. The paper sets certain rules to coordinate the modifications to obtain a general improvement. For the modifications that bring overall improvements, they are put into the root; when there is a trade-off between several tasks, that knowledge is then put into a leaf. If there is no gain at all, then the knowledge will be get rid off.

Critical strengths
  1. The paper identifies a very interesting and applicable problem in designing skills, that is how to keep the best balance between the shared skill and the specialized skill.
  2. The proposed method is easy to implement, and is applicable to the existing skill optimizers. The actual structure is also indeed simpler than the competing method.
  3. Theoretical guarantees on robustness are provided to support the method design, and the experimental results are straightforward to validate the method.
Critical weaknesses
  1. There is a lack of alignment of skill editing resources in comparing the methods. Thus the contribution of the proposed method is hard to validate. Skill-bench is an important benchmark, yet the method is built on the optimized skills of CoEvoSkills, there is no fair and direct comparison with the baseline.
  2. The complete cost of building the skill is not presented. The end-to-end token expenditure comparison between the proposed method and baselines should be provided to validate that the proposed method is more efficient. The existing results only indicate that the cost will be smaller, but are not concrete.
  3. The improvements over the strong baseline is not evident enough. The error bar on multiple experiments should be provided to validate that the proposed method is really effective. The current experiments only show subtle improvements on most tasks.
  4. The proposed root-leaf mechanism is not showing its combined effectiveness. Compared to the two ablated versions, the current structure is gaining from the root only or the leaf only. The improvements are only concentrated on one specific domain.
Questions for authors
  1. How is the proposed theory guiding the proposed method? From the presentation, the theory is more like a make-up on the already proposed method. Is the theory validated through experiments?
  2. When there are vast application scenarios, how does the proposed method scale in such situations? How should you decide the application coverage of the root? And what should the method change when there are tasks that share very little knowledge, and forcing the use of the root harms performance?
  3. How to guarantee the scalability of the method when there are a lot of tasks and the skill has to wriite a lot of rules? The conflict from different leaves could oftenly happen.
6 comments
AuthorsOpenReview ↗

Response to W1 W2 and W3

We thank the reviewer for the constructive suggestions. We address the fair-comparison, cost, replication, mechanism, and scalability questions below.

Response to W1: Controlled comparisons reveal what shared placement contributes

The added baseline comparison tests whether further editing alone explains the gain. Table 1 (Appendix D.5) starts native CoEvoSkills continuation, Local-only, and ScopeEvo from the same released library, evaluates all arms with GPT-5.6-sol on the same 17 tasks, and uses ten fresh runs per task. ScopeEvo and Local-only also share the editor, starting evidence, and leaf attempts. ScopeEvo's first evolution matches continued CoEvoSkills at 71.2% J17 while using 21 rather than 289 editor calls; unlike target-only leaf edits, its shared roots raise non-target passes from 101/120 to 105/120.

We separately control the placement decision. Table 6 (Appendix D.5) replays alternative admission rules on the same candidates and paired executions, holding proposal and decision-stage costs fixed. Total native-loop budgets differ, as reported under W2 below; we make no equal-compute end-to-end superiority claim.

Response to W2: Editing is cheaper, while subtree validation is more expensive

The cost breakdown now includes editor and acceptance-validation tokens. Table 1 and Appendix D.5 report the overall editing and validation costs. The input/output token breakdown is summarized below (millions of tokens):

CostCoEvoSkills continuedLocal-onlyScopeEvo
Editor calls2892121
Editor tokens, input/output (M)16.8 / 0.321.13 / 0.101.41 / 0.13
Validation executions1061224
Validation tokens, input/output (M)5.2 / 0.04109.5 / 0.59328.5 / 1.70
Tasks covered by validation5517
Validation executions per covered task2.012.213.2
Editor + validation tokens (M)≥22.4111.3331.7

These totals cover editing and admission validation; the 84 initial evidence runs shared by ScopeEvo and Local-only, and final evaluation executions, are disclosed separately and excluded from the admission-cost totals. CoEvoSkills' unrecorded hidden reasoning tokens also make its figure a lower bound. The resulting comparison is explicit and does not equate lower editing cost with lower total token cost.

The additional validation covers more affected tasks rather than re-editing the same targets. Continued CoEvoSkills spends at least 17.1M editor tokens and 289 calls, but validates each of five targets about twice. ScopeEvo uses about 1.5M editor tokens and 21 calls, like Local-only's 1.2M and 21, but spends 180 of its 224 validation executions on root checks and covers 17 rather than five tasks. Per covered task, ScopeEvo and Local-only use 13.2 versus 12.2 validation executions and 19.3M versus 21.9M input tokens; about 96% of this input is cached. The total ScopeEvo cost (331.7M tokens, roughly 95% cached executor input) is the price of checking every consumer of a shared update. Figure 4(b) separately shows that on QA it matches SkillOpt with 25% of its editor calls and 59% of its editor tokens. ScopeEvo economizes on editing but pays a validation cost approximately linear in descendants and paired repeats (Lemma 1, Corollary 1).

Response to W3: Reliability gains recur across independent evolutions

We added independent evolution runs, not just more scoring repeats. To address the small main-result margins and missing error bars, Table 1 evaluates each arm over 170 fresh executions and reports two separately evolved ScopeEvo libraries, with fresh editor samples and gate executions. Their J17 values are 71.2% and 75.9% (mean ± SD: 73.5 ± 3.3%), and their pass^5 values are 50.3% and 54.6% (52.4 ± 3.0%); the best one-run baseline gives 71.2% J17 and 48.9% pass^5. Both independent evolutions reproduce the reliability advantage in pass^5; the mean-performance evidence is descriptive with only two evolution runs.

The improvement extends beyond one descendant. Both evolutions improve four of the 12 non-target descendants, yielding 105/120 and 107/120 passes versus 101/120 for the baselines. Since leaf editing is confined to the five target tasks, non-target gains come through shared roots; Table 2 isolates this contribution for the first evolution. This repeats the cross-descendant effect, without establishing gains in every skill family.

AuthorsOpenReview ↗

Response to W4 Q1 Q2 and Q3

Response to W4: Root and leaf edits provide complementary, not uniformly additive, benefits

Figure 5 shows complementary roles for both scopes within one loop: shared-root transfer and task-specific leaf gains. New Table 2 (Appendix D.5) holds the leaves fixed when adding ScopeEvo roots. The four-pass non-target gain is the same observation across leaf configurations, since these tasks retain the same leaves; target effects depend on compatibility and can be negative. With CoEvoSkills continued's leaves, J17 rises from 71.2% to 78.8% in a component combination, not a single evolution run. The consistent root contribution is cross-descendant reach, while leaf effects remain task-specific.

Root and leaf edits contribute at different stages. In the GPT-5.6-sol run, two promoted roots add four passes among non-target descendants, and the accepted leaf edits add one net target pass (Table 2). In the original Opus 4.6 run, two promoted roots add three target passes without an observed loss, followed by three more from an accepted leaf (Appendix D.5). The spreadsheet root's 11 descendants gain seven passes in the ten-run audit (Table 3). Both scopes contribute, but their effects are not always additive.

Response to Q1: The theory specifies the gate's scope and safeguards

Each analysis result motivates a concrete design choice. To explain how §5 and Appendix C guide the method, Lemma 1 and Corollary 1 establish that a root edit can affect every inheriting descendant whereas a private leaf can affect only its own task; hence ROOTGATE checks the whole subtree and leaf gates check only their task. Theorem 1(c) establishes the validation-coverage limit; consequently, unusable scores yield UNDECIDED, and a root with all descendants at validation ceiling yields NOHEADROOM. Proposition 4 shows the false-refusal risk of zero-margin regression checks on stochastic neutral descendants, motivating the two-pass tolerance and adaptive 5→7-repeat protocol (Appendix B.2). The theory determines which tasks to check and which failure modes the gate must handle.

The added experiments probe the predicted limits. In Table 3, ten fresh runs reveal a PDF-root regression on court-form-filling (10/10 → 6/10) missed by the shorter gate. Table 4 evaluates the actual adaptive rule under the run model, complementing Theorem 1's fixed-budget, zero-margin analysis. Theorem 1(c) concerns absent validation inputs or inputs that pass with probability one under both libraries; Appendix F illustrates this coverage limit. These results distinguish sampling limits from coverage limits.

Response to Q2 + Q3: Placement in a fixed hierarchy and conflict handling

Validation scales linearly, but the algorithm does not force a shared root. The existing hierarchy determines which tasks inherit each root; ScopeEvo selects where to place an edit, not which tasks belong to the root. A root event uses 5–7 paired repeats per descendant (4–6 for shock-analysis-supply); LOCALIZE reuses the root outcomes and separately checks benefiting leaves. Table 4 models 2–13 descendants. For a 0.8→0.3 or 0.9→0.5 drop in one descendant, with all others unchanged at p=0.9, promotion has probability 0.02–0.06. If another descendant also improves, the relevant upper bound is the corresponding regression-miss probability. In its beneficial-edit case with neutral descendants at p=0.9, refusal rises from 0.133 to 0.665, and redirection to LOCALIZE from 0.079 to 0.622. New Table 5 applies the rule post hoc to a ten-descendant PDF subtree, where one regresses and three benefit; the edit is redirected to three leaf rewrites, only one of which passes its leaf gate. Placement therefore follows observed descendant-level compatibility within the given hierarchy.

When common knowledge is scarce, leaves remain the effective scope. On QA, just one of 16 candidate root edits is promoted by the replayed gate and the delivered leaf-first library accepts no root, matching Local-only (Appendix D.1). New Table 7 shows that forcing candidate R8 into the shared root would reduce DocVQA by 1.6 pp on test. Not promoting a root is an intended outcome, not a forced loss of task-specific knowledge.

Conflicting rules are isolated and tested before release. Private leaves are delivered only to their tasks (Lemma 1); on QA, more specific leaf guidance takes precedence, and a leaf is validated under its actual inherited root (Appendix B.1). The leaf gate rejects observed conflicts: leaf candidates 8 and 9 conflict with existing rules and are rejected (Appendix D.4). Root rules have explicit applicability conditions, edit count/text growth is bounded, and each accepted library becomes the next validation parent (§4). These safeguards contain observed conflicts, though deeper hierarchies and larger libraries remain outside the present evaluation.

Dear Reviewer dFwh, thank you again for your thoughtful review and constructive suggestions. We have added controlled comparisons and independent evolution runs, reported editing and validation costs separately, and clarified the complementary roles of root and leaf edits and the connection between the theoretical analysis and gate design. Your feedback is very important to us. We would be grateful for your assessment of whether the additional evidence and explanations address your concerns, and would be happy to discuss any points that need further clarification.

Reviewer_zsKnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies where to place execution-driven skill edits in a hierarchical agent library, since editing a shared root enables cross-task transfer but risks breaking sibling tasks. To prevent such regressions, the proposed ScopeEvo framework uses a validation gate to test each candidate root edit across all inheriting descendants, promoting the edit to the root only when the subtree improves without hurting any descendant. Conversely, when a root edit helps some descendants but harms others, ScopeEvo instead rewrites the edit into the private leaves of the benefiting descendants and validates those leaf edits separately. Because this gating decision relies on finite validation runs, the paper further analyzes the sampling and coverage limitations of the gate. Finally, to verify the framework in practice, the authors evaluate ScopeEvo through additional optimization of CoEvoSkills libraries on SkillsBench, QA experiments on SearchQA and DocVQA, and diagnostic studies of localization and transfer.

Critical strengths
  1. Equations (2) - (4) allow edits that help one descendant but harm another to be relocated into private leaves, recovering useful knowledge that binary root admission rules would discard. Because localization rewrites the candidate edit for the target leaf, the gain observed at the root may not survive the rewrite, which makes the subsequent leaf validation step essential. By pairing three-way placement with dedicated leaf validation, ScopeEvo turns the conceptual distinction between shared and task-specific knowledge into an actionable update procedure.

  2. Theorem 1 separates regressions missed due to limited execution sampling from those hidden by incomplete validation coverage, explaining why adding more runs cannot fix every gating failure. Complementing this analysis of missed regressions, Proposition 4 examines the opposite risk by showing how unchanged descendants can trigger false refusals. Because Appendix C clearly distinguishes these idealized bounds from the practical adaptive protocol, the theoretical analysis offers a transparent account of both the guarantees and the limits of execution gating.

  3. The diagnostic studies directly link placement decisions to behavioral improvements, showing that promoted power-flow edits transfer to withheld tasks while rejected QA root edits still yield gains after leaf localization. In addition, the repeated-run evaluation on SkillsBench shows that root promotion improves execution consistency across runs. Together, these results clarify how shared root updates and local leaf updates complement each other, even though the empirical evidence remains concentrated in a small number of tasks.

Critical weaknesses
  1. In Appendix D.1, the reported leaf-first QA library is identical to Local-only because no root edit was accepted, and Figure 6 shows the same test gain with 65 editor calls for ScopeEvo versus 48 for Local-only. While this outcome confirms compatibility with single-skill optimization, it shows no incremental gain from edit placement. Similarly, the main SkillsBench experiment applies ScopeEvo on top of an already evolved CoEvoSkills library without comparing against a resource-matched continuation of CoEvoSkills. As a result, neither headline benchmark cleanly isolates the benefit of edit placement from ordinary leaf optimization or extra editing budget.

  2. The paper provides limited evidence that ScopeEvo generalizes to unseen inputs across skill families on SkillsBench. Because the main SkillsBench benchmark evaluates final libraries on the exact same instances seen during optimization, generalization to new inputs is tested only on the separate placement testbed. On that testbed, however, Figure 7 shows that the entire aggregate gain comes from just two power-flow tasks. Moreover, the reported aggregate mixes 13 tasks with genuinely unseen inputs and five unchanged reference tasks scored on previously seen inputs, which further obscures true generalization performance.

Questions for authors
  1. Under a fixed initial library, identical editing constraints, and a matched total budget that includes validation executions, how does ScopeEvo compare against continued CoEvoSkills and Local-only optimization? In addition, please break down the resulting gains across ordinary leaf edits, root promotions, and localizations to isolate the specific contribution of edit placement.

  2. What specific rule triggers extending validation from five to seven repeats, and how was the two-pass regression tolerance chosen? To clarify how this tolerance behaves in practice, can you evaluate frozen candidates with more repeats to estimate how often the gate misses true regressions or falsely rejects beneficial root edits as the number of descendants grows?

7 comments
AuthorsOpenReview ↗

Response to W1 + Q1 and W2

We thank the reviewer for the detailed comments. We respond below to the comparison, generalization, and gate-calibration concerns.

Response to W1 + Q1: Placement is isolated, with full-loop costs reported separately

The controlled comparison separates shared-root gains from additional leaf editing. To address the absence of a continuation baseline, new Table 1 (Appendix D.5) evaluates CoEvoSkills continued, Local-only, and ScopeEvo from the same released library with the same GPT-5.6-sol executor, on 17 tasks with 10 fresh runs each. ScopeEvo and Local-only use identical editing constraints, initial evidence, and leaf attempts. ScopeEvo matches continued CoEvoSkills on first-run J17 (71.2%) and has higher pass^5 (50.3% versus 45.6%); its roots raise non-target descendants from 101/120 to 105/120 passes. This improvement reaches tasks that target-only leaf editing cannot reach.

The QA result is compatibility and filtering, not an extra score gain. Figure 6 and Appendix D.1 already show that leaf-first ScopeEvo delivers the same QA library and test scores as Local-only, with 65 versus 48 editor calls, because no new root is accepted. The QA domains share little rule-level knowledge. New Table 7 (Appendix D.5) tests the two additional root proposals: forcing R8 into the root lowers DocVQA by 1.6 pp without helping SearchQA, whereas R5 is close to the delivered library and slightly higher on this test. The placement layer prevents some inappropriate sharing; it does not guarantee an accuracy gain when there is no shared rule to promote.

Fixed leaves and fixed proposals identify placement separately from generation. Table 2 (Appendix D.5) shows that the promoted roots add four passes to non-target descendants under every fixed leaf configuration; their target effects depend on compatibility, while adding them to CoEvoSkills continued's leaves raises J17 from 71.2% to 78.8%. The accepted ScopeEvo leaves add one net target pass after the promoted roots in the GPT-5.6-sol run; in the original Opus run, the roots contribute three target passes and a subsequent leaf adds three more (Appendix D.5). Localizations are evaluated separately on QA (Figure 4(d), Appendix D.4) and the ten-descendant PDF subtree (Table 5).

Decision-stage costs are exactly matched; total native-loop costs are not. Table 6 (Appendix D.5) replays adapted admission-rule proxies on identical root candidates using the same editor calls and paired executions. Table 1 reports that the native loops allocate resources differently: ScopeEvo uses 21 editor calls and 224 validation executions; Local-only uses 21 and 61; continued CoEvoSkills uses 289 and 10. We did not run an equal-total-budget full-loop comparison and therefore do not claim equal-compute superiority. The fixed-candidate comparison isolates the placement rule, while the full-loop comparison shows its benefits and resource trade-off.

Response to W2: Generalization is measured on genuinely unseen inputs and tasks

We now report unseen-input results separately from seen reference cases. To address the confounding in Figure 5(a), new Table 8 (Appendix D.5) separates all 13 genuinely unseen-input tasks from the five unchanged tasks evaluated on seen inputs:

Task subsetInitial (%)Root-only (%)Local-only (%)ScopeEvo (%)
13 unseen-input tasks67.380.878.880.8
2 power-flow tasks12.5100.087.5100.0
Other 11 unseen-input tasks77.377.377.377.3
5 seen-input reference tasks100.0100.0100.0100.0

The unseen-input gain (67.3% → 80.8%) is concentrated in the two power-flow tasks, which share a computational failure; the other 11 remain unchanged and no root consumer loses a pass on its unseen instance. The evidence supports transfer of a common correction, not a broad gain across unrelated task families.

Whole-task holdout shows transfer in two different families. Table 8 also extends Figure 4(c)'s targets to 12 paired repeats, with neither target ever included in editor evidence or gate validation: protein-expression-analysis (spreadsheet) improves 9/12 → 12/12; grid-dispatch-operator (power-flow) improves 3/12 → 7/12. Local-only cannot update either withheld task. Unseen-task transfer spans both families, while the unseen-input gain is concentrated in power-flow.

AuthorsOpenReview ↗

Response to Q2

Response to Q2: The adaptive gate trades false refusals for regression detection

The 5→7 trigger and the two-pass tolerance are now explicit. To address the operational question, Appendix B.2 specifies five paired parent/candidate repeats per descendant (four for shock-analysis-supply). If any descendant is down by exactly one pass or the family's net change is exactly +1, all descendants are extended to seven repeats (six for shock-analysis-supply), and the gate is read again. A drop of at least two passes counts as regression; an increase of at least one counts as benefit. The tolerance guards against refusing a beneficial root because of noise on unchanged descendants: Proposition 4 shows the high false-refusal risk of a zero-margin gate. Under the actual adaptive rule, a neutral descendant at p=0.9 is falsely flagged with probability 0.085, compared with 0.27 for a zero-margin five-repeat check; degradation 0.9 → 0.5 is detected with probability 0.80 (Table 4). This is an explicit detection–false-refusal compromise, not a guarantee of zero misses.

Ten-run checks show both types of finite-sample errors. New Table 3 checks frozen edits with ten runs per affected task; its root rows compare the released library with the library carrying both promoted roots. Of 21 root–descendant pairs, the spreadsheet root has no observed regression, while the PDF root's court-form-filling loss (10/10 → 6/10) is missed by the original gate; Table 4 gives a 0.69 detection probability for a comparable 0.92 → 0.60 degradation. Eight rejected leaves fall collectively from 30/80 to 17/80 passes, so refusal prevents 13 net lost passes, although one rejected leaf improves in the deeper check. Conversely, an accepted shock-analysis-supply leaf declines from 6/10 to 3/10. Table 5 shows that deeper root counts would redirect the PDF edit to three benefiting leaves, of which only paper-anonymizer passes subsequent leaf validation. The same sampling limit can cause either missed harm or rejected benefit.

Increasing descendant count mainly increases false root refusals, not the per-descendant miss probability. New Table 4 evaluates the actual adaptive gate under the run model for 2–13 descendants:

Modeled eventk = 2k = 13
Degradation 0.8 → 0.3 not flagged0.1100.110
Such an edit still promoted0.0210.025
Degradation 0.9 → 0.5 not flagged0.1950.197
Such an edit still promoted0.0360.045
Beneficial edit 0.4 → 0.9 refused (other descendants p=0.9)0.1330.665
Beneficial edit refused but localized0.0790.622

The miss probability for one degrading descendant is approximately stable with k under these tested settings, while stochastic neutral descendants make root promotion harder. Many refusals become LOCALIZE opportunities rather than lost edits. No root was rejected in this run, so root-level false-refusal probabilities are modeled, whereas the frozen-candidate test directly observes missed degradation and leaf-level false refusal. The gate's reliability is therefore bounded both by repeat count and by validation coverage, as the revised analysis states.

Thank you for the detailed responses and additional experiments. The new comparisons help clarify the contribution of shared root edits beyond leaf optimization, and separating the unseen-input tasks from the reference cases makes the transfer results easier to interpret. The deeper checks also provide useful evidence about cases where the validation gate misses a performance drop.

Could you clarify whether the library evaluated in Table 1 still includes the promoted roots and the accepted shock-analysis-supply leaf examined in Table 3? This is relevant to understanding how Table 5 relates to the main results. Does Table 5 only show how additional validation runs would have changed the placement decisions, or were those changes applied to the final library? If the library was changed, please identify which results use the original version and which use the revised version. Please also clarify whether the revised library was evaluated using fresh runs, separate from those used to decide which edits to keep.

Reviewer_c4BnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks where an edit to a hierarchical agent skill library should live. SCOPEEVO tests proposed root edits across descendants, then promotes them, localizes benefits to selected leaves, or discards them. The distinction between a useful edit and one placed too broadly is sensible. But does the mechanism add value?

Critical strengths

The distinction between an edit that is wrong and an edit that is correct but written at too broad a scope is a useful one, and validating a shared edit on every consumer rather than only on the tasks that proposed it is the right response to it. Scoring is programmatic throughout: verifiers, exact match, ANLS, with no LLM judge anywhere in the paper, which eliminates the LLM induced confound or bias. The appendices are well-presented and, in places, supply the counterexample to the paper’s own headline: Appendix F reports that the promoted spreadsheet root lowered two consumers on their training instances, and notes that recomputed there the per-descendant rule returns localize rather than promote.

Critical weaknesses
  1. The reliability and transfer evidence is thin. The main SkillsBench gain from 79.3% to 83.3% is six additional successful executions out of 150, and the cross-executor result is one out of 90. The unseen-task transfer experiment has two targets at six paired repeats each. The pass^k curve is informative but is computed from the same pool of repeats rather than from independently repeated evolution runs. I accept the authors’ reasoning in Appendix B.4 for reporting counts rather than intervals, but that argument makes the case for replication rather than against it, particularly since reliability is presented as a headline finding.

  2. Failure modes in the appendices are understated in the main text. Beyond Appendix F, Figure 4(c) shows the grid-dispatch transfer target losing one repeat and the shock-analysis-supply control falling from 3/6 to 1/6, neither of which Section 6.3 mentions while concluding that each family’s root repaired its transfer target. Appendix D.2 reports that the full run’s power-flow root contains the optimal values of two tasks’ training instances, so those two tasks are not scored on training instances under the libraries carrying it. That an instance-specific value entered a shared root at all, and that a regression gate cannot detect it, seems directly relevant to a paper arguing that roots should hold broadly applicable knowledge. An audit of accepted root texts, not only of their execution outcomes, would strengthen the claim considerably. Relatedly, the abstract states that refused root edits “are recovered in the leaves rather than lost,” while Appendix D.4 reports six of nine localizations failing their leaf gate; the worked example, leaf candidate 1, is identified in Appendix E as coming from early exploratory experiments, where it was refused on a training-set check before reaching ROOTGATE, and was validated on a 64-item subset rather than the full set.

  3. Novelty relative to behavioral scope validation. SkillCommit already decides the permissible scope of a skill from execution evidence, expanding a patch only where replay supports it; SCOPEEVO runs the same logic in the opposite direction. GRASP already gates skill edits under a regression budget. The promote/localize/discard policy is a reasonable variation, but the empirical comparisons are against single-skill optimizers and against restricted variants of SCOPEEVO itself, not against a method that also performs scope selection. I would also note that with two descendants the LOCALIZE exit reduces to “edit the leaf that improved”, and both the QA subtree and the power-flow family have exactly two, leaving the spreadsheet root as the only place where the three-way decision is meaningfully exercised.

Questions for authors
  1. Can the effect of placement be isolated with candidates and budgets held fixed; the same proposed edits, routed by different placement rules, at matched editor-call and execution cost?
  2. How does SCOPEEVO compare directly against SkillCommit-style behavioral scope expansion? What distinguishes the contribution beyond reversing the order of generalization and specialization?
  3. Can every accepted root be audited for instance-specific content, given the memorized optimal values in Appendix D.2?
6 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the careful assessment and constructive suggestions. We address each concern in turn, with the corresponding new results and manuscript revisions.

Response to W1: Independent evolution and transfer support the reliability claim

A controlled full-loop comparison identifies gains beyond ordinary leaf editing. To address the limited execution evidence and missing continuation baseline, we added Table 1 (Appendix D.5): four arms start from the same released CoEvoSkills library, use the same GPT-5.6-sol executor and evaluation, and are scored on 17 SkillsBench tasks (5 targets and 12 other descendants), with 10 fresh runs per task. CoEvoSkills continued uses its native evolution loop; ScopeEvo and Local-only share the editor, initial evidence, and leaf-edit attempts.

LibraryJ17 (%)pass^5 (%)Other descendants (/120)Editor callsValidation runs
CoEvoSkills (released)69.448.9101——
CoEvoSkills continued71.245.610128910
Local-only70.048.91012161
ScopeEvo71.250.310521224
ScopeEvo, two runs (mean ± SD)73.5 ± 3.352.4 ± 3.0106.0 ± 1.4——

The first ScopeEvo run matches CoEvoSkills continued on J17 with 21 rather than 289 editor calls, yields higher pass^5, and improves descendants that no leaf edit targets. We report validation cost separately rather than claiming equal total computation.

Reliability and cross-descendant gains recur across independent evolutions. To separate repeated execution of one library from replication of evolution itself, we independently evolved ScopeEvo twice, using fresh editor samples and gate executions; each evolved baseline above represents one evolution run. ScopeEvo obtains pass^5 of 50.3% and 54.6% (best baseline: 48.9%) and 105/120 and 107/120 passes on other descendants (all baselines: 101/120), with four descendants improving in each run. Across the two evolutions, J17 is 73.5 ± 3.3% and pass^5 is 52.4 ± 3.0%. The reliability and cross-descendant improvements therefore recur across separately evolved libraries, not just repeated scoring of one library.

Both root families transfer to tasks absent from optimization. To strengthen Figure 4(c)'s six-repeat results, Table 8 (Appendix D.5) adds six paired repeats per target: protein-expression-analysis improves from 9/12 to 12/12 and grid-dispatch-operator from 3/12 to 7/12. Neither task appeared in editor inputs or gate runs, so Local-only cannot change its library. The existing cross-executor result (Figure 3(d)) tests the Opus-evolved library unchanged on GPT-5.2; Table 1 complements it with full evolution under GPT-5.6-sol, used for every arm. These are separate forms of evidence: cross-executor reuse, independent evolution, and transfer across two families.

AuthorsOpenReview ↗

Response to W2

Response to W2: Failure modes and selective localization are now explicit

The main text now reports regressions alongside net transfer gains. To address the omissions, we revised §6.3 to state that the grid-dispatch target gains four repeats but loses one (net 1/6 → 4/6), while the shock-analysis-supply control falls from 3/6 to 1/6 (Figure 4(c)). Neither control improves. We also revised the abstract, contributions, and conclusion: refused root edits can be recovered through validated leaf updates, rather than being automatically recovered. The claim is net transfer with an explicit validation limit, not regression-free transfer.

Coverage failures and finite-repeat misses are different limitations. Appendix F identifies two regressing spreadsheet-root consumers that the validation sets cannot diagnose: shock-analysis-supply has no validation instance, and invoice-fraud-detection is at ceiling under both libraries. Reapplying the rule to their training instances, where the losses are observable, gives LOCALIZE; LE-src, LE-agg, the relaxed source-mean proxy, and the pooled-budget proxies with tolerances of five or ten would admit the root (Table 6). By contrast, new ten-run frozen-candidate evaluations (Table 3) reveal a sampling miss: the promoted PDF root lowers court-form-filling from 10/10 to 6/10, which the original 5–7-repeat gate did not flag. Table 4 quantifies the detection risk, and Table 5 shows that the deeper counts lead to LOCALIZE. More repeats can expose sampling misses; missing or saturated validation inputs require better coverage (Theorem 1(c)).

Localization is validated, not presumed successful, and its provenance is documented. Section 6.3 and Appendix D.4 now state that only three of nine QA leaf rewrites pass the leaf gate; six are refused, leaving the parent unchanged. Four DocVQA rewrites lack sufficient validation gain, and two SearchQA rewrites conflict with existing leaf rules. The two rejected QA leaves score 79.1% on SearchQA versus the delivered library's 79.6% (new Table 7); the rejected jpg-ocr-stat rewrite falls from 3/5 to 0/5 under its actual parent (Table 5).

Appendices B.1 and D.4 now trace all nine candidates: 1, 2, 4, and 5 arose in early exploratory runs using a training-set-only root check; 3, 6, and 7 rewrite R12, R10, and R13 from the root-first run; 8 and 9 rewrite R8 and R5 from the leaf-first run. Candidate 1 (Appendix E, Example 1) never reached the later ROOTGATE validation protocol: its leaf was checked on a 64-item subset (46 → 49 correct) and improves test-set J by 0.786 pp. Figure 4(d)'s caption now says “refused root candidates,” without implying identical provenance. The accepted-root text audit is reported in Appendix D.5 (Table 9) and discussed under Q3 below.

Together, these results show that useful edits can be recovered through separately validated localization, while the remaining failures clarify the limits of finite validation and incomplete coverage.

AuthorsOpenReview ↗

Response to W3 + Q1 + Q2

Response to W3 + Q1 + Q2: Descendant-wide validation changes the placement decision

Novelty is in edit placement, not behavioral validation itself. We strengthened Related Work (§2) and added a systematic comparison (Appendix H, Table 10). SkillCommit expands source-supported abstractions through replay and can split unsupported groups; GRASP gates updates using pooled regression budgets and trigger refinement; GSE consolidates skills through relation-aware replay. ScopeEvo instead decides where an edit belongs in a fixed inheritance hierarchy. Because a root edit reaches even non-proposing descendants, ROOTGATE checks each evaluated descendant and either promotes the shared edit, attempts separately gated localization into benefiting leaves, or discards it. The difference is the deployment decision based on descendant-wise evidence, not simply reversing scope expansion.

Fixed candidates isolate placement decisions (Q1–Q2). New Table 6 (Appendix D.5) applies adapted SkillCommit-inspired source rules, GRASP-inspired pooled-budget rules, LE-src, LE-agg, and ROOTGATE to the same 16 QA root candidates and spreadsheet-root candidate, using identical paired executions and editor calls. These are decision-level proxies, not native end-to-end SkillCommit or GRASP implementations; Appendix H (Table 10) compares the original mechanisms.

RuleQA roots admitted (/16)Lowering a domainSpreadsheet candidate
LE-src96Root
LE-agg63Root
Source mean (relaxed)96Root
Source no-loss003 source leaves
Pooled budget, ≤200Discard
Pooled budget, ≤521Root
Pooled budget, ≤1074Root
ROOTGATE10Localize

Eleven QA candidates move the domains in opposite directions. Source-mean admits nine roots, six lowering DocVQA on validation; looser pooled budgets also admit such candidates. ROOTGATE promotes one without observed domain regression and localizes the eleven conflicting candidates; source no-loss admits none. Table 6 isolates placement decisions; Table 7 separately examines the held-out consequences for two candidates.

QA test results show both benefits and limits. Table 7 (Appendix D.5) evaluates leaf-first candidates R8 and R5 in the root versus the SearchQA leaf using the official test harness.

LibrarySearchQA (%)DocVQA (%)Mean (%)
ScopeEvo, delivered79.691.285.4
R8 in root79.489.684.5
R5 in root79.992.085.9
R8 in SearchQA leaf79.191.285.1
R5 in SearchQA leaf79.191.285.2

Sharing R8 lowers DocVQA by 1.6 pp without helping SearchQA; ROOTGATE localizes it, and the leaf gate rejects the rewrite. R5 scores somewhat better when shared on this test, so the gate also forgoes a small test gain.

We also examined LOCALIZE on a larger subtree. Table 5 (Appendix D.5) reanalyzes the PDF-root candidate on ten descendants, using the ten-run evaluation counts from Table 2’s first two rows. The original 5–7-run gate promoted it; deeper evidence reveals court-form-filling declining 10/10→6/10 while three descendants benefit, so the rule would return LOCALIZE. The conflict is visible even on PDF-only consumers (court-form-filling −4; paper-anonymizer +2). Of three proposed leaf rewrites, only paper-anonymizer passes its separate gate (0/7→1/7); jpg-ocr-stat (3/5→0/5) and sales-pivot-analysis (5/5→5/5) are rejected. For tasks reading both roots, leaf validation retains the spreadsheet root and removes the PDF edit. This frozen-candidate check illustrates the placement decision and separate leaf acceptance beyond two descendants.

Fixed leaves isolate root contributions. Table 2 (Appendix D.5) adds ScopeEvo roots to multiple leaf configurations:

LibraryTargets (/50)Other descendants (/120)J17 (%)
CoEvoSkills (released)1710169.4
+ two ScopeEvo roots1510570.6
+ four accepted leaves (ScopeEvo)1610571.2
Local-only1810170.0
Local-only + ScopeEvo roots1210568.8
CoEvoSkills continued2010171.2
CoEvoSkills continued + ScopeEvo roots2910578.8

The four-pass gain on the 12 non-target descendants is the same observation across rows, since their leaves are unchanged. Target effects depend on root–leaf compatibility, including the unfavorable Local-only combination. With continued CoEvoSkills leaves, J17 rises from 71.2% to 78.8% in a component combination, not a single evolution run. Table 1 separately reports native-loop costs; we do not claim equal-total-budget end-to-end superiority. Together, these comparisons isolate placement decisions and root contributions under fixed leaves.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful evaluations and constructive feedback. The common concerns involve fair computational comparisons, novelty relative to existing scope-validation methods, reliability and generalization, and the limits of scope-aware validation.

To address these concerns, we conducted additional experiments covering: (1) controlled comparisons with continued CoEvoSkills and Local-only optimization; (2) independent evolution runs and expanded evaluations; (3) root–leaf contribution analysis and fixed-candidate comparisons with alternative admission rules; (4) deeper gate evaluation, multi-descendant localization, and scalability analysis; and (5) a systematic audit of accepted root texts.

We have also revised the manuscript to clarify the evaluation protocols, editing and validation budgets, and gate thresholds. In particular, we strengthened Related Work (Section 2) to clarify the distinctions from closely related methods and added a systematic method comparison in Appendix H (Table 10). We also added the new experimental results, expanded failure-case analyses, and made the limitations and provenance of localized edits more explicit.

The additional evidence supports our central claim: ScopeEvo's contribution is not simply additional skill optimization, but deciding where edits should reside based on their effects across inheriting descendants. The results demonstrate reliability gains, transfer through shared roots, and selective localization when sharing is inappropriate. We also explicitly characterize the computational costs and finite-validation limitations of this approach. We hope these revisions address the reviewers' concerns and clarify the contribution of ScopeEvo.

AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers and Area Chair for their time and constructive feedback. We have comprehensively addressed the reviewers' concerns through additional experiments, analyses, and manuscript revisions.

To address concerns that the gains merely reflect additional optimization, we added continued CoEvoSkills and Local-only baselines and fixed-leaf comparisons. Shared roots improve non-target descendants (101/120 → 105/120 passes), demonstrating gains beyond targeted leaf editing. Two independent evolutions reproduce reliability improvements (pass^5: 50.3% and 54.6%, versus 48.9% for the best baseline), while transfer across two skill families further supports generalization. To address novelty concerns, fixed-candidate comparisons distinguish descendant-wise edit placement from alternative admission rules, and a ten-descendant analysis demonstrates how conflicting root edits can be localized. Root audits and removal ablations confirm that the observed transfer does not depend on identified training-instance optima. We also clarified computational costs, gate failures, localization outcomes, and validation and scalability limits.

Reviewer zsKn confirmed that our follow-up clarifications answered their questions and raised their recommendation. We appreciate Reviewer dFwh's acknowledgment that our experiments resolved their questions about limited knowledge sharing and leaf-level conflicts. Although we welcomed further discussion, the final assessment arrived a few hours before the discussion deadline, leaving limited opportunity for further clarification. Nevertheless, the remaining concerns are addressed by evidence already presented: controlled comparisons and independent evolutions support benefits beyond targeted leaf optimization, while the manuscript explicitly reports computational costs, non-additive root–leaf effects, and scalability limits. Our paper does not claim lower end-to-end computational cost, uniformly additive gains, or demonstrated scalability to arbitrarily large hierarchies. Our central contribution is descendant-wise edit placement, with its limitations explicitly characterized. We sincerely thank all reviewers for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.