- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ScopeEvo, studying how the skill should be modified when a new line of experience is summarized from real tasks. It proposes that the skill should be structured as a tree; its root is responsible for the general skills and knowledge, while its leaves are responsible for specific tasks. The two levels of skills combine to finish the task. The authors points out that locating the skill modifications to either the root or a certain leaf is the challenging part. The paper sets certain rules to coordinate the modifications to obtain a general improvement. For the modifications that bring overall improvements, they are put into the root; when there is a trade-off between several tasks, that knowledge is then put into a leaf. If there is no gain at all, then the knowledge will be get rid off.
Critical strengths
- The paper identifies a very interesting and applicable problem in designing skills, that is how to keep the best balance between the shared skill and the specialized skill.
- The proposed method is easy to implement, and is applicable to the existing skill optimizers. The actual structure is also indeed simpler than the competing method.
- Theoretical guarantees on robustness are provided to support the method design, and the experimental results are straightforward to validate the method.
Critical weaknesses
- There is a lack of alignment of skill editing resources in comparing the methods. Thus the contribution of the proposed method is hard to validate. Skill-bench is an important benchmark, yet the method is built on the optimized skills of CoEvoSkills, there is no fair and direct comparison with the baseline.
- The complete cost of building the skill is not presented. The end-to-end token expenditure comparison between the proposed method and baselines should be provided to validate that the proposed method is more efficient. The existing results only indicate that the cost will be smaller, but are not concrete.
- The improvements over the strong baseline is not evident enough. The error bar on multiple experiments should be provided to validate that the proposed method is really effective. The current experiments only show subtle improvements on most tasks.
- The proposed root-leaf mechanism is not showing its combined effectiveness. Compared to the two ablated versions, the current structure is gaining from the root only or the leaf only. The improvements are only concentrated on one specific domain.
Questions for authors
- How is the proposed theory guiding the proposed method? From the presentation, the theory is more like a make-up on the already proposed method. Is the theory validated through experiments?
- When there are vast application scenarios, how does the proposed method scale in such situations? How should you decide the application coverage of the root? And what should the method change when there are tasks that share very little knowledge, and forcing the use of the root harms performance?
- How to guarantee the scalability of the method when there are a lot of tasks and the skill has to wriite a lot of rules? The conflict from different leaves could oftenly happen.
6 comments
Response to W4 Q1 Q2 and Q3
Response to W4: Root and leaf edits provide complementary, not uniformly additive, benefits
Figure 5 shows complementary roles for both scopes within one loop: shared-root transfer and task-specific leaf gains. New Table 2 (Appendix D.5) holds the leaves fixed when adding ScopeEvo roots. The four-pass non-target gain is the same observation across leaf configurations, since these tasks retain the same leaves; target effects depend on compatibility and can be negative. With CoEvoSkills continued's leaves, J17 rises from 71.2% to 78.8% in a component combination, not a single evolution run. The consistent root contribution is cross-descendant reach, while leaf effects remain task-specific.
Root and leaf edits contribute at different stages. In the GPT-5.6-sol run, two promoted roots add four passes among non-target descendants, and the accepted leaf edits add one net target pass (Table 2). In the original Opus 4.6 run, two promoted roots add three target passes without an observed loss, followed by three more from an accepted leaf (Appendix D.5). The spreadsheet root's 11 descendants gain seven passes in the ten-run audit (Table 3). Both scopes contribute, but their effects are not always additive.
Response to Q1: The theory specifies the gate's scope and safeguards
Each analysis result motivates a concrete design choice. To explain how §5 and Appendix C guide the method, Lemma 1 and Corollary 1 establish that a root edit can affect every inheriting descendant whereas a private leaf can affect only its own task; hence ROOTGATE checks the whole subtree and leaf gates check only their task. Theorem 1(c) establishes the validation-coverage limit; consequently, unusable scores yield UNDECIDED, and a root with all descendants at validation ceiling yields NOHEADROOM. Proposition 4 shows the false-refusal risk of zero-margin regression checks on stochastic neutral descendants, motivating the two-pass tolerance and adaptive 5→7-repeat protocol (Appendix B.2). The theory determines which tasks to check and which failure modes the gate must handle.
The added experiments probe the predicted limits. In Table 3, ten fresh runs reveal a PDF-root regression on court-form-filling (10/10 → 6/10) missed by the shorter gate. Table 4 evaluates the actual adaptive rule under the run model, complementing Theorem 1's fixed-budget, zero-margin analysis. Theorem 1(c) concerns absent validation inputs or inputs that pass with probability one under both libraries; Appendix F illustrates this coverage limit. These results distinguish sampling limits from coverage limits.
Response to Q2 + Q3: Placement in a fixed hierarchy and conflict handling
Validation scales linearly, but the algorithm does not force a shared root. The existing hierarchy determines which tasks inherit each root; ScopeEvo selects where to place an edit, not which tasks belong to the root. A root event uses 5–7 paired repeats per descendant (4–6 for shock-analysis-supply); LOCALIZE reuses the root outcomes and separately checks benefiting leaves. Table 4 models 2–13 descendants. For a 0.8→0.3 or 0.9→0.5 drop in one descendant, with all others unchanged at p=0.9, promotion has probability 0.02–0.06. If another descendant also improves, the relevant upper bound is the corresponding regression-miss probability. In its beneficial-edit case with neutral descendants at p=0.9, refusal rises from 0.133 to 0.665, and redirection to LOCALIZE from 0.079 to 0.622. New Table 5 applies the rule post hoc to a ten-descendant PDF subtree, where one regresses and three benefit; the edit is redirected to three leaf rewrites, only one of which passes its leaf gate. Placement therefore follows observed descendant-level compatibility within the given hierarchy.
When common knowledge is scarce, leaves remain the effective scope. On QA, just one of 16 candidate root edits is promoted by the replayed gate and the delivered leaf-first library accepts no root, matching Local-only (Appendix D.1). New Table 7 shows that forcing candidate R8 into the shared root would reduce DocVQA by 1.6 pp on test. Not promoting a root is an intended outcome, not a forced loss of task-specific knowledge.
Conflicting rules are isolated and tested before release. Private leaves are delivered only to their tasks (Lemma 1); on QA, more specific leaf guidance takes precedence, and a leaf is validated under its actual inherited root (Appendix B.1). The leaf gate rejects observed conflicts: leaf candidates 8 and 9 conflict with existing rules and are rejected (Appendix D.4). Root rules have explicit applicability conditions, edit count/text growth is bounded, and each accepted library becomes the next validation parent (§4). These safeguards contain observed conflicts, though deeper hierarchies and larger libraries remain outside the present evaluation.
Dear Reviewer dFwh, thank you again for your thoughtful review and constructive suggestions. We have added controlled comparisons and independent evolution runs, reported editing and validation costs separately, and clarified the complementary roles of root and leaf edits and the connection between the theoretical analysis and gate design. Your feedback is very important to us. We would be grateful for your assessment of whether the additional evidence and explanations address your concerns, and would be happy to discuss any points that need further clarification.
Response to W1 W2 and W3
We thank the reviewer for the constructive suggestions. We address the fair-comparison, cost, replication, mechanism, and scalability questions below.
Response to W1: Controlled comparisons reveal what shared placement contributes
The added baseline comparison tests whether further editing alone explains the gain. Table 1 (Appendix D.5) starts native CoEvoSkills continuation, Local-only, and ScopeEvo from the same released library, evaluates all arms with GPT-5.6-sol on the same 17 tasks, and uses ten fresh runs per task. ScopeEvo and Local-only also share the editor, starting evidence, and leaf attempts. ScopeEvo's first evolution matches continued CoEvoSkills at 71.2% J17 while using 21 rather than 289 editor calls; unlike target-only leaf edits, its shared roots raise non-target passes from 101/120 to 105/120.
We separately control the placement decision. Table 6 (Appendix D.5) replays alternative admission rules on the same candidates and paired executions, holding proposal and decision-stage costs fixed. Total native-loop budgets differ, as reported under W2 below; we make no equal-compute end-to-end superiority claim.
Response to W2: Editing is cheaper, while subtree validation is more expensive
The cost breakdown now includes editor and acceptance-validation tokens. Table 1 and Appendix D.5 report the overall editing and validation costs. The input/output token breakdown is summarized below (millions of tokens):
These totals cover editing and admission validation; the 84 initial evidence runs shared by ScopeEvo and Local-only, and final evaluation executions, are disclosed separately and excluded from the admission-cost totals. CoEvoSkills' unrecorded hidden reasoning tokens also make its figure a lower bound. The resulting comparison is explicit and does not equate lower editing cost with lower total token cost.
The additional validation covers more affected tasks rather than re-editing the same targets. Continued CoEvoSkills spends at least 17.1M editor tokens and 289 calls, but validates each of five targets about twice. ScopeEvo uses about 1.5M editor tokens and 21 calls, like Local-only's 1.2M and 21, but spends 180 of its 224 validation executions on root checks and covers 17 rather than five tasks. Per covered task, ScopeEvo and Local-only use 13.2 versus 12.2 validation executions and 19.3M versus 21.9M input tokens; about 96% of this input is cached. The total ScopeEvo cost (331.7M tokens, roughly 95% cached executor input) is the price of checking every consumer of a shared update. Figure 4(b) separately shows that on QA it matches SkillOpt with 25% of its editor calls and 59% of its editor tokens. ScopeEvo economizes on editing but pays a validation cost approximately linear in descendants and paired repeats (Lemma 1, Corollary 1).
Response to W3: Reliability gains recur across independent evolutions
We added independent evolution runs, not just more scoring repeats. To address the small main-result margins and missing error bars, Table 1 evaluates each arm over 170 fresh executions and reports two separately evolved ScopeEvo libraries, with fresh editor samples and gate executions. Their J17 values are 71.2% and 75.9% (mean ± SD: 73.5 ± 3.3%), and their pass^5 values are 50.3% and 54.6% (52.4 ± 3.0%); the best one-run baseline gives 71.2% J17 and 48.9% pass^5. Both independent evolutions reproduce the reliability advantage in pass^5; the mean-performance evidence is descriptive with only two evolution runs.
The improvement extends beyond one descendant. Both evolutions improve four of the 12 non-target descendants, yielding 105/120 and 107/120 passes versus 101/120 for the baselines. Since leaf editing is confined to the five target tasks, non-target gains come through shared roots; Table 2 isolates this contribution for the first evolution. This repeats the cross-descendant effect, without establishing gains in every skill family.