- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes GraphSkillAA, a framework for optimizing external skills while keeping the underlying language models frozen. Skills are represented as nodes with applicability, execution, and exclusion fields, connected by typed relations. A teacher model compares successful and failed executions to propose targeted edits. These edits undergo local regression testing and whole-pool validation before being committed or rolled back. The paper reports improvements on SearchQA, LiveMath, and DocVQA, together with component ablations and an additional ALFWorld experiment.
Critical strengths
- The paper addresses a relevant problem. Updating reusable skills without unnecessarily damaging previously successful behavior is significant.
- The idea of two-stage validaiton seems useful.
Critical weaknesses
- The presentation makes the method unnecessarily difficult to understand. The manuscript repeatedly emphasizes “localize, edit, validate, and rollback,” but spends less effort explaining the operational details that distinguish the method. The interaction between attribution routing, joint synthesis, atomic patch groups, and the two validation gates requires substantial cross-referencing between the main text and the appendix.
- The methodological novelty appears limited and is not sufficiently isolated. The main contribution therefore appears to be the integration of field-level attribution, graph-scoped retesting, and rollback into an existing skill optimization pipeline. This may be a useful engineering contribution, but the paper does not clearly establish what fundamentally new capability requires the proposed graph formulation.
- The experimental scope and controls are insufficient for the breadth of the claims. The reported final graphs contain only 20–37 nodes. A comparison under matched optimization budgets, together with actual token usage and runtime, is needed to determine whether the gains justify this additional complexity.
Questions for authors
See the weakness. Clarify the novelty and justify the experimental settings.
7 comments
Thanks for the reply. My concerns about the writing, novelty and evaluation remain, and thus my rating stays unchanged.
Second response to Reviewer Uj6W
Dear Reviewer, thank you for taking another look at our paper.
We have provided a new PDF and substantially revised the paper in response to your comments. We rewrote the method and added comparisons to examine what the graph contributes. Since your latest comment does not identify which changes you found insufficient, we explain them below.
W1. Making the method easier to follow
Sections 3.1-3.4 and Tables 1-2 now explain the steps of one update in order. The teacher first identifies a node field or relation to change. Joint synthesis then groups changes that need to work together. For example, a new rule and its connecting edge form one repair group: both are kept if the group passes, and both are removed if it fails.
Local Gate tests each group on a copy of the graph from the start of the round. It compares the results with those obtained before editing, using source failures, previous users of the edited objects, related update examples, and examples with missing or invalid usage records. The accepted groups are then merged, and Big Gate tests the resulting graph before it is saved for the next round.
Table 2 explains which diagnosis leads to which edit, including when to return NO_PATCH. Section 3.4 gives the acceptance rules: Local Gate requires more fixes than broken cases plus unresolved source failures; Big Gate requires more fixes than regressions. A positive net gain can include some regressions; it does not mean every previously correct answer stays correct. Appendices C.3-C.8 and D.1 provide the algorithm and examples.
W2. What the graph adds, and how we test it
GraphSkillAA builds on SkillOpt, as we state in Section 3.1. What changes is how rules and their relationships can be edited and checked. For example, we can change a prerequisite relation while leaving both rules intact, or change when a skill applies without rewriting its steps. The IDs of the edited objects identify previous executions to retest. Changes that depend on one another are also kept or undone together. Other structured representations could encode similar information; the experiments evaluate our particular design.
Section 4.4, Table 5, and Appendices F.6-F.7 examine these choices separately. Results below use GPT-5.6-sol, with datasets ordered as SearchQA/LiveMath/DocVQA; accuracy differences are percentage points.
Do relations help when node content is unchanged? Removing edges while keeping every node and field lowers accuracy by 4.0/4.6/1.2. Keeping edges but hiding their explanations lowers it by 3.0/3.2/2.2. Does choosing the edit target matter? Appendix F.6 replaces the diagnosed target with a random object of the same compatible type. The models, initial graph, evidence-construction rules, editing templates, patch budget, and Gate rules stay the same. Using the diagnosed targets improves test accuracy by 3.2/6.0/1.5. Net repairs on the update sets are 128 versus 100, 216 versus 110, and 72 versus 40. This directly tests the choice of edit location. It does not establish equal total token use. Does local checking help beyond Big Gate? Skipping Local Gate while retaining Big Gate lowers accuracy by 3.5/17.7/1.7. Appendix C.7 also examines errors introduced outside the locally tested examples. Those statistics come from archived runs with subset evaluation, not new full-pool runs. What does Top-6 retrieval miss? Appendix F.7 reports node coverage of 84.6/69.8/88.2% and edge coverage of 72.3/56.4/78.6%, measured against Full's usage records. This helps explain the retrieval results. Full's usage is not ground truth, and Top-6 shows fewer rules. These experiments test more than whether the complete system beats a baseline. They examine the roles of relations, edit location, and local checking separately.
W3. Graph size and optimization cost
The 20-37-node graphs fit in context and allow us to test these changes while retaining all rules. They do not demonstrate scalability to large skill libraries. Appendix H.2 discusses that limitation and the additional problems posed by longer tasks.
We agree that reporting extra tokens per accuracy point does not answer whether SkillOpt could reach the same performance with the same budget. We also need to correct our earlier reply: the 25.09/53.18M figures and corresponding entries are budget estimates, not measured API totals. Having one student call per question does not imply similar deployment costs, since prompt and output lengths differ. Runtime also needs a controlled comparison. We have not yet provided an equal-total-budget experiment.
We understand that this budget comparison is still missing. For the writing and novelty concerns, could you point to the explanation or comparison in the revised PDF that you still find inadequate? We would appreciate your assessment of the specific changes and results above.
Response to Reviewer Uj6W
W1
Thank you for pointing out the unclear method description. We sincerely apologize for the difficulty this caused. Space limits make the main text a brief outline. Appendix D combines construction and execution examples with prompts for attribution, repair and audit, so readers can start there to understand the operations. In the workflow, attribution identifies edit targets, joint synthesis forms repair groups, Local Gate screens those groups, and Big Gate checks the merged graph. Sections 3.1–3.4 outline the framework, graph representation, execution evidence, attribution and updates. Table 1 summarizes stage inputs and outputs, Table 2 maps failure causes to edits, and Appendix C provides the complete algorithm, validation criteria and rollback rules.
W2
GraphSkillAA records skill rules and how they work together. Related rule and link edits are checked as a group and reverted together if rejected.
Figure 5 SearchQA example (repair illustrated). Asked the Nautilus's vessel type, suppose the model answers “ship” instead of “submarine”.
The new rule and edge form one repair group. Source failures and cases selected by the retesting rules are rerun. If Local Gate ultimately rejects the group, both additions are removed. Other accepted edits proceed to merging and Big Gate.
Table 5, GPT-5.6-sol accuracy (%):
The graph connects relation-specific edits, affected-example tracing, and joint validation and restoration of dependent changes.
W3
GraphSkillAA is training-free: model parameters stay frozen. We report actual API tokens for one complete GPT-5.6-sol optimization run per method. GraphSkillAA counts student execution, teacher attribution, joint synthesis, refinement and both Gates, excluding upstream SkillOpt optimization. Accuracy averages three tests in Table 3. SkillOpt denotes Flat skill using its released best skills. SearchQA/LiveMath/DocVQA have 200/117/200 test cases.
Token Ratio is relative to SkillOpt. Extra Tokens / pp divides the difference between the two methods' optimization tokens by their accuracy difference (M/pp). Both use model APIs, so elapsed time varies with server load, network conditions and concurrency. Tokens cover each full optimization run.
Measured deployment costs are similar. Most deployment steps are shared: both use the same student, inference settings and test inputs, fixed skills and one student inference round per question. Teacher attribution, patch generation and both Gates run only during optimization, adding no deployment calls.
Starting from SkillOpt's released best skills, GraphSkillAA improves accuracy by 9.0/24.5/3.5 pp, with optimization token ratios of 2.12/3.56/3.87×. The differences between the two workflows' token use, divided by their accuracy differences, are 3.121/0.052/0.926M per point. LiveMath has the largest accuracy gain and lowest ratio. Finer-grained attribution and targeted repair thus raise attainable performance without model updates. We consider this optimization cost worthwhile for performance-first applications.