acceptodds
Under review as a conference paper at ICLR 2027

GraphSkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback

12 views

Abstract

External skills provide domain knowledge and procedures without updating model parameters, but flat collections obscure skill applicability, dependencies, and composition. Graphs organize skills into addressable nodes and explicit relations, supporting selection and composition. Yet existing skill-graph methods use this structure mainly for retrieval, rather than to localize updates, scope retesting, or precisely roll back rejected changes. We introduce GRAPHSKILLAA (GraphSkill Abductive Attribution), which uses one addressable graph for skill selection, execution, failure attribution, targeted updating, validation, and rollback. Nodes separate applicability, execution, and exclusion conditions; typed edges encode prerequisite and enhancement relations. The frozen student records used nodes and edges, while the teacher contrasts related successes and failures to route each supported repair to the smallest relevant field or relation; execution lapses or insufficient evidence leave the graph unchanged. A Local Gate retests affected examples, while a Big Gate evaluates the merged graph on the complete update pool; rejected changes are rolled back. With GPT-5.6-sol, GRAPHSKILLAA reaches 81.5%, 66.7%, and 91.2% on SearchQA, LiveMath, and DocVQA, respectively, and attains the highest observed mean in every main setting. These results show that object-level attribution and graph-scoped validation make a skill graph a locally optimizable, testable, and reversible external state.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, Skill graphs, External skill optimization, Failure attribution, Targeted validation, Rollback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Uj6WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes GraphSkillAA, a framework for optimizing external skills while keeping the underlying language models frozen. Skills are represented as nodes with applicability, execution, and exclusion fields, connected by typed relations. A teacher model compares successful and failed executions to propose targeted edits. These edits undergo local regression testing and whole-pool validation before being committed or rolled back. The paper reports improvements on SearchQA, LiveMath, and DocVQA, together with component ablations and an additional ALFWorld experiment.

Critical strengths
  1. The paper addresses a relevant problem. Updating reusable skills without unnecessarily damaging previously successful behavior is significant.
  2. The idea of two-stage validaiton seems useful.
Critical weaknesses
  1. The presentation makes the method unnecessarily difficult to understand. The manuscript repeatedly emphasizes “localize, edit, validate, and rollback,” but spends less effort explaining the operational details that distinguish the method. The interaction between attribution routing, joint synthesis, atomic patch groups, and the two validation gates requires substantial cross-referencing between the main text and the appendix.
  2. The methodological novelty appears limited and is not sufficiently isolated. The main contribution therefore appears to be the integration of field-level attribution, graph-scoped retesting, and rollback into an existing skill optimization pipeline. This may be a useful engineering contribution, but the paper does not clearly establish what fundamentally new capability requires the proposed graph formulation.
  3. The experimental scope and controls are insufficient for the breadth of the claims. The reported final graphs contain only 20–37 nodes. A comparison under matched optimization budgets, together with actual token usage and runtime, is needed to determine whether the gains justify this additional complexity.
Questions for authors

See the weakness. Clarify the novelty and justify the experimental settings.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Uj6W

W1

Thank you for pointing out the unclear method description. We sincerely apologize for the difficulty this caused. Space limits make the main text a brief outline. Appendix D combines construction and execution examples with prompts for attribution, repair and audit, so readers can start there to understand the operations. In the workflow, attribution identifies edit targets, joint synthesis forms repair groups, Local Gate screens those groups, and Big Gate checks the merged graph. Sections 3.1–3.4 outline the framework, graph representation, execution evidence, attribution and updates. Table 1 summarizes stage inputs and outputs, Table 2 maps failure causes to edits, and Appendix C provides the complete algorithm, validation criteria and rollback rules.

W2

GraphSkillAA records skill rules and how they work together. Related rule and link edits are checked as a group and reverted together if rejected.

Figure 5 SearchQA example (repair illustrated). Asked the Nautilus's vessel type, suppose the model answers “ship” instead of “submarine”.

  • Existing rule: Q011 identifies the requested answer type: a vessel category.
  • Added rule: If specificity checking is missing, add Q029: return the specific category supported by evidence, not a broader one. Here, use “submarine”, not “ship”.
  • Link: E007 connects Q011 to Q029. Identifying the answer type helps check whether the answer is too broad.

The new rule and edge form one repair group. Source failures and cases selected by the retesting rules are rerun. If Local Gate ultimately rejects the group, both additions are removed. Other accepted edits proceed to merging and Big Gate.

Table 5, GPT-5.6-sol accuracy (%):

SettingSearchQALiveMathDocVQA
GraphSkillAA (Full)81.566.791.2
Nodes only77.562.190.0
No edge-rationale78.563.589.0
No structured attribution73.238.586.7
No Local Gate78.049.089.5
  1. Relations help execution. At evaluation, Nodes only keeps all nodes/fields but removes edges, losing 4.0/4.6/1.2 pp. Hiding only edge explanations loses 3.0/3.2/2.2 pp. Relations and their explanations thus add value to identical structured node content.
  2. Object-level repair and validation help optimization. Within the same graph framework and downstream process, replacing structured attribution with generic failure summaries loses 8.3/28.2/4.5 pp. Skipping Local Gate and sending merged edits directly to Big Gate loses 3.5/17.7/1.7 pp. These controls support object-level diagnosis and group-level local validation, respectively.

The graph connects relation-specific edits, affected-example tracing, and joint validation and restoration of dependent changes.

W3

GraphSkillAA is training-free: model parameters stay frozen. We report actual API tokens for one complete GPT-5.6-sol optimization run per method. GraphSkillAA counts student execution, teacher attribution, joint synthesis, refinement and both Gates, excluding upstream SkillOpt optimization. Accuracy averages three tests in Table 3. SkillOpt denotes Flat skill using its released best skills. SearchQA/LiveMath/DocVQA have 200/117/200 test cases.

DatasetMethodActual Tokens (M)Token RatioAccuracy (%)Gain (pp)Extra Tokens / pp (M)
SearchQASkillOpt25.091.00×72.5——
GraphSkillAA53.182.12×81.5+9.03.121
LiveMathSkillOpt0.501.00×42.2——
GraphSkillAA1.783.56×66.7+24.50.052
DocVQASkillOpt1.131.00×87.7——
GraphSkillAA4.373.87×91.2+3.50.926

Token Ratio is relative to SkillOpt. Extra Tokens / pp divides the difference between the two methods' optimization tokens by their accuracy difference (M/pp). Both use model APIs, so elapsed time varies with server load, network conditions and concurrency. Tokens cover each full optimization run.

Measured deployment costs are similar. Most deployment steps are shared: both use the same student, inference settings and test inputs, fixed skills and one student inference round per question. Teacher attribution, patch generation and both Gates run only during optimization, adding no deployment calls.

Starting from SkillOpt's released best skills, GraphSkillAA improves accuracy by 9.0/24.5/3.5 pp, with optimization token ratios of 2.12/3.56/3.87×. The differences between the two workflows' token use, divided by their accuracy differences, are 3.121/0.052/0.926M per point. LiveMath has the largest accuracy gain and lowest ratio. Finer-grained attribution and targeted repair thus raise attainable performance without model updates. We consider this optimization cost worthwhile for performance-first applications.

Thanks for the reply. My concerns about the writing, novelty and evaluation remain, and thus my rating stays unchanged.

AuthorsOpenReview ↗

Second response to Reviewer Uj6W

Dear Reviewer, thank you for taking another look at our paper.

We have provided a new PDF and substantially revised the paper in response to your comments. We rewrote the method and added comparisons to examine what the graph contributes. Since your latest comment does not identify which changes you found insufficient, we explain them below.

W1. Making the method easier to follow

Sections 3.1-3.4 and Tables 1-2 now explain the steps of one update in order. The teacher first identifies a node field or relation to change. Joint synthesis then groups changes that need to work together. For example, a new rule and its connecting edge form one repair group: both are kept if the group passes, and both are removed if it fails.

Local Gate tests each group on a copy of the graph from the start of the round. It compares the results with those obtained before editing, using source failures, previous users of the edited objects, related update examples, and examples with missing or invalid usage records. The accepted groups are then merged, and Big Gate tests the resulting graph before it is saved for the next round.

Table 2 explains which diagnosis leads to which edit, including when to return NO_PATCH. Section 3.4 gives the acceptance rules: Local Gate requires more fixes than broken cases plus unresolved source failures; Big Gate requires more fixes than regressions. A positive net gain can include some regressions; it does not mean every previously correct answer stays correct. Appendices C.3-C.8 and D.1 provide the algorithm and examples.

W2. What the graph adds, and how we test it

GraphSkillAA builds on SkillOpt, as we state in Section 3.1. What changes is how rules and their relationships can be edited and checked. For example, we can change a prerequisite relation while leaving both rules intact, or change when a skill applies without rewriting its steps. The IDs of the edited objects identify previous executions to retest. Changes that depend on one another are also kept or undone together. Other structured representations could encode similar information; the experiments evaluate our particular design.

Section 4.4, Table 5, and Appendices F.6-F.7 examine these choices separately. Results below use GPT-5.6-sol, with datasets ordered as SearchQA/LiveMath/DocVQA; accuracy differences are percentage points.

Do relations help when node content is unchanged? Removing edges while keeping every node and field lowers accuracy by 4.0/4.6/1.2. Keeping edges but hiding their explanations lowers it by 3.0/3.2/2.2. Does choosing the edit target matter? Appendix F.6 replaces the diagnosed target with a random object of the same compatible type. The models, initial graph, evidence-construction rules, editing templates, patch budget, and Gate rules stay the same. Using the diagnosed targets improves test accuracy by 3.2/6.0/1.5. Net repairs on the update sets are 128 versus 100, 216 versus 110, and 72 versus 40. This directly tests the choice of edit location. It does not establish equal total token use. Does local checking help beyond Big Gate? Skipping Local Gate while retaining Big Gate lowers accuracy by 3.5/17.7/1.7. Appendix C.7 also examines errors introduced outside the locally tested examples. Those statistics come from archived runs with subset evaluation, not new full-pool runs. What does Top-6 retrieval miss? Appendix F.7 reports node coverage of 84.6/69.8/88.2% and edge coverage of 72.3/56.4/78.6%, measured against Full's usage records. This helps explain the retrieval results. Full's usage is not ground truth, and Top-6 shows fewer rules. These experiments test more than whether the complete system beats a baseline. They examine the roles of relations, edit location, and local checking separately.

W3. Graph size and optimization cost

The 20-37-node graphs fit in context and allow us to test these changes while retaining all rules. They do not demonstrate scalability to large skill libraries. Appendix H.2 discusses that limitation and the additional problems posed by longer tasks.

We agree that reporting extra tokens per accuracy point does not answer whether SkillOpt could reach the same performance with the same budget. We also need to correct our earlier reply: the 25.09/53.18M figures and corresponding entries are budget estimates, not measured API totals. Having one student call per question does not imply similar deployment costs, since prompt and output lengths differ. Runtime also needs a controlled comparison. We have not yet provided an equal-total-budget experiment.

We understand that this budget comparison is still missing. For the writing and novelty concerns, could you point to the explanation or comparison in the revised PDF that you still find inadequate? We would appreciate your assessment of the specific changes and results above.

Reviewer_vrwNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces GRAPHSKILLAA, which uses one addressable graph for skill selection, execution, failure attribution, targeted updating, validation, and rollback. Nodes separate applicability, execution, and exclusion conditions; typed edges encode prerequisite and enhancement relations. The frozen student records used nodes and edges, while the teacher contrasts related successes and failures to route each supported repair to the smallest relevant field or relation; execution lapses or insufficient evidence leave the graph unchanged. A Local Gate retests affected examples, while a Big Gate evaluates the merged graph on the complete update pool; rejected changes are rolled back. With GPT-5.6-sol, GRAPHSKILLAA reaches 81.5%, 66.7%, and 91.2% on SearchQA, LiveMath, and DocVQA, respectively, and attains the highest observed mean in every main setting.

Critical strengths
  • The research problem is clearly defined, and the motivation behind the work is well explained.
  • The writing is generally clear and concise, making the paper easy to follow.
Critical weaknesses
  1. The teacher works from the trace and usage records that the model generates itself. Even when the two agree, that only shows the records are internally consistent. It does not show that the model actually failed because of the identified node. An edit that appears to fix the error may instead work by adding a reminder, prompting extra verification, or changing the overall prompt. The authors should provide stronger evidence of causality, for example through controlled ablations or counterfactual interventions on the identified node.

  2. Limiting the teacher to modifying one field at a time is an editing constraint. It does not show that no smaller, equally effective modification exists. The paper should either avoid presenting the edits as minimal or provide an argument or experiment that supports minimality.

  3. Both Gates accept updates that introduce some new errors, as long as the gains from fixes outweigh them. For example, the Big Gate would accept an update that fixes 10 cases and breaks 9. What the mechanism actually delivers is a net accuracy gain on the update set. It does not keep all previously correct cases correct, and it does not prevent regression on unseen samples. The authors should state this clearly and report how many cases each update breaks (both counts and examples), together with performance on held-out data.

  4. The full method has access to the entire active graph, while the retrieval-based baseline may see only a subset of skills. If the retrieval baseline misses key nodes or dependency edges, its lower performance may come from missing information rather than from weaknesses of the method itself. Please control for the information each variant can access, or analyze retrieval coverage of the relevant nodes and edges.

  5. The ALFWorld results are not enough to show that the NO_PATCH decisions are correct. To show that leaving the graph unchanged is the better choice, the authors need to compare the actual outcomes of NO_PATCH against efficiency-oriented skill modifications and against other execution-level improvements.

Questions for authors

Please refer to the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer vrwN

W1

Trace/usage checks consistency. GPT-5.6-sol accuracy (%, Tables 3/5):

SettingSearchQALiveMathDocVQA
G075.044.288.0
G0 + Trace/Usage76.251.388.2
GraphSkillAA (Full)81.566.791.2
No structured attribution73.238.586.7
No when-to-use & avoid70.542.784.3
  1. Full gains 5.3/15.4/3.0 pp over G0 + Trace/Usage, beyond recording alone.
  2. Generic summaries lose 8.3/28.2/4.5 pp for GPT-5.6-sol. Table 5 uses each student's self-teacher Full: GPT-5.6-sol 81.5/66.7/91.2%, GPT-5.4-mini 75.7/23.9/86.3% (SearchQA/LiveMath/DocVQA). All six controls decline, averaging 8.3 pp.
  3. Hiding when_to_use/avoid in the frozen graph loses 11.0/24.0/6.9 pp. Local Gate tests patches by execution (C.7).
  4. Shuffled Attribution randomizes type-compatible targets before synthesis. Edit types, graph constraints, GPT-5.6-sol student/teacher, evidence construction, prompts, patch budget and Gates stay fixed (F.6, Table 13).

Both start from G0 + Trace/Usage: three epochs, seeds 42/43/44. Fixed/Broken sum update-pool wrong→right/right→wrong changes from initial to final committed graphs across seeds. Net = Fixed - Broken. Test accuracy is mean ± half-range.

DatasetAttributionFixedBrokenNetTest (%)
SearchQAOriginal240112+12881.5±1.3
SearchQAShuffled200100+10078.3±1.3
LiveMathOriginal350134+21666.7±2.1
LiveMathShuffled260150+11060.7±2.1
DocVQAOriginal10533+7291.2±1.0
DocVQAShuffled7535+4089.7±1.3

With the same optimization settings, original targets outperform random type-compatible targets in fixes, net gains and test accuracy. SearchQA has more regressions (112 vs. 100), but a higher net gain.

W2

We ask the teacher to make changes as small as possible while achieving an effective repair. A complete repair may require several coordinated operations, so this objective does not mean editing only one field or performing only one operation. We agree that “minimal edits” overstates the claim and will replace it with “aiming to minimize edits.” This is an instruction to the teacher rather than a guarantee of global minimality. The degree to which changes can be reduced depends on the teacher model's diagnostic and generation capabilities.

W3

The Gates require net gains, not zero regressions on known or unseen cases. Local rollback rejects failing repair groups separately, while Big Gate checks their combination. The existing subset-Gate audit gives:

DatasetOutside cases (mean per round)Local Gate coverageBig Gate outside regression rate
SearchQA352.541.25%3.40%
LiveMath253.527.78%9.86%
DocVQA5626.33%3.02%

In DocVQA round 1, Local Gate rejected one group that fixed 1 case, broke 1 and left 2 source failures unresolved. Three other groups were committed.

Table 5 tests the value of this screening on held-out data (GPT-5.6-sol, accuracy %):

DatasetGraphSkillAA(Full)No Local GateGain (pp)
SearchQA81.578.0+3.5
LiveMath66.749.0+17.7
DocVQA91.289.5+1.7

These results support checking repair groups before commitment.

W4

GPT-5.6-sol. Full uses the whole graph. Top-6 selects six nodes by similarity and retains their internal edges. Coverage is mean per-question recall of Full-logged nodes/edges (F.7, Table 14). Zero denominators are excluded per metric only, not from other metrics.

DatasetRetrieved NodesNode CoverageEdge CoverageS--Q AccuracyGraphSkillAA (Full) Accuracy
SearchQA684.6%72.3%73.5%81.5%
LiveMath669.8%56.4%42.5%66.7%
DocVQA688.2%78.6%86.0%91.2%

Nodes only keeps all nodes/fields, removing edges: losses are 4.0/4.6/1.2 pp. Hiding only when_to_use/avoid, retaining all nodes/edges, loses 11.0/24.0/6.9 pp (Table 5). These controls separate relations and activation from coverage. Full versus S--Q compares whole-graph and Top-6 presentation.

W5

ALFWorld emphasizes retaining effective skills rather than updating for its own sake. In Table 6, both skills reach 100% success within 100 steps, and failures at 50 steps are timeouts. Skill edits may improve efficiency, but a timeout alone does not establish a faulty rule. When no skill defect has been identified, further rewriting may introduce errors and disrupt capabilities that already work. GraphSkillAA therefore proposes edits when it identifies a reusable skill defect and otherwise returns NO_PATCH, preserving existing capabilities.

AuthorsOpenReview ↗

Follow-up on Reviewer Discussion

Dear Reviewer,

I hope you are doing well. With fewer than three days remaining in the discussion period, we wanted to kindly follow up and ask whether our responses have adequately addressed your concerns. If you have any further questions or feedback, we would greatly appreciate hearing from you.

Thank you again for your time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

2nd Response to Reviewer vrwN

Dear Reviewer,

Thank you for your careful assessment and constructive feedback. We appreciate your recognition of the clarity of our presentation and the practical value of the proposed framework.

We have revisited the five concerns in your review and focus this response on the remaining methodological questions: the causal relevance of attributed graph objects, the scope of the minimal-edit claim, the regressions introduced by accepted updates, the information available to retrieval-based baselines, and the justification for NO_PATCH decisions.

Below, we explain the results and how they address each concern.

W1. Behavioral Relevance of Object-Level Attribution

The central question is whether attribution identifies useful repair targets beyond the benefit of adding execution records or general instructions. We test this through component ablations, target shuffling, and interventions on attributed objects.

1.1 Trace and usage records versus structured attribution

Table R1. Existing controlled ablations (GPT-5.6-sol accuracy, %).

SettingSearchQALiveMathDocVQA
G075.044.288.0
G0 + Trace/Usage76.251.388.2
GraphSkillAA Full81.566.791.2
No Structured Attribution73.238.586.7
No when_to_use & avoid70.542.784.3

Adding trace and usage records alone improves performance over G0 by 1.2, 7.1, and 0.2 percentage points. The complete method provides further improvements of 5.3, 15.4, and 3.0 percentage points over G0 + Trace/Usage.

This comparison indicates that recording execution information alone does not account for the complete method's gains.

Removing structured attribution lowers accuracy by 8.3, 28.2, and 4.5 points relative to Full. Removing when_to_use and avoid lowers it by 11.0, 24.0, and 6.9 points. These ablations show that identifying repair targets and stating when rules apply both contribute to the gains, especially on LiveMath.

1.2 Original versus shuffled attribution

Table R2. Existing target-shuffling experiment.

DatasetOriginal AttributionShuffled AttributionGain
SearchQA81.578.3+3.2 pp
LiveMath66.760.7+6.0 pp
DocVQA91.289.7+1.5 pp

Both configurations start from G0 + Trace/Usage and use three optimization epochs with seeds 42, 43, and 44.

The original targets outperform randomly selected type-compatible targets while keeping the student and teacher models, evidence construction, prompts, patch budget, and Gate rules fixed.

These results support the practical value of attribution-guided target selection.

1.3 Repair and regression statistics

Table R3. Existing update-pool outcomes.

DatasetAttributionFixedBrokenNet GainTest Accuracy
SearchQAOriginal240112+12881.5 ± 1.3
SearchQAShuffled200100+10078.3 ± 1.3
LiveMathOriginal350134+21666.7 ± 2.1
LiveMathShuffled260150+11060.7 ± 2.1
DocVQAOriginal10533+7291.2 ± 1.0
DocVQAShuffled7535+4089.7 ± 1.3

For each seed, Fixed and Broken compare the initial G0 + Trace/Usage graph with the final committed graph on the update pool. Counts are summed across three seeds; Net Gain equals Fixed minus Broken. Test accuracy is reported as mean ± half-range, where half-range is (maximum minus minimum)/2.

Original attribution yields 28, 106, and 32 additional net corrections. On SearchQA, it fixes 40 more cases and breaks 12 more; on LiveMath and DocVQA, it fixes more cases while breaking fewer. Together with R2, this shows that better target selection improves both update-pool repair outcomes and held-out accuracy.

Reviewer_YhwHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GraphSkillAA organizes skill documents into an addressable skill graph, uses contrasting successful and failed trajectories to identify edit targets, and filters patches through Local and Big Gates with atomic rollback. Experiments cover three main benchmarks and multiple models. The main contribution is a unified graph representation connecting execution records, targeted updates, and validation.

Critical strengths
  1. Practical framework. Node fields and relations provide explicit edit targets, making skill updates and rollback easier to track.
  2. Reasonable update controls. The method distinguishes skill defects from execution lapses and insufficient evidence, allowing NO_PATCH to avoid unsupported persistent changes.
Critical weaknesses
  1. Graph construction and execution semantics need clarification. The paper describes deterministic skill-to-graph conversion but provides limited detail on node decomposition, field assignment, and relation construction. Some field boundaries and execution constraints remain unclear, affecting understanding and reproducibility.
  2. Object-level attribution has limited validation. The results support the usefulness of structured diagnosis, but trace/usage validation mainly checks record consistency, while joint synthesis permits edits to multiple objects. Successful patches therefore do not fully establish the reliability of the initial localization.
Questions for authors
  1. Graph construction and execution semantics: How are initial skills converted into graphs? Please provide the conversion rules and a complete example, clarify the distinction between avoid and how_to_use, and explain whether edges represent transitions, prerequisites, or advisory relations. Do the execution sequences in Figure 5 omit intermediate steps where they do not directly follow the displayed edges?
  2. Attribution and local validation: Could you report how closely final edits follow the initial attribution targets, the proportion of examples covered by the Local Gate, and regressions detected by the Big Gate outside that scope? These results would help assess the practical value of object-level localization and local validation.
  3. Baseline provenance and efficiency: Which version or configuration produced the GPT-5.5 SkillOpt results in Table 3, which differ from the publicly available original paper? Please explain the discrepancy and provide cost comparisons under matched models, data, scorers, and optimization budgets, particularly total optimization tokens and deployment-time token usage.
10 comments
AuthorsOpenReview ↗

Response to Reviewer YhwH

W1 & Q1

Our earlier wording did not clearly distinguish semantic construction from deterministic rendering. We clarify in §3.1 and Appendix D.1 that an LLM constructs the graph semantically. “Deterministic” refers to rendering the saved graph.

Construction. GraphSkillAA initializes a skill document, here using SkillOpt's released best skill. The LLM may group rules sharing conditions and purpose, while independent checks become separate nodes. when_to_use states applicability, how_to_use gives actions, and avoid gives exclusions or prohibited actions. Conditions tied to a step may stay in how_to_use; unspecified fields remain empty.

The LLM also identifies relations: if B requires A first, it adds A→B as prereq; if A assists B, as enhance. co_occur represents joint use without ordering. Nodes and edges have stable IDs.

Example: reading a table (DocVQA, paraphrased).

The source skill says: find the requested row and column, check nearby lookalikes, copy exactly without changing characters, and return only the needed text.

The LLM makes four nodes for lookup, checking, copying and answer selection, merging exact-copy and spelling/punctuation rules. “For tables” becomes when_to_use, lookup steps become how_to_use, and copying prohibitions become avoid.

Lookup must precede checking (prereq). Checking and exact copying each help answer selection (enhance), giving three edges. The four nodes and three edges are saved and rendered into the student's prompt. Appendix D.1 provides their IDs and full fields.

Execution. The program validates fields, IDs, endpoints and prerequisite acyclicity. Runs reuse G0, rendered in fixed order. Edges express prerequisites or coordination, not automatic state transitions: the student interprets them in the prompt. Later edits use execution feedback and pass both Gates before commitment. Co-occurrence relations remain fixed.

Figure 5. Upper arrows show graph relations; lower arrows show presentation order, with intermediate checks omitted. Q029 is enhanced by Q011 through E007. Showing it after Q008 does not imply a Q008→Q029 edge. Thus, the lower panels summarize decisions rather than a complete edge-by-edge execution path, as explained in Appendix D.1.

W2

Joint synthesis treats coordinated edits as one repair. Several objects or error types may require changes, even when edits are minimized; individual edits need not work alone. Local Gate compares the same affected cases before and after repair, retaining the group if accepted and otherwise reverting it.

Trace/usage checks record consistency; controlled ablations test attribution. Table 5 compares each student's ablations with its self-teacher Full: GPT-5.6-sol, 81.5/66.7/91.2%; GPT-5.4-mini, 75.7/23.9/86.3% (SearchQA/LiveMath/DocVQA). Generic failure summaries lower all six results by 8.3 pp on average. Table 13 further shows original targets outperform shuffled targets by 3.2/6.0/1.5 pp in held-out accuracy. These results support attribution-guided repair.

Q2

Edits follow the initial plan's object scope: 100% adherence, not 100% attribution accuracy.

Big Gate subset results:

DatasetOutside cases (mean per round)Local Gate coverageBig Gate outside regression rate
SearchQA352.541.25%3.40%
LiveMath253.527.78%9.86%
DocVQA5626.33%3.02%

On SearchQA, Local Gate covers 41.25% of the update pool. Of 352.5 outside cases per round, 3.40% change from correct to incorrect (about 12), counted against acceptance.

Q3

Table 4 (formerly 3) averages three GPT-5.5 reevaluations of SkillOpt's released best skills. Both retain original splits, cases and scorer.

Costs: GPT-5.6-sol/Table 3, tokens per full optimization run and three-test mean accuracy (SearchQA/LiveMath/DocVQA: 200/117/200 cases). SkillOpt = Flat skill.

DatasetMethodActual Tokens (M)Token RatioAccuracy (%)Gain (pp)Extra Tokens / pp (M)
SearchQASkillOpt25.091.00×72.5——
GraphSkillAA53.182.12×81.5+9.03.121
LiveMathSkillOpt0.501.00×42.2——
GraphSkillAA1.783.56×66.7+24.50.052
DocVQASkillOpt1.131.00×87.7——
GraphSkillAA4.373.87×91.2+3.50.926

GraphSkillAA counts student execution, teacher attribution, synthesis, refinement and both Gates, excluding upstream SkillOpt. Ratios use SkillOpt. Extra Tokens / pp = (GraphSkillAA tokens − SkillOpt tokens) / accuracy gain.

Measured deployment costs are similar: both use the same student, settings and test inputs, fixed skills and one inference round per question. Teacher calls, patching and Gates run only during optimization.

Gains of 9.0/24.5/3.5 pp over SkillOpt's best skills without model updates justify extra optimization for performance-first use.

AuthorsOpenReview ↗

Follow-up on Reviewer Discussion

Dear Reviewer,

I hope you are doing well. With fewer than three days remaining in the discussion period, we wanted to kindly follow up and ask whether our responses have adequately addressed your concerns. If you have any further questions or feedback, we would greatly appreciate hearing from you.

Thank you again for your time and consideration.

Best regards, The Authors

Thank you for the clarifications. Some concerns remain unresolved. In particular, the GPT-5.5 SkillOpt results still differ substantially from those in the original paper: on LiveMath, the reported score is 51.1%, compared with 66.9% in SkillOpt, a gap of 15.8 percentage points. Re-evaluating the released skill explains where the number came from, but not why the discrepancy is so large under the stated matched protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.