acceptodds
Under review as a conference paper at ICLR 2027

Skill Transfer: Adapting Agent Skills from Strong to Weak Agents

Abstract

Agent skills are modular bundles of instructions, code, and resources that an agent loads on demand, increasingly authored and validated on frontier models and shared as an open standard. The same skill behaves differently across models: a strong model turns it into a large performance gain, while a weak model realizes little of that gain and sometimes does worse than with no skill at all. Cost and resource constraints, however, often restrict real deployments to exactly these weaker agents. We hypothesize that the strong model silently supplies procedural steps the skill leaves implicit, which the weak model omits, and formalize closing this gap as Skill Transfer: holding the skill's capability ceiling and the model's parameters fixed, we adapt the skill so a specific weak agent can execute it well. Our method abstracts strong and weak trajectories into typed execution structures, localizes their structural divergence with a graph edit distance (GED), and repairs the skill through an iterative diagnoser-patcher loop. Across three benchmarks and two model families, Skill Transfer raises pass rate over the unmodified skill significantly and well over other skill-evolution baselines. The adapted skill also improves the strong agent it was never adapted for, and transfers out of distribution to an unseen task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI agents, agent skills, LLM, transfer learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_96HCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Skill Transfer, a training-free framework that adapts skills written for strong agents to weaker agents. It abstracts paired strong and weak trajectories into typed execution structures, identifies procedural gaps between them and converts them into structured patches to the skill. Experiments show that the adapted skills consistently improve weak-agent performance over the original skills and generally outperform skill-evolution baselines.

Critical strengths
  • The paper studies an important and practically relevant problem: how to make skills developed for strong agents usable by weaker and cheaper models without retraining them.
  • The empirical results are strong and consistent across multiple benchmarks and both GPT and Qwen model families, with clear improvements over the original skills and the skill-evolution baseline.
  • The core idea is intuitive and well motivated: explicitly identifying the procedural steps that strong agents implicitly supply, and then incorporating those missing steps into the skill, is a reasonable way to close the strong–weak agent gap.
Critical weaknesses

This paper seems to provide a nice solution to the important problem of adapting skills developed for strong agents to weaker agents, but I have several concerns about the current method and evaluation:

(1) The role of GED in gap localization is unclear. Although GED is presented as a central component for comparing strong and weak execution structures, it appears to provide only a global scalar measure of trajectory divergence. The actual localization of missing_node, weak_node, order_difference, etc. is performed through separate label matching and LLM-based comparison of node details. As a result, it is unclear whether GED itself contributes to localization at all, or is mainly an auxiliary similarity metric. The paper also does not provide an ablation that isolates GED from the rest of the framework.

(2) The skill coverage is limited. The evaluation only considers three benchmark-level skill configurations, built from a small number of spreadsheet, finance/retrieval, and data-analysis skills. The paper does not demonstrate whether the method generalizes to a broader range of skills with substantially different structures, domains, lengths, or levels of procedural complexity, which limits the strength of the claim that Skill Transfer is broadly applicable to agent skills.

Questions for authors

(1) The current evaluation covers only a small number of skill configurations. Could the authors provide more characterization of the evaluated skills, and ideally evaluate on a broader and more diverse set of skills?

(2) Can all trajectory steps and strong–weak differences be reliably captured by the six fixed Tier-1 categories and the five predefined gap types, and how does the framework handle ambiguous, multi-role, or out-of-taxonomy behaviors?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 96HC 1/2

We appreciate the reviewer's constructive feedback and respond to each weakness and question below. The revised paper highlights all changes in teal.


W1. The role of GED and its ablation

Thank you for raising this important point. GED is a global scalar and does not localize gaps. Gaps are localized and typed by the label matching and the node details (§3.3). GED summarizes the structural differences in one number and has two roles. During training, the diagnoser sees each task's normalized GED per iteration as an auxiliary progress signal. Its prompt states that a GED decreasing across iterations means the structural gap is closing, and that a flat or increasing GED means the current approach is not working. At evaluation time, GED measures how far the weak agent's procedure on held-out tasks is from the strong agent's (§4.3).

We added the ablation that isolates GED (w/o GED). It removes GED from the diagnoser's case listing, case history and prompt, and keeps abstraction, validator reports, gap taxonomy, cross-iteration statuses and patching rules unchanged. We also report the raw-trajectory arm of Appendix A (w/o structure), which removes the whole structured pipeline.

Table R9. Test pass rate (%). Each ablation is one training run. The original skill and Skill Transfer are means over 5 runs.

SettingOriginal skill S0S_0w/o structure (raw trajectory)w/o GEDSkill Transfer
SpreadsheetBench / gpt-4.1-mini (200 tasks)25.731.042.544.4
DABench / gpt-4.1-mini (150 tasks)69.767.381.381.7
OfficeQA / gpt-5.4-mini (70 tasks)17.124.324.328.6

Removing GED lowers the pass rate on SpreadsheetBench (44.4 → 42.5) and OfficeQA (28.6 → 24.3), so GED improves training to some extent, although its main role is that of an evaluation metric. Removing the whole structured pipeline costs considerably more on SpreadsheetBench and DABench (13.4 and 14.4 points). We have revised §3.3 and Appendix A accordingly.


W2 & Q1. Broader skill coverage

We appreciate this suggestion. Following it, we added a fourth benchmark, whose skill library is larger and comes from a different domain. SeqQA2, from LABBench2 [1], consists of molecular-biology sequence tasks of 20 types, such as primer design, restriction digests and codon optimization, graded by the benchmark's official validators. The initial skills are six bioinformatics skills from the third-party bioSkills library [2] (sequence reading, primer design, primer validation, restriction-enzyme selection, restriction sites, and transcription and translation). With these skills, the strong agent (gpt-5.4) solves 71.7% of the test tasks, against 67.5% without skills. We use 20 training, 20 validation and 120 test tasks.

Table R10 characterizes the four skill configurations. They cover four domains, contain one to six skills, and range from a 1.7k-character skill to skills of over 30k characters, and they differ in whether they bundle scripts and reference files. With several skills, the agent must also choose which ones to activate.

Table R10. The evaluated skill configurations.

BenchmarkDomainSkillsSKILL.md length (characters)Bundled scriptsReference files
SpreadsheetBenchSpreadsheet editing111.5k130
DABenchData analysis and statistics213.5k, 31.3k34
OfficeQAFinancial analysis and document search212.1k, 1.7k04
SeqQA2Molecular-biology sequence analysis610.0k to 22.1k146

Table R11. Test pass rate (%) on SeqQA2.

Weak modelNo skillOriginal librarySkill Transfer
gpt-5.4-mini43.360.069.2
gpt-4.1-mini42.540.862.5

Skill Transfer improves both weak models over the original library, by 9.2 and 21.7 points. With gpt-4.1-mini the original library brings no gain over using no skill (40.8 vs. 42.5), while the adapted library reaches 62.5%. In both runs, Skill Transfer edited five of the six skills, demonstrating Skill Transfer is effective on broader coverage of skills. These results are in Appendix I of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 96HC 2/2

Q2. Coverage of the Tier-1 categories and gap types

Tier-1 categories. Table R12 gives the Tier-1 distribution of all nodes abstracted during training on the four benchmarks, including the new SeqQA2 domain. Only 6 nodes (under 0.1%) fall into other. Each of them is the final node of a run and contains no tool call. It is a note that the abstractor adds to describe how the run ended, for example that the agent stopped before writing the result file, so none of them is a step the agent performed. A step that serves several purposes is split into one node per purpose, and domain-specific detail goes into the free-text Tier-2 label, so the fixed Tier-1 set only has to capture the procedural role.

Table R12. Share of nodes (%) in each Tier-1 category.

Benchmarksetupexecutevalidaterecoveroutputother
SpreadsheetBench40.519.413.29.117.80.1
DABench45.825.31.35.222.30.0
OfficeQA38.832.72.94.820.60.1
SeqQA243.322.53.19.121.90.1

Gap types. An execution structure is a labelled path graph, and the four operations of graph edit distance [3, 4] suffice to transform any such graph into any other. With the strong run as the reference, each operation is one gap type. A deleted node is a missing_node, a substituted node (matched, but executed to a lower bar) is a weak_node, an inserted node is an extra_wrong_node, and an edited edge (two matched steps in opposite order) is an order_difference. No structural difference therefore falls outside these four. The fifth type, wrong_skill_routing, is designed for skill use. When the weak agent does not activate a skill the strong agent used, that skill's description is edited so that the weak agent selects it. These are in §3.3 and Appendix J of the revised paper.


References

[1] Laurent et al. LABBench2: An Improved Benchmark for AI Systems Performing Biology Research. arXiv:2604.09554, 2026.

[2] GPTomics. bioSkills: a library of bioinformatics skills for AI agents. Open-source skill library, 2026.

[3] Sanfeliu and Fu. A distance measure between attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, 1983.

[4] Gao et al. A survey of graph edit distance. Pattern Analysis and Applications, 2010.

Thank you for the clarification. My concerns have been addressed, and I will raise my rating accordingly. I encourage the authors to incorporate these additional experiments into the final version of the paper.

We sincerely thank the reviewer for the positive feedback and for raising the rating. The additional experiments are already included in the revised paper. The ablation of GED is reported in Appendix A. The SeqQA2 experiment and the characterization of the evaluated skills are in Appendix I. The coverage of the Tier-1 categories is in Appendix J, and the correspondence between the gap types and the graph edit operations is described in §3.3. We will keep all of these additions in the final version.

Reviewer_W2HEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the issue of inconsistent applicability of Agent Skills across different capability models, and proposes Skill Transfer. Without updating the model parameters, this method adapts the skills designed for the strong model to the weak model. By comparing the execution processes of the strong and weak models, this approach identifies the implicit process knowledge in the skills and transforms it into clear guidelines that the weak model can follow through iterative diagnosis and rewriting.

Critical strengths
  1. The paper focuses on the differences in the applicability of existing skills between strong and weak models, which are closely related to the costs and resource constraints in actual deployment. The research motivation is clear.

  2. The paper compares the execution trajectories of strong and weak models structurally, breaks down the general task failures into specific process defects, and provides clear basis for skill modification.

  3. The paper's method does not require updating model parameters, but rather modifies the readable and checkable skill content, facilitating understanding of the modification reasons and reusing existing skill resources.

Critical weaknesses
  1. Several improvements over EvoSkill are small, and there is a lack of explanation for the fluctuations in the experiments. For instance, GPT-5.4-mini's improvement over EvoSkill on SpreadsheetBench was 1.5 percentage points, corresponding to 3 out of 200 questions; on OfficeQA, the improvement was approximately corresponding to 2 out of 70 questions. The paper did not provide repeated runs, confidence intervals, or paired significance tests, and the method itself contains multiple random generation steps. Therefore, cases with significant improvements are more convincing, but it is still unclear whether the advantages in some configurations are stable.

  2. The experiments do not distinguish model-specific skill transfer from general skill improvement. The edits in Appendix E correct existing rules and introduce concrete solution procedures. These changes can improve performance without supporting the proposed explanation of recovering implicit steps. The contribution of target-model-specific adaptation remains unclear beyond general improvements to skill quality.

  3. The contribution over closely related work is not sufficiently established. MASA (https://arxiv.org/html/2605.30723v1) already performs model-specific skill adaptation. SkillGrad (https://arxiv.org/html/2605.27760v1) combines contrastive diagnosis with iterative skill patching. MemCollab (https://arxiv.org/html/2603.23234v2) extracts reusable guidance from cross-model trajectory comparisons. Without comparisons against these methods, the added value of the proposed structural alignment remains unclear.

  4. The paper overinterprets lower GED as evidence for its proposed mechanism. On SpreadsheetBench, the original skill reduces GPT-5.4-mini’s GED from 0.227 to 0.111, while its pass rate falls from 48.0% to 39.5%. Greater similarity to the teacher’s execution therefore does not establish a more effective procedure. GED reduction alone cannot support the claim that performance gains result from closing procedural gaps.

Questions for authors
  1. The raw-trajectory ablation removes several components simultaneously. Does removing only GED reduce performance when trajectory abstraction, gap diagnosis, and iterative skill patching are kept unchanged?

  2. SkillGrad already includes contrastive trajectory diagnosis and iterative skill patching. How does Skill Transfer compare with it under the same initial skill, task split, and adaptation budget?

  3. A weak agent may solve a task correctly while following different steps from the strong agent. How does the method decide whether these differences actually need correction? What prevents it from treating a different but correct approach as an error?

4 comments
AuthorsOpenReview ↗

Response to Reviewer W2HE 1/3

We are grateful to the reviewer for the detailed and thoughtful comments. Our responses to each weakness and question follow. We have also revised the paper, with the changes marked in teal.


W1. Robustness of the smaller improvements

We conducted repeated experiments. Skill Transfer and EvoSkill each have 5 full runs, and no skill and the original skill have 5 evaluations. Table R3 reports mean ± std. Its last column is the difference between Skill Transfer and EvoSkill with its 95% CI, obtained by a two-level bootstrap (runs resampled within each method, test tasks shared, 10,000 draws). We can see Skill Transfer is above the original skill in all six settings, and is ahead against EvoSkill in five settings. Table 3 of the revised paper now reports these five-run means.

Table R3. Test pass rate (%), mean ± std over 5 runs. The last column is the difference in percentage points.

SettingNo skillOriginalEvoSkillSkill TransferSkill Transfer − EvoSkill [95% CI]
SpreadsheetBench / gpt-4.1-mini26.6 ± 2.225.7 ± 2.735.1 ± 1.144.4 ± 3.0+9.3 [+4.2, +14.7]
SpreadsheetBench / gpt-5.4-mini46.8 ± 0.645.3 ± 3.047.5 ± 1.252.4 ± 1.2+4.9 [+0.8, +9.2]
DABench / gpt-4.1-mini74.5 ± 1.869.7 ± 2.176.1 ± 2.481.7 ± 1.7+5.6 [+1.1, +10.4]
DABench / gpt-5.4-mini80.4 ± 1.082.4 ± 1.083.7 ± 1.586.8 ± 0.5+3.1 [+0.1, +6.3]
OfficeQA / gpt-4.1-mini20.0 ± 3.917.7 ± 5.517.7 ± 0.822.9 ± 3.7+5.1 [−2.0, +12.6]
OfficeQA / gpt-5.4-mini16.0 ± 3.917.1 ± 3.628.6 ± 2.328.6 ± 1.00.0 [−8.3, +8.3]

Q1. Removing only GED

Thank you for proposing this ablation, which we have added. GED is removed from the diagnoser's case listing, case history and prompt, while abstraction, validator reports, gap taxonomy, cross-iteration statuses and patching rules stay unchanged. We also report the raw-trajectory arm of Appendix A (w/o structure), now also run on DABench and OfficeQA, which removes the whole structured pipeline.

Table R4. Test pass rate (%). Each ablation is one training run. The original skill and Skill Transfer are the means over 5 runs of Table R3.

SettingOriginal skill S0S_0w/o structure (raw trajectory)w/o GEDSkill Transfer
SpreadsheetBench / gpt-4.1-mini (200 tasks)25.731.042.544.4
DABench / gpt-4.1-mini (150 tasks)69.767.381.381.7
OfficeQA / gpt-5.4-mini (70 tasks)17.124.324.328.6

During training, GED serves the diagnoser as an auxiliary progress signal. Its prompt states that a GED decreasing across iterations means the structural gap is closing, and that a flat or increasing GED means the current approach is not working. Removing GED lowers the pass rate on SpreadsheetBench (44.4 → 42.5) and OfficeQA (28.6 → 24.3), so it improves training to some extent, although its main role is that of an evaluation metric. Removing the whole structured pipeline costs considerably more on SpreadsheetBench and DABench (13.4 and 14.4 points). Table R4 is in Appendix A of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer W2HE 2/3

W2. Model-specific transfer vs. general skill improvement

This is an insightful question. Recovering implicit steps does not conflict with general improvement, because some of these steps help every model. Every edit in Skill Transfer starts from a gap backed by quoted evidence from both trajectories (§3.3). Most gaps are steps that the strong agent performs and the weak agent either omits or performs poorly. Writing such a step into the skill makes explicit for the weak agent what the strong agent supplies implicitly. Which steps need to be made explicit depends on the target model.

To test model specificity directly, we swap adapted skills across model families. The skill adapted for one weak model is evaluated on the other, and compared with the skill adapted for that model itself. We use gpt-4.1-mini and Qwen3.5-4B for this experiment, and results are in Table R5.

Table R5. Test pass rate (%) on SpreadsheetBench with skills adapted for different target models, mean ± std. The original skill and the own adapted skill of gpt-4.1-mini are taken from Table R3, and the other cells are five evaluations of one skill.

ExecutorOriginal skillOwn adapted skillSkill adapted for the other executor
gpt-4.1-mini25.7 ± 2.744.4 ± 3.036.5 ± 0.7
Qwen3.5-4B11.3 ± 2.123.7 ± 1.618.5 ± 2.7

On both executors, the skill adapted for the executor itself scores 5 to 8 points above the skill adapted for the other model. Part of the adaptation is therefore specific to the target model. The skill adapted for the other model still improves on the original skill, by 7 to 11 points, so part of the content is general.

As a model-specific example, the skill adapted for Qwen3.5-4B forbids inline multi-line Python in shell calls and requires a script file, while the one adapted for gpt-4.1-mini allows a heredoc. gpt-4.1-mini uses a heredoc in only 9 of 1,000 runs, but with the same skill Qwen3.5-4B attempts one in 818 runs and malforms the call in 756, which wastes turns and raises the runs without an output from 147 to 218. As a general example, both adapted skills add a mandatory inspection of the workbook before any edit, which the original skill lacks. On gpt-4.1-mini, the share of runs that inspect first rises from 28% to 95% (own skill) and 61% (skill adapted for Qwen3.5-4B), and failures with a formula error in the answer cells fall from 74 to 26 and 22 per 1,000 runs. These results are in Appendix H of the revised paper.


W3 & Q2. Comparison with MASA, SkillGrad and MemCollab

We thank the reviewer for pointing us to these works. Table R6 summarizes how the three methods differ from Skill Transfer in their goal and in the signal they use. MASA and SkillGrad observe a single agent, so they diagnose its failures without a reference execution. MemCollab also contrasts different models, but keeps what they share, whereas Skill Transfer keeps what is specific to the weak agent. MASA also relies on a model card that lists fields such as the number of layers and the hidden size, which cannot be obtained for closed models. Skill Transfer needs no model card and trains no component.

Table R6. Skill Transfer and the three related methods.

MethodGoalTrajectories comparedWhat is kept
MASAA model-specific skill for each backboneNone: only the target's failed trajectories, with a model cardRewrites that raise the target's reward
SkillGradHigher general quality of a skillFailed vs. successful attempts of the same agentText gradients, accumulated with momentum
MemCollabA memory shared by different modelsDifferent models on the same taskConstraints that hold across models; model-specific content is filtered out
Skill TransferTransfer of an existing skill from a strong agent to a weak oneA strong vs. a weak agent executing the same skill on the same taskSteps specific to the weak agent (omitted or poorly executed), written into the skill

We also compare Skill Transfer with SkillGrad on SpreadsheetBench, starting from the same initial skill and using the same settings. Its executor is the weak model (gpt-4.1-mini or gpt-5.4-mini) and its optimizer roles use gpt-5.4, as in Skill Transfer. The results demonstrate that Skill Transfer outperforms SkillGrad on this task, as shown in Table R7.

Table R7. Test pass rate (%) of Skill Transfer and SkillGrad on SpreadsheetBench.

ExecutorOriginal skillSkillGradSkill Transfer
gpt-4.1-mini25.736.544.4
gpt-5.4-mini45.349.052.4
AuthorsOpenReview ↗

Response to Reviewer W2HE 3/3

W4. Lower GED does not by itself imply a more effective procedure

We thank the reviewer for this valuable observation. We agree that GED measures only how far the weak agent's sequence of steps is from the strong agent's. It does not capture whether each step is executed correctly. A lower GED raises the pass rate only when the weak agent fails because it takes the wrong steps. It does not help when the agent takes the right steps but makes errors inside them. Table R8 shows the test pass rate by GED, pooled over all four methods.

Table R8. Test pass rate (%) of the weak agents by GED to the strong agent. No skill, original skill, EvoSkill and Skill Transfer; gpt-4.1-mini and gpt-5.4-mini pooled; tasks solved by the strong agent.

GEDSpreadsheetBenchDABenchOfficeQA
= 062.796.249.0
(0, 0.1]59.587.346.2
(0.1, 0.2]57.185.833.9
(0.2, 0.3]46.786.222.2
> 0.331.770.715.2

The two kinds of failure separate SpreadsheetBench from DABench and OfficeQA. On SpreadsheetBench, the pass rate rises substantially as GED falls to 0.2, but then stays around 60%. Once the structure is aligned, the bottleneck moves inside the steps (for example, computing the right cell values). On DABench and OfficeQA, the pass rate keeps rising as GED falls, even when GED is already low, because wrong steps remain a major cause of failure on these benchmarks.

This explains the reviewer's SpreadsheetBench example in the original Table 3, where the original skill lowers GED but also lowers the pass rate. On SpreadsheetBench, aligning the steps is not enough, because the remaining failures occur inside the steps. Skill Transfer can also repair such failures. We have revised §4.3 and added Appendix G accordingly.


Q3. Different but correct steps

A structural difference is never treated as an error by itself. Skill Transfer repairs only the gaps that affect the weak agent's performance or threaten its robustness.

This is built into the method. A gap is admitted for repair only if it appears in at least one task the weak agent failed, or in at least two tasks it passed (§3.4). In addition, the diagnoser's prompt requires that a patch must not break the tasks the weak agent already solves.

Large differences are also uncommon among correct runs. Of the weak agents' passing runs in Table R8, only 11% have a GED above 0.3.

As a result, Skill Transfer rarely breaks what already works. Across the six settings, it breaks 23 of the 383 tasks that the weak agent solves with the original skill (only 6.0%), while it newly solves 107.

Thank you again for your review. As the discussion period is closing soon, we would like to kindly ask whether our response and the revised paper have addressed your concerns. If so, we would greatly appreciate it if you could reconsider your assessment. We are glad to clarify anything further in the remaining time.

Reviewer_12utOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to adapt an existing skill that works well for a strong agent to a weaker agent without changing model parameters. It abstracts paired strong and weak trajectories into typed step sequences, characterizes structural differences through alignment and graph edit distance, and iteratively edits the skill through separate diagnosis and patching agents. A validation set selects the final version. Experiments cover SpreadsheetBench, OfficeQA, and DABench with GPT and Qwen models, supplemented by a raw-trajectory rewriting comparison, reuse by the strong agent, and transfer to WikiTableQuestions.

Critical strengths
  1. The problem is clearly defined and the method addresses it directly. The paper turns differences in skill usability across executors into inspectable issues such as missing steps, inadequate execution, and ordering differences, while restricting adaptation to the skill. The regressions caused by original skills in several weak-model settings make the problem practically relevant. Separating diagnosis from patching also makes the adaptation process easier to inspect (Sections 3.2–3.6; Tables 3–5).

  2. Multiple settings and a useful control support the complete system. The method achieves the best or tied-best pass rate in all eight weak-model configurations. Appendix A compares against raw-trajectory rewriting with the same teacher, training tasks, and iteration budget. On SpreadsheetBench with gpt-4.1-mini, the full method reaches 48.5%, versus 31.0% for raw rewriting and 25.5% for the original skill; it fixes 48 tasks and breaks two relative to the original skill. This directly supports the value of the structured workflow as a whole. The WikiTableQuestions results provide additional evidence of transfer to a related task (Tables 3–5, 7–8).

Critical weaknesses
  1. The individual contribution of the structural components remains unresolved. Appendix A simultaneously removes abstraction, GED, validator reports, the gap taxonomy, cross-iteration statuses, and strategy mapping. It therefore supports the complete pipeline without isolating which components matter. Moreover, Section 3.3 distinguishes the order-preserving alignment used for GED from the matching M used for gap diagnosis; the operational influence of the GED value or optimal alignment on patching needs clarification. The decrease in GED accompanying improved accuracy does not by itself establish the causal mechanism claimed in Section 4.3 and the conclusion.

  2. The robustness of the smaller improvements is unclear. Tables 3–5 do not provide repeated-run results or paired confidence intervals for the main comparisons. Some gains over EvoSkill are modest: gpt-5.4-mini improves by 1.5 percentage points on SpreadsheetBench and ties its pass rate on DABench. Claims of significant improvement therefore need statistical support or narrower wording. The raw-trajectory comparison also covers only one model–benchmark combination, leaving its broader stability uncertain.

Questions for authors
  1. Where does the GED scalar or its optimal alignment affect diagnosis and patching? Can you remove only GED or its alignment signal while retaining the abstract representations, validator reports, and patching rules, to establish the incremental contribution of this component?

  2. Can you report independent repetitions of the full adaptation process and paired improvements with confidence intervals against EvoSkill and raw-trajectory rewriting? The smaller differences in Table 3 and replication of Appendix A on another model or benchmark would be particularly informative.

The paper defines a useful adaptation problem and presents reasonably consistent results across tasks and models, with a meaningful raw-trajectory control supporting the complete system. Although component attribution and the robustness of smaller gains need strengthening, the existing evidence supports accepting the work as a useful skill-adaptation procedure. Claims about GED and the causal mechanism should be moderated.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 12ut

We thank the reviewer for the careful reading and for recommending acceptance. Below we answer both weaknesses and both questions with new experiments. The paper has been revised accordingly, and all changes are marked in teal.


W1 & Q1. Where GED enters the loop, and the contribution of individual components

We thank the reviewer for raising this point, which helped us state the role of GED more precisely.

How GED is used. GED plays two roles, while gap localization itself is done by label matching.

During training, GED is an auxiliary progress signal for the diagnoser. The diagnoser sees each training task's normalized GED per iteration in its case listing and case history. The prompt states two rules: a decreasing GED means the structural gap is closing and the patch direction is correct, and a flat or increasing GED means the current approach is not working. The optimal order-preserving alignment is used only to compute this scalar. Gaps are localized and typed with the label matching MM and the node details (§3.3), and the patcher acts on the diagnoser's gap report.

At evaluation time, GED is the metric that quantifies how far the weak agent's procedure on held-out tasks is from the strong agent's (§4.3).

Ablations. We add a GED-only ablation (w/o GED). It removes GED from the case listing, the case history and the prompt, and keeps abstraction, validator reports, gap taxonomy, cross-iteration statuses and patching rules unchanged. We also run the raw-trajectory arm of Appendix A (w/o structure), which removes the whole structured pipeline, on DABench and OfficeQA. We run w/o structure and w/o GED once because of the resource limit.

Table R1. Test pass rate (%). Each ablation is one training run. The original skill and Skill Transfer are the means over 5 runs of Table R2.

SettingOriginal skill S0S_0w/o structure (raw trajectory)w/o GEDSkill Transfer
SpreadsheetBench / gpt-4.1-mini (200 tasks)25.731.042.544.4
DABench / gpt-4.1-mini (150 tasks)69.767.381.381.7
OfficeQA / gpt-5.4-mini (70 tasks)17.124.324.328.6

The structured pipeline matters most. Without it, the pass rate drops by 13.4, 14.4 and 4.3 points on SpreadsheetBench, DABench and OfficeQA. These components are coupled, since the taxonomy is defined on the abstracted structures and statuses are computed per typed gap, so w/o structure measures their joint contribution. GED improves training to some extent. Removing it lowers the pass rate on SpreadsheetBench (44.4 → 42.5) and OfficeQA (28.6 → 24.3).

We have revised §3.3, §4.3, the conclusion and Appendix A accordingly.


W2 & Q2. Repeated runs and paired confidence intervals

Thank you for this suggestion. For every setting we repeated the full adaptation 5 times with independent seeds. Skill Transfer and EvoSkill have 5 full runs, and no skill and the original skill have 5 evaluations. Table R2 reports mean ± std. Its last column is the difference between Skill Transfer and EvoSkill with its 95% CI, obtained by a two-level bootstrap (runs resampled within each method, test tasks shared, 10,000 draws). We can see Skill Transfer is above the original skill in all six settings, and is ahead against EvoSkill in five settings.

Table R2. Test pass rate (%), mean ± std over 5 runs. The last column is the difference in percentage points.

SettingNo skillOriginalEvoSkillSkill TransferSkill Transfer − EvoSkill [95% CI]
SpreadsheetBench / gpt-4.1-mini26.6 ± 2.225.7 ± 2.735.1 ± 1.144.4 ± 3.0+9.3 [+4.2, +14.7]
SpreadsheetBench / gpt-5.4-mini46.8 ± 0.645.3 ± 3.047.5 ± 1.252.4 ± 1.2+4.9 [+0.8, +9.2]
DABench / gpt-4.1-mini74.5 ± 1.869.7 ± 2.176.1 ± 2.481.7 ± 1.7+5.6 [+1.1, +10.4]
DABench / gpt-5.4-mini80.4 ± 1.082.4 ± 1.083.7 ± 1.586.8 ± 0.5+3.1 [+0.1, +6.3]
OfficeQA / gpt-4.1-mini20.0 ± 3.917.7 ± 5.517.7 ± 0.822.9 ± 3.7+5.1 [−2.0, +12.6]
OfficeQA / gpt-5.4-mini16.0 ± 3.917.1 ± 3.628.6 ± 2.328.6 ± 1.00.0 [−8.3, +8.3]

Table 3 of the revised paper now reports these five-run means.

The replication of Appendix A on DABench and OfficeQA is the w/o structure column of Table R1. The raw-trajectory control stays below Skill Transfer on all three benchmarks.

Thank you again for your review. As the discussion period is closing soon, we would like to kindly ask whether our response and the revised paper have addressed your concerns. We are glad to clarify anything further in the remaining time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.