acceptodds
Under review as a conference paper at ICLR 2027

SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown

Abstract

Executable Agent Skills combine natural-language instructions and scripts into reusable packages for LLM agents, and revising them requires fixing errors without breaking correct behavior. Existing benchmarks do not systematically distinguish documentation repair, script repair, and preservation when evaluating skill self-evolution. We introduce SkillScriptBench, a 350-task benchmark designed to evaluate these capabilities separately. From a survey of over 35,000 GitHub-hosted Skill roots, we select 100 packages and construct 150 repair tasks. Each task pairs a package containing injected script faults with a maintenance request and executable checks of the required behavior. A complementary controlled track contains 200 tasks from 50 packages, each evaluated under the same maintenance request in four states: clean, documentation faults, script faults, and faults in both. Across four LLMs, methods that edit both documentation and scripts can repair script faults but do not consistently outperform Markdown-only revision on documentation repair or preservation. We therefore introduce AST-Guided Skill Revision, which uses abstract syntax trees and calling relationships to link maintenance requirements to relevant code locations. It restricts script edits to these locations and updates the documentation to match the revised scripts. Averaged across models, this revision stage yields absolute gains in repair success of 21.9% for Raw Package and 27.7% for CoEvoSkills on faulty packages. Absolute gains in the proportion of tasks solved in all three runs reach 20.8% and 31.5%, respectively, indicating more consistent repair success across repeated runs.

Primary area:
datasets and benchmarks
Keywords:
Agent Skills, Skill Self-Evolution, Benchmarking, Program Repair, Abstract Syntax Trees

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t4ChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillScriptBench, a benchmark targeting the self-evolution capability of executable agent skill packages, containing SKILL.md documentation and scripts/ executable scripts, motivated by the observation that existing work fails to distinguish three independent dimensions: "documentation repair," "script repair," and "correct behavior preservation." The benchmark contains 350 tasks: an In-the-Wild Repair track and a Controlled Repair track. The paper also contributes an AST-Guided Skill Revision method: behavioral requirements in maintenance requests are bound via an LLM to language-native AST/CST nodes, combined with call-relationship context across functions, and script edits are constrained by strict node-count/byte-count limits and interface-signature preservation checks, followed by reference-aligned editing of the Markdown. Experiments on four model backbones compare Markdown-only, Raw Package, CoEvoSkills, and their respective versions with AST-Guided Revision added.

Critical strengths

The paper systematically disentangles three previously conflated evaluation dimensions, documentation repair / script repair / behavior preservation, and reveals a trade-off relationship of real practical significance. The Controlled Repair track, through a strictly controlled design that fixes the same maintenance request across four states, demonstrates that the repair-capability gains from "full package access" come at the cost of behavior preservation on Clean tasks (a gap of up to 7.8/3.7 percentage points)—a previously underappreciated finding with real implications for skill maintenance system design.

Critical weaknesses
  1. The sample size for the Controlled Repair subcategory is too small (some repair categories have only 3-9 tasks). The authors need to discuss the impact on the robustness of fine-grained conclusions.

  2. Were the benchmarks built from over 35,000 GitHub repositories rigorously filtered to ensure no data leakage? The tested model corpus may contain overlapping contamination; the authors did not provide details on how this was handled.

  3. The core steps of AST-Guided Skill Revision still heavily rely on an LLM to complete inference decisions in the prompt. The authors should provide more details on the effectiveness of this design and more ablation experiments to verify the innovativeness of this part of the method.

Questions for authors
  1. Has the overlap between the 35,554 GitHub skill repositories crawled to build the benchmark and the pretraining corpora of the evaluated LLMs been assessed and filtered?

  2. Could a case study or error analysis be provided to demonstrate that the improvement from AST-Guided Revision mainly comes from the AST structural constraint itself?

  3. The Single-Target/Multi-Target/Cross-Call partitions in Controlled Repair have only 20/15/15 base packages respectively, and some repair categories have extremely small sample sizes (e.g., 3 tasks)—how statistically significant are the fine-grained percentage-point improvements reported in §5.2 given such small samples?

4 comments

We thank the reviewer for the positive feedback on the importance of our task and for the valuable suggestions. Our responses are provided below.

Q1 & W2. Has the overlap between the 35,554 GitHub skill repositories crawled to build the benchmark and the pretraining corpora of the evaluated LLMs been assessed and filtered?

We did not screen the GitHub packages for overlap with model pretraining data. Raw Package and AST-guided revision use the same model on the same faulty package. If familiarity with the upstream code helps Raw repair a fault, that repair already counts toward the baseline, leaving less room for AST-guided revision to improve on it. If familiarity benefits Raw more than it benefits AST-guided revision, the gain from adding AST revision could be larger on unfamiliar packages.

Q2 & W3. Could a case study or error analysis be provided to demonstrate that the improvement from AST-Guided Revision mainly comes from the AST structural constraint itself?

In the VCP stock-analysis case, GPT-5.6-Sol’s Raw Package left six argument-binding errors across four calls. One illustrates why the fault was easy to miss. The user reported that changing one setting affected an unrelated calculation. Raw had the full code, but the symptom did not identify the faulty call. When analyze_stock called calculate_vcp_pattern, it passed min_contraction_days to both min_contraction_days and lookback_days. Both values represent days, so the call ran without an error even though the user’s lookback setting never reached the calculator.

The AST analysis resolved the call and traced each argument to its source in analyze_stock. This exposed the mismatch: both parameters received the same variable even though the caller had a separate lookback_days value. The structural feedback showed the editor the call site, the value passed, and the receiving parameter. The editor then corrected the six bindings at the call sites. On the same price data, shortening the lookback window from 120 to 30 days reduced the number of detected contractions as requested. The repaired package passed 10/10 behavioral checks, up from 4/10 with Raw.

The ablation shows why caller–callee context matters beyond this example. Starting from the same Raw Package proposals on Controlled Repair’s Joint tasks, an additional text-based revision achieved 56.7% success. AST node editing without caller–callee context achieved 54.7%, compared with 81.3% for full AST-guided revision. Removing call context while retaining node editing reduced success by 26.6 percentage points.

Q3 & W1. The Single-Target/Multi-Target/Cross-Call partitions in Controlled Repair have only 20/15/15 base packages respectively, and some repair categories have extremely small sample sizes (e.g., 3 tasks)—how statistically significant are the fine-grained percentage-point improvements reported in §5.2 given such small samples?

We compared Raw Package and Raw Package + AST on the same base packages in each Controlled Repair partition. For each package, we paired outcomes by fault state, model, and run, and averaged the 36 paired differences (three fault states × four models × three runs). We then resampled base packages within each partition 10,000 times to estimate a 95% bootstrap interval for the mean gain.

PartitionBase packagesMean gain (percentage points)95% bootstrap intervalPackages improved
Single-Target20+32.8[19.7, 46.5]19/20
Multi-Target15+27.6[21.7, 33.3]15/15
Cross-Call15+24.1[14.4, 32.4]12/15

AST-guided revision improved success on 46 of the 50 base packages. The other four comprised two ties and two declines. In all three partitions, the 95% interval for the mean gain lay above zero. For repair categories with only three tasks, we report the observed success rates descriptively.

Thank you for your reply, I will keep my score.

Dear Reviewer t4Ch,

Thank you for taking the time to consider our response. We would appreciate knowing whether we have fully addressed your concerns. If any point remains unresolved, please let us know. We would be happy to address it in more detail.

Best regards,
The Authors of SkillScriptBench

Thank you for your response. After careful consideration, I have decided to keep the score.

Reviewer_kjdUOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SkillScriptBench, a benchmark for maintaining executable agent skill packages that contain both Markdown instructions and scripts. It comprises 150 repair tasks constructed by injecting faults into repository-sourced packages and 200 controlled tasks covering matched clean, documentation-fault, script-fault, and joint-fault states. Evaluation combines behavioral tests with documentation-driven execution checks. The paper also proposes AST-Guided Skill Revision, an additional revision stage that grounds requirements in code structure, constrains script edits, and aligns documentation with the revised implementation. Across four LLM backbones, the complete revision pipeline improves repair success and preservation relative to Raw Package and an adapted CoEvoSkills baseline.

Critical strengths
  1. Useful separation of repair and preservation. The matched four-state construction is a meaningful benchmark design choice. Holding the package and maintenance request fixed while varying the faulty artifact helps distinguish documentation repair, script repair, coordinated repair, and unnecessary regressions. These distinctions provide information that aggregate task success alone would obscure.

  2. Consistent improvements from the complete pipeline. AST-augmented methods improve Avg, Pass@3, and Hit3 across all four backbones for both base methods. The reported Overall Avg gains of 21.9 and 27.7 percentage points are substantial. Paired source-package bootstrap intervals, correction/regression analyses, and alternative task-weighting analyses strengthen the evidence that the observed gains are not confined to a single model or dominant task group.

Critical weaknesses
  1. The baselines limit the scope of the comparative conclusions. Markdown-only differs from full-package methods in both visible information and editing permissions: it cannot inspect scripts, even when repairing documentation. A full-package-read, Markdown-only-edit baseline would better isolate the effect of editing scope. In addition, the CoEvoSkills adaptation allows only one verification and at most one revision, and its executable surrogate checks are restricted to file presence, textual conditions, and syntax validity. These checks offer limited direct evidence about the behavioral faults emphasized by the benchmark. The results therefore support comparisons against this particular adaptation, rather than a broader conclusion about competitive skill-evolution or execution-guided repair methods.

  2. External validity remains limited by the constructed fault distribution. Despite the In-the-Wild name, both tracks use injected faults; the real-world provenance applies to the packages rather than naturally occurring failures. Local parameter substitutions, field mappings, and helper-call faults are well aligned with the proposed constrained editing procedure. The source distribution is also uneven, with Python accounting for 115 of the 150 In-the-Wild tasks and schema/field repairs accounting for 57.

Questions for authors

The most significant questions are already listed under weaknesses. In addition, I would be interested to get a clarification on the following point:

  1. How much of the improvement remains when independently removing Markdown alignment, preservation constraints, or interface checks?

  2. Is there evidence that the gains transfer to naturally occurring skill failures or improve downstream agent task success after the repaired packages are reused?

3 comments

We thank the reviewer for the positive feedback on the importance of our task and for the valuable suggestions. Our responses are provided below.

W1.1. A full-package-read, Markdown-only-edit baseline would better isolate the effect of editing scope.

We agree that the original comparison changed both what the agent could read and what it could edit. We based our Markdown-only baseline on how SkillOpt and SkillAdaptor revise skills. Both methods use execution feedback to update the written instructions without reading the underlying scripts during revision.

We therefore ran the control you suggest with GPT-5.6-Sol on all 150 In-the-Wild script-repair tasks. We gave the agent access to the full package while keeping edits restricted to SKILL.md. The agent revised 74 documents, but script-repair success remained at 0/150, the same as in the original Markdown-only setting. With the full package visible in both settings, allowing script edits in Raw Package raised success to 102/150 (68%) in repeat 1.

W1.2. In addition, the CoEvoSkills adaptation allows only one verification and at most one revision, and its executable surrogate checks are restricted to file presence, textual conditions, and syntax validity. These checks offer limited direct evidence about the behavioral faults emphasized by the benchmark.

Our adaptation repairs an existing package against a maintenance request. An independent verifier inspects the package and checks file presence, text requirements, and script syntax. A failed check or verifier judgment triggers one revision. These checks offer limited evidence about runtime behavior. One AST-guided revision of the resulting packages raised average repair success from 48.8% to 76.5% across 300 faulty tasks, four models, and three runs per model.

To test whether CoEvoSkills benefits from further iterations, we ran two additional rounds on Controlled Repair using the same verifier. CoEvoSkills passed 133/200 tasks when we rescored its first-round outputs, 143/200 after the second round, and 144/200 (72.0%) after the third. Starting from the same first-round outputs, a single AST-guided revision passed 170/200 tasks (85.0%).

W2. External validity remains limited by the constructed fault distribution.

We injected the same fault types we observed in GitHub Skill scripts, including passing the wrong argument and skipping a required helper call. Injection let us verify each In-the-Wild case with a paired check. The same test passed on the original package and failed after we introduced the fault. Q2 reports results on 50 naturally occurring faults.

Since 57 of the 150 tasks involve schema/field mapping, we also gave all eight repair categories equal weight. AST revision improved on Raw Package by 13.5 percentage points with equal category weights and 15.2 with equal task weights. For CoEvoSkills, the corresponding gains were 22.2 and 22.9 points.

Python was the most common language in both the GitHub survey (56.8% of script files, 7,353/12,943) and In-the-Wild Repair (76.7% of tasks, 115/150). Starting from Raw Package, AST revision raised average success from 64.9% to 78.0% on Python and from 55.5% to 79.9% on the 29 JavaScript tasks. All four models gained more on JavaScript.

Q1. How much of the improvement remains when independently removing Markdown alignment, preservation constraints, or interface checks?

On 200 Controlled Repair tasks with GPT-5.6-Sol, Raw Package passed 132 tasks and AST-guided revision passed 175 in the paper. We ran each ablation on the same 200 Raw Package proposals. The last column reports each ablation’s gain over Raw as a fraction of the 43-task gain reported for full AST revision.

ConfigurationTasks passedSuccess rateNet gain over Raw (tasks)Gain retained
Raw Package132/20066.0%0—
Without final Markdown alignment134/20067.0%+24.7%
Without explicit behavior-preservation constraints144/20072.0%+1227.9%
Without public-interface signature checks145/20072.5%+1330.2%
Raw Package + AST175/20087.5%+43100%

Q2 & W2. Is there evidence that the gains transfer to naturally occurring skill failures or improve downstream agent task success after the repaired packages are reused?

Thank you for raising this question. We evaluated GPT-5.6-Sol on 50 naturally occurring faults, each from a different GitHub Skill package. Raw Package repaired 26 of the 50 faults (52%). With AST-guided revision, nine additional faults were repaired, bringing the total to 35/50 (70%).

To evaluate downstream reuse, we had DeepSeek V4 Pro perform the same user task with each Skill before and after fixes addressing downstream failures. Across all 50 Skills, single-run success (pass@1) rose from 87/150 (58.0%) to 122/150 (81.3%). The proportion of tasks completed at least once in three attempts (pass@3) rose from 35/50 (70%) to 44/50 (88%).

Dear Reviewer kjdU,

Thank you again for your detailed and constructive feedback. We have posted our responses to your comments, including clarifications, additional analyses, and new experimental results.

When convenient, we would greatly appreciate it if you could let us know whether our responses address your concerns. If you have any further questions or suggestions, we would be happy to discuss them.

Thank you for your time and consideration.

Best regards,
The Authors of SkillScriptBench

Dear Reviewer kjdU,

Thank you again for your constructive feedback. We would like to summarize the additional experiments we conducted in response to your suggestions.

1. Reading scripts versus editing them. We ran the control you suggested on all 150 In-the-Wild script-repair tasks. With the entire package visible but edits restricted to SKILL.md, GPT-5.6-Sol revised 74 documents but passed none of the 150 tasks. With the same read access and permission to edit scripts, Raw Package passed 102/150 tasks (68%). On these script faults, reading the implementation did not substitute for being able to change it.

2. Additional CoEvoSkills rounds and component ablations. Starting from the same 200 first-round CoEvoSkills outputs, two additional rounds with the original verifier increased the number of tasks passed from 133 to 144. A single AST-guided revision of those first-round outputs passed 170 tasks. We also ran the three ablations you requested, each starting from the same 200 Raw Package proposals. Without final Markdown alignment, 134 tasks passed; without explicit behavior-preservation constraints, 144 passed; and without interface checks, 145 passed. For reference, Raw Package passed 132/200 tasks, and our AST-guided method passed 175/200.

3. Fault distribution and naturally occurring faults. To check whether the most common repair category drove the overall result, we gave all eight categories equal weight. The gain over Raw Package was 13.5 percentage points, close to the 15.2-point gain when each task received equal weight. We also observed gains on both Python and JavaScript tasks. Separately, we evaluated 50 naturally occurring faults from 50 distinct GitHub Skill packages. Raw Package repaired 26/50 faults (52%). With follow-up AST-guided revision, nine more faults were repaired, for a total of 35 out of 50 (70%).

4. Downstream reuse. We evaluated DeepSeek V4 Pro on the same 50 user tasks using the Skill packages before and after targeted fixes based on downstream failure traces, with three attempts per task. Pass@1 rose from 58.0% (87/150) to 81.3% (122/150), and pass@3 rose from 70% (35/50) to 88% (44/50).

As the rebuttal period draws to a close, we would appreciate hearing whether these experiments address your concerns, particularly about the effect of script-editing access and whether repairs benefit an agent that later uses the Skill. If any question remains, or if there is a comparison you would like us to clarify, we would be happy to follow up.

Best regards,
The Authors of SkillScriptBench

Reviewer_CZCJOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that executable agent skills (SKILL.md plus scripts) need an evaluation that separates three abilities: repairing documentation, repairing scripts, and preserving behavior that is already correct. SkillScriptBench is built from real GitHub skill packages and has two tracks: an in-the-wild track with injected script faults, and a controlled track that poses the same maintenance request to a package in clean, documentation-faulty, script-faulty, and jointly faulty states. Building on what this reveals, the authors propose AST-Guided Skill Revision, which uses syntax trees and call relationships to localize script edits to requirement-relevant code and then re-aligns the documentation with the revised scripts.

Critical strengths

The controlled four-state design poses the same maintenance request to clean, documentation-faulty, script-faulty, and jointly faulty versions of a package, which cleanly separates repair from preservation. It surfaces a useful finding: editing the whole package can hurt packages that were already correct, compared with Markdown-only revision. The evaluation is careful, with repeated runs per setting, consistency metrics, and bootstrap confidence intervals.

Critical weaknesses
  • The AST-guided method appears to have been developed on the same tasks it is evaluated on, and the injected fault types (e.g., swapped arguments, bypassed helper calls) are well suited to node-level edits. Describing how the prompts were developed and reporting results on a held-out subset or on real historical fixes would make the gains more convincing.
  • Compute is not matched across methods: the Raw Package baseline is a single call, while the AST stage adds several calls plus validation and fallback. An equal-budget baseline and token costs would clarify how much of the gain comes from the structure itself.
Questions for authors
  1. Does the AST-guided stage keep a clear advantage over an equal-budget baseline, such as repeated self-revision or best-of-n with the same checks?
  2. Were the method prompts tuned on the benchmark tasks, and how does the method perform on tasks not used for tuning?
  3. How are documentation faults assessed by the executable checks, and how were those checks validated?
3 comments

We thank the reviewer for the positive feedback on the importance of our task and for the valuable suggestions. Our responses are provided below.

Q1 & W2. Does the AST-guided stage keep a clear advantage over an equal-budget baseline, such as repeated self-revision or best-of-n with the same checks?

On 200 Controlled Repair tasks with GPT-5.6-Sol, the shared Raw Package proposals achieved 63.7% success. Both methods then received a revision stage. In the w/o Structural Feedback ablation, the model had an additional opportunity to repair its scripts and update SKILL.md. This raised success to 69.2%, using an average of 98.0k input and 11.2k output tokens per revision. AST-guided revision reached 84.8% from the same proposals, averaging 45.7k input and 3.5k output tokens. These averages use provider-reported usage across all 600 runs per configuration. AST-guided revision thus achieved higher success with fewer recorded tokens.

We also took the 200 saved CoEvoSkills outputs from Controlled Repair and ran two additional rounds with the original verifier. After the two extra rounds, CoEvoSkills passed 144/200 tasks (72.0%). A single AST-guided revision of the same starting outputs passed 170/200 (85.0%).

Q2 & W1. Were the method prompts tuned on the benchmark tasks, and how does the method perform on tasks not used for tuning?

Our survey of executable Skill packages on GitHub informed the fault types in In-the-Wild Repair. We injected faults such as incorrect argument passing and skipped helper calls into repository-sourced scripts, then verified with executable checks that each fault broke the required behavior.

These cases motivated Controlled Repair, which asks whether a method can repair a faulty component while preserving correct behavior elsewhere. We hold the package and maintenance request fixed across four states: documentation faults, script faults, both, or neither.

We developed AST-Guided Skill Revision to address this problem and tested the core idea on 20 pilot tasks, none of which appears in the 350-task evaluation. The early prompt asked the model to use AST findings to make a minimal edit while preserving public interfaces and default behavior. We later made the editing instructions more specific. The model checks proposed edits against the request and source, edits only designated nodes, and keeps the original proposal when the evidence does not justify a change. We did not use failure cases or scores from the 350 evaluation tasks to tune the prompts used in the main experiments.

We also tested GPT-5.6-Sol on 50 naturally occurring faults from 50 distinct GitHub Skill packages. For each fault, we confirmed that a regression test failed on the faulty version and passed after the maintainer’s fix. The follow-up AST edits drew on failures observed in this evaluation. Raw Package repaired 26 of the 50 faults (52%). With AST-guided revision, nine additional faults were repaired, bringing the total to 35/50 (70%).

Q3. How are documentation faults assessed by the executable checks, and how were those checks validated?

We evaluate a documentation repair by following the instructions in the revised SKILL.md. The evaluator extracts the documented command and its arguments, fills in placeholders with test inputs, and runs the command against the revised scripts. It checks whether the command produces the outputs and artifacts required by the task. On documentation-fault tasks, it also runs the command from the reference documentation against the same scripts to verify that previously correct behavior still works.

We validated the executable checks on all 50 documentation-fault tasks in Controlled Repair. For each task, we ran the same checks on two packages that differed only in SKILL.md. All 50 packages passed with the reference instructions and failed with the faulty instructions. For example, changing --all to --all-results in a documented search command still produced an exit code of zero, but the check detected that the number and ranking of the returned results were incorrect.

Dear Reviewer CZCJ,

Thank you again for your detailed and constructive feedback. We have posted our responses to your comments, including clarifications, additional analyses, and new experimental results.

When convenient, we would greatly appreciate it if you could let us know whether our responses address your concerns. If you have any further questions or suggestions, we would be happy to discuss them.

Thank you for your time and consideration.

Best regards,
The Authors of SkillScriptBench

Dear Reviewer CZCJ,

Thank you again for your constructive feedback. We would like to highlight three points from our response.

First, to test whether the gains simply reflect another opportunity to revise, we started from the same 200 first-round CoEvoSkills outputs on Controlled Repair. With its original verifier, CoEvoSkills passed 133 tasks after the first round, 143 after the second, and 144 after the third. A single AST-guided revision of those same first-round outputs passed 170 tasks. Additional rounds helped CoEvoSkills, but one AST-guided revision still passed 26 more tasks than two further CoEvoSkills rounds.

Second, the 20 pilot tasks used to develop the core method were disjoint from the 350 evaluation tasks. We did not use failures or scores from those 350 tasks to tune the prompts used in the main experiments. To address your question about real faults, we also evaluated 50 naturally occurring faults from 50 distinct GitHub Skill packages. Raw Package alone repaired 26 of the 50 faults (52%). With follow-up AST-guided revision, nine more faults were repaired, for a total of 35 out of 50 (70%).

Third, we validated the documentation checks by changing only SKILL.md and holding the scripts fixed. On each of the 50 documentation-fault tasks, the same executable check passed with the correct instructions and failed with the faulty ones. For example, when a flag in a documented search command was changed, the command still exited successfully but returned the wrong number of results in the wrong order. The check detected both errors.

As the rebuttal period draws to a close, we would appreciate your thoughts on whether the additional experiments and clarifications address your concerns, particularly whether the advantage of AST-guided revision persists when CoEvoSkills is given more rounds to revise. We also hope our response makes clear how the development tasks were kept separate from the evaluation tasks and how we validated the checks for documentation faults. If any of these points remains unclear, or if there is another aspect you would like us to discuss, we would be happy to follow up.

Best regards,
The Authors of SkillScriptBench

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for your careful reading and constructive feedback. SkillScriptBench is the first benchmark to systematically evaluate the self-evolution of complete executable Skill packages, testing whether agents can repair faults in documentation and scripts, keep the documentation consistent with the implementation, and preserve correct behavior. All three reviewers (CZCJ, kjdU, and t4Ch) highlighted the value of the matched four-state design in distinguishing these abilities. Reviewers CZCJ and t4Ch emphasized a practical trade-off it reveals: giving agents broader editing access can help them repair faulty Skills while increasing the risk of breaking working ones. This finding underscores the importance of evaluating the complete Skill package as a whole. Reviewer kjdU highlighted the consistent gains from AST-guided revision across four models and both base methods. Reviewer CZCJ noted the consistency of our results across repeated runs.

We have responded to each reviewer’s questions individually. Below, we summarize the main clarifications and the additional experiments and analyses reported in Appendix E of the revised manuscript.

  • Script read and edit permissions. Following Reviewer kjdU’s suggestion, we gave GPT-5.6-Sol access to the full package on all 150 In-the-Wild script-repair tasks. When edits were restricted to SKILL.md, the model revised 74 documents, but none of the 150 packages passed the script behavior checks. With the same read access and permission to edit scripts, Raw Package repaired 102/150 cases (68.0%) (Appendix E.1).

  • What AST guidance adds beyond additional revision. In response to Reviewers CZCJ and kjdU, we compared AST-guided revision with two ways of giving GPT-5.6-Sol further opportunities to repair packages on Controlled Repair. Starting from the same 200 first-round CoEvoSkills outputs, two additional CoEvoSkills rounds with the original verifier passed 144/200 tasks (72.0%), while one AST-guided revision passed 170/200 (85.0%). In a separate comparison starting from the same Raw Package proposals, one round of text-based revision achieved 69.2% success, compared with 84.8% for one AST-guided revision. Across 600 runs per configuration, AST-guided revision also used fewer tokens per revision on average (45.7k input and 3.5k output, compared with 98.0k and 11.2k for text-based revision). Both comparisons are reported in Appendix E.1. On the 200 Controlled Repair tasks, AST-guided revision passed 175 tasks, while the ablations without final Markdown alignment, behavior-preservation constraints, or public-interface signature checks passed 134, 144, and 145 tasks, respectively (Appendix E.2).

  • Repairing naturally occurring faults and reusing Skills downstream. In response to Reviewers CZCJ and kjdU, we evaluated GPT-5.6-Sol on 50 naturally occurring faults, each from a different GitHub Skill package. Raw Package repaired 26/50 faults (52%), and follow-up AST-guided revision repaired nine more, bringing the total to 35/50 (70%). In the downstream reuse experiment, DeepSeek V4 Pro used Skill packages based on GPT-5.6-Sol’s AST-guided revisions. From pre-repair to post-repair, pass@1 rose from 58.0% (87/150) to 81.3% (122/150), and pass@3 rose from 70% (35/50) to 88% (44/50) (Appendix E.3).

  • Robustness across packages and repair partitions. Reviewer t4Ch asked how reliable the gains were given the small number of packages in each Controlled Repair partition. For each of the 50 base packages, we compared Raw Package with AST-guided revision under the same fault state, model, and run, then averaged the 36 paired differences per package. Success improved on 46 packages, was unchanged on two, and declined on two. We resampled base packages within each partition 10,000 times to estimate 95% bootstrap confidence intervals. The mean gain was 32.8 percentage points for Single-Target (20 packages, 95% CI [19.7, 46.5]), 27.6 points for Multi-Target (15 packages, [21.7, 33.3]), and 24.1 points for Cross-Call (15 packages, [14.4, 32.4]). All three intervals lay above zero (Appendix E.4).

We thank the reviewers for the suggestions that led to these additional experiments and analyses. We hope the results in Appendix E address the questions raised during the discussion. In the next revision, we will incorporate the key findings into the main text. We appreciate the time and care that the Area Chair and reviewers have devoted to our paper.

Best regards,

The Authors of SkillScriptBench

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.