- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SkillScriptBench, a benchmark targeting the self-evolution capability of executable agent skill packages, containing SKILL.md documentation and scripts/ executable scripts, motivated by the observation that existing work fails to distinguish three independent dimensions: "documentation repair," "script repair," and "correct behavior preservation." The benchmark contains 350 tasks: an In-the-Wild Repair track and a Controlled Repair track. The paper also contributes an AST-Guided Skill Revision method: behavioral requirements in maintenance requests are bound via an LLM to language-native AST/CST nodes, combined with call-relationship context across functions, and script edits are constrained by strict node-count/byte-count limits and interface-signature preservation checks, followed by reference-aligned editing of the Markdown. Experiments on four model backbones compare Markdown-only, Raw Package, CoEvoSkills, and their respective versions with AST-Guided Revision added.
Critical strengths
The paper systematically disentangles three previously conflated evaluation dimensions, documentation repair / script repair / behavior preservation, and reveals a trade-off relationship of real practical significance. The Controlled Repair track, through a strictly controlled design that fixes the same maintenance request across four states, demonstrates that the repair-capability gains from "full package access" come at the cost of behavior preservation on Clean tasks (a gap of up to 7.8/3.7 percentage points)—a previously underappreciated finding with real implications for skill maintenance system design.
Critical weaknesses
-
The sample size for the Controlled Repair subcategory is too small (some repair categories have only 3-9 tasks). The authors need to discuss the impact on the robustness of fine-grained conclusions.
-
Were the benchmarks built from over 35,000 GitHub repositories rigorously filtered to ensure no data leakage? The tested model corpus may contain overlapping contamination; the authors did not provide details on how this was handled.
-
The core steps of AST-Guided Skill Revision still heavily rely on an LLM to complete inference decisions in the prompt. The authors should provide more details on the effectiveness of this design and more ablation experiments to verify the innovativeness of this part of the method.
Questions for authors
-
Has the overlap between the 35,554 GitHub skill repositories crawled to build the benchmark and the pretraining corpora of the evaluated LLMs been assessed and filtered?
-
Could a case study or error analysis be provided to demonstrate that the improvement from AST-Guided Revision mainly comes from the AST structural constraint itself?
-
The Single-Target/Multi-Target/Cross-Call partitions in Controlled Repair have only 20/15/15 base packages respectively, and some repair categories have extremely small sample sizes (e.g., 3 tasks)—how statistically significant are the fine-grained percentage-point improvements reported in §5.2 given such small samples?
4 comments
Thank you for your reply, I will keep my score.
Dear Reviewer t4Ch,
Thank you for taking the time to consider our response. We would appreciate knowing whether we have fully addressed your concerns. If any point remains unresolved, please let us know. We would be happy to address it in more detail.
Best regards,
The Authors of SkillScriptBench
Thank you for your response. After careful consideration, I have decided to keep the score.
Response
We thank the reviewer for the positive feedback on the importance of our task and for the valuable suggestions. Our responses are provided below.
We did not screen the GitHub packages for overlap with model pretraining data. Raw Package and AST-guided revision use the same model on the same faulty package. If familiarity with the upstream code helps Raw repair a fault, that repair already counts toward the baseline, leaving less room for AST-guided revision to improve on it. If familiarity benefits Raw more than it benefits AST-guided revision, the gain from adding AST revision could be larger on unfamiliar packages.
In the VCP stock-analysis case, GPT-5.6-Sol’s Raw Package left six argument-binding errors across four calls. One illustrates why the fault was easy to miss. The user reported that changing one setting affected an unrelated calculation. Raw had the full code, but the symptom did not identify the faulty call. When
analyze_stockcalledcalculate_vcp_pattern, it passedmin_contraction_daysto bothmin_contraction_daysandlookback_days. Both values represent days, so the call ran without an error even though the user’s lookback setting never reached the calculator.The AST analysis resolved the call and traced each argument to its source in
analyze_stock. This exposed the mismatch: both parameters received the same variable even though the caller had a separatelookback_daysvalue. The structural feedback showed the editor the call site, the value passed, and the receiving parameter. The editor then corrected the six bindings at the call sites. On the same price data, shortening the lookback window from 120 to 30 days reduced the number of detected contractions as requested. The repaired package passed 10/10 behavioral checks, up from 4/10 with Raw.The ablation shows why caller–callee context matters beyond this example. Starting from the same Raw Package proposals on Controlled Repair’s Joint tasks, an additional text-based revision achieved 56.7% success. AST node editing without caller–callee context achieved 54.7%, compared with 81.3% for full AST-guided revision. Removing call context while retaining node editing reduced success by 26.6 percentage points.
We compared Raw Package and Raw Package + AST on the same base packages in each Controlled Repair partition. For each package, we paired outcomes by fault state, model, and run, and averaged the 36 paired differences (three fault states × four models × three runs). We then resampled base packages within each partition 10,000 times to estimate a 95% bootstrap interval for the mean gain.
AST-guided revision improved success on 46 of the 50 base packages. The other four comprised two ties and two declines. In all three partitions, the 95% interval for the mean gain lay above zero. For repair categories with only three tasks, we report the observed success rates descriptively.