- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies why procedural agent skills, represented primarily as natural-language instructions and optional scripts, often fail when supplied to small open-weight language-model executors.
The paper evaluates six executors from two model families—Qwen3.5 at 4B, 9B, and 27B, and Gemma 4 at E4B, 12B, and 31B—on 20 families and 80 held-out instances from SkillLearnBench. It compares five delivery conditions:
- no skill;
- the official on-demand skill channel;
- the official skill text placed directly in context;
- compiled tools;
- compiled tools plus a short tool card.
The main empirical findings are:
- Small Qwen executors rarely invoke the on-demand skill channel.
- Putting the skill directly in context helps, but leaves the smallest executor far below the larger models.
- The authors observe failures in self-produced intermediate results, such as incorrect file manipulations, unverified assumptions about PDF forms, and truncated or incomplete data extraction.
- A prefix intervention suggests that the next step is more likely to succeed when the previous intermediate result is clean.
- In a handoff experiment, a 27B executor performs worse after inheriting a 4B executor’s partial state than after producing the partial state itself.
- Compiling each skill into registered tools substantially increases pass rates and reduces the number of actions.
- The effect replicates on a second model family and is much stronger for registered executable tools than for a textual description of the same nominal procedure.
The paper’s proposed method is a one-time compilation pipeline: a strong Claude Opus producer reads the public skill package and a development instance, builds tools under a generation contract, an independent reviewer checks them, the producer revises once, and the frozen suite is then evaluated on held-out instances.
The raw benchmark result is substantial. The methodological interpretation is less secure because the compiled tools add task-specific computation and a specialized interface, while the baseline skill is not provided with equivalent executable affordances or an equal-information prompt representation.
Critical strengths
Strength 1: Strong empirical engineering and triangulation
The paper does more than report a single pass-rate comparison. It combines:
- skill-channel versus in-context comparisons;
- trajectory inspection;
- prefix interventions;
- a preregistered handoff experiment;
- compiled-tool ablations;
- a second model-family replication;
- per-family breakdowns;
- a contamination and integrity audit.
This is substantially stronger than a conventional “prompt A versus prompt B” agent paper. The central benchmark effect is large and appears in both Qwen and Gemma conditions.
Strength 2: Clear diagnosis of an important practical failure mode
The distinction between procedural knowledge and the ability to construct reliable intermediate artifacts is practically meaningful. The examples involving PDF forms, large files, video frames, and enterprise search make the failure mode concrete. The observation that registered tools reduce both failure rates and action counts could be useful to practitioners building local agents under strict compute or context budgets.
Critical weaknesses
Weakness 1: The main comparison is confounded by privileged, task-specific executable solvers
The paper describes the method as compiling skills into tools, but the experimental treatment is considerably stronger than that description suggests. The producer is a strong Claude Opus 5 agent that receives:
- the public skill package;
- a development instance;
- a family-specific note about likely failure points and tool decomposition;
- a generation contract;
- a runtime description;
- the ability to inspect files and run code;
- a review and revision stage.
The resulting MCP tools can fill forms, parse files, perform computations, render artifacts, verify requirements, and return compact reports. In some families, this likely removes much of the hardest deterministic work from the executor. The court-form tool, for example, performs the mapping from case facts to PDF fields and then checks the completed document.
Thus, the key comparison is partly between an unoptimized textual procedure and a privileged executable task solution with a highly engineered interface. The result is important, but it does not establish that the causal mechanism is specifically “clean intermediate results” or that the missing ingredient is tools rather than knowledge. A fair assessment would require matched controls such as an official skill exposed through the same MCP interface, the original skill’s scripts registered as tools, an equal-length textual summary with an equivalent card, a generic code executor, and tools generated without the development instance or family note.
Weakness 2: Limited generalization and overconfident statistical interpretation
Only 5 of the 20 families, totaling 13 held-out instances, were fixed in advance as evaluation families for method design. The other 15 families were used while developing and iterating the contract and family notes. This is acceptable as an exploratory development protocol, but it does not justify broad claims about skill execution in general.
The statistical evidence also needs a more careful unit of analysis. There is usually one stochastic trial per instance and only 1–5 instances per family. Instances from the same family are likely correlated because they share task templates, file layouts, and tool behavior. The handoff analysis is especially problematic: the same 30 instances are evaluated at both and , but the paper pools the 23 versus 5 discordant outcomes into a sign test with . The two conditions are repeated measurements on the same instances, so a cluster-aware or repeated-measures analysis is needed. The prefix intervention also treats repeated samples from the same prefix as if they provided independent evidence unless the analysis explicitly clusters by prefix.
The paper should report family-clustered confidence intervals and tests, all seeds or repeated trials for every central condition, per-family effect sizes, and a corrected handoff analysis.
Questions for authors
Clarification requests that could materially change my assessment
-
What fraction of each tool suite is task-solving computation?
Please report tool source code size, execution time, number of file operations, and a per-tool classification of generic mechanics, deterministic transformation, verification, and task-specific solution logic. For the court-form and similar tasks, how many grader-relevant operations are performed entirely inside the tool? -
How were the family notes produced?
Who wrote the family notes, how much effort did they require, and were any held-out results or failed held-out trajectories used to write or revise them? The abstract describes compilation from the public skill package and one development instance, but the method also gives the producer a family-specific note about likely derailment points. This additional supervision should be precisely characterized. -
Can the authors provide matched interface baselines?
In particular:- the official skill registered through the same MCP mechanism;
- the official skill’s own scripts exposed as registered tools;
- a concise skill summary with an equal token budget and equivalent card;
- a generic code-execution tool;
- a tool suite generated without the development instance and family note.
These controls would separate executable computation, discoverability, prompt compression, and tool schema effects.
-
Can the statistical analysis be repeated with family-level clustering?
Please report a family-clustered analysis for all main comparisons, including the five evaluation families separately. For the handoff study, please analyze and separately or use a repeated-instance permutation or mixed-effects test that respects the reuse of the same 30 instances. -
How much does the proxy alter the benchmark behavior?
The proxy folds mid-conversation system messages, changes PDF observations, clamps tokens, and translates between APIs. Have the skill-channel access results been reproduced under the native harness or another implementation? If not, the claim that small executors generally fail to access skills should be limited to this proxy and harness. -
Why is the development gate allowed to substitute a 9B executor when the 4B executor does not call the tools?
How many families required this substitution, and do the conclusions change if every compiled suite must first be adopted by the target executor itself? -
How much wall-clock and compute budget is actually available after the first handoff segment?
The handoff uses the same remaining turn and wall-clock budgets, but the two model sizes may spend different amounts of time generating and executing actions. Please report elapsed time and token usage by segment and show that the handoff result is not explained by unequal remaining compute.
Suggestions for future work
These are not requests for a positive score in the current version, but they would make the contribution substantially stronger:
- Evaluate unseen task families and at least one independent skill benchmark.
- Replicate all central conditions across multiple seeds or repeated runs.
- Randomize clean and messy intermediate states while matching context length and semantics.
- Separate deterministic artifact production from judgment-bearing work using an explicit tool-computation budget.
- Test reverse handoffs, such as a larger executor producing state for a smaller executor.
- Compare native tool registration with simpler automatic script execution under matched action budgets.
3 comments
Response to Reviewer MdRM (2/2)
Question 1: Tool computation profile
Claude Opus and Sonnet independently classified all 52 tools (about 36k non-comment lines; median 1.4k lines per suite, median local tool runtime 6.8 s per trial excluding network-API families) with line-level evidence (role agreement kappa = 0.79):
- Line breakdown (Opus / Sonnet): generic mechanics 31% / 27%, deterministic transformation 28% / 26%, verification 16% / 20%, task-specific solution logic 26% / 27%.
- Source of the remaining (non-generic) code: skill package 17% / 18%, development-instance prompt 27% / 28%, general domain/library knowledge 48% / 46%, development-instance defaults 7%.
- Grader-relevant operations: Pooled across all three Qwen executors (4B/9B/27B, 182 passing Tools + card trials total), 59% contain zero own Bash/file actions by the executor, and in 17% the executor writes the deliverable itself. In Court-form filling (1,456 lines, 1.1 s tool runtime), the executor performs 0 file actions in all 13/13 passing Qwen trials across the three sizes: it extracts a median of 21 case facts from the prompt, and the tool maps them to PDF fields and writes /root/sc100-filled.pdf. We will add the per-suite table to the appendix.
Question 2: How the family notes were produced
Before each producer session, an automated Claude Opus 5 call generated the family note and producer prompt from the development instance (instance 1) and public skill package, with no human authoring or editing of individual notes. Six development families received a standalone note file and 14 received it inside the producer prompt; in 10 families, including the 5 evaluation families, that prompt also proposed tool names and roles. All prompts are in the supplementary material, and the General Response discloses the two development prompts that leaked held-out examples. We will state this automated step in the abstract and Section 3.1, and list compiling without the note or development instance as future work.
Question 3: Matched interface baselines
- Official scripts registered as MCP tools (pre-registered Nov 7, Qwen 4B): Across the 20 official packages, 11 ship no script, 4 ship generic or missing-dependency scripts, and 5 families (23 held-out instances) ship task-relevant runnable scripts. Mechanically registering every entry-point script as an MCP tool alongside Skill in context raises tool calls to 15/23 trials (vs. 12/23 running a script via Bash in Skill in context), but passes 4/23 vs. 5/23 for Skill in context (2:3, p = 1.00) and 16/23 for Tools + card (13:1, p = 0.0018; 13/18 vs. 4/18 without Weighted GDP). Official scripts cover a median of only 5–8% of a compiled suite's logic.
- Generic code execution: Every condition runs in Claude Code with Bash and file-editing tools enabled.
Question 5: Proxy effects on skill access
Under the same API translation proxy and Claude Code harness, the executors call registered MCP tools in 96–100% of trials, vs. 1/80, 1/80, and 8/80 Skill-tool calls at Qwen 4B/9B/27B (29/80, 13/80, and 23/80 on Gemma). Since message folding could still interact with Skill prompts, we will scope Section 2.2's access claim explicitly to this proxy and harness.
Question 6: 9B substitution at the development gate
Logs from the pre-freeze check on development instance 1 confirm that the 4B target executor itself called the tools in all 20 frozen suites; the Appendix A note about 9B referred to two superseded suites and will be corrected. Excluding the 3 families where 4B called the tools on instance 1 without passing its tests leaves 4B held-out results unchanged (51/71 vs. 7/71, family-clustered p = 1.5e-5).
Question 7: Remaining budget after the first handoff segment
Across the 30 handoff instances (1,800 s and 100-turn cap per trial), 4B spends less wall-clock time in segment 1 than 27B at both K=4 (median 25 s vs. 33 s; 877 vs. 574 output tokens) and K=8 (67 s vs. 150 s; 1,927 vs. 2,284 output tokens), leaving the 27B receiver more remaining wall-clock time after 4B. No trial reached the 100-turn limit. In segment 2, the receiver works longer and consumes more tokens after 4B than after 27B (K=8: median 140 s, 10.5 turns, 309k input / 2.7k output tokens vs. 63 s, 3.5 turns, 118k input / 1.1k output tokens; K=4: 216 s, 417k / 3.6k tokens vs. 201 s, 289k / 2.5k tokens) and hits the 1,800 s timeout in 2/30 (K=4) and 4/30 (K=8) trials vs. 0/30 after 27B. Unequal remaining compute does not explain the gap.
Brief Update: Full n=80 Controls and ReAct Demonstration Baseline
Two quick updates on the controls from our earlier response before discussion closes.
We finished the remaining 4 GitHub analytics instances for the neutral-wording CLI script control on Qwen 4B, bringing every row to the full n=80: neutral-wording scripts pass 32/80 (run 1) and 35/80 (run 2) vs. 53/80 for Tools + card (discordant pairs 25:4, p = 0.00010, family-clustered p = 0.0043; run 2: 24:6, p = 0.0014, family-clustered p = 0.011; without DBSCAN tuning and Weighted GDP: 47/71 vs. 30/71 and 34/71, family-clustered p = 0.013 and 0.039).
We also ran a ReAct trajectory demonstration baseline across all 80 held-out instances for Reviewer 9Kkn (using the same official skill package and development instance 1). It passes 20/80 on Qwen 4B and 23/80 on Qwen 9B, compared to 53/80 and 60/80 for Tools + card (family-clustered p = 0.0011 and 6.1e-5). Across matched controls on Qwen 4B, passes go from 9/80 (Skill in context) to 20/80 (ReAct demo) to 32-35/80 (neutral-wording CLI scripts) to 53/80 (Tools + card).
Happy to answer any remaining questions before the deadline.
Response to Reviewer MdRM (1/2)
Thank you for the clear, rigorous review and for specifying the exact analyses and controls needed to evaluate our claims.
Weakness 1: Privileged task-specific solvers vs. tool interface
We agree that compiled tools add substantial executable computation and that the paper's closing claim should reflect both factors. We separated code content from interface registration with two controls:
First, same producer code, script vs. MCP tool interface. In Table 3, row (c), where the exact same compiled code is offered as CLI scripts instead of registered MCP tools, Qwen 4B/9B/27B pass 26/39, 33/48, and 57/67 trials when they run a script, and 0/41, 1/32, and 6/13 when they do not. Logs showed that row (c)'s "# Skill installed..." heading in CLAUDE.md diverted 4B into calling Claude Code's built-in Skill tool instead of running the scripts in 36/80 trials. We ran a pre-registered 4B control changing only those three framing lines to "# Scripts installed..." across two runs (n = 76; the 4 GitHub analytics instances failed in November reruns due to an expired external GH_TOKEN in the container image):
Neutral framing raises script execution to 75–80%, yet Tools + card still leads by 16–18 passes (discordant pairs 22:4, p = 0.0005, and 22:6, p = 0.0037; without DBSCAN and Weighted GDP: 44/67 vs. 30/67 and 33/67). In the remaining gap, shell callers pass malformed JSON arguments without schema validation and abandon the script, or run only the checker script.
Second, gains stratified by tool autonomy. In our code audit (Question 1, Claude Opus and Sonnet), both models independently labeled whether calling a suite's tools without reading the task input can pass all, some, or none of a family's held-out instances. On the 19 agreed families (76 instances), 4B passes (Tools + card vs. Skill in context) are 10 vs. 0 for "all" (12 instances), 29 vs. 6 for "some" (47 instances), and 11 vs. 3 for "none" (17 instances: Poster design, Financial analysis, GitHub analytics, Chinese poem). On the 20th family (Dependency check, labeled "all" vs. "some"), 4B passes 3/4 vs. 0/4.
Revision change. We will revise the conclusion to state two separate findings supported by the controls: small executors need executable step-level computation beyond prose or thin helpers, and typed MCP registration improves both adoption and call reliability over CLI scripts.
Weakness 2 & Question 4: Family-level clustering and repeated-measures handoff
We repeated all statistical tests at the family or prefix level: