acceptodds
Under review as a conference paper at ICLR 2027

Small Executors Derail on Intermediates They Produced Themselves: Compiling Skills into Tools They Can Act On

Abstract

Agent skills package procedural knowledge as natural-language instructions that an LLM agent loads at inference time, and are expected to carry over to any model without retraining. We find that they largely fail to reach small open executors. Offered through the standard skill channel, skills are often left unopened; placed directly in context, they are read and still fail. Small executors derail not on understanding the procedure but on the intermediate results they must produce themselves to carry it out. In controlled interventions, once these results are messy, the next step fails for executors of every size, and in a pre-registered handoff, a larger executor that takes over a small executor's state fails more often than after its own turns. We therefore compile each skill into tools that return clean intermediate results, once per task family from the public skill package and a single development instance, while the executor still decides which tools to call and how to finish. On SkillLearnBench, compiled tools raise a 4B executor from 9 to 53 of 80 held-out instances, above a 27B executor given the skill in context, and the gains replicate on a second model family. Written out as text, the same suite does not help: what small executors lack is not the knowledge in a skill, but tools they can act on.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent skills, small language models, tool use, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MdRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why procedural agent skills, represented primarily as natural-language instructions and optional scripts, often fail when supplied to small open-weight language-model executors.

The paper evaluates six executors from two model families—Qwen3.5 at 4B, 9B, and 27B, and Gemma 4 at E4B, 12B, and 31B—on 20 families and 80 held-out instances from SkillLearnBench. It compares five delivery conditions:

  • no skill;
  • the official on-demand skill channel;
  • the official skill text placed directly in context;
  • compiled tools;
  • compiled tools plus a short tool card.

The main empirical findings are:

  1. Small Qwen executors rarely invoke the on-demand skill channel.
  2. Putting the skill directly in context helps, but leaves the smallest executor far below the larger models.
  3. The authors observe failures in self-produced intermediate results, such as incorrect file manipulations, unverified assumptions about PDF forms, and truncated or incomplete data extraction.
  4. A prefix intervention suggests that the next step is more likely to succeed when the previous intermediate result is clean.
  5. In a handoff experiment, a 27B executor performs worse after inheriting a 4B executor’s partial state than after producing the partial state itself.
  6. Compiling each skill into registered tools substantially increases pass rates and reduces the number of actions.
  7. The effect replicates on a second model family and is much stronger for registered executable tools than for a textual description of the same nominal procedure.

The paper’s proposed method is a one-time compilation pipeline: a strong Claude Opus producer reads the public skill package and a development instance, builds tools under a generation contract, an independent reviewer checks them, the producer revises once, and the frozen suite is then evaluated on held-out instances.

The raw benchmark result is substantial. The methodological interpretation is less secure because the compiled tools add task-specific computation and a specialized interface, while the baseline skill is not provided with equivalent executable affordances or an equal-information prompt representation.

Critical strengths

Strength 1: Strong empirical engineering and triangulation

The paper does more than report a single pass-rate comparison. It combines:

  • skill-channel versus in-context comparisons;
  • trajectory inspection;
  • prefix interventions;
  • a preregistered handoff experiment;
  • compiled-tool ablations;
  • a second model-family replication;
  • per-family breakdowns;
  • a contamination and integrity audit.

This is substantially stronger than a conventional “prompt A versus prompt B” agent paper. The central benchmark effect is large and appears in both Qwen and Gemma conditions.

Strength 2: Clear diagnosis of an important practical failure mode

The distinction between procedural knowledge and the ability to construct reliable intermediate artifacts is practically meaningful. The examples involving PDF forms, large files, video frames, and enterprise search make the failure mode concrete. The observation that registered tools reduce both failure rates and action counts could be useful to practitioners building local agents under strict compute or context budgets.

Critical weaknesses

Weakness 1: The main comparison is confounded by privileged, task-specific executable solvers

The paper describes the method as compiling skills into tools, but the experimental treatment is considerably stronger than that description suggests. The producer is a strong Claude Opus 5 agent that receives:

  • the public skill package;
  • a development instance;
  • a family-specific note about likely failure points and tool decomposition;
  • a generation contract;
  • a runtime description;
  • the ability to inspect files and run code;
  • a review and revision stage.

The resulting MCP tools can fill forms, parse files, perform computations, render artifacts, verify requirements, and return compact reports. In some families, this likely removes much of the hardest deterministic work from the executor. The court-form tool, for example, performs the mapping from case facts to PDF fields and then checks the completed document.

Thus, the key comparison is partly between an unoptimized textual procedure and a privileged executable task solution with a highly engineered interface. The result is important, but it does not establish that the causal mechanism is specifically “clean intermediate results” or that the missing ingredient is tools rather than knowledge. A fair assessment would require matched controls such as an official skill exposed through the same MCP interface, the original skill’s scripts registered as tools, an equal-length textual summary with an equivalent card, a generic code executor, and tools generated without the development instance or family note.

Weakness 2: Limited generalization and overconfident statistical interpretation

Only 5 of the 20 families, totaling 13 held-out instances, were fixed in advance as evaluation families for method design. The other 15 families were used while developing and iterating the contract and family notes. This is acceptable as an exploratory development protocol, but it does not justify broad claims about skill execution in general.

The statistical evidence also needs a more careful unit of analysis. There is usually one stochastic trial per instance and only 1–5 instances per family. Instances from the same family are likely correlated because they share task templates, file layouts, and tool behavior. The handoff analysis is especially problematic: the same 30 instances are evaluated at both K=4K=4 and K=8K=8, but the paper pools the 23 versus 5 discordant outcomes into a sign test with p=0.0009p=0.0009. The two KK conditions are repeated measurements on the same instances, so a cluster-aware or repeated-measures analysis is needed. The prefix intervention also treats repeated samples from the same prefix as if they provided independent evidence unless the analysis explicitly clusters by prefix.

The paper should report family-clustered confidence intervals and tests, all seeds or repeated trials for every central condition, per-family effect sizes, and a corrected handoff analysis.

Questions for authors

Clarification requests that could materially change my assessment

  1. What fraction of each tool suite is task-solving computation?
    Please report tool source code size, execution time, number of file operations, and a per-tool classification of generic mechanics, deterministic transformation, verification, and task-specific solution logic. For the court-form and similar tasks, how many grader-relevant operations are performed entirely inside the tool?

  2. How were the family notes produced?
    Who wrote the family notes, how much effort did they require, and were any held-out results or failed held-out trajectories used to write or revise them? The abstract describes compilation from the public skill package and one development instance, but the method also gives the producer a family-specific note about likely derailment points. This additional supervision should be precisely characterized.

  3. Can the authors provide matched interface baselines?
    In particular:

    • the official skill registered through the same MCP mechanism;
    • the official skill’s own scripts exposed as registered tools;
    • a concise skill summary with an equal token budget and equivalent card;
    • a generic code-execution tool;
    • a tool suite generated without the development instance and family note.

    These controls would separate executable computation, discoverability, prompt compression, and tool schema effects.

  4. Can the statistical analysis be repeated with family-level clustering?
    Please report a family-clustered analysis for all main comparisons, including the five evaluation families separately. For the handoff study, please analyze K=4K=4 and K=8K=8 separately or use a repeated-instance permutation or mixed-effects test that respects the reuse of the same 30 instances.

  5. How much does the proxy alter the benchmark behavior?
    The proxy folds mid-conversation system messages, changes PDF observations, clamps tokens, and translates between APIs. Have the skill-channel access results been reproduced under the native harness or another implementation? If not, the claim that small executors generally fail to access skills should be limited to this proxy and harness.

  6. Why is the development gate allowed to substitute a 9B executor when the 4B executor does not call the tools?
    How many families required this substitution, and do the conclusions change if every compiled suite must first be adopted by the target executor itself?

  7. How much wall-clock and compute budget is actually available after the first handoff segment?
    The handoff uses the same remaining turn and wall-clock budgets, but the two model sizes may spend different amounts of time generating and executing actions. Please report elapsed time and token usage by segment and show that the handoff result is not explained by unequal remaining compute.

Suggestions for future work

These are not requests for a positive score in the current version, but they would make the contribution substantially stronger:

  • Evaluate unseen task families and at least one independent skill benchmark.
  • Replicate all central conditions across multiple seeds or repeated runs.
  • Randomize clean and messy intermediate states while matching context length and semantics.
  • Separate deterministic artifact production from judgment-bearing work using an explicit tool-computation budget.
  • Test reverse handoffs, such as a larger executor producing state for a smaller executor.
  • Compare native tool registration with simpler automatic script execution under matched action budgets.
3 comments
AuthorsOpenReview ↗

Response to Reviewer MdRM (1/2)

Thank you for the clear, rigorous review and for specifying the exact analyses and controls needed to evaluate our claims.

Weakness 1: Privileged task-specific solvers vs. tool interface

We agree that compiled tools add substantial executable computation and that the paper's closing claim should reflect both factors. We separated code content from interface registration with two controls:

First, same producer code, script vs. MCP tool interface. In Table 3, row (c), where the exact same compiled code is offered as CLI scripts instead of registered MCP tools, Qwen 4B/9B/27B pass 26/39, 33/48, and 57/67 trials when they run a script, and 0/41, 1/32, and 6/13 when they do not. Logs showed that row (c)'s "# Skill installed..." heading in CLAUDE.md diverted 4B into calling Claude Code's built-in Skill tool instead of running the scripts in 36/80 trials. We ran a pre-registered 4B control changing only those three framing lines to "# Scripts installed..." across two runs (n = 76; the 4 GitHub analytics instances failed in November reruns due to an expired external GH_TOKEN in the container image):

Qwen 4B on 76 paired instancesCalls Skill firstRuns at least 1 scriptTotal passes
Row (c) scripts, original / rerun36 / 3639 / 3926 / 23
Neutral-wording scripts, run 1 / run 23 / 757 / 6132 / 34
Tools + card (registered MCP tools)----50

Neutral framing raises script execution to 75–80%, yet Tools + card still leads by 16–18 passes (discordant pairs 22:4, p = 0.0005, and 22:6, p = 0.0037; without DBSCAN and Weighted GDP: 44/67 vs. 30/67 and 33/67). In the remaining gap, shell callers pass malformed JSON arguments without schema validation and abandon the script, or run only the checker script.

Second, gains stratified by tool autonomy. In our code audit (Question 1, Claude Opus and Sonnet), both models independently labeled whether calling a suite's tools without reading the task input can pass all, some, or none of a family's held-out instances. On the 19 agreed families (76 instances), 4B passes (Tools + card vs. Skill in context) are 10 vs. 0 for "all" (12 instances), 29 vs. 6 for "some" (47 instances), and 11 vs. 3 for "none" (17 instances: Poster design, Financial analysis, GitHub analytics, Chinese poem). On the 20th family (Dependency check, labeled "all" vs. "some"), 4B passes 3/4 vs. 0/4.

Revision change. We will revise the conclusion to state two separate findings supported by the controls: small executors need executable step-level computation beyond prose or thin helpers, and typed MCP registration improves both adoption and call reliability over CLI scripts.

Weakness 2 & Question 4: Family-level clustering and repeated-measures handoff

We repeated all statistical tests at the family or prefix level:

Main comparison (Tools + card vs. Skill in context)Qwen 4BQwen 9BQwen 27BGemma E4BGemma 12BGemma 31B
Family-clustered p (20 families, n=80)1.5e-51.5e-40.00491.5e-50.00106.1e-5
95% family-bootstrap CI, points (n=80)[42, 68][33, 63][14, 51][45, 72][19, 54][33, 64]
5 evaluation families, passes (Table 7, n=13)8 vs. 38 vs. 49 vs. 55 vs. 08 vs. 410 vs. 5
  • Repeated runs: Averaging the two independent Qwen runs gives family-clustered p between 4e-6 and 2e-4.
  • Evaluation families (Table 7): Pooled across executors by family, 4 of 5 families improve and 0 worsen (95% CI [15.5, 59.3] points). Because 5 families have a minimum two-sided sign-flip p of 0.0625, we will move Table 7 to the main text explicitly as directional evidence on untouched families.
  • Handoff (Section 2.4, 30 instances, 11 families): Analyzed separately, K=8 is 24/30 vs. 13/30 (discordant 13:2, p = 0.0074, 0.015 Bonferroni; family-clustered p = 0.031); K=4 is 26/30 vs. 19/30 (10:3, p = 0.092). Averaging over K=4 and K=8 so each of the 30 instances contributes once: 15 favor 27B->27B, 4 favor 4B->27B, 11 tie (exact sign-flip p = 0.0052; family-clustered p = 0.027). We will replace the pooled p = 0.0009 test with these numbers.
  • Prefix intervention (Section 2.3): Clustering at the prefix level (6 clean, 19 messy trajectories across 4 instances of Enterprise search) and permuting labels within instance gives p = 0.021, 0.045, 0.009 at 4B/9B/27B. We will note in the main text that clean vs. messy in this single-family study uses an observation-length cutoff (under 3,500 chars, separating compact extracted post lists from raw or truncated table dumps).
AuthorsOpenReview ↗

Response to Reviewer MdRM (2/2)

Question 1: Tool computation profile

Claude Opus and Sonnet independently classified all 52 tools (about 36k non-comment lines; median 1.4k lines per suite, median local tool runtime 6.8 s per trial excluding network-API families) with line-level evidence (role agreement kappa = 0.79):

  • Line breakdown (Opus / Sonnet): generic mechanics 31% / 27%, deterministic transformation 28% / 26%, verification 16% / 20%, task-specific solution logic 26% / 27%.
  • Source of the remaining (non-generic) code: skill package 17% / 18%, development-instance prompt 27% / 28%, general domain/library knowledge 48% / 46%, development-instance defaults 7%.
  • Grader-relevant operations: Pooled across all three Qwen executors (4B/9B/27B, 182 passing Tools + card trials total), 59% contain zero own Bash/file actions by the executor, and in 17% the executor writes the deliverable itself. In Court-form filling (1,456 lines, 1.1 s tool runtime), the executor performs 0 file actions in all 13/13 passing Qwen trials across the three sizes: it extracts a median of 21 case facts from the prompt, and the tool maps them to PDF fields and writes /root/sc100-filled.pdf. We will add the per-suite table to the appendix.

Question 2: How the family notes were produced

Before each producer session, an automated Claude Opus 5 call generated the family note and producer prompt from the development instance (instance 1) and public skill package, with no human authoring or editing of individual notes. Six development families received a standalone note file and 14 received it inside the producer prompt; in 10 families, including the 5 evaluation families, that prompt also proposed tool names and roles. All prompts are in the supplementary material, and the General Response discloses the two development prompts that leaked held-out examples. We will state this automated step in the abstract and Section 3.1, and list compiling without the note or development instance as future work.

Question 3: Matched interface baselines

  • Official scripts registered as MCP tools (pre-registered Nov 7, Qwen 4B): Across the 20 official packages, 11 ship no script, 4 ship generic or missing-dependency scripts, and 5 families (23 held-out instances) ship task-relevant runnable scripts. Mechanically registering every entry-point script as an MCP tool alongside Skill in context raises tool calls to 15/23 trials (vs. 12/23 running a script via Bash in Skill in context), but passes 4/23 vs. 5/23 for Skill in context (2:3, p = 1.00) and 16/23 for Tools + card (13:1, p = 0.0018; 13/18 vs. 4/18 without Weighted GDP). Official scripts cover a median of only 5–8% of a compiled suite's logic.
  • Generic code execution: Every condition runs in Claude Code with Bash and file-editing tools enabled.

Question 5: Proxy effects on skill access

Under the same API translation proxy and Claude Code harness, the executors call registered MCP tools in 96–100% of trials, vs. 1/80, 1/80, and 8/80 Skill-tool calls at Qwen 4B/9B/27B (29/80, 13/80, and 23/80 on Gemma). Since message folding could still interact with Skill prompts, we will scope Section 2.2's access claim explicitly to this proxy and harness.

Question 6: 9B substitution at the development gate

Logs from the pre-freeze check on development instance 1 confirm that the 4B target executor itself called the tools in all 20 frozen suites; the Appendix A note about 9B referred to two superseded suites and will be corrected. Excluding the 3 families where 4B called the tools on instance 1 without passing its tests leaves 4B held-out results unchanged (51/71 vs. 7/71, family-clustered p = 1.5e-5).

Question 7: Remaining budget after the first handoff segment

Across the 30 handoff instances (1,800 s and 100-turn cap per trial), 4B spends less wall-clock time in segment 1 than 27B at both K=4 (median 25 s vs. 33 s; 877 vs. 574 output tokens) and K=8 (67 s vs. 150 s; 1,927 vs. 2,284 output tokens), leaving the 27B receiver more remaining wall-clock time after 4B. No trial reached the 100-turn limit. In segment 2, the receiver works longer and consumes more tokens after 4B than after 27B (K=8: median 140 s, 10.5 turns, 309k input / 2.7k output tokens vs. 63 s, 3.5 turns, 118k input / 1.1k output tokens; K=4: 216 s, 417k / 3.6k tokens vs. 201 s, 289k / 2.5k tokens) and hits the 1,800 s timeout in 2/30 (K=4) and 4/30 (K=8) trials vs. 0/30 after 27B. Unequal remaining compute does not explain the gap.

AuthorsOpenReview ↗

Brief Update: Full n=80 Controls and ReAct Demonstration Baseline

Two quick updates on the controls from our earlier response before discussion closes.

We finished the remaining 4 GitHub analytics instances for the neutral-wording CLI script control on Qwen 4B, bringing every row to the full n=80: neutral-wording scripts pass 32/80 (run 1) and 35/80 (run 2) vs. 53/80 for Tools + card (discordant pairs 25:4, p = 0.00010, family-clustered p = 0.0043; run 2: 24:6, p = 0.0014, family-clustered p = 0.011; without DBSCAN tuning and Weighted GDP: 47/71 vs. 30/71 and 34/71, family-clustered p = 0.013 and 0.039).

We also ran a ReAct trajectory demonstration baseline across all 80 held-out instances for Reviewer 9Kkn (using the same official skill package and development instance 1). It passes 20/80 on Qwen 4B and 23/80 on Qwen 9B, compared to 53/80 and 60/80 for Tools + card (family-clustered p = 0.0011 and 6.1e-5). Across matched controls on Qwen 4B, passes go from 9/80 (Skill in context) to 20/80 (ReAct demo) to 32-35/80 (neutral-wording CLI scripts) to 53/80 (Tools + card).

Happy to answer any remaining questions before the deadline.

Reviewer_9KknOpenReview ↗
Rating
4(+3)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper makes a good observations that small language models derail early from skill defined workflows for agentic tasks, and it is hard for them to recover.

They also find that the models are not good at invoking the skills in the first place.

The paper proposes offering skill workflow steps as executable tools to the models.

They find that this improves model's task completion and workflow following.

Critical strengths

The observations that small language models:

  1. Don't invoke skills when available via a skill tool or file access
  2. If they invoke they don't follow it well.
  3. They derail early in the trajectory and are not good at recovering

Are all really good observations. While practitioners have generally known this, it is good to quantify to what degree this happens.

Critical weaknesses
  1. The paper is extremely poorly written. Terms are used before defining them.
  • 'small executors' : Is used first, defined later. The authors mean a local LLM by it, but a small executor could be a deterministic static piece of code also, unless defined.
  • 'standard skill channel' : For a person reading the paper, to gain knowledge of how to use skills this is very vague. Please define early on what the standard skill channel is. It is only much later that it is defined to be the equivalent of claude code, but even that is a weak explanation.
  • 'Public skill package' : There is no apparent justification for this.
  • 'Compiled tools': No justification while using it.
  1. The study basis is wrong. I have addressed this in questions for authors in greater details:
  • the producer uses a hand crafted note on the environment that the authors don't justify how htey got or how its procurement can be automated. its effect is also not discussed.
  • Only one benchmark is used (SkillLearnBench). While this can be overlooked as the field is new, it would be good to include more contemporary benchmarks. eg Skillsbench etc.
  • in the prefix interventions the code is removed form the skill. and then code is provided as tools and apparent gains are observed. This is not a fair comparison, code should be available in skill prefix.
  • How do the tools look after compilation, an analysis on this is missing.
Questions for authors

Note: The following points are my own handwritten annotations while studying the paper and have been paraphrased with AI to ground them in the paper and give more context to the authors while addressing the questions.

Line 15: “Standard skill channel” is not defined until Section 2.1 / Table 1. Please define it at first use.

Line 15: “Small open executors” is only defined at line 37 (“the models one runs locally”). Please define it in the abstract where is is first used.

Line 22: “Public skill package” is not defined. Does it mean should say it refers to SkillLearnBench’s official per-family package ? If so, it should be defined earlier.

Lines 40–41: Compiled tools are never introduced. They are talked about many times. First occurrence at line 40 and generally defined in line 94. Leaving the reader confused.

Line 44: This reads as contradicting line 039, which says skills are almost never opened. I assume line 39 refers to the skill channel and line 44 to the skill-in-context condition. Or it could be rarely opened, but when opened rarely followed. Please make this clear.

Line 96: This leaks environmental and procedural information. The producer sees the development instance’s container and input files, and Appendix E (lines 908–914) says the contract and family notes were iterated using results on all instances of the 15 development families, including held-out ones. Only the 5 evaluation families (13 instances) are clean. Please report the headline results on those families separately and prominently.

Line 97: If the bottleneck is information on intermediate outcomes and instruction following, a good baseline would be ReAct + adding 1–2 demonstrations of the whole trajectory in the skill.

Line 141: Also use SkillsBench (cited, but not evaluated on). SkillLearnBench measures skill learning/synthesizing, which is relevant. SkillsBench measures utilization of skills. Augmenting tools there would also be good.

Line 208: Why remove code blocks from the skill in the prefix interventions? The main experiments use the official package with its scripts, so the intervention tests a different, weaker skill than the one the main results compare against.

Lines 288–289: Where does the family note on where the small executor is likely to fail come from? Is it human-written or learnable? Appendix E suggests the authors wrote and iterated the notes using held-out results on development families, so how much of the gain depends on them? An ablation without the note would help. Does the coding agent find it? The requirement for this family note seems critical to the performance of the coding agent, and this is not addressed in the paper.

Lines 372–373: Not sure the experiments address this claim. Ablation (d) shows the knowledge as text doesn’t help, but the producer’s code itself is still better informed than the official package’s scripts. A fairer control would register the official package’s own scripts as tools.

Lines 472–473: A study on the producer strength requirement would be useful, as that is an integral step.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9Kkn

Thank you for the concrete annotations and for highlighting the three empirical observations on skill invocation, following, and early derailment. We address each point directly below.

  • Line 15 ("small open executors" and "standard skill channel"): We will define both at first use in the abstract: small open executors are open-weight LLMs (4B–31B parameters) run locally to drive a coding agent scaffold such as Claude Code; the standard skill channel (Skill channel) lists installed skills only by name and short description, loading a skill's instructions only when the model calls the Skill tool.
  • Line 22 ("public skill package"): Yes, this means SkillLearnBench's official human-written package (SKILL.md plus any bundled scripts and reference files) for each task family. We will state this at line 22.
  • Lines 40–41 ("compiled tools"): We will define compiled tools at first mention in the introduction as the 2–3 callable Python tools (registered via MCP) generated once per task family from the official skill package and one development task from that family.
  • Line 44 vs. Line 39: Your reading is exact: line 39 refers to Skill channel (rarely opened) and line 44 refers to Skill in context (placed in CLAUDE.md, read on every turn, yet still failing). We will name the condition in both sentences.
  • Line 96 (environmental leakage and the 5 evaluation families): We audited all generation prompts against held-out instructions (see General Response). The 5 pre-frozen evaluation families (Table 7, 13 held-out instances) contain zero held-out information, and Tools + card beats Skill in context on every executor (8/13, 8/13, 9/13 vs. 3/13, 4/13, 5/13 on Qwen 4B/9B/27B; 5/13, 8/13, 10/13 vs. 0/13, 4/13, 5/13 on Gemma E4B/12B/31B). We will move Table 7 to the main text, remove the two development families whose prompts contained held-out examples (DBSCAN tuning and Weighted GDP) from the headline totals, and report the 18-family results (p from 6.1e-5 to 0.0049).
  • Line 208 (removing code blocks in prefix interventions): Section 2.3 never uses compiled tools: it tests only how continuing from a clean vs. messy intermediate result affects the next step under the same plain-text skill (with code snippets omitted only so the next action must use the intermediate result rather than copy a snippet verbatim). Every benchmark comparison in Sections 2.2 and 3 keeps the full official skill package with all of its code blocks and scripts intact. We will make this explicit at line 208.
  • Lines 288–289 (where the family note comes from): The notes were not human-written. Before each producer session, an upstream Claude Opus 5 call generated the family note and prompt automatically from the single development task (instance 1) and the official skill package (and in 10 families, including the 5 evaluation families, also proposed the tool names and roles). We will state this in Section 3.1 and include all prompts in the supplement.
  • Lines 372–373 & tool analysis (official scripts as tools, and what compiled tools look like):
    • We ran the exact control you suggested on Qwen 4B (details in General Response and MdRM Q3): 11 of 20 official packages ship no script at all, and on the 5 families with runnable task scripts (23 held-out instances), registering those official scripts as MCP tools passes 4/23 vs. 5/23 for Skill in context and 16/23 for Tools + card (p = 0.0018).
    • Profiling all 52 compiled tools (about 36k lines) shows roughly 29% generic mechanics, 27% deterministic transformation, 18% verification, and 26% task-specific logic; official scripts cover a median of only 5–8% of this logic. We will add a per-suite summary table to the appendix.
  • Lines 97, 141, 472–473 (trajectory demonstrations, SkillsBench, producer strength): We agree all three are useful extensions. Because compiling and running container suites across a second benchmark or multiple producer sizes requires substantial compute beyond the discussion window, we will state these explicitly as limitations and future work in Section 5.
Reviewer_9KknOpenReview ↗

Response to rebuttal

I still find the usage of a single benchmark(SkillLearnBench) a major limitation of the paper (even though it covers various domains, it comes from the same inductive bias of the creators, which is mitigated by using various benchmarks).

A lot of my other concerns are adressed well by the authors. The writing changes will also improve the flow of the paper for the reader.

Can the authors adress the omission of the react demonstration baseline?

I am updating my score, as per the currently adressed points.

AuthorsOpenReview ↗

Follow-up to Reviewer 9Kkn: ReAct Demonstration Baseline Results

Thank you for updating your score and for following up. We ran the ReAct demonstration baseline (Line 97) on Qwen3.5 4B and Qwen3.5 9B across all 20 task families (80 held-out instances per model).

To match the inputs used for tool compilation, we used each family's official skill package and its single development task (instance 1). Claude Opus 5 solved instance 1 inside the task container with the test grading folder removed before the run. All 20 trajectories passed the grading tests afterward. We formatted each trajectory step by step (Thought, Action, and Observation; 2 to 24 steps, median 7) and appended it directly to the skill text in CLAUDE.md. Everything else (container, sampling, and the 100-turn / 1,800 s limits) is the same as Skill in context.

ConditionQwen 4B (passed)Qwen 9B (passed)
No skill1 / 80 (1.3%)1 / 80 (1.3%)
Skill in context9 / 80 (11.3%)21 / 80 (26.3%)
Skill + ReAct demonstration (new)20 / 80 (25.0%)23 / 80 (28.8%)
Tools + card53 / 80 (66.3%)60 / 80 (75.0%)

Demonstrations help the 4B model follow workflow order when intermediate outputs are short and clean. On 4B, task completion goes up from 9/80 to 20/80 (p = 0.043), which is close to 9B with plain Skill in context (21/80). Most gains come on fixed-step tasks such as offer-letter generation (0/5 to 4/5), video counting (1/4 to 4/4), and dependency checks (0/4 to 3/4). However, demonstrations stop helping around 25-29% on both models. On 9B, adding the demonstration barely changes the score (23/80 vs. 21/80, p = 0.83). Tools + card is much higher on both 4B (53/80, per-family p = 0.0011) and 9B (60/80, per-family p = 6.1e-5). Excluding DBSCAN tuning and Weighted GDP gives the same gap (47/71 vs. 20/71 on 4B, and 54/71 vs. 23/71 on 9B).

The failed logs show why a clean trajectory from instance 1 is not enough on new tasks. First, large intermediate outputs still fill the context window: when a task involves big search results or repeated trials, context fills up and triggers automatic history summarization in 39/80 (4B) and 44/80 (9B) runs, compared to 21/80 and 12/80 with Tools + card, and 1,800 s timeouts stay high at 14/80 and 39/80 vs. 5/80 and 8/80. Second, small mistakes at the end go unchecked: among the 57 failed 4B runs on tasks with itemized grading checks, 33 already passed at least half of the checks (for example, court-form filling is 1/5 vs. 5/5 because missed PDF fields are not caught without a check tool). Third, in 2/80 runs on 4B, the model confused the example trajectory in CLAUDE.md with its own past turns and stopped immediately with 0 tool calls. We will add this baseline and log analysis to Section 3 (Table 3) and Appendix C.

On the single-benchmark limitation, we agree that using SkillLearnBench alone is a real limitation because all tasks come from the same inductive bias of the creators in how skills and tests are structured. Evaluating on additional benchmarks such as SkillsBench is needed to test whether the gains hold more broadly. We will state creator inductive bias clearly as a main limitation in Section 5, and narrow our claims in the abstract, introduction, and conclusion to SkillLearnBench.

Reviewer_QbPxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies and tackles a problem that LLM agents powered by small executor models do not leverage skills (i.e., SKILL.md's) effectively. First, the authors analyze small executors and find that they rarely open/read skills, and even when they do, the execution often derails in the middle. As a remedy, the authors propose to "compile" each skill package into a set of tools using strong model(s) (Claude Opus 5) as the producer and the reviewer. This basically turns some of the steps that appear in each skill into executable code-based tools that can be invoked by the agent. Empirically, on SkillLearnBench, by compiling the skills using the development task and leveraging the results on the evaluation tasks in each task family, the authors demonstrate performance gains with small Qwen3.5 and Gemma 4 models.

Critical strengths

None

Critical weaknesses
  • I have a concern regarding the practicality and scalability of the proposed approach. In practical systems, multiple skills may be available simultaneously, and compiling them into tools would expand the executor's action space on top of existing tools. This could introduce a different type of burden for small executors, which would need to select appropriate tools based on the current task state, potentially among many tools with overlapping/similar functionalities. Also, complex skills may involve branching workflows and dependencies across multiple steps, requiring the executor to reason about previous results and coordinate tool calls accordingly. However, the experiments only provide a small set of tools (usually 2–3 tools) tailored to each task family, leaving it unclear whether the approach remains effective in more realistic settings.
Questions for authors

Please refer to the Critical Weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer QbPx

We thank the reviewer for the review and for raising two questions about practical deployment.

Action-space size, overlapping tools, and state-based selection.

Removing the one-line tool summary card in CLAUDE.md (Table 3, row a: Tools without card vs. Tools + card) measures the cost of finding compiled tools when they sit undocumented alongside Claude Code's roughly 20 built-in tools. Qwen3.5 4B drops from 53/80 with the card to 39/80 without it (p = 0.0043, family-clustered p = 0.016), whereas Qwen3.5 9B and 27B (55/80 and 71/80 without card vs. 60/80 and 69/80 with card) and all three Gemma 4 executors (54, 52, 71 without card vs. 47, 51, 72 with card) are unaffected; in a new control registering up to 7 official-package scripts per family as MCP tools (about 27 tools total; see General Response), 4B still called the registered tools in 15/23 trials.

Two rules in our compilation contract (Section 3.1) also address overlapping functionalities and state-dependent selection directly. Rule (v) ties each tool name to its task artifact rather than a generic verb (for example, fill_small_claims_form_sc100 vs. fill_offer_letter_template, and check_regulated_verse_requirements vs. check_simpo_reproduction_requirements), and rule (iii) ends every tool's return report with the next step or tool to call, so subsequent calls follow the prior tool's output instead of re-searching the full tool pool on each turn. Even so, our containers mount one family's suite at a time rather than all 52 compiled tools from all 20 families together. We will state this boundary in Section 5: when many skills are installed in one system, skill-level routing or tool retrieval should select the relevant skill first and expose only its 2–3 compiled tools rather than loading every skill's tools into one global action space.

Branching workflows and dependencies across steps.

Several families already require the executor to decide next calls from earlier outputs rather than follow a fixed sequence. In Temperature simulation, passing executors calibrate the lake model by calling the simulation tool a median of 27.5 times, guided by residual reports. In Travel planning, passing trials contain a median of 11 of the executor's own shell or file actions between tool calls. In Enterprise search (5 held-out instances), the executor writes /root/answer.json itself from three retrieval tools' reports in all 15/15 passing trials across Qwen 4B/9B/27B. Pooled across all 20 families and the three Qwen executors (182 passing trials), 41% contain own shell or file actions by the executor between or after tool calls, while in 59% one main tool plus the checker produces the final deliverable directly; we will add this per-family split to the appendix and list long open-ended branching workflows as a limitation in Section 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for their detailed and constructive reviews. For context, SkillLearnBench has 20 task families, each with an official skill package, 1 development instance used once by Claude Opus to compile 2–3 tools (52 total), and 1–5 held-out test instances (80 total). Across the reviews, the main questions are: (1) whether gains hold when instances are clustered by task family and repeated handoff runs on the same instance are not counted twice, (2) how much comes from the step code Opus wrote versus registering it as a tool, and (3) how much the tools solve on their own and how the per-family notes on likely failure points (family notes) were created. We re-analyzed all 3,693 held-out trials, audited all 52 tools and generation prompts, and ran two pre-registered controls on Qwen3.5 4B (263 new trials):

  1. Family-clustered tests confirm the main effect. Using the task family as the unit of analysis (exact sign-flip permutation test over the 20 family pass-rate differences), compiled tools with a one-line summary card (Tools + card) beat placing the official skill text directly in context (Skill in context) with p from 1.5e-5 to 0.0049 across all six Qwen3.5 (4B–27B) and Gemma 4 (E4B–31B) executors, and every 95% family-bootstrap CI excludes zero.
  2. Both task-specific step code and MCP tool registration are required.
    • Official scripts registered as MCP tools (same tool interface, official package code): 11 of the 20 official skill packages ship no script. On the 5 families with task-relevant runnable scripts (23 held-out instances), registering every entry-point official script as an MCP tool gives 4/23 passes at 4B vs. 5/23 for Skill in context (p = 1.00) and 16/23 for Tools + card (p = 0.0018), even though 4B calls the registered official-script tools in 15/23 trials.
    • Compiled code run as CLI scripts with neutral wording (same compiled code, script interface): In our ablation offering the exact same compiled code as CLI scripts (Table 3, row c), CLAUDE.md introduced those scripts under the heading "# Skill installed...", which led 4B to call Claude Code's built-in Skill tool instead of running the scripts. Re-running with the neutral heading "# Scripts installed..." (on the 76 of 80 instances not blocked by an expired external GitHub API token in November) raises 4B script execution from 39/76 to 57/76 and 61/76 across two runs, but passes rise only from 26/76 to 32/76 and 34/76, still well below 50/76 for Tools + card (p = 0.0005 and 0.0037).
  3. The handoff effect holds without pooling repeated instances. In our 30-instance handoff study, where a 27B executor takes over after turn K=4 or K=8 from either 4B (4B->27B) or itself (27B->27B), averaging each instance across K=4 and K=8 (n = 30) still favors 27B->27B with exact sign-flip p = 0.0052 (family-level p = 0.027; K=8 alone: p = 0.0074), and 27B actually has more wall-clock time left after 4B than after 27B.
  4. Two independent code audits of all 52 compiled tools (~36k lines, by Claude Opus and Sonnet) quantify generic mechanics, transformation, verification, and task-specific logic, and show gains even in families where calling the tools without reading the task input passes zero instances (11/17 vs. 3/17 at 4B).

Prompt audit correction. During method design, 15 of the 20 families served as development families while iterating the compilation pipeline, and 5 families (Table 7, 13 held-out instances) were frozen in advance as untouched evaluation families. Auditing every generation prompt against held-out instructions showed an omission in Appendix E for two of the 15 development families: the producer prompt for DBSCAN tuning included distance formulas from held-out instances 2–4, and the prompt for Weighted GDP named the derived formula of held-out instances 2 and 5. A 4-gram and phrase scan found zero held-out content in the 5 evaluation families. In the revision, we will exclude both affected families from the headline summary, report them separately, and correct Appendix E. On the remaining 18 clean families (71 held-out instances), all conclusions hold:

Metric (18 clean families, 71 instances)Qwen 4BQwen 9BQwen 27BGemma E4BGemma 12BGemma 31B
Skill in context9213601931
Tools + card475462414565
Family-clustered permutation p6.1e-55.8e-40.00496.1e-50.00312.4e-4

Excluding the entire initial 6-family development batch that contained those two families (leaving 14 families, 54 held-out instances) also remains significant for every executor (p from 4.9e-4 to 0.020).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.