acceptodds
Under review as a conference paper at ICLR 2027

When Skills Fail to Compose: Benchmarking and Compiling Agent Skills

Abstract

Agent skills provide reusable procedures and resources, but supplying relevant skills does not ensure that an agent can coordinate them to complete a shared task. We introduce SkillsBench-Compose, a benchmark of 20 human-authored tasks that evaluates component requirements and cross-component relationships separately. This separation exposes failures in integrating otherwise successful components. We also propose SkillCompiler, which jointly compiles the supplied skills into one task-specific package before execution. It generates a coordinated workflow grounded in critical source rules, records guidance and resource choices in a structured intermediate representation, and packages the guidance with selected original resources. Across three language models, SkillCompiler improves on the highest-scoring baseline in each setting by 2.9–4.8 reward points on SkillsBench and 3.3–8.3 percentage points in full completion on SkillsBench-Compose, while using fewer total model tokens, including preparation. Ablations show gains over task-only guidance and independently rewritten skills, including when the rewritten skills are consolidated into a single package.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Skills, Skill Composition, Skill Compilation, Compositional Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9WH7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces SkillsBench-Compose, a benchmark composed of 20 tasks combining two or three components which studies whether agents can satisfy both component requirement and their integration. It introduces also SkillCompiler, which rewrites the supplied skills into one joint task-specific package, retaining the most critical rules and selected resources through a structured intermediate representation. The authors evaluate GPT-5.4 mini, GPT-5.4, and Qwen3.7-plus on 87 tasks from SkillsBench and 20 from Compose. The findings are:
1\ passing individual components does not guarantee integration;
2\ SkillCompiler improves the strongest baselines up to 4.8 points on SkillsBench and up to 8.3 points in Compose completion;
3\ SkillCompiler's joint rewriting beats independent rewriting even after concatenation;
4\ total tokens are lower than the highest-scoring baseline (including preparation).

Critical strengths

1\ separating component and integration checks exposes failures that an otherwise aggregate task would hide;
2\ the validation of the benchmark seems to be have been done carefully (e.g., the authors claim three independent reviewers per task, having references solutions, constructing examples that pass components which, however, fail integration, ...);
3\ attribution studies are done carefully, in particular, the independent rewrite vs. rewrite + concat comparison holds the generated content fixed, and allows attribution to be controlled;
4\ the positive gains seem to generalize across models and benchmarks;
5\ having access to detailed prompts and schemas is useful for reproducibility.

Critical weaknesses

Novelty: limited. The work is essentially a task-conditioned joint rewrite of exisiting instructions, with on top of that resource selection and packaging. It seems like the cited SkVM, AgentSkillOS, and SkillRAE already cover related preparation or orchestration ides that make this side of the paper less novel. The comparisons do not establish which parts of the compiler are actually necessary, and that would be important. One experiment I would have appreciated is to set a baseline like a simple joint plan from all source skills, keeping original resources. Overall, the most novel contribution is the benchmark itself, but having only 20 tasks makes its coverage relatively small.

Realism: very limited. There are only 20 tasks, of which 17 with only two components. 14 of these are basically artifact handoffs, and only 1 is a joint-constraint task. Using only OpenHands and 3 models is also a big constraint, particularly given that these models are not very recent (e.g., GPT-5.4 dates back to March). In my opinion such an empirical contribution needs more a broader model roaster to better see how improvements generalize. A limited roaster would be fine only if those models tested are at the frontier, which is not the case. Also, all the methods get already relevant skills, so this cannot test skill retrieval or robustness to irrelevant (or missing) skills.

More general soundness complaints:
1\ statistics: 1 run per SkillsBench task, 3 runs on only 20 Compose tasks, no CIs, ... The Compose gains over the strongest baselines are up to 5 successful executions out of 60 exeuctions over 20 tasks (and, moreover, these are not independent). This limits my confidence in the results, which, given the empirical nature of this contribution, is a fundamental necessary condition.
2\ control of instruction length and resource selection: joint rewriting indeed changes also the instruction length and resource selection, in fact Table 8 shows 2068 instruction tokens for SkillCompiler vs. 8785 for independent rewriting on Compose. I would argue that a control on the joint summary length and some ablations of critical rules selection would have been more informative than the concat control (although these are disjoint and both provide a useful, yet different, causal control mechanism).
3\ unexplained integration failures: in the GPT-5.4 runs, both Concat and SkillCompiler have six component-successful runs which failed integration (this is a substantial amount, particularly for SkillCompiler, given that it accounts for 27% of the of them). Given that coordinating skills is the method's central purpose, they should have been analyzed. Do they arise from omitted or incorrect coordination in the package, are they due to an executor failure to follow correct guidance, or why are there so many failures?

Presentation-wise, I believe some key information is missing, while other less important details receive too much attention. For example:
1\ A table of the tasks, their source components, and integration checks is needed for a reviewer to judge how tasks differ between each other, how components are reused, and so on.
2\ Figure 3 does not acknowledge that tokens alone do not establish cost savings, and it would have been more informative to have monetary savings or latency metrics instead. Input, output, and cacehd tokens have differnet costs, time execution depends greatly on tool use and parallelism, and so on. This is acknowledged in the appendix, but I should have read this in the main paper instead.
3\ I was forced to read the appendix to understand the compiler description. Something like a small "source skills to compiled" workflow example in the main paper would have made the overall setting more clear from the start.

Questions for authors

Dear authors, please answer my weakness if you have anything to rebut or if my understanding of the work seems incorrect to you. I have two additional questions:
1\ were the compiler prompt and method choices fixed before evaluating Compose, or was a separate development set, or similar, used?
2\ what causes the SkVM-AOT preparation failures you mention, and does the advantage remain on tasks where both the methods prepare successfully?

Reviewer_3pGSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of skill composition in LLM agents: supplying relevant, high-quality skill packages does not guarantee that an agent can coordinate them toward a shared goal. The authors first provide empirical evidence on SkillsBench that the benefits of supplied skills degrade — and can even reverse — as the number of skills grows to four or more. To make the source of such failures measurable, they introduce SkillsBench-Compose, a benchmark of 20 human-authored tasks (2–3 components each) that retains component-level acceptance checks while adding separate integration checks over artifact handoffs, shared objects/state, and joint output constraints; full completion requires passing both. Motivated by this decomposition, they propose SkillCompiler, which jointly compiles the supplied skill set into a single task-specific package before execution: the model produces a structured intermediate representation (coordinated workflow, critical source rules with task-specific applications, and resource selections), which is programmatically validated and packaged for an unchanged executor. Across three backbones (GPT-5.4 Mini, GPT-5.4, Qwen3.7-plus), SkillCompiler improves over the strongest baseline in each setting by 2.9–4.8 reward points on SkillsBench and 3.3–8.3 points in Compose full completion, while using fewer total tokens including preparation. Ablations show that joint compilation outperforms task-only guidance and independently rewritten skills, even when the latter are consolidated into a single package.

Critical strengths
  • Well-motivated and cleanly decomposed problem formulation. The separation of component correctness from cross-component integration correctness is a genuinely useful conceptual contribution; the diagnostic on SkillsBench (Table 1) convincingly shows that aggregate rewards mask multi-skill degradation, and the "passes all component checks but fails integration" negative examples validate that the benchmark measures what it claims.
  • Sound benchmark construction protocol. Human-authored tasks with three independent reviewers, a frozen task set before method evaluation, retained original acceptance criteria, and reference-solution validation give SkillsBench-Compose reasonable credibility despite its small size.
  • Principled method design with a favorable cost profile. The IR-based separation of semantic decisions (model) from validation and packaging (program) is elegant, and the finding that joint compilation reduces total tokens relative to the strongest baseline in all six settings with preparation at only 1.25–3.28% of total usage, which makes the approach practically attractive.
  • Informative ablations. The Per-Skill Rewrite + Concat comparison is exactly the right control: it isolates the value of joint preparation from the mere convenience of a single package, and the consistent residual gap (6.67–11.67 points in full completion) substantiates the paper's central claim.
Critical weaknesses
  • 20 tasks with at most 3 components, all derived from a single source benchmark (SkillsBench), limits statistical power and generality. With three runs per task, an 8.33-point full-completion difference corresponds to a handful of executions; no confidence intervals or significance tests are reported, so it is hard to tell how robust the headline gains are.
  • Improvements of 2.9–4.8 reward points on SkillsBench and a best full-completion rate of only ~27% suggest the method nudges rather than solves skill composition; the paper could do more to characterize which failure classes remain (the single case study in Appendix D.5 is thin).
  • All experiments use OpenHands with three models, two of which are from the same family; whether joint compilation transfers to other harnesses, open-weight models, or weaker backbones where compiled guidance might matter most is untested.
Questions for authors

See weakness.

Reviewer_HK4oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies tasks in which several agent skills must agree on shared artifacts, state, or output constraints. SkillsBench-Compose contains 20 such tasks and evaluates component completion separately from integration. SkillCompiler prepares the supplied skills jointly, producing coordinated instructions and a structured record of source rules and selected resources. It then packages these resources for the existing executor. Across three backbones, it obtains the highest reported primary scores on SkillsBench and Compose. The ablations compare joint preparation with task-only guidance, independent skill rewriting, and concatenation of the rewritten skills.

Critical strengths
  1. The separate integration checks in Section 3.3 are useful. Appendix B includes reference solutions and constructed cases that pass every component check but fail integration. This failure also occurs in the agent results: six GPT-5.4 Concat executions pass all components and still fail integration (Section 5.2).

  2. The comparison between Per-Skill Rewrite and Per-Skill Rewrite + Concat holds the generated guidance and resources fixed while changing their packaging. SkillCompiler improves further over the consolidated version for every backbone on both benchmarks (Table 3). This is a convincing control for whether consolidation alone explains the gains.

  3. The reporting includes preparation costs, the single-skill identity cases, and executions where compilation fails to help. The token savings do not hold against Raw in every setting, and the authors say so.

Critical weaknesses
  1. My main reservation is the uncertainty in the performance differences. SkillsBench uses one execution per task and condition. Compose has three runs on each of 20 tasks, but the paper gives neither paired task-level differences nor run dispersion. The improvements over the strongest Compose baseline amount to three additional completions for Mini, five for GPT-5.4, and two for Qwen out of 60 (Table 2). These margins could depend on a few tasks or on execution variability. Please report a paired analysis of the existing results, clustering Compose repetitions by task, so that readers can judge both the uncertainty and how widely the gains are distributed.

  2. The external baseline results depend on adaptation choices that need more explanation. Appendix D.1 uses SkVM's AOT-only mode and a modified SkillRAE pipeline, with at most three cues per package and a 384-token coordination budget. SkVM-AOT also falls back to matched Raw results after preparation fails in 17 to 26 of 60 Compose runs (Table 6). What causes these failures, and how were the SkillRAE limits chosen? Preparation robustness is part of system performance, but these results alone do not establish how the complete original systems would compare. The paper's own rewriting ablations remain persuasive evidence for joint preparation.

Questions for authors
  1. Can the authors provide paired per-task results and uncertainty estimates from the existing runs? For Compose, the analysis should account for the three repetitions of each task. I would especially like to know whether the additional full completions are spread across tasks.

  2. Please explain the SkVM-AOT preparation failures and the choice of SkillRAE cue and coordination limits. Which failures come from the preparation method, and which arise from adapting it to this setting?

  3. Where can the full frozen Compose task manifest, verifiers, reference solutions, and validation cases be inspected? Appendix B describes the validation procedure, but assessing and reusing the benchmark requires the complete artifacts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.