- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work introduces SkillsBench-Compose, a benchmark composed of 20 tasks combining two or three components which studies whether agents can satisfy both component requirement and their integration. It introduces also SkillCompiler, which rewrites the supplied skills into one joint task-specific package, retaining the most critical rules and selected resources through a structured intermediate representation. The authors evaluate GPT-5.4 mini, GPT-5.4, and Qwen3.7-plus on 87 tasks from SkillsBench and 20 from Compose. The findings are:
1\ passing individual components does not guarantee integration;
2\ SkillCompiler improves the strongest baselines up to 4.8 points on SkillsBench and up to 8.3 points in Compose completion;
3\ SkillCompiler's joint rewriting beats independent rewriting even after concatenation;
4\ total tokens are lower than the highest-scoring baseline (including preparation).
Critical strengths
1\ separating component and integration checks exposes failures that an otherwise aggregate task would hide;
2\ the validation of the benchmark seems to be have been done carefully (e.g., the authors claim three independent reviewers per task, having references solutions, constructing examples that pass components which, however, fail integration, ...);
3\ attribution studies are done carefully, in particular, the independent rewrite vs. rewrite + concat comparison holds the generated content fixed, and allows attribution to be controlled;
4\ the positive gains seem to generalize across models and benchmarks;
5\ having access to detailed prompts and schemas is useful for reproducibility.
Critical weaknesses
Novelty: limited. The work is essentially a task-conditioned joint rewrite of exisiting instructions, with on top of that resource selection and packaging. It seems like the cited SkVM, AgentSkillOS, and SkillRAE already cover related preparation or orchestration ides that make this side of the paper less novel. The comparisons do not establish which parts of the compiler are actually necessary, and that would be important. One experiment I would have appreciated is to set a baseline like a simple joint plan from all source skills, keeping original resources. Overall, the most novel contribution is the benchmark itself, but having only 20 tasks makes its coverage relatively small.
Realism: very limited. There are only 20 tasks, of which 17 with only two components. 14 of these are basically artifact handoffs, and only 1 is a joint-constraint task. Using only OpenHands and 3 models is also a big constraint, particularly given that these models are not very recent (e.g., GPT-5.4 dates back to March). In my opinion such an empirical contribution needs more a broader model roaster to better see how improvements generalize. A limited roaster would be fine only if those models tested are at the frontier, which is not the case. Also, all the methods get already relevant skills, so this cannot test skill retrieval or robustness to irrelevant (or missing) skills.
More general soundness complaints:
1\ statistics: 1 run per SkillsBench task, 3 runs on only 20 Compose tasks, no CIs, ... The Compose gains over the strongest baselines are up to 5 successful executions out of 60 exeuctions over 20 tasks (and, moreover, these are not independent). This limits my confidence in the results, which, given the empirical nature of this contribution, is a fundamental necessary condition.
2\ control of instruction length and resource selection: joint rewriting indeed changes also the instruction length and resource selection, in fact Table 8 shows 2068 instruction tokens for SkillCompiler vs. 8785 for independent rewriting on Compose. I would argue that a control on the joint summary length and some ablations of critical rules selection would have been more informative than the concat control (although these are disjoint and both provide a useful, yet different, causal control mechanism).
3\ unexplained integration failures: in the GPT-5.4 runs, both Concat and SkillCompiler have six component-successful runs which failed integration (this is a substantial amount, particularly for SkillCompiler, given that it accounts for 27% of the of them). Given that coordinating skills is the method's central purpose, they should have been analyzed. Do they arise from omitted or incorrect coordination in the package, are they due to an executor failure to follow correct guidance, or why are there so many failures?
Presentation-wise, I believe some key information is missing, while other less important details receive too much attention. For example:
1\ A table of the tasks, their source components, and integration checks is needed for a reviewer to judge how tasks differ between each other, how components are reused, and so on.
2\ Figure 3 does not acknowledge that tokens alone do not establish cost savings, and it would have been more informative to have monetary savings or latency metrics instead. Input, output, and cacehd tokens have differnet costs, time execution depends greatly on tool use and parallelism, and so on. This is acknowledged in the appendix, but I should have read this in the main paper instead.
3\ I was forced to read the appendix to understand the compiler description. Something like a small "source skills to compiled" workflow example in the main paper would have made the overall setting more clear from the start.
Questions for authors
Dear authors, please answer my weakness if you have anything to rebut or if my understanding of the work seems incorrect to you. I have two additional questions:
1\ were the compiler prompt and method choices fixed before evaluating Compose, or was a separate development set, or similar, used?
2\ what causes the SkVM-AOT preparation failures you mention, and does the advantage remain on tasks where both the methods prepare successfully?