- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Skill Compilation, implemented by SIGIL, for converting natural-language agent skills and their supporting resources into executable agent harnesses.
The compiler performs four main steps:
- It extracts procedural requirements, conditions, ordering constraints, prohibitions, executable bindings, and knowledge resources from a skill package.
- It represents these elements using an Agent Instruction Set and a typed Agent Intermediate Representation.
- It validates the representation and repairs detected structural or interpretation-dependent errors.
- It deterministically lowers the validated representation into Jac code, preserving semantic operations as model calls while encoding executable operations and procedural control flow in the harness.
The intended separation is between:
- Executable operations, such as tools, scripts, database calls, and external actions;
- Semantic computations, such as classification or extraction that still require model judgment; and
- Delegated subprocedures, whose internal orchestration is deliberately left open to a runtime agent.
The evaluation has three research questions:
- RQ1: Does compilation improve adherence to skill requirements?
- RQ2: Does compilation reduce runtime token use?
- RQ3: Does compilation improve end-to-end task performance?
For RQ1 and RQ2, the authors use 11 public compliance-critical skills, three tasks per skill, and three executions per task. Four runtime models are evaluated: Claude Opus 5, GLM-5.2, GPT-4o, and Llama-3.1-8B. Three compile-time models are used: Claude Opus 5, GLM-5.3, and GPT-5.6-Sol.
The main reported results are:
- Direct
SKILL.mdexecution achieves 26.2–54.0% mean Skill Adherence. - Opus-compiled harnesses achieve 100% measured mean Skill Adherence across all four runtime models.
- GLM-5.3- and GPT-5.6-Sol-compiled harnesses achieve 96.4–100%.
- GEPA reaches 37.2% mean adherence and SkVM reaches 31.7% in the reported GPT-4o comparison.
- SIGIL reduces total runtime tokens by 21–45% for three runtime models, but increases total tokens by 2.13× for Llama-3.1-8B because the direct baseline often terminates before completing the prescribed procedure.
- On 87 SkillsBench tasks, Task Macro Pass Rate increases by 20.8 percentage points for Claude Opus 5, 35.7 for GPT-4o, 34.1 for Llama-3.1-8B, and 3.5 for GLM-5.2.
The paper’s validation therefore demonstrates that compiled harnesses can substantially improve procedural execution on a selected set of procedure-heavy skills. It does not yet demonstrate source-level semantic correctness, general compliance guarantees, or broad generalization to the wider skill ecosystem.
Critical strengths
Strength 1: A clear and useful systems abstraction
The separation between procedural structure and semantic model decisions is the strongest conceptual aspect of the work. AIS and AG-IR give the compiler an explicit vocabulary for requirements, data dependencies, control flow, executable bindings, semantic operations, and delegated subprocedures. This is a meaningful systems contribution because it identifies where deterministic execution can reduce procedural failures without attempting to eliminate model judgment entirely.
Strength 2: Broad runtime-model coverage with a concrete end-to-end benchmark
The evaluation uses four runtime models and three compiler models, rather than demonstrating the method only on the same model used for compilation. The additional SkillsBench evaluation is valuable because it tests whether compiled skills help an outer agent complete tasks that require more than a single isolated procedure. The reported gains are substantial enough to motivate further investigation, even though the experimental controls are currently insufficient.
Critical weaknesses
Weakness 1: The evaluation does not isolate the cause of the reported gains
The comparison is between direct textual skill execution and a substantially different compiled runtime. SIGIL provides deterministic control flow, scoped inputs and knowledge, explicit tool bindings, structured state, and an MCP-accessible harness. The paper does not include the ablations needed to determine whether the gains come from:
- the AIS/AG-IR representation;
- deterministic control flow;
- scoped model context;
- reduced prompt length;
- explicit tool wrappers;
- a stronger outer-agent interface;
- Jac execution; or
- compilation itself.
The GPT-5-based adherence judge further weakens the claim that 100% represents independently verified behavioral correctness. A compiled harness can structurally contain a required model call while the model still produces an incorrect semantic result. This is especially consequential for the clinical, HIPAA, GDPR, and compliance examples.
Weakness 2: The evidence is too narrow and incomplete for the reliability and efficiency claims
The 11-skill suite is curated toward procedure-heavy compliance tasks and does not establish generality. The submission provides no repeated-compilation study, no uncertainty estimates, no complete cost accounting, and no detailed release or reproducibility information. The token results exclude compilation cost and count a baseline that often stops early, making the headline efficiency claim difficult to interpret without a quality-normalized amortization analysis. The paper also lacks direct empirical comparisons with several closely related executable-skill and workflow-compilation systems.
Questions for authors
The following are clarification requests whose answers could materially change my assessment.
-
Evaluation independence: What proportion of Skill Adherence judgments has been independently checked by human experts or an independent automated verifier? Please provide inter-rater agreement, judge calibration results, the exact judge prompt, and examples of disagreements between GPT-5 and human reviewers.
-
Baseline parity: Do the direct
SKILL.mdagents receive exactly the same tools, supporting resources, system instructions, model-call opportunities, input data, and outer-agent capabilities as SIGIL? If not, please quantify each difference and provide ablations with matched scoped context, matched tool wrappers, and an identical controller. -
Ablations: What are the results for a manually authored harness, a harness with deterministic control flow but no AG-IR, a harness with scoped prompts but no procedural compilation, and a generated workflow without typed carries? These experiments are necessary to attribute the gains to the proposed method.
-
Reference and task construction: Who selected the 11 skills and 33 tasks, how many domain experts reviewed each reference DAG, and were any authors involved in creating or approving the evaluation references? Please provide the complete skill packages, task prompts, reference DAGs, and per-mandate labels.
-
Cost and reproducibility: What are compilation token counts, repair rounds, compile failures, wall-clock latency, monetary cost, and the break-even number of runtime executions at which SIGIL becomes cheaper than direct execution? Please also report model versions, temperatures, seeds, tool settings, and variance over multiple independent compilations.
-
Semantic correctness and safety: For the clinical and compliance skills, how do you evaluate the correctness of model-generated classifications and recommendations, rather than only whether the required calls and ordering constraints were executed? In particular, how are false negatives, incorrect PHI classification, unsafe clinical outputs, and prompt-injection attempts handled?
4 comments
Compilation cost and reuse across tasks.
Compilation is a separate, one-time cost per skill harness. SIGIL compiles the skill’s procedure without task inputs, allowing the completed harness to be reused for repeated runs of the same task and for different tasks within the skill’s scope, without task-specific training or recompilation (Figure 3 in the paper). Validation and internal repairs are part of this initial process.
Figure 7 in the revised paper compares this reuse with SKILL.md + ReAct + supervisor using GPT-4o. Both methods complete 15 runs of each of three increasingly difficult tasks before moving to the next. The baseline retains corrective feedback from Claude Code running Claude Opus 5 across runs and tasks. SIGIL reuses the unchanged harness with fresh runtime state for every execution.
Figure 7(b) in the revised paper shows 100% measured Skill Adherence (SA) for SIGIL in all executions, including each new task’s first run. Even retrospectively selecting the baseline’s best run for each skill–task pair yields only 73.4% mean SA across skills on the final task.
Section 4.2 in the revised paper reports a mean compilation cost of 4.457M tokens per skill. Figure 7(a) includes compilation, internal repairs, all executions and baseline supervisor calls. SIGIL’s mean cumulative token use is already lower than the supervised baseline’s at the second task checkpoint (30 runs per skill) and is 35.3% lower by the third. The revised paper therefore reports cost and adherence together, demonstrating amortization under this repeated-use protocol.
The runtime-only comparison in Figure 6 of the paper also distinguishes lower token use from completed execution. For Llama-3.1-8B, SIGIL uses additional tokens to complete steps omitted by direct execution. Figure 7 in the revised paper separately evaluates compilation-inclusive token use alongside measured adherence.
How execution and semantic correctness are evaluated.
A required model call can execute while producing an output that fails the requirement. The adherence results in Figure 4 of the paper and Figure 7 of the revised paper therefore check both whether the required code ran and whether the resulting behavior satisfied the reference requirements.
GPT-5 compares the task, execution trace and relevant code against fixed, expert-reviewed reference DAGs. These references are derived from the source skills independently of the compiler. For applicable deterministic mandates, runtime instrumentation records whether the required code ran, including ordinary code between model calls. A deterministic mandate automatically receives zero credit if the trace does not confirm the required execution, as clarified in §4.1.1 of the revised paper. GPT-5 judges the required actions, conditions, ordering and semantic results.
In the HIPAA example in Figure 3 of the paper, PHI classification remains a model judgment. If an incorrect classification causes an applicable BAA check to be skipped, the affected requirements fail under the scoring rules. An incorrect branch does not make an otherwise applicable obligation disappear (Appendix D.3 in the paper). Thus, the presence of a model call alone does not establish adherence.
Figure 5 in the paper separately evaluates requirement preservation in compiled artifacts, including omissions and differences in structural encoding. Expert review establishes the reference requirements; GPT-5 performs the artifact and execution assessments. The reported 100% is measured SA under this evaluation procedure.
Task construction and broader task performance.
In the revised paper, §4.1 clarifies that we constructed evaluation tasks for public source skills and that an independent domain expert reviewed each skill’s reference DAGs against the source and supporting resources. No author served as an expert reviewer. This review established the references used in the adherence evaluation in Figure 4.
The two benchmarks measure complementary outcomes. In the compliance-critical suite, required checks, actions and ordering are part of correctness; a plausible final output is insufficient. SkillsBench evaluates broader tasks using native task verifiers, where successful completion does not necessarily require a particular prescribed procedure. Figure 8 in the paper demonstrates that the benefit of compiling skills extends beyond procedural adherence to broader task performance.
What the comparisons establish.
Thank you for the thoughtful feedback.
SIGIL’s contribution is automatically deriving a reusable execution harness from an authored skill. Figure 3 in the paper shows how AG-IR represents the required operations, data dependencies, control flow and bindings for validation and deterministic lowering. The resulting harness preserves the procedure while leaving semantic decisions to the runtime model.
In the paper, Figure 4(a) evaluates reuse of the same compiled artifacts across runtime models. Figure 4(b) compares SIGIL with instruction optimization and adaptation using GPT-4o and the same adherence evaluation. These comparisons assess the benefit of the complete compilation approach.
For the SkillsBench comparison in Figure 8 of the paper, both conditions use OpenHands and the same runtime model, which also powers SIGIL’s internal model calls. The baseline receives the original SKILL.md, while SIGIL exposes compiled harnesses through MCP (§4.3). The shared framework and runtime model provide the comparison setting; the generated procedure, bindings, state and scoped calls constitute the compiled approach being evaluated.