acceptodds
Under review as a conference paper at ICLR 2027

SIGIL: Skill Compilation for Reliable and Efficient Agent Execution

Abstract

Agent skills describe reusable procedures, but runtime models must still interpret their instructions and coordinate execution. We introduce Skill Compilation, which translates the procedure prescribed by an authored skill into an executable harness while preserving decisions left to the model. Our compiler, SIGIL, translates skills and their resources into a typed intermediate representation, validates it, and deterministically generates the harness. We evaluate SIGIL on 11 compliance-critical skills, where following the prescribed procedure is part of correctness. SIGIL improves adherence across all four runtime models, achieving up to 100% measured mean Skill Adherence compared with 26.2-54.0% for direct SKILL.md execution. On this suite, SIGIL reduces total runtime tokens by 21-45% for three of the four models. On SkillsBench, SIGIL also improves task performance across all four models, with absolute gains of 3.5-35.7 percentage points. These findings suggest that compiling reusable skill procedures into executable harnesses can improve adherence, task completion, and runtime efficiency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, agent skills, skill compilation, procedural compliance, agent harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_twPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Skill Compilation, implemented by SIGIL, for converting natural-language agent skills and their supporting resources into executable agent harnesses.

The compiler performs four main steps:

  1. It extracts procedural requirements, conditions, ordering constraints, prohibitions, executable bindings, and knowledge resources from a skill package.
  2. It represents these elements using an Agent Instruction Set and a typed Agent Intermediate Representation.
  3. It validates the representation and repairs detected structural or interpretation-dependent errors.
  4. It deterministically lowers the validated representation into Jac code, preserving semantic operations as model calls while encoding executable operations and procedural control flow in the harness.

The intended separation is between:

  • Executable operations, such as tools, scripts, database calls, and external actions;
  • Semantic computations, such as classification or extraction that still require model judgment; and
  • Delegated subprocedures, whose internal orchestration is deliberately left open to a runtime agent.

The evaluation has three research questions:

  • RQ1: Does compilation improve adherence to skill requirements?
  • RQ2: Does compilation reduce runtime token use?
  • RQ3: Does compilation improve end-to-end task performance?

For RQ1 and RQ2, the authors use 11 public compliance-critical skills, three tasks per skill, and three executions per task. Four runtime models are evaluated: Claude Opus 5, GLM-5.2, GPT-4o, and Llama-3.1-8B. Three compile-time models are used: Claude Opus 5, GLM-5.3, and GPT-5.6-Sol.

The main reported results are:

  • Direct SKILL.md execution achieves 26.2–54.0% mean Skill Adherence.
  • Opus-compiled harnesses achieve 100% measured mean Skill Adherence across all four runtime models.
  • GLM-5.3- and GPT-5.6-Sol-compiled harnesses achieve 96.4–100%.
  • GEPA reaches 37.2% mean adherence and SkVM reaches 31.7% in the reported GPT-4o comparison.
  • SIGIL reduces total runtime tokens by 21–45% for three runtime models, but increases total tokens by 2.13× for Llama-3.1-8B because the direct baseline often terminates before completing the prescribed procedure.
  • On 87 SkillsBench tasks, Task Macro Pass Rate increases by 20.8 percentage points for Claude Opus 5, 35.7 for GPT-4o, 34.1 for Llama-3.1-8B, and 3.5 for GLM-5.2.

The paper’s validation therefore demonstrates that compiled harnesses can substantially improve procedural execution on a selected set of procedure-heavy skills. It does not yet demonstrate source-level semantic correctness, general compliance guarantees, or broad generalization to the wider skill ecosystem.

Critical strengths

Strength 1: A clear and useful systems abstraction

The separation between procedural structure and semantic model decisions is the strongest conceptual aspect of the work. AIS and AG-IR give the compiler an explicit vocabulary for requirements, data dependencies, control flow, executable bindings, semantic operations, and delegated subprocedures. This is a meaningful systems contribution because it identifies where deterministic execution can reduce procedural failures without attempting to eliminate model judgment entirely.

Strength 2: Broad runtime-model coverage with a concrete end-to-end benchmark

The evaluation uses four runtime models and three compiler models, rather than demonstrating the method only on the same model used for compilation. The additional SkillsBench evaluation is valuable because it tests whether compiled skills help an outer agent complete tasks that require more than a single isolated procedure. The reported gains are substantial enough to motivate further investigation, even though the experimental controls are currently insufficient.

Critical weaknesses

Weakness 1: The evaluation does not isolate the cause of the reported gains

The comparison is between direct textual skill execution and a substantially different compiled runtime. SIGIL provides deterministic control flow, scoped inputs and knowledge, explicit tool bindings, structured state, and an MCP-accessible harness. The paper does not include the ablations needed to determine whether the gains come from:

  • the AIS/AG-IR representation;
  • deterministic control flow;
  • scoped model context;
  • reduced prompt length;
  • explicit tool wrappers;
  • a stronger outer-agent interface;
  • Jac execution; or
  • compilation itself.

The GPT-5-based adherence judge further weakens the claim that 100% represents independently verified behavioral correctness. A compiled harness can structurally contain a required model call while the model still produces an incorrect semantic result. This is especially consequential for the clinical, HIPAA, GDPR, and compliance examples.

Weakness 2: The evidence is too narrow and incomplete for the reliability and efficiency claims

The 11-skill suite is curated toward procedure-heavy compliance tasks and does not establish generality. The submission provides no repeated-compilation study, no uncertainty estimates, no complete cost accounting, and no detailed release or reproducibility information. The token results exclude compilation cost and count a baseline that often stops early, making the headline efficiency claim difficult to interpret without a quality-normalized amortization analysis. The paper also lacks direct empirical comparisons with several closely related executable-skill and workflow-compilation systems.

Questions for authors

The following are clarification requests whose answers could materially change my assessment.

  1. Evaluation independence: What proportion of Skill Adherence judgments has been independently checked by human experts or an independent automated verifier? Please provide inter-rater agreement, judge calibration results, the exact judge prompt, and examples of disagreements between GPT-5 and human reviewers.

  2. Baseline parity: Do the direct SKILL.md agents receive exactly the same tools, supporting resources, system instructions, model-call opportunities, input data, and outer-agent capabilities as SIGIL? If not, please quantify each difference and provide ablations with matched scoped context, matched tool wrappers, and an identical controller.

  3. Ablations: What are the results for a manually authored harness, a harness with deterministic control flow but no AG-IR, a harness with scoped prompts but no procedural compilation, and a generated workflow without typed carries? These experiments are necessary to attribute the gains to the proposed method.

  4. Reference and task construction: Who selected the 11 skills and 33 tasks, how many domain experts reviewed each reference DAG, and were any authors involved in creating or approving the evaluation references? Please provide the complete skill packages, task prompts, reference DAGs, and per-mandate labels.

  5. Cost and reproducibility: What are compilation token counts, repair rounds, compile failures, wall-clock latency, monetary cost, and the break-even number of runtime executions at which SIGIL becomes cheaper than direct execution? Please also report model versions, temperatures, seeds, tool settings, and variance over multiple independent compilations.

  6. Semantic correctness and safety: For the clinical and compliance skills, how do you evaluate the correctness of model-generated classifications and recommendations, rather than only whether the required calls and ordering constraints were executed? In particular, how are false negatives, incorrect PHI classification, unsafe clinical outputs, and prompt-injection attempts handled?

4 comments
AuthorsOpenReview ↗

What the comparisons establish.

Thank you for the thoughtful feedback.

SIGIL’s contribution is automatically deriving a reusable execution harness from an authored skill. Figure 3 in the paper shows how AG-IR represents the required operations, data dependencies, control flow and bindings for validation and deterministic lowering. The resulting harness preserves the procedure while leaving semantic decisions to the runtime model.

In the paper, Figure 4(a) evaluates reuse of the same compiled artifacts across runtime models. Figure 4(b) compares SIGIL with instruction optimization and adaptation using GPT-4o and the same adherence evaluation. These comparisons assess the benefit of the complete compilation approach.

For the SkillsBench comparison in Figure 8 of the paper, both conditions use OpenHands and the same runtime model, which also powers SIGIL’s internal model calls. The baseline receives the original SKILL.md, while SIGIL exposes compiled harnesses through MCP (§4.3). The shared framework and runtime model provide the comparison setting; the generated procedure, bindings, state and scoped calls constitute the compiled approach being evaluated.

AuthorsOpenReview ↗

Compilation cost and reuse across tasks.

Compilation is a separate, one-time cost per skill harness. SIGIL compiles the skill’s procedure without task inputs, allowing the completed harness to be reused for repeated runs of the same task and for different tasks within the skill’s scope, without task-specific training or recompilation (Figure 3 in the paper). Validation and internal repairs are part of this initial process.

Figure 7 in the revised paper compares this reuse with SKILL.md + ReAct + supervisor using GPT-4o. Both methods complete 15 runs of each of three increasingly difficult tasks before moving to the next. The baseline retains corrective feedback from Claude Code running Claude Opus 5 across runs and tasks. SIGIL reuses the unchanged harness with fresh runtime state for every execution.

Figure 7(b) in the revised paper shows 100% measured Skill Adherence (SA) for SIGIL in all executions, including each new task’s first run. Even retrospectively selecting the baseline’s best run for each skill–task pair yields only 73.4% mean SA across skills on the final task.

Section 4.2 in the revised paper reports a mean compilation cost of 4.457M tokens per skill. Figure 7(a) includes compilation, internal repairs, all executions and baseline supervisor calls. SIGIL’s mean cumulative token use is already lower than the supervised baseline’s at the second task checkpoint (30 runs per skill) and is 35.3% lower by the third. The revised paper therefore reports cost and adherence together, demonstrating amortization under this repeated-use protocol.

The runtime-only comparison in Figure 6 of the paper also distinguishes lower token use from completed execution. For Llama-3.1-8B, SIGIL uses additional tokens to complete steps omitted by direct execution. Figure 7 in the revised paper separately evaluates compilation-inclusive token use alongside measured adherence.

AuthorsOpenReview ↗

How execution and semantic correctness are evaluated.

A required model call can execute while producing an output that fails the requirement. The adherence results in Figure 4 of the paper and Figure 7 of the revised paper therefore check both whether the required code ran and whether the resulting behavior satisfied the reference requirements.

GPT-5 compares the task, execution trace and relevant code against fixed, expert-reviewed reference DAGs. These references are derived from the source skills independently of the compiler. For applicable deterministic mandates, runtime instrumentation records whether the required code ran, including ordinary code between model calls. A deterministic mandate automatically receives zero credit if the trace does not confirm the required execution, as clarified in §4.1.1 of the revised paper. GPT-5 judges the required actions, conditions, ordering and semantic results.

In the HIPAA example in Figure 3 of the paper, PHI classification remains a model judgment. If an incorrect classification causes an applicable BAA check to be skipped, the affected requirements fail under the scoring rules. An incorrect branch does not make an otherwise applicable obligation disappear (Appendix D.3 in the paper). Thus, the presence of a model call alone does not establish adherence.

Figure 5 in the paper separately evaluates requirement preservation in compiled artifacts, including omissions and differences in structural encoding. Expert review establishes the reference requirements; GPT-5 performs the artifact and execution assessments. The reported 100% is measured SA under this evaluation procedure.

AuthorsOpenReview ↗

Task construction and broader task performance.

In the revised paper, §4.1 clarifies that we constructed evaluation tasks for public source skills and that an independent domain expert reviewed each skill’s reference DAGs against the source and supporting resources. No author served as an expert reviewer. This review established the references used in the adherence evaluation in Figure 4.

The two benchmarks measure complementary outcomes. In the compliance-critical suite, required checks, actions and ordering are part of correctness; a plausible final output is insufficient. SkillsBench evaluates broader tasks using native task verifiers, where successful completion does not necessarily require a particular prescribed procedure. Figure 8 in the paper demonstrates that the benefit of compiling skills extends beyond procedural adherence to broader task performance.

Reviewer_6KnvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SIGIL, a framework that compiles compliance-critical agent skills before runtime, rather than leaving the LLM to interpret natural-language skill descriptions at execution time. The authors define an intermediate representation (AG-IR) and an instruction set (AIS) as compilation targets. Skills are compiled through an iterative loop in which an LLM judge detects errors and fixes are applied. SIGIL was evaluated on an internal benchmark of compliance-critical skills and on SkillBench, an external benchmark, and it improved performance on both.

Critical strengths
  1. The motivation for addressing compliance-critical skills through pre-runtime skill compilation is well-founded.
  2. The structure (AG-IR) and instruction set (AIS) required for skill compilation are reasonably and clearly defined.
  3. The method is evaluated on both an external benchmark (SkillBench) and an internal benchmark (11 self-collected compliance-critical skills), and it shows performance gains on both.
  4. In particular, the very large gain for a weak model (Llama 3.1 8B) strongly supports the need for skill compilation. I especially appreciate this contribution.
Critical weaknesses
  1. Although the paper uses both an internal benchmark and an external benchmark (SkillBench), nearly all of the main experiments and ablation studies are conducted on the internal benchmark; the SkillBench results appear only in RQ3 and Figure 7. On the internal benchmark, it seems that compliance would almost inevitably be high as long as the skill is compiled correctly, regardless of how it is executed. This is most apparent in Figure 4, where Llama 3.1 8B achieves Skill Adherence comparable to frontier models. As a result, I can only interpret the RQ1 results as “a sentence that has been run through a spell checker shows a high spelling-compliance rate.” To properly address RQ1, the authors should have paid more attention to the independence between compilation and evaluation.
  2. Demonstrating SIGIL’s improvement on an external benchmark (SkillBench) is a strength, but the experiments on that benchmark are not thorough enough. The diverse and interesting analyses conducted on the internal benchmark (e.g., differences across the LLMs used for compilation, comparisons with existing methods such as GEPA and SkVM, token usage, and skill adherence analysis) should also have been carried out on the external benchmark.
  3. The quality of skill compilation appears to be critical, yet there is no analysis of how many tokens/iterations the compilation requires. According to Algorithm 1, skill compilation is not a single step. It is a loop: compile, use an LLM judge to detect errors, and revise if errors are found. The paper should report how many iterations this loop runs on average, and the cost of this process should have been included in RQ2.
Questions for authors
  1. Is compilation performed only once per skill? What was the success rate (yield) for obtaining a successful compilation?
  2. Did the authors consider conducting a human evaluation to verify that the compiled outputs faithfully capture all the content of the original skill?
5 comments
AuthorsOpenReview ↗

Compilation cost and reuse across tasks.

Thank you for the thoughtful feedback.

Compilation is a separate, one-time cost per skill harness. SIGIL compiles the skill’s procedure without task inputs, so the completed harness can be reused for repeated runs of the same task and for different tasks within the skill’s scope, without task-specific training or recompilation. Figure 3 and §§3.3–3.4 in the paper describe this compilation process. That cost is amortized across subsequent executions.

Figure 7 in the revised paper compares this reuse with SKILL.md + ReAct + supervisor on the compliance suite using GPT-4o. Both methods complete 15 runs of each of three increasingly difficult tasks before moving to the next. The baseline retains corrective feedback from Claude Code running Claude Opus 5 in its context across runs and tasks, while each SIGIL run starts with fresh runtime state.

Figure 7(b) in the revised paper shows 100% measured Skill Adherence (SA) for SIGIL in all executions, including each new task’s first run. Even selecting the baseline’s best run retrospectively for each skill–task pair yields only 73.4% mean SA across skills on the final task. The unchanged harness therefore maintains full measured adherence without supervisor feedback or task-specific refinement.

Figure 7(a) in the revised paper includes all executions, baseline supervisor calls, and SIGIL’s one-time compilation and internal repairs. Despite the upfront cost, SIGIL’s mean cumulative token use falls below the baseline’s by the second task and is 35.3% lower by the third. Section 4.2 thus shows that repeated use amortizes compilation while maintaining full measured adherence under the evaluated protocol.

As clarified in §4.1.1 of the revised paper, the one-time compilation includes bounded validation and repair before the completed artifact is reused (Figure 3). Compilation can fail if errors remain; the overall compilation success rate and distribution of repair rounds are not reported.

AuthorsOpenReview ↗

How adherence is evaluated.

A harness can pass the compiler’s internal checks while still omitting a required step or encoding the wrong conditions or execution order. Adherence is therefore evaluated against fixed reference DAGs constructed independently of the compiler and reviewed by domain experts. These references are derived from the source skills, not from the compiled harnesses. The compiler does not receive these DAGs; it validates its own source-derived requirements, as described in Appendices A.2 and C of the paper and clarified in §3.3 of the revised paper. Expert review thus applies to the evaluation references, not to the compiler’s output.

Figure 4 in the paper and Figure 7 in the revised paper assess actual task executions. For applicable deterministic mandates, runtime instrumentation records whether the required code ran; a mandate automatically receives zero credit if the trace does not confirm that execution, regardless of the judge’s assessment. Section 4.1.1 in the revised paper makes this execution check explicit. GPT-5 evaluates the execution trace and relevant code against the reference DAG to determine whether the required action was performed under the correct conditions and in the required order. A required model call may execute while still producing an output that fails the requirement.

The separate fidelity analysis in Figure 5 of the paper identifies omitted reference requirements in harnesses compiled by GPT-5.6-Sol and differences in how compilers encode requirements (§4.1.3). Requirement preservation is therefore evaluated separately, rather than assumed from successful compilation.

Llama-3.1-8B’s high SA in Figure 4(a) of the paper is consistent with the intended role of the harness: it controls the procedure instead of relying on the runtime model to remember every required step. The score still depends on whether the resulting execution satisfies the independently reviewed requirements.

AuthorsOpenReview ↗

Why we use two benchmarks and where we compare GEPA and SkVM.

We use two benchmarks to distinguish procedural adherence from general task performance. In the 11-skill compliance-critical suite, following the prescribed procedure is part of correctness. A plausible final output is insufficient if required checks, actions or ordering are skipped. We therefore assess execution against the skill’s requirements to measure SA, as reported in Figure 4 of the paper (§4.1).

GEPA and SkVM address a specific question within this evaluation: whether instruction optimization or adaptation can achieve the procedural adherence obtained through compilation. Figure 4(b) in the paper compares these methods using the same GPT-4o runtime and adherence evaluation, with GEPA explicitly optimized for SA (§4.1.2). This directly tests the adherence benefit against optimization and adaptation baselines.

SkillsBench evaluates broader tasks where a correct result does not necessarily require following a particular prescribed procedure. Figure 8 in the paper compares compiled harnesses with direct execution of the original skills across four runtime models, using native task verifiers (§4.3). These complementary evaluations establish the procedural-adherence benefit on compliance-critical skills and demonstrate the broader task-performance benefit of compiling reusable skills.

Reviewer_yLACOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an approach that mimics program compilation, translating a program into a collection of instruction sets, to the area of prompting in large language models.

The proposed approach takes in skills (skill.md) like a program and translates or compiles it into a set of agent instruction set, like instruction sets, and showing that such compiled representation with a pre-defined set of instruction sets shown in Table 1 improves the performance of the task.

In the experiment, three questions are addressed. (1) Does this approach follow requirements better? (2) Does this approach reduce token usage? (3) Does this approach improve performance? The reported results in the submission show improvements.

Critical strengths

The strength of this submission is that (1) the proposed approach demonstrates consistent improvements in the experiment section.

In the paper, the runtime models for executing a compiled version are Claude Opus 5, GLM-5.2, GPT-4o, and Llama-3.1-8B, and the compile-time models are Claude Opus 5, GLM-5.3, and GPT-5.6-Sol. This shows that a smaller-scale model (Llama-3.1-8B) could be used with frontier models for compilation.

Critical weaknesses

The paper doesn't show how this is achieved. Section 3 is the main part of the paper that defines concepts like instruction sets and intermediate steps, as we have them in programming languages. However, I cannot find how it is done.

Questions for authors

How does an input skill translate to a collection of instruction sets that are robust to be executed across different models, given a fixed set of elementary instruction sets?

4 comments
AuthorsOpenReview ↗

How the source skill becomes an executable harness.

SIGIL turns the skill’s prescribed procedure into executable control flow while retaining model judgment. Figure 3 in the paper shows how the compiler extracts requirements, source spans and applicability conditions, separates supporting knowledge from executable bindings, and represents the operations and their dependencies in AG-IR (§3.3).

In the HIPAA example in Figure 3 of the paper, SELECT classifies PHI. On the PHI path, OBSERVE obtains Business Associate Agreement (BAA) status, and BRANCH permits the transmission ACT only when that status is established; otherwise it returns a blocked result. Typed values connect the operations, with explicit ordering and path constraints (§§3.1–3.2).

Validation checks requirement realization, types, references, bindings and control constraints, with bounded repair and failure if errors remain. Deterministic lowering to Jac maps semantic operations to typed model calls, bound operations to executable calls, and branches to control flow, without further source interpretation. In the revised paper, §3.4 expands the Figure 3 walkthrough; Appendix A in the paper provides the validation algorithm and lowering rules.

AuthorsOpenReview ↗

Why the fixed instruction set can express different skills.

Thank you for the thoughtful feedback.

AIS expresses different skills through parameters and composition. Typed inputs and outputs, knowledge and bindings give each instruction its skill-specific meaning; AG-IR composes these instances. SELECT can classify PHI, as in Figure 3, or make another semantic choice in a different skill. DELEGATE retains model-directed orchestration where the skill leaves a subprocedure’s internal steps open. In the revised paper, §3.1 makes this distinction explicit: the instruction vocabulary is fixed, while its instances and their composition vary.

AuthorsOpenReview ↗

What is retained across tasks and runtime models.

Reuse preserves the procedure across new task inputs and runtime models. Changing the model can change PHI classification, but it preserves the encoded BAA check on the PHI path, as illustrated in Figure 3 of the paper. Source interpretation and runtime semantic outputs remain model-dependent (§3.3; Appendix A.2).

The same compiled artifacts are reused across four runtime models (Figure 4(a) in the paper; §4.1.1). In the revised paper, §4.2 adds a GPT-4o experiment in which harnesses compiled without task inputs are reused unchanged on increasingly difficult tasks, with fresh runtime state for every run. They achieve 100% measured Skill Adherence in all executions, including each new task’s first run (Figure 7(b)). This demonstrates that adherence persists without task-specific recompilation or supervised refinement.

Thanks very much for your response.

Agent instruction set defines instruction sets for model calls, tool calls, control flow and delegation. This paper shows a kind of domain-specific language that translates or compiles the natural language prompt or skill into this. However, we don't see how those instruction sets were chosen and whether existing set is sufficent to capture the nautral langauge skills. On the 11 selected compliance skills, 16~25 percent of reference mandates remain as runtime instructions rather than being structurally encoded in Figure 5. It is not clear whether this is a limitation of AIS or the LLM that compiles. All the compilation process is performed by LLMs, so the LLM rewrites skill in the proposed DSL. The verification is also decided by LLMs. The core question is how this translation is done.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.