acceptodds
Under review as a conference paper at ICLR 2027

HEXIS: Compiling Agent Skills into Extended Finite State Machines

17 views

Abstract

Agent skills provide instructions and reusable knowledge, yet agents have to frequently infer which action to take next. This skill's execution paradigm combines task reasoning with control decisions, which could cause the agent to act improperly or omit necessary steps. To improve agent compliance for skills, we introduce HEXIS, which compiles agent skills into extended finite state machines (FSM) that separate knowledge from control flow. Skill knowledge is incorporated into local instructions that guide reasoning and generation within states. The machine records execution progress and intermediate results, while explicit transition conditions determine subsequent operations. Our incremental compiler first maps skill clauses and tool interfaces to state operations, local instructions, data bindings, and transitions. It then aligns development traces with existing states to identify missing operations and dependencies. These are incorporated by adding or reusing states and refining their connections. Updates are accepted only after static checks and replay of the current and all previously accepted traces. Across four benchmarks and four executors, HEXIS improves success over Skill + ReAct by 16.2 percentage points on average. Qwen3.8-27B reduces execution tokens by 38.4-88.9% across benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, skill compilation, automata learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P33BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

HEXIS turns a written agent skill into a state machine, so the runtime handles the procedure and the LLM reasons within each step. The machine is then refined using execution traces. The paper reports improvements across four benchmarks and four executors.

Critical strengths

I like the motivation. Asking an LLM to repeatedly work out a procedure that could be enforced by the runtime seems wasteful and error-prone. The comparison against using the compiled machine as a prompt is useful, and I find the transfer results encouraging.

Critical weaknesses

My main concern is that I still cannot tell how much the proposed compiler contributes. Appendix B.1 convinced me that executing a workflow can work better than describing it in a prompt. But how much of the gain is already there in the initial machine, before any trace-guided updates? That comparison is missing. Given the closely related ideas in Formal Skill, TraceCompiler, and Compile, Then Page, I need a clearer reason to prefer this compilation procedure. The current results leave me unsure what the incremental refinement and replay actually buy us.

I also found problems in the theoretical analysis that make me less confident in its conclusions. The algorithm discards traces that fail validation, yet Proposition A.2 treats the accepted traces as iid samples from the original distribution. Proposition A.3 uses H(B | Z, F=1) <= H(B), which does not hold in general. I do not think Appendix A currently justifies the claimed connection between replay and the representation objective. The empirical gains remain interesting, but this theoretical argument needs to be corrected.

Finally, I think the evaluation tells us too little about whether the skills were compiled faithfully. There are only four source skill documents. Higher task success and better output compliance do not tell me whether the compiler dropped a condition or ruled out a valid branch. I appreciate that the authors acknowledge this limitation, but preserving the meaning of the skill is central to calling this a compiler. I would like to see that checked directly, rather than inferred from downstream scores.

Questions for authors
  1. How well does the initial validated machine perform before trace refinement? Does archive replay make a meaningful difference on held-out tasks?
  2. What does HEXIS gain over a simpler executable-skill approach under the same setup? I am particularly interested in the benefit of the proposed compiler.
  3. How would you correct the sampling argument in A.2 and the entropy inequality in A.3? Which conclusions would still hold?
  4. Have you checked the compiled machines against the original skills for missing conditions and lost branches? What did that check find?
6 comments

I received the committee’s message about possible AI-generated reviews and would like to clarify my use of AI.

My initial concerns came from my own reading of the paper. I used AI to discuss these concerns, check whether I had misunderstood anything or made unfair judgments, and help put my comments into English. I have included the conversation to make this process transparent. The final review reflects my judgment, and I take responsibility for it.

I am open to the authors’ replies and will update my assessment if they address my concerns. I will follow the discussion and respond promptly.

AuthorsOpenReview ↗

Response to Reviewer P33B, Part I

We thank the reviewer for the detailed comments and for the positive view of the motivation, the Prompt-only comparison, and the transfer results.

Q1. Performance of the initial machine, and the effect of archive replay.

We agree that this comparison is missing. Table 3 fixes the final machine and varies how it is executed, so it cannot separate the initial compilation from trace refinement. The comparison bears on the main idea of the method, so we state that idea first.

Native execution loses part of what the skill specifies. The control requirements are in the context, but every decision has to recover them again, and each decision can deviate (Eqs. (2) and (3)). Initial compilation is lossy too. M0M_0 is drafted from the document alone and can omit operations, dependencies, and branches that executing the skill requires. Incremental compilation exists to recover this loss. Each native trace exposes something the current machine lacks, the compiler adds it as states, bindings, and guarded transitions, and replay of all accepted traces keeps it in the machine. A native run that performs a required step does not make the next run more likely to perform it. The machine retains it.

The development tasks are processed in groups of 10. Each group is executed natively with Skill + ReAct, and its traces are then used for incremental compilation. Before a group's traces are compiled in, the machine obtained so far is evaluated on the test tasks. We show LiveMath, the benchmark with the largest gain in Table 1. Its 487 development tasks form 49 groups, and no update changed the machine after group 20, so groups 21–49 were all evaluated with the final machine.

Table R1. LiveMath success by compilation stage, qwen3.6-flash.

MachineTasksMachineSkill + ReAct, same tasks
M0M_0 (no traces)development group 148/121 (39.7%)none
Machines during refinementdevelopment groups 572/121 (59.5%)none
Machines during refinementdevelopment groups 1086/121 (71.1%)none
Machines during refinementdevelopment groups 2096/121 (79.3%)none
Final machine (Table 1)held-out test93/121 (76.9%)54/121 (44.6%)

M0M_0 does no better than native execution (39.7%, against 44.6% for Skill + ReAct on test) and is far below the final machine. This is the pattern the method predicts: the initial machine is lossy, the loss is recovered within the first groups of traces, and the machine then keeps what it has recovered.

AuthorsOpenReview ↗

Response to Reviewer P33B, Part II

Q2. What does the compiler add compared with a simpler executable-skill approach?

The simplest executable-skill approach in our setup is M0M_0: one compilation from the skill document, with static checks and no traces. Our results separate what each further component adds.

StepWhat changesResult with qwen3.6-flash
Skill + ReAct → Prompt-onlyThe model receives the final machine as text but still controls execution44.1% → 47.3% on average. LiveMath drops from 44.6% to 33.9% (Table 3)
Prompt-only → HEXISThe runtime executes the same machine47.3% → 64.1% on average, +16.8 points (Table 3)
M0M_0 → final machineTrace refinement with replayLiveMath: M0M_0 is at the level of native execution; the final machine reaches 76.9% (Table R1)

A compiled description alone adds little, and runtime execution of the final machine adds 16.8 points. The third row is the reason to prefer incremental compilation over a one-shot executable skill. One-shot compilation is itself lossy: on LiveMath, M0M_0 under the same runtime does no better than native execution. Runtime control helps only when the machine contains the operations the task needs, and incremental compilation is how the machine acquires them from native traces and keeps them.

Two further results concern the procedure itself. With Sonnet 5 at low effort as the compiler, the machines score 2.0–4.2 points below those compiled by Fable 5.1 and stay above Skill + ReAct on all four benchmarks (Table 4), so the gains do not depend on one compiler model. AFlow is the baseline in Table 1 that also produces an executable workflow under the same executor and tools, obtained by search over execution feedback and not from the skill document. HEXIS is higher in 13 of 16 settings, tied in 1, and lower in 2. AFlow's workflows fall to 51.0% and 15.7% on DABench with the two local executors, while HEXIS machines are reused across executors without recompilation.

The related methods differ from HEXIS in their input and in what they keep from it:

  • Formal Skill makes control explicit, but the program is written by hand. HEXIS generates the machine from an existing natural-language skill.
  • TraceCompiler mines agent traces into mostly deterministic workflows. HEXIS starts from the skill document and keeps its knowledge in state prompts, so the model still reasons inside each state. Traces are used only to add missing operations and dependencies.
  • Compile, Then Page compiles SOP constraints that are already machine-readable. HEXIS must first extract the rules from free text (Section 4.2) and check each one against a quotation from the document.

HEXIS therefore covers a setting the other three do not: it starts from an existing natural-language skill and needs neither a hand-written program nor machine-readable constraints. Formal Skill and Compile, Then Page require inputs that the four skill documents do not provide, so they cannot be run on our benchmarks as they stand. TraceCompiler works from traces without the skill document, and a direct comparison on our development traces is a natural extension. We will state this positioning explicitly in Section 2.

Reviewer_CHE6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To improve agent compliance with skills, the paper introduces HEXIS, which compiles agent skills into extended finite state machines (FSMs) that sep- arate knowledge from control flow. It then aligns development traces with existing states to identify missing operations and dependencies.

Critical strengths

This is the first time I have seen a method which converts reasoning/steps into state machines which an agent can use to improve task performance.

Critical weaknesses
  1. The paper is too difficult to understand at many points.
  2. The use of a finite state machine to control how the agent is too unscalable and difficult to manage in real-world settings. For instance, you need a construction model to edit previous reasoning traces. As such, I'm unsure if it has any true research value in the real-world...
Questions for authors
  1. How would section 4.2 and 4.3 look like from a real world POV of agents? can you give us a concrete example of what happens in a particular LLM task?
5 comments
AuthorsOpenReview ↗

Reponse to Reviewer CHE6, Part I

We thank the reviewer for the time and comments. We respond to each weakness and question below.

W1. "The paper is too difficult to understand at many points."

We would be grateful if the reviewer could point out the specific parts that are difficult. The review mentions Sections 4.2 and 4.3, so our answer to Q1 walks through them step by step with a real example.

W2. "The use of a finite state machine to control how the agent is too unscalable and difficult to manage in real-world settings. For instance, you need a construction model to edit previous reasoning traces. As such, I'm unsure if it has any true research value in the real-world..."

We respectfully disagree this comment. HEXIS has two separate phases, offline compilation and online execution, and the example in the comment concerns only the offline phase.

Offline compilation. The construction model and the traces are used only in this phase, once for each skill before deployment. Traces are only read, and the compiler modifies a copy of the machine, as Section 4.3 describes. A skill is written for work that is repeated many times with inputs of a defined format, so this one-time cost is shared by all later executions. The baselines AWM, ReasoningBank, SkillOpt, and AFlow also rely on executions of the development tasks.

Online execution. At run time the machine is fixed, and the construction model and the traces are no longer used. Each task runs as a separate instance of the machine with its own state and variables, so executing many tasks is the same as running many independent agent sessions.

How a machine runs inside a harness. A compiled machine runs inside an existing agent harness, and the harness keeps providing the tools, the model access, and the working directory. In ordinary skill execution the model chooses the next tool call at every turn. With HEXIS a small runtime loop takes this role. It looks up the current state in the machine file and performs the operation of that state. For a tool state it passes the tool call to the harness, which validates and executes the call with its own tools and returns the result. For a model state or a judge state it sends the prompt of that state and the variables it reads to the execution model. The runtime then stores the outputs in the variables and follows the first transition whose condition is true, as Equation 8 defines. In our experiments the harness is OpenCode, and Table 6 shows the calls to its bash and read tools. When a machine reaches its fallback state, control returns to ordinary skill execution in the same harness.

Scalability. The runtime keeps no shared state between tasks. A run consists of the current state and a table of variables, and the machine file is only read. Each task is therefore an independent instance with its own state and variables, and any number of tasks can run in parallel like ordinary agent sessions.

Management. A machine is one JSON file with limited states, and the same runtime executes all machine files, so supporting a new skill adds one file. Every run records the states it visited, so a problem can be traced to a specific state. In our tests, 276 of 280 runs needed no retry and no fallback.

Practical value. Moving control flow out of the model and into an explicit program is the direction that major agent platforms took in 2026. Anthropic added dynamic workflows to Claude Code, where a script decides what runs next and the model works inside each step. Its documentation contrasts this with skills, where the model decides what runs next by following the prompt [1]. Google rebuilt its Agent Development Kit around graph workflows. Its engineering blog states that model-driven control flow is unreliable in production, and that ADK 2.0 separates routing, which a graph decides with explicit conditions, from reasoning, which stays in the model [2]. Microsoft Agent Framework defines workflows as graphs of executors and edges [3], and the Agent Service of Scale AI documents state machines with states and transitions for building agents [4]. These systems rely on a developer or the model to write the control program for each task. The usual difficulty with state machines is that people have to write and maintain them by hand, and HEXIS addresses this difficulty because the compiler generates the machine automatically from an existing skill document. And that's why we think this issue should be well discussed in an academic way. The problem that HEXIS targets is also well documented, since SOPBench [5] and AgentIF [6] report that LLM agents have difficulty following procedural requirements.

AuthorsOpenReview ↗

Reponse to Reviewer CHE6, Part II

Q1. "How would section 4.2 and 4.3 look like from a real world POV of agents? can you give us a concrete example of what happens in a particular LLM task?"

We walk through the spreadsheet skill from our experiments, which Figure 3 also illustrates. The compiled machine is the file examples/machines/spreadsheet/machine.json in our anonymous repository.

The skill. The skill is a natural-language document named structure-preserving-spreadsheet-edits. Its workflow has three parts, which are Inspect Before Editing, Edit Narrowly, and Verify The Saved Workbook. Each part mixes knowledge with control. The second part explains how to change only the requested cells, which is knowledge. The document also requires that inspection comes before editing and that the saved workbook is verified before the task is reported as done, which is control. A task gives the agent a request, an input workbook, and an output path.

Section 4.2, step 1: extract rules. The construction model reads the skill and the description of the bash tool, and it lists the control requirements, each with a sentence quoted from the skill. For this skill the rules state that the workbook is inspected before it is edited and that a run may end as verified only after the saved workbook has been checked. A rule whose quotation cannot be found in the skill is discarded.

Section 4.2, step 2: draft the states. The construction model writes one state for each operation and puts the knowledge of the skill into the prompt of that state. The first part of the skill becomes s1, a model state that writes an inspection command, and s2, a tool state that runs it. Two further states, s3 and s4, determine which cells must be edited and judge whether the inspection is sufficient. The second part becomes s5, which plans the edit and writes the edit command, and s6, which runs it. The third part becomes s7g, which writes a verification program, s7, which runs it, and s8, a judge state that reads the verification report and outputs pass, wrong_cells, or boundary_wrong.

Section 4.2, step 3: add the transitions. The control requirements become transitions with conditions over variables. After s2, a successful command leads to the next state, and a failed command leads back to s1 and increases a counter that is limited to three. After s8, the label pass leads to the final report and the verified end state, the label wrong_cells leads back to s5, and the label boundary_wrong leads back to s3.

Section 4.2, step 4: run the static checks. The checker examines the draft before it is accepted. It confirms that every state is reachable and has a default transition, that each variable such as the verification report is written before a state reads it, and that the verified end state cannot be reached without passing through s7 and the label pass of s8. If a check fails, the errors are returned to the construction model, which drafts the machine again. The result is the initial machine in the first panel of Figure 3.

Section 4.3, step 5: read a trace. A development trace is the log of one Skill + ReAct run. The trace in the second panel of Figure 3 contains a command that reads the workbook, a command that writes it, two verification commands, and a final answer. The compiler merges the two consecutive verification commands into one event, which gives the events read, write, verify, answer, and end.

AuthorsOpenReview ↗

Reponse to Reviewer CHE6, Part III

Section 4.3, step 5: read a trace. A development trace is the log of one Skill + ReAct run. The trace in the second panel of Figure 3 contains a command that reads the workbook, a command that writes it, two verification commands, and a final answer. The compiler merges the two consecutive verification commands into one event, which gives the events read, write, verify, answer, and end.

Section 4.3, step 6: align the events and modify a copy. The compiler aligns each event with a state that performs the same operation. The read event matches s2, the write event matches s6, the verify event matches s7, and the answer matches the report state. The initial machine runs the verification once, while the trace runs it twice. The compiler therefore modifies a copy of the machine and adds a judge state s7_loop after s7, which decides whether the verification must be run again, together with a counter that bounds this loop. Other traces lead to changes of the same kind. One trace ended after the edit without verification, while earlier traces continued to verification, so two continuations followed s6 under the same condition. The compiler then inserted a judge state s6_j that chooses between them, as Equation 13 describes.

Section 4.3, step 7: check and replay. The modified copy must pass the static checks of step 4, and it must replay this trace and every trace accepted earlier. During replay each state returns the output recorded in the trace, so no model or tool is called. If the copy passes, it replaces the machine, and otherwise the machine is kept unchanged. After all development traces, the machine has 17 states and 35 transitions, as Table 5 reports.

Execution of a new task. The compiled machine runs the states in the order described above. The model writes the commands and the judgments inside the states, and the runtime decides the order. Inspection therefore always precedes editing, a failed command is retried a bounded number of times, and a result is reported as verified only after the judge outputs pass. Appendix B.4 gives a complete recorded run of the LiveMath machine with every state on its path.

We hope this reply addresses the reviewer's concerns. We would be glad to answer any questions during the discussion period.

References

[1] Orchestrate subagents at scale with dynamic workflows. Claude Code Docs, Anthropic. Feature introduced May 28, 2026. https://code.claude.com/docs/en/workflows

[2] Why we built ADK 2.0. Google Developers Blog, July 1, 2026. https://developers.googleblog.com/en/why-we-built-adk-20/

[3] Microsoft Agent Framework Workflows. Microsoft Learn, October 2026. https://learn.microsoft.com/en-us/agent-framework/workflows/workflows

[4] State Machines. Scale AI Agent Service documentation. https://docs.gp.scale.com/docs/agents/state-machines

[5] Zekun Li et al. SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints. arXiv:2503.08669, 2025.

[6] Yunjia Qi et al. AGENTIF: Benchmarking Large Language Models Instruction Following Ability in Agentic Scenarios. NeurIPS 2025, Datasets and Benchmarks Track.

Reviewer_8ASCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to turn natural-language agent skills into executable procedures. HEXIS represents each skill as an extended finite state machine with local model and tool operations, typed variables, and guarded transitions. A construction model drafts the machine and refines it using development trajectories, accepting updates after static checks and replay of the current and previously accepted traces. Experiments on four benchmarks and four executors report higher success than Skill + ReAct in 15 of 16 settings, with additional comparisons examining runtime execution and transfer across executors.

Critical strengths
  1. HEXIS provides a concrete route from existing skill documents to executable workflows, reducing the need to implement their control logic manually. Local model calls retain task-dependent reasoning, while typed variables and guarded transitions make the encoded ordering and data dependencies enforceable by the runtime.

  2. Table 2 provides useful evidence that skill optimization and compiled execution are complementary. On SpreadsheetBench, HEXIS raises success with the SkillOpt-optimized document from 59.6% to 84.2%, while optimizing the document also improves HEXIS over compiling the original skill, which achieves 75.4%.

Critical weaknesses
  1. The benefit of trajectory-guided refinement over the initial compilation is not quantified. Table 3 compares ways of executing the final machine, but the paper does not report the performance of M0M_0 under the same runtime.

  2. Static validation checks the extracted rules RDR_D, while RFC measures request and interface compliance. It is not clear how completely these checks cover the source skill’s conditional and ordering requirements.

  3. The proof of Proposition A.3 uses H(B∣ZM,F=1)≤H(B)H(B\mid Z_M,F=1)\le H(B), which need not hold for a particular failure event. The information-loss bound is therefore not established by the given argument.

Questions for authors
  1. How does the initial machine M0M_0 compare with the final machine when both use the same executor and runtime?

  2. For one evaluated skill, which procedural requirements are covered by the static checks and RFC, and which are not assessed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8ASC, Part I

We thank the reviewer for the careful reading, and in particular for checking the proof in Appendix A.

W1 / Q1. Performance of M0M_0 under the same runtime.

We agree that this comparison is missing. Table 3 fixes the final machine and varies how it is executed, so it cannot separate the initial compilation from trace refinement. The comparison bears on the main idea of the method, so we state that idea first.

Native execution loses part of what the skill specifies. The control requirements are in the context, but every decision has to recover them again, and each decision can deviate (Eqs. (2) and (3)). Initial compilation is lossy too. M0M_0 is drafted from the document alone and can omit operations, dependencies, and branches that executing the skill requires. Incremental compilation exists to recover this loss. Each native trace exposes something the current machine lacks, the compiler adds it as states, bindings, and guarded transitions, and replay of all accepted traces keeps it in the machine. A native run that performs a required step does not make the next run more likely to perform it. The machine retains it.

The development tasks are processed in groups of 10. Each group is executed natively with Skill + ReAct, and its traces are then used for incremental compilation. Before a group's traces are compiled in, the machine obtained so far is evaluated on the test tasks. We show LiveMath, the benchmark with the largest gain in Table 1. Its 487 development tasks form 49 groups, and no update changed the machine after group 20, so groups 21–49 were all evaluated with the final machine.

Table R1. LiveMath success by compilation stage, qwen3.6-flash.

MachineTasksMachineSkill + ReAct, same tasks
M0M_0 (no traces)development group 148/121 (39.7%)none
Machines during refinementdevelopment groups 572/121 (59.5%)none
Machines during refinementdevelopment groups 1086/121 (71.1%)none
Machines during refinementdevelopment groups 2096/121 (79.3%)none
Final machine (Table 1)held-out test93/121 (76.9%)54/121 (44.6%)

M0M_0 does no better than native execution (39.7%, against 44.6% for Skill + ReAct on test) and is far below the final machine. This is the pattern the method predicts: the initial machine is lossy, the loss is recovered within the first groups of traces, and the machine then keeps what it has recovered.

W3. The inequality H(B∣ZM,F=1)≤H(B)H(B \mid Z_M, F=1) \le H(B) in Proposition A.3.

We thank the reviewer for catching this step. The bound holds once H(B)H(B) is replaced by the size of the alphabet, and the conclusion of Proposition A.3 is unchanged.

BB is the next recorded operation, so it takes values in a finite set A\mathcal{A} of operation types (tool or state type, plus termination). Then

H(B∣ZM)≤H(B∣ZM,F)+H(F)=Pr⁡[F=1],H(B∣ZM,F=1)+h2(err(M)),H(B \mid Z_M) \le H(B \mid Z_M, F) + H(F) = \Pr[F=1], H(B \mid Z_M, F=1) + h_2(\mathrm{err}(M)),

since H(B∣ZM,F=0)=0H(B \mid Z_M, F=0) = 0 as in the original proof. Because H(B∣ZM,F=1)≤log⁡∣A∣H(B \mid Z_M, F=1) \le \log |\mathcal{A}| for any conditioning event,

Hμ(B∣ZM)≤err(M)log⁡∣A∣+h2(err(M)).H_\mu(B \mid Z_M) \le \mathrm{err}(M) \log|\mathcal{A}| + h_2(\mathrm{err}(M)).

This is a Fano-type bound, and the loss still goes to zero as the replay error goes to zero. Reviewer P33B raised a related point on the sampling assumption of Proposition A.2. Our response to P33B (Q3) gives that fix as well, and Appendix A will be revised with both.

AuthorsOpenReview ↗

Response to Reviewer 8ASC, Part II

W2 / Q2. Which requirements are covered by the static checks and RFC.

The two checks have different scopes. The breakdown below is for the spreadsheet skill (structure-preserving-spreadsheet-edits). Clause numbers refer to the sections of the skill document.

Table R2. Coverage of the spreadsheet skill's requirements.

Requirement in the skillHow the machine represents itChecked by
Inspect the workbook before choosing the target range (2.1.1)Inspection states s1/s2 precede the boundary state s3 and the edit state s6Static check, Eq. (10), ordering
Infer the table boundary first and reuse the same range for editing and auditing (Overview, 2.2.1)s3 is the only state that writes boundary_model; the edit state s5 and the audit state s7g read itStatic check, Eqs. (9) and (10)
Reopen the saved workbook and compare it with the input before reporting (2.3.1–2.3.3)Audit s7g/s7 and judge s8; END_VERIFIED is reachable only through label passStatic check, Eq. (10), required operation
If another sheet supplies names or row structure, inspect it too (2.1.5)Label insufficient of judge s4 returns to s1Guard. Not checked statically; replay checks it on recorded paths
If verification shows the edit matched the wrong cells, discard the output, regenerate from the original workbook, and retry (2.2.5)s8 → s5 on wrong_cells, s8 → s3 on boundary_wrong; the edit always starts from the input fileGuard, as above. Bounded cycles
Change only cells in the requested scope and preserve existing formulas (2.2.2, 2.3.4)Prompt of s5; the audit program diffs input and output, flags changes outside the span, overwritten cells, and a lower formula count; s8 sends any flag to revisionRun-time check on every run that reaches END_VERIFIED
Check merged ranges, styles, and features such as charts or macros when layout is at risk (2.3.5, 3.3)Formula counts and cell differences onlyNot covered beyond formula counts
Write the output file to the given path in the required format (task interface)Tool state s6RFC (file delivery, output format)

Static checks. Eq. (10) covers required operations, orderings, and prohibitions. These are properties of the graph, so they hold on every run that follows stored transitions, not only on recorded traces, and Check is re-applied to every accepted update (Lemma A.1). We also confirmed the first three rows independently on the released machine, by exploring every reachable combination of state and counter values with all model outputs left free. Among the 280 test runs of Appendix B.3, 276 never entered the recovery hub and therefore followed stored transitions only.

Run-time checks and guards. Scope and formula preservation are not static properties, but they are not left to the model either: the audit program compares the two workbooks on every run that reaches the verified terminal, and any flag blocks the label pass. Conditional requirements of the form "if cc then do oo" are encoded as guards. The guard and its target are in the graph; whether the condition is recognized is decided by the judge, and replay checks it on the paths that development traces cover.

RFC. RFC is a request-level metric. It checks file delivery, output format, and tool use, and it does not inspect the order of operations in a trace. Procedural requirements are covered by the mechanisms in Table R2, not by RFC.

In the revision we will state in Section 5.2 that RFC measures request compliance, and add Table R2 to the appendix. Our response to Reviewer P33B (Q4) reports the same audit, clause by clause, for all four skills.

Reviewer_8ASCOpenReview ↗

Comments from Reviewer 8ASC

Thank you for the additional results. The LiveMath comparison addresses my question about the benefit of trace refinement, and the spreadsheet audit explains which requirements are checked. My concern is whether the compiled machine captures the skill’s procedural requirements completely. Static checks cannot detect requirements missing from the extracted rules, and replay cannot reveal requirements absent from the recorded traces. The audit makes these omissions visible, but the method still relies on the compiler and the available traces to recover them. I appreciate the performance gains, but am not fully convinced by the evidence that refinement adequately addresses this gap.

AuthorsOpenReview ↗

Follow-up response to Reviewer 8ASC

Thank you for the follow-up. Your remaining question is whether procedural requirements of the skill are still missing after incremental compilation (the trace refinement in your comment). We measured this directly.

**After incremental compilation, very little of the skill is missing. ** Static checks and replay are not our only evidence. In the clause-level audit (Table R4 in our response to Reviewer P33B), we compare the text of each skill with its final machine, clause by clause. This comparison uses neither the extracted rules nor the traces. Across the four skills, 5 of 140 clauses (3.6%) are not represented, and none of them is a workflow step: an advisory note on charts and macros, a heuristic for classification questions, loading of reference files, a second-source rule for sports cut-offs, and the option to answer that a value cannot be verified. Every step of the main workflow of every skill is enforced by the structure of its machine. We checked this over all states and counter values of each machine, for any model output, so it holds on every run, including runs that no trace covers.

**Incremental compilation adds procedural requirements that the initial compilation misses. ** In the released spreadsheet machine, three states are labeled with origin trace: s6_loop repeats the edit with new parameters, s6_j decides after the edit whether verification can start, and s7_loop regenerates the audit when its report is unusable. The initial compilation from the document did not contain them. They were added from traces, and by Lemma A.1 every later machine keeps them. Table R1 shows the gain in success.

**What is still missing can be listed and added. ** With native execution, all 140 clauses are in the context, but none of them is guaranteed to be followed. In Table 3, Skill + ReAct reaches full request compliance (FRC) on 65.9% of tasks on average, and the compiled machines on 96.5%. On LongSeal the difference is 33.3% against 96.1%. In a compiled machine, each requirement is an explicit rule or state. If one is missing, the audit shows which one, and after it is added, Check enforces it on every path. The remaining 3.6% can therefore be added without waiting for a trace.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To the Area Chair and all Reviewers

Dear Area Chair and Reviewers,

Thank you for the time and care you have put into our submission. Your comments and the discussion that followed have been very helpful to us, and they have made the paper clearer and more complete.

We have uploaded a revised version of the paper. It incorporates our responses to your questions, the points raised during the discussion, and the corresponding changes to the text. All revisions are marked in blue, so they can be located easily.

We would be grateful if you could take a look at the revised version when you have time. If any point remains unclear, or if there is anything further you would like us to address, please let us know. We will be glad to respond.

Thank you again for your thoughtful feedback and for your continued engagement with our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.