- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
HEXIS turns a written agent skill into a state machine, so the runtime handles the procedure and the LLM reasons within each step. The machine is then refined using execution traces. The paper reports improvements across four benchmarks and four executors.
Critical strengths
I like the motivation. Asking an LLM to repeatedly work out a procedure that could be enforced by the runtime seems wasteful and error-prone. The comparison against using the compiled machine as a prompt is useful, and I find the transfer results encouraging.
Critical weaknesses
My main concern is that I still cannot tell how much the proposed compiler contributes. Appendix B.1 convinced me that executing a workflow can work better than describing it in a prompt. But how much of the gain is already there in the initial machine, before any trace-guided updates? That comparison is missing. Given the closely related ideas in Formal Skill, TraceCompiler, and Compile, Then Page, I need a clearer reason to prefer this compilation procedure. The current results leave me unsure what the incremental refinement and replay actually buy us.
I also found problems in the theoretical analysis that make me less confident in its conclusions. The algorithm discards traces that fail validation, yet Proposition A.2 treats the accepted traces as iid samples from the original distribution. Proposition A.3 uses H(B | Z, F=1) <= H(B), which does not hold in general. I do not think Appendix A currently justifies the claimed connection between replay and the representation objective. The empirical gains remain interesting, but this theoretical argument needs to be corrected.
Finally, I think the evaluation tells us too little about whether the skills were compiled faithfully. There are only four source skill documents. Higher task success and better output compliance do not tell me whether the compiler dropped a condition or ruled out a valid branch. I appreciate that the authors acknowledge this limitation, but preserving the meaning of the skill is central to calling this a compiler. I would like to see that checked directly, rather than inferred from downstream scores.
Questions for authors
- How well does the initial validated machine perform before trace refinement? Does archive replay make a meaningful difference on held-out tasks?
- What does HEXIS gain over a simpler executable-skill approach under the same setup? I am particularly interested in the benefit of the proposed compiler.
- How would you correct the sampling argument in A.2 and the entropy inequality in A.3? Which conclusions would still hold?
- Have you checked the compiled machines against the original skills for missing conditions and lost branches? What did that check find?
6 comments
Response to Reviewer P33B, Part I
We thank the reviewer for the detailed comments and for the positive view of the motivation, the Prompt-only comparison, and the transfer results.
Q1. Performance of the initial machine, and the effect of archive replay.
We agree that this comparison is missing. Table 3 fixes the final machine and varies how it is executed, so it cannot separate the initial compilation from trace refinement. The comparison bears on the main idea of the method, so we state that idea first.
Native execution loses part of what the skill specifies. The control requirements are in the context, but every decision has to recover them again, and each decision can deviate (Eqs. (2) and (3)). Initial compilation is lossy too. is drafted from the document alone and can omit operations, dependencies, and branches that executing the skill requires. Incremental compilation exists to recover this loss. Each native trace exposes something the current machine lacks, the compiler adds it as states, bindings, and guarded transitions, and replay of all accepted traces keeps it in the machine. A native run that performs a required step does not make the next run more likely to perform it. The machine retains it.
The development tasks are processed in groups of 10. Each group is executed natively with Skill + ReAct, and its traces are then used for incremental compilation. Before a group's traces are compiled in, the machine obtained so far is evaluated on the test tasks. We show LiveMath, the benchmark with the largest gain in Table 1. Its 487 development tasks form 49 groups, and no update changed the machine after group 20, so groups 21–49 were all evaluated with the final machine.
Table R1. LiveMath success by compilation stage, qwen3.6-flash.
| Machine | Tasks | Machine | Skill + ReAct, same tasks |
|---|---|---|---|
| (no traces) | development group 1 | 48/121 (39.7%) | none |
| Machines during refinement | development groups 5 | 72/121 (59.5%) | none |
| Machines during refinement | development groups 10 | 86/121 (71.1%) | none |
| Machines during refinement | development groups 20 | 96/121 (79.3%) | none |
| Final machine (Table 1) | held-out test | 93/121 (76.9%) | 54/121 (44.6%) |
does no better than native execution (39.7%, against 44.6% for Skill + ReAct on test) and is far below the final machine. This is the pattern the method predicts: the initial machine is lossy, the loss is recovered within the first groups of traces, and the machine then keeps what it has recovered.
Response to Reviewer P33B, Part II
Q2. What does the compiler add compared with a simpler executable-skill approach?
The simplest executable-skill approach in our setup is : one compilation from the skill document, with static checks and no traces. Our results separate what each further component adds.
| Step | What changes | Result with qwen3.6-flash |
|---|---|---|
| Skill + ReAct → Prompt-only | The model receives the final machine as text but still controls execution | 44.1% → 47.3% on average. LiveMath drops from 44.6% to 33.9% (Table 3) |
| Prompt-only → HEXIS | The runtime executes the same machine | 47.3% → 64.1% on average, +16.8 points (Table 3) |
| → final machine | Trace refinement with replay | LiveMath: is at the level of native execution; the final machine reaches 76.9% (Table R1) |
A compiled description alone adds little, and runtime execution of the final machine adds 16.8 points. The third row is the reason to prefer incremental compilation over a one-shot executable skill. One-shot compilation is itself lossy: on LiveMath, under the same runtime does no better than native execution. Runtime control helps only when the machine contains the operations the task needs, and incremental compilation is how the machine acquires them from native traces and keeps them.
Two further results concern the procedure itself. With Sonnet 5 at low effort as the compiler, the machines score 2.0–4.2 points below those compiled by Fable 5.1 and stay above Skill + ReAct on all four benchmarks (Table 4), so the gains do not depend on one compiler model. AFlow is the baseline in Table 1 that also produces an executable workflow under the same executor and tools, obtained by search over execution feedback and not from the skill document. HEXIS is higher in 13 of 16 settings, tied in 1, and lower in 2. AFlow's workflows fall to 51.0% and 15.7% on DABench with the two local executors, while HEXIS machines are reused across executors without recompilation.
The related methods differ from HEXIS in their input and in what they keep from it:
- Formal Skill makes control explicit, but the program is written by hand. HEXIS generates the machine from an existing natural-language skill.
- TraceCompiler mines agent traces into mostly deterministic workflows. HEXIS starts from the skill document and keeps its knowledge in state prompts, so the model still reasons inside each state. Traces are used only to add missing operations and dependencies.
- Compile, Then Page compiles SOP constraints that are already machine-readable. HEXIS must first extract the rules from free text (Section 4.2) and check each one against a quotation from the document.
HEXIS therefore covers a setting the other three do not: it starts from an existing natural-language skill and needs neither a hand-written program nor machine-readable constraints. Formal Skill and Compile, Then Page require inputs that the four skill documents do not provide, so they cannot be run on our benchmarks as they stand. TraceCompiler works from traces without the skill document, and a direct comparison on our development traces is a natural extension. We will state this positioning explicitly in Section 2.
I received the committee’s message about possible AI-generated reviews and would like to clarify my use of AI.
My initial concerns came from my own reading of the paper. I used AI to discuss these concerns, check whether I had misunderstood anything or made unfair judgments, and help put my comments into English. I have included the conversation to make this process transparent. The final review reflects my judgment, and I take responsibility for it.
I am open to the authors’ replies and will update my assessment if they address my concerns. I will follow the discussion and respond promptly.