acceptodds
Under review as a conference paper at ICLR 2027

Optimizing Agent Execution through Just-in-Time Skill Compilation

Abstract

Agent skills describe procedures that an LLM agent unfolds at run time into sequences of reasoning steps, tool calls, and observations. Much of this work recurs across runs, yet writing the procedure back into a skill as text still leaves the agent to carry it out step by step. We present a just-in-time compiler for skills that treats execution traces as profiling information, compiles recurring deterministic segments into parameterized scripts, and delivers them as an ordinary skill. Our central observation is that savings depend on the handoff: a script helps only if the agent calls it where it applies and trusts what it returns. The compiler therefore records each script's assumptions and tests them on subsequent seen inputs; a runtime interface discloses these assumptions on demand and returns a checkable execution summary. We evaluate on 20 SkillLearnBench tasks and a 100-task SpreadsheetBench subset, using held-out inputs. On SkillLearnBench our system raises correctness from 60% to 80% and lowers median delivery cost by 37.7% relative to bare agents. On SpreadsheetBench, it raises correctness from 72% to 78% and reduces median delivery cost by 24.7% relative to static compilation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent efficiency, skill learning, just-in-time compilation, neuro-symbolic execution, agent harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UQjcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Just-in-Time Skill Compilation framework to improve the execution efficiency of LLM agents. The framework extracts recurring deterministic segments from execution traces and compiles them into reusable, parameterized scripts. It explicitly maintains each script's assumptions and their sources, proactively tests and revises these assumptions on subsequent inputs, and provides a runtime interface that discloses the script's applicability conditions and returns an execution summary. This helps the agent decide when to invoke a script and whether to trust its output. On SkillLearnBench, the proposed method improves correctness from 60% to 80% while reducing the median delivery cost by 37.7% compared to bare agents. On SpreadsheetBench, JIT compilation maintains 78% correctness while reducing the median delivery cost by 24.7% compared to static compilation.

Critical strengths
  1. Compiling deterministic procedures into executable scripts can reduce the overhead associated with agent reasoning, code generation, and tool calls. This is a practically relevant research direction with potential benefits for real-world agent systems.
  2. The paper provides a focused analysis of how agents interact with compiled artifacts, identifying several failure modes, including Ignored, Redone, Re-checked, and Ineffective Fold.
  3. Explicitly tracking semantic assumptions and iteratively testing them against new inputs is a potentially useful design choice, particularly for identifying and revising assumptions that may not generalize beyond previously seen inputs.
Critical weaknesses
  1. Limited novelty and substantial overlap with existing skill compilation methods. SkVM has already introduced Adaptive Recompilation and Code Solidification, enabling skills to be continuously optimized using execution feedback and historical traces. Similarly, SkillSmith has proposed Boundary Contracts, a boundary-aware runtime, validation evidence, and runtime fallback mechanisms. These capabilities appear to overlap with a substantial portion of the proposed framework. The main remaining distinctions seem to be explicit Assumption Tracking and iterative Boundary Refinement. However, whether these incremental mechanisms constitute a sufficiently significant methodological contribution requires further justification.
  2. Lack of ablation studies that isolate the core contributions. Assumption Tracking and Boundary Refinement appear to be the key mechanisms distinguishing this work from SkVM and SkillSmith. However, the paper does not provide direct ablation results that remove these components individually, making it difficult to assess their independent contributions.
  3. Incomplete evaluation of compilation costs and amortized benefits. The paper primarily reports the model usage cost of solving tasks with compiled skills. However, compiling and continuously updating these skills also requires additional LLM calls. Although the appendix provides a cost model for amortizing compilation overhead, the paper does not report the actual break-even number of reuses or a long-term cost analysis that accounts for the full compilation and update overhead.
  4. Unclear reliability and generalization of semantic assumption checking. The assumptions maintained by the framework are largely generated and evaluated by an LLM, rather than enforced through sound, machine-checkable correctness conditions. It remains unclear how reliably the system can identify violated assumptions and generalize to previously unseen inputs.
Questions for authors
  1. The paper compares Static, JIT, and JIT + Interface, but these configurations introduce multiple design changes simultaneously. Could the authors clarify how much of the performance improvement comes from iterative script refinement, assumption checking, execution summaries, or runtime guidance? More fine-grained ablations would help isolate the contributions of these components.
  2. The paper states that SkVM and SkillSmith were evaluated under the same agent host and model. However, these systems use different optimization triggers and compilation procedures. For example, SkVM's Adaptive Recompilation is primarily triggered by recurring failures, whereas the proposed method performs a fold after each solving session on a seen input. Were the different methods given comparable opportunities for optimization, including sufficient execution histories to activate their respective adaptation mechanisms? Clarifying this would help rule out the possibility that part of the observed performance gap stems from differences in how effectively each system can leverage historical execution traces.
  3. How do the compilation and ongoing update costs of the proposed method compare with those of the baselines? In particular, could the authors provide an end-to-end cost comparison that includes the overhead of compiling and continuously refining skills, rather than only the cost of executing tasks with the resulting compiled artifacts?
Reviewer_TfeGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a just-in-time skill compiler that transforms recurring deterministic segments of agent execution traces into reusable, parameterized scripts to reduce redundant reasoning and tool calls. By explicitly tracking script assumptions and exposing them through a runtime interface, the framework enables agents to make more informed decisions about script applicability and result reliability, improving execution efficiency and task correctness.

Critical strengths
  • The framework records each compiled script's assumptions and their provenance, enabling systematic updates when assumptions are invalidated by new inputs.
  • The framework combines textual instructions with executable scripts while leaving invocation decisions to the agent.
Critical weaknesses
  • Lack of repeated runs and statistical significance tests makes it difficult to assess the reliability of the reported improvements, especially because the compiled scripts evolve through sequential folding, the order and diversity of seen inputs may affect the final package.

  • The paper does not isolate the contributions of important designs like assumption tracking, provenance recording, or per-fold assumption validation. It remains unclear which components are essential to the observed improvements.

Questions for authors
  • While the authors analyze 398 SpreadsheetBench tasks in Section 4.1, the main correctness and cost evaluation is restricted to a 100-task subset. It would be helpful to clarify how this subset was selected.
Reviewer_nTp7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a JIT compilation framework that converts recurring segments of agent execution traces into reusable scripts. The main observation is that compilation saves work only when the agent correctly decides when to use a script and whether to accept its result. The system records script assumptions and their provenance, revisits them during subsequent compilation and exposes applicability information and execution summaries through a runtime interface.

Critical strengths

The paper presents an approach to turning past execution into script. It allows agents to benefit from previous work while retaining the flexibility to handle new inputs.

Critical weaknesses
  1. The inference-enable and inference-disable conditions use independently compiled packages, and disabling the interface also changes whether scripts contain help information and execution summaries. Consequently, differences may be from changes in script quality, interface presentation or both. A controlled experiment using the same executable scripts is needed.

  2. The paper introduces a break-even model but does not measure how many reuses are needed to recover compilation and subsequent update costs. Reporting the cumulative costs over repeated executions are needed.

  3. SKVM already supports runtime code solidification and adaptive recompilation, web agent JIT synthesizes reusable tools from execution traces. An comparision with Web Agent JIT would also help clarify the scope of novelty.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.