acceptodds
Under review as a conference paper at ICLR 2027

From Traces to Guarded Programs: Evidence-Gated Compilation of Recurrent Agent Workflows

Abstract

Tool-using agents often invoke a model at every step, even along recurrent read-only paths. Replacing repeated reasoning with deterministic execution could reduce inference cost, but recurrence alone does not make substitution safe: tool arguments may depend on prior observations, apparently read-only operations may hide effects, and replaying the same calls may still alter the final answer. Recurrence identifies an optimization opportunity, not permission to remove reasoning. We introduce guarded agentic compaction (GAC), an evidence-gated approach that compiles recurrent execution paths into deterministic programs only when substitution is justified. GAC compiles initial read-only prefixes only: it grounds tool arguments in observable state, treats writes, approvals, handoffs, and unknown effects as hard barriers, protects compiled execution with input, output, and environment guards, and admits specialization only when finite-sample evidence satisfies configured risk and confidence requirements. Otherwise, it retains the largest justified prefix when possible or falls back to the unchanged agent. Admission certificates are exact finite-sample bounds under i.i.d. calibration groups, compiler-wide where one candidate reached calibration and per-candidate where two did (two of three primary families, whose corrected bound is 0.057). Across three live GitHub workflow families and 90 unseen test cases, GAC satisfies 90/90 exact task contracts and the unchanged agent 89/90, a difference of one stochastic baseline miss, while reducing model requests by 66.6%, tokens by 63.1%, latency by 64.2%, and estimated cost by 58.7%. A time-forward evaluation across five frozen public repositories specializes four and refuses one. On four public trace benchmarks, NESTFUL, API-Bank, and BFCL v4 contain recurrent, replayable traces but insufficient qualifying evidence to satisfy GAC's default 0.05 risk bound at the required confidence, whereas AppWorld provides sufficient evidence for admission. On a second model family and a second provider, the unchanged pipeline re-derives the same artifacts from each model's own traces on two of the three families and refuses the third. GAC reframes agent optimization as evidence-gated specialization: compile only what is justified, dispatch only when runtime conditions remain valid, and preserve model reasoning everywhere else.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
tool-using agents, agent workflow compilation, trace-derived specialization, provenance, effect systems, selective prediction, finite-sample risk control, conformal admission, program synthesis, guarded execution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HJTFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This study introduces guarded agentic compaction: a repeated, initially read-only sequence of agent actions may be converted into a deterministic program once checks on provenance, side effects, contracts, and calibration evidence have passed. Across three GitHub workflow families, the authors observe exact agreement with all 90 held-out contracts and 66.6% fewer model requests. Experiments on external traces also document cases in which the method declines compilation because the available calibration evidence is inadequate.

Critical strengths
  1. The admission criteria address distinct sources of error, including arguments without supporting evidence, unintended effects, execution beyond the allowed prefix, and disagreement with the output contract. The examples involving issues #4420 and #6602 illustrate why these checks matter (Figures 1–3).
  2. The empirical account includes both admitted programs and rejected candidates, together with their resource implications. Table 1 relates exact contract agreement to cost, and Table 2 demonstrates that recurring patterns in outside traces may still lack enough observations for admission.
Critical weaknesses
  1. The broad safety language is stronger than the reported statistical result. According to Section 5.1, calibrating two candidates in each of two main families yields a selection-adjusted upper bound of 0.0569, which misses the 0.05 goal. Section 7 further finds that using creation day or author as the independent unit eliminates every family. The main claims should state these limits directly; additional independent calibration data would also make the evidence more persuasive.
  2. The evidence does not yet show that the learned gate assigns higher scores to genuinely safer cases. At 92 calibration groups, the analysis in Section 3 and Figure 6 permits certification only at zero or complete coverage under the selected bound; the forward-in-time experiment likewise certifies no intermediate level. Testing a larger cohort that supports several certified coverage levels, and comparing with a gate based only on support at equal coverage, would clarify the contribution of the learned score.
Questions for authors
  1. Does any main or supplementary cohort contain enough independent units to establish a 5% bound for the entire candidate-selection process? If none does, how will the abstract and conclusion be revised to reflect that limitation?
  2. Could you compare the learned gate with a support-based gate on one held-out population at two or more positive, certified coverage levels? Such evidence would be important for evaluating the selective component.
Reviewer_71S8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Guarded Agentic Compaction (GAC), which converts recurrent initial read-only agent workflows into deterministic programs. Compilation requires grounded tool arguments, admissible effects, runtime guards, and finite-sample risk admission. Unsupported candidates are rejected, while runtime checks support fallback to the original agent.

On three GitHub workflows with 90 held-out records, the paper reports 90/90 exact-contract successes versus 89/90 for the baseline, with 66.6% fewer model requests and 63.1% fewer tokens. Additional evaluations examine cross-repository transfer, different models and providers, external trace benchmarks, and refusal behavior.

Critical strengths
  1. A meaningful problem with explicit safety boundaries. The paper recognizes that recurring tool sequences alone do not justify removing model decisions. Argument provenance, effect declarations, execution position, and downstream answer fidelity are relevant concerns. The restricted formulation and conditional finite-sample admission certificate provide a concrete foundation for studying when specialization is admissible. The certificate's dependence on i.i.d. calibration groups and candidate-selection assumptions must remain explicit.

  2. Concrete measurements and transparent reporting of unfavorable results. Paired evaluations on identical records provide credible evidence of resource savings for the tested workflows. Reporting handwritten comparators, unsuccessful admissions, continuation failures, and unfavorable ablations allows readers to assess the method's actual boundaries. This transparency strengthens confidence in the experimental record.

Critical weaknesses
  1. The workloads do not sufficiently establish the practical value of automatic guarded compilation. The primary tasks largely involve metadata-derived classification and verbatim extraction: issue categories follow labels, PR outcomes follow status fields, and backlog routing follows assignee and discussion information. These tasks are readily amenable to predefined deterministic workflows. The handwritten comparator achieves the same correctness and request reduction, with somewhat better aggregate token and cost results (Table 9), while the value of automatically discovering and maintaining such workflows remains unquantified. Cross-repository experiments retain an even simpler PR-outcome task, rather than demonstrating specialization within workflows containing substantial adaptive decision-making. Consequently, the headline savings establish the benefit of programmatic retrieval more strongly than the distinctive value of the proposed system.

  2. The independent contribution of the central guarding and learned-selection mechanisms remains unestablished in the current experiments. In Table 19, barrier-free replay achieves the same 30/30 correctness with fewer requests and tokens than GAC, showing no measured quality benefit from the additional restrictions on that cohort. Disabling multiple barriers simultaneously and changing the compiled region also prevents attribution to individual components. The learned-gate comparison in Table 23 likewise does not establish risk-ranking value: scores are constant within repositories, and the calibration-pool size cannot certify intermediate coverage. The comparison additionally changes the risk budget and threshold grid, limiting component attribution. The proposed drift comparison could address an important part of this gap, but remains unexecuted. These limitations are consequential because evidence-gated admissibility, rather than deterministic execution alone, is the paper's central distinguishing contribution.

Questions for authors
  1. Clarification — task requirements: Could you provide a complete primary-task specification and identify which decisions require semantic interpretation or adaptive branching beyond deterministic metadata processing? This would help assess whether the workloads exercise the intended specialization problem.

  2. Clarification — calibration evidence: How are the calibration violation labels L(A,x)L(A,x) generated? Do they come from executing the frozen compiled prefix followed by the model continuation, or from replaying existing trajectories? Please also clarify how failed or inexact discovery runs are handled before the train/development/calibration split, and whether calibration sampling precedes filtering on baseline success or occurrence of the mined region. These details materially affect the interpretation of the admission guarantees.

  3. Suggested additional validation — requires new experimentation: Could a targeted evaluation compare full GAC, unguarded replay, and a handwritten or composite-tool baseline within one workflow containing both recurrent retrieval and genuinely adaptive decisions? Reporting task success, optimization coverage, fallback costs, and behavior under realistic changes would materially affect my assessment of the method's incremental value.

Reviewer_1FScOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper intorduces "guarded agentic compaction", a technique to allow "mature" agents to not have to repeat key decision-making steps unnecessarily. It adapts guarded tracing compilation to initial read-only agent prefixes, combining argument provenance, effect declarations and induced contracts with statistical admission (Sections 1-3). Clean failures return to the unchanged agent. The paper gives a conditional finite-sample selective-risk bound and reports 66.6% fewer provider requests and 63.1% fewer tokens across 90 paired GitHub cases, with 90/90 compiled and 89/90 baseline exact contracts (Table 1). External benchmarks establish compilation feasibility or refusal (without live-agent efficiency measurements).

Critical strengths
  1. The paper connects guarded tracing compilation and bounded synthesis to agent-specific substitution hazards (Sections 1 and 6). Argument witnesses, declared effects and entry-position checks prevent distinct failures documented in Figure 1 and Table 20. This gives practitioners a useful account of which recurring decisions can be replaced and where the agent must retain control. In the workflow-reuse literature, Agent Workflow Memory induces routines to guide subsequent generations. GAC contributes a complementary deployment criterion for replacing those generations with executable programs. Connecting program synthesis to statistical admission is a useful idea.
  2. Automatic derivation of useful programs, together with explicit failures and refusals, constitutes a practical contribution even when hand-written bundles match runtime performance (Table 9).
Critical weaknesses
  1. The introduction is hard to follow and feels very LLM written. Concepts are introduced without context (e.g. ``held out issue #4420'': what kind of issue is this? is merely an example or it's part of a dataset that's introduced elsewhere? what's defined in this issue? what does it have to do at the start of the third paragraph without mentioning the notion of issue or why it matters to the paper?)
  2. The benchmarks do not compare against alternative agentic workflow compilation methods that have been proposed in the literature (e.g. Guarded Programs: EvoC2F and Agent JIT Compilation). A comparison should assess the cost–quality trade-off against other compilers. The results in Table 2 are difficult to interpret and don't allow judging the quality of the compiled segments.
  3. Section 2 includes downstream answer errors in the certified loss (p. 3), whereas the supplied calibration helper labels tool-output agreement after verifier success without evaluating the continuation. Section 7 acknowledges continuation limits, and Appendix F.2 shows that correct replay can precede an incorrect answer. It's not clear how the reported labels support the end-to-end certificate, which is central to the admissibility contribution (even perfect replay labels cannot bound unmeasured continuation errors). The authors should align the loss, labels and claims throughout: a replay-only certificate would support a narrower contribution, while the stated episode-level guarantee requires calibration of the substituted agent's final outcomes.
  4. The calibration population also needs accounting. The primary study drivers select successful discovery traces before splitting, and the compiler constructs calibration samples from matching family windows. Every dispatch-eligible member sampled before recurrence filtering requires labels (p. 29): it's not clear whether omitted cases could pass the deployed guard. If they could, the calibration bound would describe a selected population rather than the claimed deployment population. If excluded records can pass the deployed guard, recalibration should sample before outcome filtering and retain failures. Enlarging the same filtered cohort would not repair this mismatch.
  5. Post-deployment lifecycle policy is underspecified. Algorithm 4 checks active status and manifest compatibility, and the supplement implements expiry, retirement and rollback. What remains unclear is when accumulating failures or task-distribution changes should trigger revision, and what evidence permits renewed admission. This limits guidance for sustained reuse. SKILL.nb provides related context on evidence-triggered maintenance and validated updates.
Questions for authors

All questions are clarifying ones.

  1. What are the complete references for the benchmarks used? Section 4.1 does not provide much detail on them.
  2. Which labels in the reported 92-group tables include downstream answer errors? Please identify the labeling path, or specify the narrower loss and guarantee these tables support after considering the continuation limitation in Section 7.
  3. Can the retained discovery and split records account for every record excluded by successful-trace or family-window filtering, including whether the deployed guard would reject it? This would resolve the population concern without a new benchmark study.
  4. Beyond manifest invalidation and preset expiry, what operational evidence triggers recompilation or retirement, and what fresh calibration is required before reactivation? Please distinguish implemented policy from future work, especially for distribution changes that pass existing guards.
  5. The notion of ``content-addressed execution manifest'' MM is introduced in Section 2, without any further nor proper definition. Explain what it is, and why it matters to the episode definition.

Comments on the paper

1 comment
Area_Chair_19PSOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.