acceptodds
Under review as a conference paper at ICLR 2027

When Can the Runtime Take Over? A Decision-Level Audit of LLM Agent Control

Abstract

LLM agents divide control between models and software runtimes, but runtime executability alone does not justify replacing a model-mediated decision. Continued model mediation can add inference work, whereas premature runtime takeover can violate unresolved behavioral obligations. We introduce Control Substitution Audit (CSA), a decision-level protocol for evaluating whether a specified runtime policy can replace an attributed model contribution under fixed boundary-visible information and a behavioral contract. CSA connects attribution of the reference model’s contribution to direct runtime intervention and executed contract checking, while reporting policy coverage and downstream outcomes separately. In six selected edit conditions executed in Writer, accepted perturbation replays support model-to-commit attribution, and all six matched direct runtime substitutions satisfy their exact contracts. On a separate controlled set, broader deterministic support increases direct-action coverage from 4/22 to 15/22 and reduces recorded proposal-model tokens by 47.1%, but introduces one direct contract violation; execution completion remains 21/24 in both policy arms. These results characterize takeover as policy- and condition-relative: reduced model work or completed execution alone does not establish valid substitution.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, agent systems, runtime control, control substitution, behavioral contracts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EVoiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when the runtime should take over control from the LLM in agent workflows. Its main motivation appears to be improving agent efficiency by analyzing when model-mediated decisions can be safely replaced by deterministic runtime execution.

However, I have several concerns about the paper. First, the problem formulation is relatively simplistic, and the experimental evaluation is limited in both scale and diversity, with most experiments focusing on a single type of agent. As a result, I am not fully convinced that the experimental evidence is sufficient to support the paper's conclusions, and I am concerned that the capabilities and generalizability of CSA may be overstated.

In addition, I find the presentation somewhat lacking in clarity. The paper provides only one relatively basic overview figure, which does not sufficiently illustrate the proposed framework, while much of the experimental evidence and detailed analysis is relegated to the appendix.

Critical strengths
  1. The requirement to connect model-to-commit attribution with a matched runtime intervention is a meaningful perspective, since neither the presence of a model call nor successful execution alone demonstrates valid substitution.
Critical weaknesses
  1. One concern I have with CSA's design is that it depends heavily on predefined runtime policies, decision boundaries, and behavioral contracts. It mainly tells us whether a particular policy works under specific tested conditions, rather than helping determine when the runtime should actually take over. Also, passing a contract check on a few executions does not necessarily mean the policy will work reliably on unseen cases.
  2. The empirical evaluation is relatively limited and lacks convincing comparisons with existing approaches. Most experiments are conducted on the Writer agent using relatively simple, deterministic text-editing tasks, which may not adequately reflect the complexity of real-world agent decision-making. Also the paper does not provide direct comparisons against simpler attribution or contract-checking baselines under the same conditions. It is difficult to assess whether the proposed framework offers substantial advantages over existing techniques or generalizes beyond the evaluated scenarios.
Questions for authors
  1. Can the authors demonstrate the generalizability of CSA beyond deterministic text editing to more complex agent decision-making tasks?
  2. How do the authors explain the disagreements between the LLM auditors and the reference labels, and what do these imply for CSA's reliability?
3 comments
AuthorsOpenReview ↗

Policy-Relative Takeover and Comparative Evidence (Part 1/2)

We thank the reviewer for recognizing the importance of connecting model-to-commit attribution with a matched runtime intervention. This connection is central to CSA. We first address the policy-relative formulation and comparative evidence.

1. How does CSA determine when the runtime should take over?

CSA evaluates a specified takeover policy at a declared decision boundary, rather than generating an optimal controller. The key question is whether that policy can replace an actual model contribution under the same available information and behavioral obligations.

Equation (4) operationalizes this criterion: the original model proposal must have influenced the committed action, the runtime policy must act directly using boundary-visible information, and the executed replacement must satisfy the same contract.

Our controlled comparison demonstrates the significance of this policy-relative evaluation (Table 2).

Across 22 proposal-reaching conditions, Literal v1 acts directly on 4/22, passing 4/4 scored contracts. Normalized v2 expands coverage to 15/22, passing 14/15 contracts.

The broader policy reduces recorded proposal-model requests from 19 to 8 and proposal tokens from 50,544 to 26,737, while both policies complete 21/24 executions. In C08, v2 incorrectly deletes content despite an explicit instruction not to delete it.

These results demonstrate why runtime coverage, execution completion, model work, and valid substitution must be distinguished. We additionally reproduced all 48 policy-condition direct-stage records using the original native implementation, obtaining results consistent with the original experiments.

2. What does CSA add beyond simpler attribution and contract checks?

Attribution and contract checking answer complementary questions.

An attribution check determines whether the original model influenced the committed decision. A contract checker evaluates whether the replacement execution satisfies its obligations.

CSA connects these forms of evidence through a matched intervention at the same decision boundary, producing a scoped conclusion about whether an actual model contribution was validly replaced (Equation 4; Table 1).

In six selected Writer conditions, accepted proposal perturbations changed committed outputs in 6/6, and matched runtime replacements satisfied exact contracts in 6/6 (§§5.1–5.2).

Our same-evidence diagnostic further clarifies the roles of individual checks: contract checking identifies the C08 violation, while attribution analysis identifies runtime-only contributions. CSA connects these findings to the same substitution claim.

The R1/R2 and v1/v2 experiments provide matched execution and policy comparisons, while Appendix A.2 provides a controlled protocol sanity check. Together, these establish the evaluated mechanisms without treating simpler components as artificially weak competitors.

AuthorsOpenReview ↗

Auditor Disagreements, Generality, and Presentation (Part 2/2)

Following our response above, we address the auditor disagreements, applicability beyond editing, and presentation concerns.

3. What explains the auditor disagreements, and what do they imply for reliability (Q2)?

The two separately initialized auditor sessions agree on all four fields across all 18 packets. Each agrees with the retained reference key on 15/18 attribution and supported-conclusion labels, and 18/18 behavior and contract labels (Table 6; Appendix A.2).

The three shared disagreements have an identifiable source.

In DS01 and DS03, the disclosed reference execution copies an existing observation; in DS02, it emits a fixed literal. These paths do not depend on the model proposal: changing that proposal leaves the committed output unchanged.

The auditors therefore identify these cases as RUNTIMEONLY, consistent with the disclosed execution evidence. The reference key instead uses source-family metadata to assign model-mediated labels.

We traced this discrepancy through the original condition generator and preserved the original key and reported 15/18 comparison.

This provides a concrete explanation for the disagreements and highlights an essential property of reliable auditing: attribution conclusions must be grounded in actual proposal-to-commit dependencies, rather than inferred from source labels or nearby model calls.

4. How broadly does CSA apply beyond deterministic editing (Q1)?

CSA is formulated around a decision boundary, rather than a particular Writer operation. The audit procedure applies wherever available observations, action-determining contributions, candidate runtime behavior, and behavioral obligations can be explicitly specified and evaluated.

The Writer experiments provide direct execution evidence for model-mediated edit construction and matched replacement. Appendix A.4 additionally instantiates CSA at Presentation routing and delegation boundaries, including direct build_outline routing and delegation to collect_materials.

These illustrate different decision types under a shared audit formulation, with the strongest executed substitution evidence currently coming from Writer.

The methodology also accommodates decisions requiring delegation: unresolved semantic obligations remain model-mediated rather than being forced into deterministic actions. Section 5.3 reports coverage and delegation behavior separately.

Thus, CSA provides a decision-level evaluation framework for different runtime policies and agent workflows without assuming that one fixed policy applies universally.

5. Presentation and experimental evidence

Figure 1 introduces the decision boundary; Equation (4) defines the substitution criterion; Table 1 distinguishes supported conclusions; and Table 2 reports the policy comparison.

Appendix Figure 2 presents the audit sequence, while Appendix A.3 and Figure 4 trace the matched Writer intervention.

We hope these connections clarify the methodological contribution, experimental findings, and interpretation of the auditor disagreements.

AuthorsOpenReview ↗

Follow-up: New Routing Experiments and Same-Evidence Baselines

Following our two earlier comments, we have completed a new experimental evaluation directly addressing your concerns about generality, policy-relative takeover, and baseline comparisons.

1. New non-editing evaluation.

We evaluated ten prospectively frozen Presentation routing conditions. Unlike the synthetic fixtures previously discussed, the new evaluation uses real model-generated proposals, accepted native transitions, and persisted workflow decisions.

All 10/10 reference proposals were commit-sensitive: changing only the accepted target under the same saved pre-decision state changed the persisted route.

Two frozen runtime policies were evaluated on the identical conditions:

  • v1: ACT 0/10; DELEGATE 10/10.
  • v2: ACT 4/10; DELEGATE 6/10.

Among v2's four direct actions, F01 and F02 passed their complete behavioral contracts, establishing two new non-Writer direct substitutions under Equation (4). F03 and F04 remained UNSCORED because semantic obligations were unresolved.

The six v2 delegations resulted in 4 PASS, 1 FAIL, and 1 UNSCORED. Both policy arms had the same overall boundary outcomes: 6 PASS, 1 FAIL, and 3 UNSCORED.

Therefore, expanding direct-action coverage from 0/10 to 4/10 produced two justified substitutions, illustrating why CSA distinguishes policy coverage from contract validity.

2. Real failure and unsafe interventions.

F07 provides an informative reference-model failure: the model chose to skip required processing of pasted material, and the native workflow accepted this route. Both policies delegated and consequently preserved the violation.

Additionally, three predeclared unsafe routing interventions were accepted natively but failed the complete contract checker. These challenge executions were not counted as additional independent conditions.

These results demonstrate that native acceptance, model delegation, and behavioral correctness can produce different outcomes.

3. Additional baseline comparison.

We evaluated model-call presence, attribution-only, complete contract-only, CSA, and a separately implemented ordinary complete-audit checklist on the same 23 execution evidence packets.

The complete contract checker identified all five observed collection-obligation violation packets, including repeated intervention arms. The ordinary complete audit agreed with CSA on 23/23 scoped conclusions.

This comparison provides a more concrete characterization of CSA's contribution: an explicit joint evidence protocol connecting original model contribution, matched runtime intervention, executed obligations, and bounded takeover conclusions. We give complete auditing and contract checking full credit for the conclusions they independently support.

4. Reliability and scope.

These results complement our earlier explanation of the DS01–DS03 auditor disagreements; the original 15/18 key comparison remains unchanged.

The new evidence establishes native, commit-sensitive control attribution and valid substitution at a second decision type. It uses one workflow engine, a bounded live proposal adapter, and programmatic contract checking supplemented by non-blinded semantic review. Unresolved obligations remain unscored.

We believe these additional results materially strengthen the empirical basis for CSA and directly address the key concerns about applicability beyond deterministic text editing.

Reviewer_PTjuOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper replaces parts of an agentic workflow with deterministic tool calls. Specifically, they confirm that the original LLM output actually influenced the action, then replace it with a pre-determined software rule, and execute the replacement and check whether it satisfied the task requirements. They show that their approach satisfies the task requirements.

Critical strengths

Their approach works and they state the limitations candidly. It also is important that they check whether the task is correct, not whether it just agreed with the LLM.

Critical weaknesses

I thought this paper was written very confusingly, and it was very hard for me to understand. The abstract is filled with so much jargon. The first sentence is "LLM agents divide control between models and software runtimes, but runtime executability alone does not justify replacing a model-mediated decision". What is runtime executability? I don't even understand what "models and software runtimes" means here. After reading the paper I assume it means the LLM doing things itself and tool calls.

The other is evaluation. They don't compare to other baseline approaches? In the limitations, they say "comparisons with ordinary audit practice or compilation systems on common natural conditions remain untested". It's also very unclear what the actual value addition of their approach is.

Questions for authors
  • Are there any baselines to compare to?
  • Can you state concretely what the benefit of using this system is?
2 comments
AuthorsOpenReview ↗

Clarifying CSA's Mechanism, Concrete Value, and Experimental Comparisons

We thank the reviewer for the careful assessment and for highlighting an important distinction: a valid replacement should satisfy the task requirements, rather than merely reproduce the LLM's output. We clarify the mechanism, concrete contribution, and experimental comparisons below.

1. What do "model," "runtime," and "runtime executability" mean?

Both execution paths use the same software to validate, apply, and save edits. The difference is who constructs the action parameters.

In the reference path (R1), an LLM proposes the old_string and new_string. In the replacement path (R2), a deterministic runtime policy constructs these parameters from already available information or delegates the decision to the model.

For example, when asked to "delete exactly the selected paragraph," the runtime can construct the edit directly from the selected text. However, when asked to "make the paragraph more concise," the replacement content remains unresolved, so the policy delegates (Figure 1; §3).

Here, runtime executability refers to whether an action can mechanically pass the guards and be applied. Executability and task correctness are distinct. The proposed distinction is therefore not between an LLM acting independently and an LLM calling tools, but between model-mediated and runtime-determined action construction at a specific decision boundary.

2. What is the concrete benefit of CSA?

CSA provides an evidence-backed assessment of whether a particular model contribution has been validly replaced, rather than merely demonstrating that a software rule can execute.

Equation (4) connects three requirements at the same decision boundary:

(i) The original model-generated information actually influenced the committed action.

(ii) The specified runtime policy acts using only information available at that boundary.

(iii) The executed replacement satisfies the original behavioral contract.

This allows CSA to distinguish supported direct substitution, valid delegation, existing runtime-only behavior, contract violations, and insufficient evidence (Table 1).

The practical value is that these outcomes lead to different conclusions about whether a model decision can be transferred to runtime control. Neither the presence of a model call nor successful execution alone establishes such a transfer.

3. What experimental evidence and baselines are available?

Our evaluation includes two complementary comparisons.

R1 versus R2: Under matched source documents, requests, localization results, guards, and application code, the proposer is changed from the LLM to a fixed runtime policy. In six selected Writer conditions, perturbing accepted model proposals changes the committed output in 6/6 reference replays. The corresponding runtime-generated edits satisfy their exact-output and non-target-preservation contracts in 6/6, with zero proposal-model requests (§§5.1–5.2; Appendix A.7). We also reproduced these results through the original native execution path.

One particularly informative case is described in Appendix A.3: the model-proposed edit removes two extra newlines, whereas the runtime-produced edit preserves them and satisfies the task contract. This demonstrates why the model's original output should not itself be treated as the correctness criterion.

Literal v1 versus Normalized v2: We compare two frozen runtime policies on the same 24 controlled conditions. Direct-action coverage increases from 4/22 to 15/22, while scored direct-contract passes are 4/4 versus 14/15. Proposal-model requests decrease from 19 to 8, while execution completion remains 21/24 for both policies (Table 2).

The C08 case further illustrates the value of the audit: a mechanically executable edit violates a negated-deletion request, demonstrating why coverage and contract validity must be evaluated separately.

Appendix A.2 additionally examines simplified evidence-checking rules. These experiments compare execution policies and audit evidence requirements; comparison against complete alternative auditing frameworks is a distinct evaluation question.

We hope this clarification makes the mechanism and its concrete research contribution easier to assess, and we welcome further questions about the criterion or its implementation.

AuthorsOpenReview ↗

Follow-up: Concrete Routing Example and Additional Audit Baselines

Following our earlier clarification of the model/runtime distinction and CSA's concrete benefit, we have completed an additional experiment that provides a new non-editing example and direct same-evidence baseline comparisons.

1. A concrete example of model-to-runtime substitution.

Consider F02, a Presentation request introducing stacks and queues.

After requirement analysis, the reference LLM selected build_outline. The native workflow accepted and persisted that choice. Under the same saved pre-decision state, changing the accepted model-selected target changed the committed route.

In the matched intervention, a frozen deterministic runtime policy independently selected and committed the route using the same boundary-visible observations. The resulting execution passed the complete behavioral contract.

This is a new instance satisfying Equation (4).

The software performs the route transition in both executions. The intervention changes who determines the route. Upstream requirement analysis remains model-generated; the audited replacement concerns the subsequent routing decision.

2. Why the additional audit evidence matters.

Across ten new, prospectively frozen Presentation conditions, all ten accepted reference proposals showed model-to-commit dependence.

The revised runtime policy acted directly on four conditions. Two passed the complete contract, while two retained unresolved semantic obligations and received no substitution credit.

The evaluation also exposed a real model error (F07): the model skipped required processing of pasted material, and the native workflow accepted the route. Both runtime policies delegated and preserved that violation.

Three additional predeclared unsafe route interventions were also accepted natively but failed contract checking.

Together, these outcomes illustrate why CSA distinguishes actual model contribution, executable replacement, and task correctness, rather than treating successful execution as sufficient evidence.

3. Newly completed baseline diagnostic.

In addition to the original R1/R2 and frozen-policy comparisons, we evaluated model-call presence, attribution-only, complete contract-only, CSA, and an independently implemented ordinary complete-audit checklist using the same new evidence.

The complete contract checker detected the observed collection violations, and the ordinary complete audit agreed with CSA on 23/23 execution packets.

This establishes the appropriate methodological positioning: CSA provides an explicit, reproducible protocol for evaluating model-to-runtime takeover claims, linking attribution, matched intervention, executed contracts, and supported conclusions.

A complete ordinary audit performing these checks can reach the same conclusion; the additional contribution is the structured decision-level evidentiary formulation.

We hope this new example and completed comparison provide a more concrete basis for assessing CSA's practical value and scientific contribution.

Reviewer_aBmCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the Control Substitution Audit (CSA), a decision-level protocol for evaluating whether a specified runtime policy can replace an attributed model contribution under fixed boundary-visible information and a behavioral contract. This is for agentic settings, where LLM agents divide control between models and software runtimes. The paper is almost fully focused on Writer, the AI Writer workflow in LazyMind (LazyAGI,2026), rather than in more generality. There are three main dimensions: "Attribute" that traces accepted model-produced information to the committed decision; "Substitute" that tests whether the given policy acts or delegates under fixed observations and checks executed consequences against the same behavioral contract; and "Bound" that determines which conclusions these local out comes support. The approach is demonstrated on several test cases.

Critical strengths

The problem that is formulated and addressed in the paper seems to be important for the use cases described.

The paper looks to be correct, in terms of the derivation of the method, and the experiments (as far as I can tell) look to be correct.

Critical weaknesses

Going back to a lot of work in business process modeling, see e.g. "Work as a Service" by Vaculin, et al., there has been numerous attempts at structures like this, but none of that literature on human work is discussed. This is a major lacuna.

The focus on just one Writer model seems to restrict the generality of things.

Questions for authors

How broadly applicable is the approach?

4 comments
AuthorsOpenReview ↗

Response on Prior Work and Generality

We thank the reviewer for recognizing the importance of the problem and the soundness of our method and experiments.

Connection to prior work. We agree that the paper should better acknowledge the broader literature on business process management and human work. Beyond Vaculín et al.'s Work as a Service (2012), relevant foundations include knowledge-intensive and flexible processes (van der Aalst et al., 2005; Vaculín et al., 2011), artifact-centric lifecycle modeling (Hull et al., 2011), and process compliance and conformance checking (Ghose and Koliadis, 2007; de Leoni et al., 2012). These studies provide important foundations for coordinating work, specifying executable control, and checking execution against requirements. We will explicitly incorporate and compare these research lines in the revised Related Work.

Distinction and contribution. CSA does not claim novelty in workflow coordination, deterministic rules, delegation, or contract checking individually. Rather, it addresses a specific evidentiary question: when does the available evidence justify replacing an actual model contribution with a specified runtime policy at a given decision boundary?

Existing process models can represent control decisions, executable rules, and compliance requirements. These capabilities, however, do not by themselves establish whether an accepted model proposal influenced a particular committed action, or whether a specified replacement policy satisfies the same behavioral obligations under matched observations.

CSA connects model-to-commit attribution, fixed-information runtime intervention, and executed contract checking, yielding a bounded substitution claim rather than a general claim of workflow equivalence or optimal automation.

Applicability beyond Writer. CSA is formulated independently of writing-specific operations. It requires identifiable boundary observations, action-determining model contributions, an executable candidate policy, and checkable behavioral obligations.

Appendix A.4 illustrates this formulation at a second workflow's routing boundary: one Presentation fixture supports direct routing to build_outline, while another correctly delegates a decision requiring additional information. These are local demonstrations, not end-to-end Presentation evaluations.

Our strongest execution evidence remains in Writer: six selected conditions establish model-to-commit dependencies through accepted proposal perturbations, and their matched runtime substitutions satisfy the exact contracts. Separately, the controlled policy comparison shows why coverage alone is insufficient: broader direct-action coverage reduces proposal-model work but introduces a contract violation.

We will clarify both the relationship to earlier process research and the distinction between the protocol's general formulation and its currently demonstrated empirical scope.

AuthorsOpenReview ↗

Follow-up: New Non-Writer Substitution Evidence

Following our earlier response on prior work and generality, we have completed an additional experiment that was not yet available when our first comment was posted.

New evidence beyond Writer.

We evaluated CSA on ten new, prospectively frozen Presentation routing conditions, covering self-contained requests, current information, organizational sources, pasted materials, mandatory collection, and ambiguous or conflicting requirements.

Each condition used a real model-generated routing proposal and the native hosted workflow's persisted transition. In 10/10 conditions, changing only the accepted proposal's target from the same saved pre-decision state changed the committed route. This provides new model-to-commit attribution evidence at a non-editing decision boundary.

Two additional valid runtime substitutions.

The original runtime policy delegated all ten decisions. A revised policy, frozen before the formal evaluation, acted directly on 4/10 conditions.

Of these, F01 (time-management instruction) and F02 (stacks-and-queues instruction) passed their complete executed contracts, satisfying Equation (4). The remaining two direct actions retained unresolved semantic obligations and were not credited as successful substitutions.

We also observed a genuine reference-model error in F07, where an accepted route skipped required processing of pasted material. Separately, three predeclared unsafe route interventions were accepted by native execution but rejected by the complete contract checker.

Implication for generality.

The original Writer experiments demonstrated valid substitution in edit construction. These new results establish two additional scoped substitutions in workflow routing, using the same attribution–intervention–contract criterion.

Thus, CSA now has empirical support across two distinct types of model-mediated control decisions within LazyMind, extending the evidence beyond the earlier synthetic Presentation fixtures.

This additional experiment directly addresses the empirical generality concern raised in your review, complementing our previous discussion of the broader business-process and delegation literature.

Reviewer_aBmCOpenReview ↗

generality

Excellent to see the new results that demonstrate more of the generality beyond Writer. Certainly strengthens the paper.

AuthorsOpenReview ↗

re-generality

Thank you for your encouraging feedback! We are glad that the additional results help demonstrate the generality of our findings beyond the Writer setting.

We are carefully revising the main manuscript to better incorporate these new results and further clarify the broader applicability of our approach.

Given your positive assessment of the additional evidence, we would sincerely appreciate it if you could reconsider your overall rating in light of these improvements. Thank you again for your constructive feedback and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.