- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates when the runtime should take over control from the LLM in agent workflows. Its main motivation appears to be improving agent efficiency by analyzing when model-mediated decisions can be safely replaced by deterministic runtime execution.
However, I have several concerns about the paper. First, the problem formulation is relatively simplistic, and the experimental evaluation is limited in both scale and diversity, with most experiments focusing on a single type of agent. As a result, I am not fully convinced that the experimental evidence is sufficient to support the paper's conclusions, and I am concerned that the capabilities and generalizability of CSA may be overstated.
In addition, I find the presentation somewhat lacking in clarity. The paper provides only one relatively basic overview figure, which does not sufficiently illustrate the proposed framework, while much of the experimental evidence and detailed analysis is relegated to the appendix.
Critical strengths
- The requirement to connect model-to-commit attribution with a matched runtime intervention is a meaningful perspective, since neither the presence of a model call nor successful execution alone demonstrates valid substitution.
Critical weaknesses
- One concern I have with CSA's design is that it depends heavily on predefined runtime policies, decision boundaries, and behavioral contracts. It mainly tells us whether a particular policy works under specific tested conditions, rather than helping determine when the runtime should actually take over. Also, passing a contract check on a few executions does not necessarily mean the policy will work reliably on unseen cases.
- The empirical evaluation is relatively limited and lacks convincing comparisons with existing approaches. Most experiments are conducted on the Writer agent using relatively simple, deterministic text-editing tasks, which may not adequately reflect the complexity of real-world agent decision-making. Also the paper does not provide direct comparisons against simpler attribution or contract-checking baselines under the same conditions. It is difficult to assess whether the proposed framework offers substantial advantages over existing techniques or generalizes beyond the evaluated scenarios.
Questions for authors
- Can the authors demonstrate the generalizability of CSA beyond deterministic text editing to more complex agent decision-making tasks?
- How do the authors explain the disagreements between the LLM auditors and the reference labels, and what do these imply for CSA's reliability?
3 comments
Auditor Disagreements, Generality, and Presentation (Part 2/2)
Following our response above, we address the auditor disagreements, applicability beyond editing, and presentation concerns.
3. What explains the auditor disagreements, and what do they imply for reliability (Q2)?
The two separately initialized auditor sessions agree on all four fields across all 18 packets. Each agrees with the retained reference key on 15/18 attribution and supported-conclusion labels, and 18/18 behavior and contract labels (Table 6; Appendix A.2).
The three shared disagreements have an identifiable source.
In DS01 and DS03, the disclosed reference execution copies an existing observation; in DS02, it emits a fixed literal. These paths do not depend on the model proposal: changing that proposal leaves the committed output unchanged.
The auditors therefore identify these cases as RUNTIMEONLY, consistent with the disclosed execution evidence. The reference key instead uses source-family metadata to assign model-mediated labels.
We traced this discrepancy through the original condition generator and preserved the original key and reported 15/18 comparison.
This provides a concrete explanation for the disagreements and highlights an essential property of reliable auditing: attribution conclusions must be grounded in actual proposal-to-commit dependencies, rather than inferred from source labels or nearby model calls.
4. How broadly does CSA apply beyond deterministic editing (Q1)?
CSA is formulated around a decision boundary, rather than a particular Writer operation. The audit procedure applies wherever available observations, action-determining contributions, candidate runtime behavior, and behavioral obligations can be explicitly specified and evaluated.
The Writer experiments provide direct execution evidence for model-mediated edit construction and matched replacement. Appendix A.4 additionally instantiates CSA at Presentation routing and delegation boundaries, including direct build_outline routing and delegation to collect_materials.
These illustrate different decision types under a shared audit formulation, with the strongest executed substitution evidence currently coming from Writer.
The methodology also accommodates decisions requiring delegation: unresolved semantic obligations remain model-mediated rather than being forced into deterministic actions. Section 5.3 reports coverage and delegation behavior separately.
Thus, CSA provides a decision-level evaluation framework for different runtime policies and agent workflows without assuming that one fixed policy applies universally.
5. Presentation and experimental evidence
Figure 1 introduces the decision boundary; Equation (4) defines the substitution criterion; Table 1 distinguishes supported conclusions; and Table 2 reports the policy comparison.
Appendix Figure 2 presents the audit sequence, while Appendix A.3 and Figure 4 trace the matched Writer intervention.
We hope these connections clarify the methodological contribution, experimental findings, and interpretation of the auditor disagreements.
Follow-up: New Routing Experiments and Same-Evidence Baselines
Following our two earlier comments, we have completed a new experimental evaluation directly addressing your concerns about generality, policy-relative takeover, and baseline comparisons.
1. New non-editing evaluation.
We evaluated ten prospectively frozen Presentation routing conditions. Unlike the synthetic fixtures previously discussed, the new evaluation uses real model-generated proposals, accepted native transitions, and persisted workflow decisions.
All 10/10 reference proposals were commit-sensitive: changing only the accepted target under the same saved pre-decision state changed the persisted route.
Two frozen runtime policies were evaluated on the identical conditions:
- v1: ACT 0/10; DELEGATE 10/10.
- v2: ACT 4/10; DELEGATE 6/10.
Among v2's four direct actions, F01 and F02 passed their complete behavioral contracts, establishing two new non-Writer direct substitutions under Equation (4). F03 and F04 remained UNSCORED because semantic obligations were unresolved.
The six v2 delegations resulted in 4 PASS, 1 FAIL, and 1 UNSCORED. Both policy arms had the same overall boundary outcomes: 6 PASS, 1 FAIL, and 3 UNSCORED.
Therefore, expanding direct-action coverage from 0/10 to 4/10 produced two justified substitutions, illustrating why CSA distinguishes policy coverage from contract validity.
2. Real failure and unsafe interventions.
F07 provides an informative reference-model failure: the model chose to skip required processing of pasted material, and the native workflow accepted this route. Both policies delegated and consequently preserved the violation.
Additionally, three predeclared unsafe routing interventions were accepted natively but failed the complete contract checker. These challenge executions were not counted as additional independent conditions.
These results demonstrate that native acceptance, model delegation, and behavioral correctness can produce different outcomes.
3. Additional baseline comparison.
We evaluated model-call presence, attribution-only, complete contract-only, CSA, and a separately implemented ordinary complete-audit checklist on the same 23 execution evidence packets.
The complete contract checker identified all five observed collection-obligation violation packets, including repeated intervention arms. The ordinary complete audit agreed with CSA on 23/23 scoped conclusions.
This comparison provides a more concrete characterization of CSA's contribution: an explicit joint evidence protocol connecting original model contribution, matched runtime intervention, executed obligations, and bounded takeover conclusions. We give complete auditing and contract checking full credit for the conclusions they independently support.
4. Reliability and scope.
These results complement our earlier explanation of the DS01–DS03 auditor disagreements; the original 15/18 key comparison remains unchanged.
The new evidence establishes native, commit-sensitive control attribution and valid substitution at a second decision type. It uses one workflow engine, a bounded live proposal adapter, and programmatic contract checking supplemented by non-blinded semantic review. Unresolved obligations remain unscored.
We believe these additional results materially strengthen the empirical basis for CSA and directly address the key concerns about applicability beyond deterministic text editing.
Policy-Relative Takeover and Comparative Evidence (Part 1/2)
We thank the reviewer for recognizing the importance of connecting model-to-commit attribution with a matched runtime intervention. This connection is central to CSA. We first address the policy-relative formulation and comparative evidence.
1. How does CSA determine when the runtime should take over?
CSA evaluates a specified takeover policy at a declared decision boundary, rather than generating an optimal controller. The key question is whether that policy can replace an actual model contribution under the same available information and behavioral obligations.
Equation (4) operationalizes this criterion: the original model proposal must have influenced the committed action, the runtime policy must act directly using boundary-visible information, and the executed replacement must satisfy the same contract.
Our controlled comparison demonstrates the significance of this policy-relative evaluation (Table 2).
Across 22 proposal-reaching conditions, Literal v1 acts directly on 4/22, passing 4/4 scored contracts. Normalized v2 expands coverage to 15/22, passing 14/15 contracts.
The broader policy reduces recorded proposal-model requests from 19 to 8 and proposal tokens from 50,544 to 26,737, while both policies complete 21/24 executions. In C08, v2 incorrectly deletes content despite an explicit instruction not to delete it.
These results demonstrate why runtime coverage, execution completion, model work, and valid substitution must be distinguished. We additionally reproduced all 48 policy-condition direct-stage records using the original native implementation, obtaining results consistent with the original experiments.
2. What does CSA add beyond simpler attribution and contract checks?
Attribution and contract checking answer complementary questions.
An attribution check determines whether the original model influenced the committed decision. A contract checker evaluates whether the replacement execution satisfies its obligations.
CSA connects these forms of evidence through a matched intervention at the same decision boundary, producing a scoped conclusion about whether an actual model contribution was validly replaced (Equation 4; Table 1).
In six selected Writer conditions, accepted proposal perturbations changed committed outputs in 6/6, and matched runtime replacements satisfied exact contracts in 6/6 (§§5.1–5.2).
Our same-evidence diagnostic further clarifies the roles of individual checks: contract checking identifies the C08 violation, while attribution analysis identifies runtime-only contributions. CSA connects these findings to the same substitution claim.
The R1/R2 and v1/v2 experiments provide matched execution and policy comparisons, while Appendix A.2 provides a controlled protocol sanity check. Together, these establish the evaluated mechanisms without treating simpler components as artificially weak competitors.