acceptodds
Under review as a conference paper at ICLR 2027

Information Flow Is Not Execution Authority: A Controlled Evaluation of LLM Multi-Agent Systems

Abstract

Safety gains in action-taking LLM multi-agent systems can arise from better proposal formation, lower execution coverage, or more effective selection among existing proposals, yet end-to-end metrics can conflate these sources. We present a controlled evaluation design to distinguish these mechanisms: we vary information flow under fixed tasks, evidence, and call budgets for proposal generation, compare authorization mechanisms on the same proposals, and separately examine candidate availability and authorization-contract validity. To enable these comparisons under explicit deployment criteria, we construct GateSim-T, a synthetic testbed comprising 192 tasks across seven model checkpoints and five information-flow configurations, yielding 6,720 paired decisions and 40,320 model calls. Within this testbed, we implement a separate authorization layer using a fixed scoped authorizer, DecisionGate, which checks proposals against an explicit contract before execution. To distinguish its selection effect from simply executing fewer proposals, we conduct a post-hoc comparison with random and confidence-based abstention, exactly matching execution counts within every model–flow stratum. At 12.26% overall execution coverage for each policy, adverse deployment on risk-eligible tasks is 0.12% under DecisionGate, compared with 8.57% for random and 7.80% for confidence abstention. A direct-rule control demonstrates that structured evidence can support effective deployment decisions without MAS-generated proposals in this testbed. In a pooled analysis of external Criteo and AgentDojo evaluations, DecisionGate occupies a lower-coverage, lower-regret operating point than a one-call LLM verifier under the specified loss. Further interventions show that explicit candidates restore useful action under trusted candidate authorization, while tested contract shifts change deployment risk. Overall, these results support treating proposal formation and execution authorization as distinct mechanisms: information flow shapes what is proposed, whereas scoped authorization determines what may be executed. Their combined effectiveness depends on candidate availability and the validity of the authorization contract.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model agents, multi-agent systems, information flow, execution authorization, selective decision-making, AI safety, tool-use security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WUsoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to separate the effects of information flow and execution authorization in LLM multi-agent systems. The authors propose a controlled evaluation framework, GateSim-T, and a rule-based authorization method called DecisionGate. Experiments across seven model checkpoints and five communication structures show that DecisionGate reduces unsafe execution compared to random and confidence-based abstention at matched coverage. The paper also evaluates topology effects, candidate availability, and authorization contract shifts. However, the large safety improvement observed in the synthetic benchmark does not clearly transfer to the external Criteo and AgentDojo evaluations.

Critical strengths
  1. The paper introduces a useful evaluation framework that separates proposal formation from execution authorization, helping researchers better understand where safety improvements come from in LLM multi-agent systems.

  2. The matched-coverage experiments provide meaningful evidence that execution selection matters beyond abstention frequency alone, supporting the value of the proposed evaluation framework.

Critical weaknesses
  1. DecisionGate uses statistical certificates and full evidence that are unavailable to the baselines (Section 3.2, Appendix Table 6). This makes it difficult to separate the benefit of the authorization rule from the advantage of having additional information.

  2. The large safety improvement in GateSim-T does not replicate in the external evaluations (Section 4.4, Table 1, Appendix Table 9). The matched-coverage differences are not statistically significant, and the original AgentDojo setting produces zero execution, weakening the generalizability of the main findings.

Questions for authors
  1. Can the authors clarify whether the existing results provide any evidence that DecisionGate outperforms alternative authorization rules under equal information access (Section 3.2, Appendix Table 6)?

  2. How do the authors explain the failure to reproduce the synthetic matched-coverage safety improvement in the external evaluations (Section 4.4, Table 1)?

  3. What assumptions ensure the validity of the permutation-based adjusted p-values and simultaneous guardrail bounds? How was the fixed critical value of 2.13 determined (Section 3.2, Eq. 2, Appendix Table 7)?

  4. The direct-rule baseline achieves higher accuracy and lower regret without MAS proposals. What additional benefit does the multi-agent proposal layer provide in this setting (Section 4.3, Appendix Table 8)?

Reviewer_1yLQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that safety gains in action-taking LLM multi-agent systems mix several effects: how proposals are formed, how often actions run, and which actions are chosen. Its main contribution is "controlled layer substitution", a protocol that freezes the multi-agent proposals and swaps only the downstream policy. The authors build GateSim-T, a synthetic testbed with 192 tasks, seven models and five communication setups (6,720 decisions). They also propose DecisionGate, a rule that lets a proposed action run only if it matches a precomputed statistical certificate. At the same execution rate (12.26%), DecisionGate's adverse deployment rate is 0.12%, versus 8.57% for random and 7.80% for confidence-based abstention. External tests on Criteo and AgentDojo do not reproduce this gap.

Critical strengths

1- A well-documented evaluation protocol. Freezing proposals and matching execution counts in each of the 35 model–flow strata (Sec. 3.3) is a sensible way to separate "running fewer actions" from "choosing better actions". Table 3 states clearly what each comparison holds fixed. This is the part of the paper most likely to be reused. 2- Good reporting of null and failed results. The paper reports that its original Phase I hypothesis failed (Appx. A.2), that H1 and H2 failed (Table 4), and that the external matched-coverage test failed (Sec. 4.4). It also marks which analyses were post hoc (Appx. H). This openness raises my trust in the numbers, even where I disagree with how they are read. Critical Weaknesses

Critical weaknesses

1- The main result is close to built in. DecisionGate keeps a proposed ship action only if it equals the certificate action c (Eq. 3). That certificate comes from full evidence cards and adjusted statistics that the random and confidence policies never see (Table 6). Applied directly with no MAS, the same rule has 0.69% adverse risk, 100% clean power and 82–95% correctness (Table 8). Its single adverse event falls on one of 144 risk-eligible tasks. So the gate's 0.12% risk mostly reflects the certificate's own quality, not better selection among MAS proposals. The authors admit this is "not an equal-information algorithm comparison" (Sec. 1). But then the headline gap of 0.12% versus 7.80% says little about authorization as a method.

2- The main result does not carry over outside the synthetic testbed. The matched-coverage comparison was frozen after earlier results and external review, so it is post hoc (Appx. H). The external test of that comparison fails: the gaps over random and confidence abstention are 0.36 pp (95% CI [−1.93, 2.40]) and 0.10 pp (95% CI [−1.67, 1.67]) (Sec. 4.4). All external execution comes from Criteo, and the original AgentDojo slice has zero coverage. In the AgentDojo recovery study, only the authorizer receives the trusted valid-candidate ID (Table 2). So "zero unsafe execution" there also rests on unequal information.

Questions for authors

1- (Clarification) Can you compare DecisionGate with a verifier or confidence policy that sees the same adjusted certificate quantities, at matched coverage? Without this, I can't tell how much of the gap comes from the rule rather than the information it sees. 2- (Clarification) The GateSim-T generator, the oracle and the certificate all follow one evidence contract (Appx. J). How would results change with a generator built independently of the certificate rule? 3- Clarification) Given that the external matched-coverage test fails, which claims do you think still hold beyond GateSim-T? Would you revise the abstract and contributions to reflect this?

Reviewer_4tVROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates safety outcomes in LLM multi-agent systems. Prior evaluations often mix the effects of communication structures with the final authorization of actions. To address this problem, the paper presents an evaluation framework called controlled layer substitution.

This framework basically separates proposal generation from the final decision to execute an action. To test this, the paper introduces GateSim-T, which is a synthetic testbed with 192 tasks. This testbed is mainly used to evaluate seven models across five information-flow architectures. The primary authorization layer uses a fixed decision rule and structured evidence rather than unstructured text evaluation to determine if a proposal is permitted (DecisionGate).

For validation, the researchers match the exact number of executed actions across different policies to isolate selection quality from simple abstention rates. At a fixed 12.26% execution rate, DecisionGate shows a 0.12% adverse deployment rate on risk-eligible tasks. Under the exact same conditions, random abstention results in 8.57% adverse deployment and confidence abstention results in 7.80%.

The paper also conducts external validation on Criteo and AgentDojo tasks. These external tests show a different relationship between coverage and regret, and they do not replicate the exact risk separation found in the synthetic environment. Overall, the results show that authorization systems require precise definitions of evidence and risk semantics to maintain validity when conditions change.

Critical strengths

(1) The paper introduces controlled layer substitution to isolate distinct functions within multi-agent systems. I see that this includes proposal formation and execution selection. Current benchmarks typically rely on end-to-end evaluations that confound these variables. The framework separates these layers to allow researchers to identify the exact origin of safety outcomes. This contribution provides a structural means that is necessary for evaluation of agent architectures.

(2) The experimental design is well thought out with the executed actions across different authorization policies. This coverage control helps ensure that safety measurements reflect actual selection quality rather than a simple increase in abstention rates. This design is highly important. This is mainly because it prevents evaluations from favoring policies that achieve low adverse deployment merely by refusing to execute actions.

Critical weaknesses

I have two main concerns regarding the evaluation.

First, I see that the paper in the evaluation compares the DecisionGate policy against baseline methods using unequal levels of information. DecisionGate evaluates structured evidence, precomputed statistical certificates, and adjusted p-values. In my opinion, however, the random and confidence-based baselines do not have access to these specific variables.

The paper states that this setup tests complete policies instead of comparing algorithms that use the exact same data. This structural imbalance makes it difficult to prove whether the lower adverse deployment rate comes from the decision rule itself or simply from its access to better statistical information. The paper would clarify this.

Second, the central claims about matched-coverage risk rely entirely on the synthetic GateSim-T environment. When the paper applies the framework to external Criteo and AgentDojo tasks, the results fail to reproduce the findings from the synthetic tests. These external tests show a different relationship between coverage and regret, and they fail to prove a statistical advantage for DecisionGate over the baseline models at exact coverage.

In addition, the AgentDojo trials produced zero executions across all policies under standard conditions, and the system required an explicit candidate identifier to function properly. This failure to replicate results on external datasets mostly restricts the practical value of the introduced framework.

Questions for authors

(1) Could you clarify whether you have tested a baseline policy that operates on the exact same structured data? This is needed because it is important to determine whether the reduced adverse deployment rate originates from the decision rule itself or solely from its exclusive access to statistical evidence.

(2) Could you provide a clear justification for how the controlled layer substitution framework generalizes to external, non-synthetic domains? This directly affects the evaluation of the framework practical utility

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Revision for the Program Chairs, Area Chairs, and Reviewers

Summary of Our Response

Dear Program Chairs, Area Chairs, and Reviewers,

Thank you for recognizing the value of controlled layer substitution. We address the shared concerns below.

What the paper does. Our contribution is a controlled evaluation framework for attributing safety outcomes in action-taking multi-agent systems. It varies information flow under fixed tasks and call budgets, compares downstream policies on shared proposals, matches execution counts, and separately tests candidate availability and contract validity. GateSim-T provides known-outcome tasks; DecisionGate is an illustrative authorization policy. This separates execution quantity from policy-specific selection.

What the reviewers asked for, and what the evidence establishes. Locations refer to the submitted PDF; brackets are 95% task-cluster confidence intervals.

  1. Information access and exact coverage (all) Location: Secs. 3.2-3.3; Fig. 3; Table 6 Each policy executes 824 proposals, matched within all 35 model-flow strata. Adverse counts are 6, 432, and 393 for DecisionGate, random, and confidence policies on the same 5,040 risk-eligible rows. Execution quantity alone cannot explain this difference. Certificate access contributes to selection; an equally informed LLM comparator has not been evaluated.

  2. Is the result built in? (1yLQ) Location: Eq. (3); Table 8; frozen-record recount Agreement with the certificate is enforced; agreement with ground truth is not. The certificate authorizes one adverse task among 144 risk-eligible tasks. All six adverse gate rows come from that task. Certificate quality drives selection and can fail; the evaluation measures that failure.

  3. External applicability (all) Location: Sec. 4.4; Tables 1 and 9 Matched-risk gaps are 0.36 pp [-1.93, 2.40] and 0.10 pp [-1.67, 1.67], so the synthetic advantage is not confirmed externally. The separate verifier comparison yields 0.1055 lower regret [0.0682, 0.1447] at 4.58% versus 25.00% coverage under the specified loss. This operating-point result does not establish matched-risk superiority.

  4. Zero execution and candidate recovery (4tVR, 1yLQ) Location: Table 2; Appendix C Original AgentDojo and semantic clarification yield zero execution; Criteo supplies all original external executions. Explicit candidates restore 33.33% raw valid-action power. Trusted-ID authorization retains that power with zero observed unsafe execution, conditional on privileged ID access, not independent semantic verification.

  5. What MAS adds (1yLQ, WUso) Location: Sec. 4.3; Table 8 The direct rule plus public monitoring reaches 94.79% correctness and 0.0286 regret, versus 74.81% and 0.1075 for proposal plus gate. Coverage is higher (20.83% vs 12.26%), as is adverse risk (0.69% vs 0.12%). It can supply actions rather than only filter proposals, so it tests MAS necessity, not authorizer superiority at matched coverage.

  6. Generator independence (1yLQ) Location: Sec. 4.5; Appendices I-J An independent generator has not been tested. The contract-shift audit holds proposals fixed and exposes stale-authorization failures; it does not establish generator independence. Within-contract evaluation and robustness beyond that contract are distinct questions.

Statistical assumptions (WUso Q3; Sec. 3.2, Eq. (2), Table 7). The 199 shared treatment-label permutations preserve the treated count across 56 benefit paths. Complete-sharp-null validity requires assignment invariance, a design-matched permutation scheme, and a fixed analysis family; strong control under partial nulls needs further conditions such as subset pivotality or null domination. Guardrail bounds use the 0.95 quantile of the permutation maximum absolute statistic over four paths. A simultaneous-confidence interpretation requires valid approximation of joint centered estimation errors; model-based standard errors and random assignment alone do not guarantee this under misspecification or heterogeneous effects. The separate fixed value 2.13 is a frozen complement/transport threshold with no derivation documented in the protocol, not a general 95% simultaneous-confidence guarantee.

Contribution and scope. Exact coverage identifies a policy-level selection difference, not rule superiority with information held equal. The synthetic matching analysis remains post hoc, inference respects task clusters, and failed preregistered tests remain failures. The external studies demonstrate the protocol's use even when a policy advantage does not replicate; the direct-rule and candidate controls identify when a proposal layer is unnecessary or lacks actionable candidates. These architectural diagnoses in the evaluated settings are not available from an end-to-end safety score alone.

Respectfully,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.