- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to separate the effects of information flow and execution authorization in LLM multi-agent systems. The authors propose a controlled evaluation framework, GateSim-T, and a rule-based authorization method called DecisionGate. Experiments across seven model checkpoints and five communication structures show that DecisionGate reduces unsafe execution compared to random and confidence-based abstention at matched coverage. The paper also evaluates topology effects, candidate availability, and authorization contract shifts. However, the large safety improvement observed in the synthetic benchmark does not clearly transfer to the external Criteo and AgentDojo evaluations.
Critical strengths
-
The paper introduces a useful evaluation framework that separates proposal formation from execution authorization, helping researchers better understand where safety improvements come from in LLM multi-agent systems.
-
The matched-coverage experiments provide meaningful evidence that execution selection matters beyond abstention frequency alone, supporting the value of the proposed evaluation framework.
Critical weaknesses
-
DecisionGate uses statistical certificates and full evidence that are unavailable to the baselines (Section 3.2, Appendix Table 6). This makes it difficult to separate the benefit of the authorization rule from the advantage of having additional information.
-
The large safety improvement in GateSim-T does not replicate in the external evaluations (Section 4.4, Table 1, Appendix Table 9). The matched-coverage differences are not statistically significant, and the original AgentDojo setting produces zero execution, weakening the generalizability of the main findings.
Questions for authors
-
Can the authors clarify whether the existing results provide any evidence that DecisionGate outperforms alternative authorization rules under equal information access (Section 3.2, Appendix Table 6)?
-
How do the authors explain the failure to reproduce the synthetic matched-coverage safety improvement in the external evaluations (Section 4.4, Table 1)?
-
What assumptions ensure the validity of the permutation-based adjusted p-values and simultaneous guardrail bounds? How was the fixed critical value of 2.13 determined (Section 3.2, Eq. 2, Appendix Table 7)?
-
The direct-rule baseline achieves higher accuracy and lower regret without MAS proposals. What additional benefit does the multi-agent proposal layer provide in this setting (Section 4.3, Appendix Table 8)?
Summary of the Revision for the Program Chairs, Area Chairs, and Reviewers
Summary of Our Response
Dear Program Chairs, Area Chairs, and Reviewers,
Thank you for recognizing the value of controlled layer substitution. We address the shared concerns below.
What the paper does. Our contribution is a controlled evaluation framework for attributing safety outcomes in action-taking multi-agent systems. It varies information flow under fixed tasks and call budgets, compares downstream policies on shared proposals, matches execution counts, and separately tests candidate availability and contract validity. GateSim-T provides known-outcome tasks; DecisionGate is an illustrative authorization policy. This separates execution quantity from policy-specific selection.
What the reviewers asked for, and what the evidence establishes. Locations refer to the submitted PDF; brackets are 95% task-cluster confidence intervals.
Information access and exact coverage (all) Location: Secs. 3.2-3.3; Fig. 3; Table 6 Each policy executes 824 proposals, matched within all 35 model-flow strata. Adverse counts are 6, 432, and 393 for DecisionGate, random, and confidence policies on the same 5,040 risk-eligible rows. Execution quantity alone cannot explain this difference. Certificate access contributes to selection; an equally informed LLM comparator has not been evaluated.
Is the result built in? (1yLQ) Location: Eq. (3); Table 8; frozen-record recount Agreement with the certificate is enforced; agreement with ground truth is not. The certificate authorizes one adverse task among 144 risk-eligible tasks. All six adverse gate rows come from that task. Certificate quality drives selection and can fail; the evaluation measures that failure.
External applicability (all) Location: Sec. 4.4; Tables 1 and 9 Matched-risk gaps are 0.36 pp [-1.93, 2.40] and 0.10 pp [-1.67, 1.67], so the synthetic advantage is not confirmed externally. The separate verifier comparison yields 0.1055 lower regret [0.0682, 0.1447] at 4.58% versus 25.00% coverage under the specified loss. This operating-point result does not establish matched-risk superiority.
Zero execution and candidate recovery (4tVR, 1yLQ) Location: Table 2; Appendix C Original AgentDojo and semantic clarification yield zero execution; Criteo supplies all original external executions. Explicit candidates restore 33.33% raw valid-action power. Trusted-ID authorization retains that power with zero observed unsafe execution, conditional on privileged ID access, not independent semantic verification.
What MAS adds (1yLQ, WUso) Location: Sec. 4.3; Table 8 The direct rule plus public monitoring reaches 94.79% correctness and 0.0286 regret, versus 74.81% and 0.1075 for proposal plus gate. Coverage is higher (20.83% vs 12.26%), as is adverse risk (0.69% vs 0.12%). It can supply actions rather than only filter proposals, so it tests MAS necessity, not authorizer superiority at matched coverage.
Generator independence (1yLQ) Location: Sec. 4.5; Appendices I-J An independent generator has not been tested. The contract-shift audit holds proposals fixed and exposes stale-authorization failures; it does not establish generator independence. Within-contract evaluation and robustness beyond that contract are distinct questions.
Statistical assumptions (WUso Q3; Sec. 3.2, Eq. (2), Table 7). The 199 shared treatment-label permutations preserve the treated count across 56 benefit paths. Complete-sharp-null validity requires assignment invariance, a design-matched permutation scheme, and a fixed analysis family; strong control under partial nulls needs further conditions such as subset pivotality or null domination. Guardrail bounds use the 0.95 quantile of the permutation maximum absolute statistic over four paths. A simultaneous-confidence interpretation requires valid approximation of joint centered estimation errors; model-based standard errors and random assignment alone do not guarantee this under misspecification or heterogeneous effects. The separate fixed value 2.13 is a frozen complement/transport threshold with no derivation documented in the protocol, not a general 95% simultaneous-confidence guarantee.
Contribution and scope. Exact coverage identifies a policy-level selection difference, not rule superiority with information held equal. The synthetic matching analysis remains post hoc, inference respects task clusters, and failed preregistered tests remain failures. The external studies demonstrate the protocol's use even when a policy advantage does not replicate; the direct-rule and candidate controls identify when a proposal layer is unnecessary or lacks actionable candidates. These architectural diagnoses in the evaluated settings are not available from an end-to-end safety score alone.
Respectfully,
The Authors