acceptodds
Under review as a conference paper at ICLR 2027

The Safety Criticality Number: Cascade Stability and Intervention Thresholds for AI Agents

Abstract

Risky actions by an AI agent may arrive in cascades: one event changes context or shared state so later events become more likely. We model this with a marked Hawkes process and separate two conflated quantities—an intrinsic cascade criticality (whether an uncontrolled cluster dies out) and operational monitorability (observation, containment, background risk, capacity). With observation probability and conditional containment , we prove within the model that control is subcritical exactly when ; we extend this to interacting agents via a controlled branching matrix and establish a tight testing rate in labeled parent events. Empirically, in a synthetic decision setting with all tools disabled and labels never executed, a prespecified auditable paired replication raises the downstream terminal unsafe rate from to over matched restrictive cases when the erroneous upstream conclusion is supplied (pp; McNemar exact ; all prompts and raw responses retained), while a multi-model panel shows heterogeneous per-edge transmission ( from 0 to ) and a realized cascade shows finite-horizon growth versus persistence consistent with . A second pre-registered study on a different task family observes no transmission on two current models; because the studies differ in more than verifiability, this contrast does not isolate verifiability as the cause and holds for the tested tasks only. A separate read-only trace study finds excess adjacency of elevated-access tool calls after conditioning on task and phase (Holm )—temporal dependence, not evidence that deployed-agent harms follow a Hawkes process. Finally, a prospective test—predictions fixed before each corrected collection on a previously examined case pool—forecast that placing an independent reviewer *last* is riskiest; the order held in both error-making models, resolved within one and pooled (McNemar ) but not the other, with a permutation-invariant baseline tying all placements. These are model-specific translations of a classical stability boundary, not a measured branching ratio or a deployment guarantee.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI agent safety, multi-agent systems, error propagation, Hawkes processes, branching processes, runtime monitoring

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RwaiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors borrow from control to create a framework for AI error monitoring that prevents cascading errors. This is a very important topic, and I think control theory is the right framework for this, but the paper is too hard to read, and has many other missed opptunitues detailed below.

Critical strengths

Cascading errors are a topic I am very interested in, so I was very excited to see this paper. I also think that transferring and translating concepts from control theory is a very good approach for this problem. I also didn't find errors in the paper.

Critical weaknesses

Despite the above, the paper does poorly as a translational paper, which is where I feel its potential contributions are.

First, it is hard to read, and I believe it may be inaccessible to those who do not already have some knowledge / background in control.

Second, but most importantly, it really doesn't do a good just making the case of where and under what conditions the framework will be useful to several issues.

-- There is an implicit underlying assumption that errors are directional, which is not made clear. -- The use of continuous time is not well justified, as no example that is not sequential is given. -- The authors do not make the case for why this is better than, for example, stopping all noticeable errors. It assumes that is not desirable but does not make the case for it, and therefore implicitly assumes that all errors are as consequential (which is not realistic). -- The limitation section is extra painful -- it is a shopping list of problems.

The biggest missed opportunity here is that I think this can work quite well for bias rather than all errors, where it makes sense we may want to allow X% bias, but not Y% -- which we may never be able to detect by only observing 1 transition at a time.

Overall, I think the theoretical contributions in themselves and the existing experiments do not merit sufficient contribution. However, adding convincing examples and making a case for where this will work better than alternatives will make a big difference here.

Questions for authors

-- Who do you think is the audience for this paper? And do you think it is clearly written for them?

-- Can you give real, clear examples of where this will work well?

-- Can you clearly articulate both assumptions and requirements for the framework to be successfully applied?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Rwai

Thank you for the thoughtful review and concrete suggestions. You asked where this framework works better than the obvious alternative, stopping every error that can be noticed. The revision answers this with measurements, and the Discussion (Sec. 6) now opens with it.

When a calibrated level is better. We see three situations.

  1. Errors cannot multiply, for example when each conclusion has a single reader. Then the lowest safe level needs the fewest inspections. Live, the certified level needed about half the inspections of stopping everything and stayed within the tolerance.
  2. Flags are noisy. Stopping a false flag also stops correct work. A judge in our earlier study had 88% recall but also flagged 43% of correct cases (Table 8).
  3. A tolerance must be met and documented. Here the average per-step error rate gives a level that is too low, because some cases are much harder than others: on held-out M2 data it let through about three times the tolerance, while the level certified from repeated readings stayed within it. This is the problem you described, one that "may never be detected by observing one transition at a time".

A concrete example (Fig. 1, App. G). An intake agent wrongly marks a refund as eligible, and three agents downstream read its conclusion. If each copies the error with probability 0.4, quarantining 17% of errors is enough to stop cascades. Now suppose half the cases are hard, where agents copy with probability 0.8, and the other half are never copied. A monitor that watches one handoff at a time still sees 0.4, but now 58% of errors must be quarantined. Only showing the same case to several agents reveals the difference. The same measurements also told us where to put an independent reviewer: last, behind agents that defer, was predicted to be riskiest, and for M2 it was (17.4% vs. 5.6% unsafe when placed first).

Q1: Audience. The paper is for engineers who build or audit multi-agent pipelines and need to decide where to put a reviewer, how much to quarantine, and how to know it is enough. The abstract and introduction now start from these decisions, Sec. 3 opens with an intuitive picture, and the abstract says "branching process" instead of "Hawkes process". We present the work as a research method, not a validated operational control.

Q3: Requirements. One needs an error that can be labeled and that something downstream uses, a way to show a handoff to fresh agents (we used 100 errors, 8 agents each), a tolerance for each error type, and a way to block a flagged conclusion. Each error type has its own tolerance, so errors need not be equally consequential. Table 1 lists each assumption and where it is relaxed or tested. The Limitations are now grouped into model assumptions, measurement scope, and external validity.

Bias (Sec. 4.3). Thank you, we adopted this suggestion. A tolerated bias becomes a tolerance on how common the biased decision is. For example, a 2% bias toward approval that later agents pass on 80% of the time settles at 10% of decisions, and keeping it at 5% requires quarantining at least a quarter of them. In M1's eight-agent chains (E7), the one-step model got this prevalence wrong in both directions, while per-case rates came close.

Direction of errors and continuous time. Errors need not flow one way: each error type has its own row in the interaction matrix, influence goes from a conclusion to whoever reads it, and feedback loops are allowed. Continuous time is there for asynchronous agents and monitor capacity; the thresholds depend only on the expected number of downstream errors, so they also hold in discrete turns. E9 tests a five-turn dialogue, where a reader sees the whole thread, not only its parent's conclusion. Errors spread 1.85 times further, so a dialogue has to be calibrated in its own format.

When stopping everything is better, and other limitations. If each conclusion is read by several agents and flags are accurate and cheap, stopping every flagged error is the best policy: live, with two or three readers per conclusion, it needed fewer inspections than any calibrated level. The framework does not help when propagation cannot be blocked without blocking the decision itself, or when the tolerance is zero. Finding out which situation applies took about 1,600 model calls per model; without measurement, assuming every error is copied gives a safe but conservative level. You were right that our summed containment rate is not a cost; the paper now calls it a proxy. Placement held for M2, not M1; one test in Table 1 failed; and two of E9's four predictions failed (App. E.7).

We believe the measured comparison and the example answer the question you raised. If you have a particular setting in mind, we would be happy to discuss how the method applies to it.

Just commenting to say I have read this response, and I am not reassured by it that the issues I mentioned can be resolved. For now, I am keeping my score the same.

Thank you for reading our response. We understand that it has not resolved your concerns. For ease of reference, the most relevant parts of the revised PDF are:

  • Practical value and comparison with simpler policies: Sec. 6 (p. 10) and App. G.4 (pp. 58–59), including when stopping every flagged error is preferable.
  • Audience and concrete example: Sec. 1 and Fig. 1 (pp. 1–2).
  • Conditions for applying the framework: end of Sec. 3 (p. 4) and Table 1 (p. 5).
  • Unequal error consequences and your bias suggestion: Prop. 8 and Sec. 4.3 (pp. 5–7).
  • Readability for readers without a control background: plain-language primer and glossary, App. G (starting on p. 56).

Could you indicate which concern remains decisive—for example, the paper’s readability, the conditions for applying the framework, or the evidence for its practical benefit?

We recognize that the empirical evidence remains synthetic and does not establish effectiveness in deployment. Your clarification would help us give a focused response during the remaining discussion.

Reviewer_tkW1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses runtime safety in AI agent systems by modeling error propagation as a self-exciting point process (Hawkes process) and branching tree. It defines the Safety Criticality Number (RSR_S) as the expected number of direct descendant risk events spawned by an uncontained parent error. The authors mathematically prove that a monitoring system with observation probability pp and conditional containment probability qq stabilizes a cascade if and only if (1−pq)RS<1(1 - pq)R_S < 1. They further decouple cascade stability from operational monitoring capacity (LSL_S), extend the theory to multi-agent networks via a spectral radius criterion (ρ(ADu)<1\rho(A D_u) < 1), and prove a tight testing sample complexity of Θ(ϵ−2)\Theta(\epsilon^{-2}). Empirically, the authors evaluate tool-call traces, test error transmission across five LLM families in synthetic decision tasks, demonstrate a +4.8 pp+4.8\text{ pp} increase in downstream unsafe decisions under an erroneous handoff, and examine workflow ordering in multi-agent pipelines.

AI Usage Disclosure "AI assistance was used to help parse cross-disciplinary mathematical terminology during the reading of this manuscript, and to help structure and polish the initial drafted review notes into final text under the reviewer's direction. The Input text and interactions with the AI are provided in "AI Documentation For Reviewing". "

Critical strengths
  1. Compelling Framing: Shifting from Action Classification to Cascade Control
    The paper addresses a critical blind spot in modern AI safety. Most current guardrails treat safety as a static, single-step classification problem (inspecting an action AtA_t in isolation). Reframing safety as dynamic cascade prevention is well-motivated and provides valuable insights into agentic systems, where early errors corrupt shared context and trigger compounding failures downstream.

  2. Actionable Architectural Guidance from Study 5
    Study 5 provides practical, concrete recommendations for multi-agent system design. Common engineering intuition suggests placing an independent auditor at the very end of an agent pipeline to catch prior errors. The paper demonstrates both theoretically and empirically that placing an independent reviewer last in a chain of deferential agents is actually the most unsafe layout (e.g., 17.5% unsafe vs. 7.2% when placed first). This is a highly valuable, counter-intuitive insight for practitioners building production agent teams.

  3. Surprising and Counter-Intuitive Mathematical Insights
    The paper uncovers several non-obvious dynamics of agent monitoring. Most notably, the analysis of operational monitoring load (LSL_S) reveals that ∂LS∂p∝(1−RS)\frac{\partial L_S}{\partial p} \propto (1 - R_S): when a system is supercritical (RS>1R_S > 1), increasing detection probability (pp) actually reduces the total long-term monitoring compute load. This occurs because catching early errors extinguishes runaway cascades before they generate large, compute-draining clusters of downstream events.

Critical weaknesses
  1. Overly Dense Presentation and Heavy Jargon Without Sufficient Intuition
    The paper is difficult to read. It incorporates dense terminology from point-process theory, branching processes, and matrix perturbation (e.g., Campbell’s identity, Poisson cluster representations, Le Cam’s two-point method, Perron eigenvectors) without providing accessible, self-contained explanations for a broader machine learning audience. Key conceptual transitions between continuous-time point processes and discrete-turn agent workflows are heavily condensed, requiring readers to spend significant time deciphering the underlying mechanics.

  2. Extreme Metric Instability Across Models and Tasks (RSR_S is Not a Stable Property)
    The empirical results reveal that the error transmission rate q^\hat{q} varies wildly—from 0.00 (completely immune) up to 0.67 (highly susceptible) across different models. More critically, on simple deterministically checkable tasks (Appendix A.4), transmission dropped to 0/229, whereas on ambiguous tasks it was substantial. Because RSR_S changes drastically with prompt framing, task ambiguity, and model checkpoints, it cannot be treated as a durable, static safety rating. In practice, engineers would have to re-benchmark and re-estimate RSR_S from scratch for every minor task or prompt modification.

  3. Limited Ecological Validity from Synthetic, Tool-Disabled Sandboxes
    To maintain experimental control, almost all empirical decision studies (Studies 2–5) evaluate synthetic business logic tasks where all tools are disabled, actions are mock-labeled, and labels are never executed. While this cleanly isolates the causal effect of text handoffs, it strips away the exact environmental mechanics where real-world agent cascades happen (e.g., file system side-effects, shell execution failures, API timeouts, and state mutations). The paper leaves unverified whether these reproduction dynamics hold in realistic, tool-executing environments.

Questions for authors

Refer to weaknesses, please

6 comments
AuthorsOpenReview ↗

Response to Reviewer tkW1

Thank you for the positive review and the helpful comments on presentation. The revision addresses each of them: the paper is simplified, transmission is now measured for each deployment, the procedure is shown to carry over to a second model, and a new experiment uses real tool calls.

W1: Presentation. The main text now explains terms such as Campbell's theorem, Le Cam's bound, the Perron root, and Poisson cluster processes in plain words where they are needed, and avoids them elsewhere. Prop. 8 and Thm. 15 are each followed by a short plain-language reading. Sec. 3 now opens with an intuitive "family tree" picture of a cascade and explains why we use continuous time: asynchronous agents and monitor capacity need a clock, but the thresholds depend only on the average number of downstream errors, so they apply unchanged to discrete turns. A new Figure 1 walks through a refund pipeline, and its panel (c) shows the reviewer-placement result you mentioned. App. G is a short primer with a glossary, a table translating continuous-time quantities into turns, and a worked example.

W2: q is not stable. We agree, and the revision treats this as a premise. On the same cases, M1's unsafe-decision rate without seeing any upstream conclusion ranged from 0% to 84%, depending only on the instruction wording. The paper therefore treats transmission as something to measure for each deployment and shows how; after any prompt or model change, the level should be re-certified from new repeated readings (Cor. 16). The procedure carries over to another model: calibrated the same way, M2's certified levels held on its held-out data and stayed within the tolerance on average when run live; the fresh-case primary test missed in one setting (upper bound 1.11). If re-measuring is not possible, the safe fallback is to assume every error is copied; that level stayed within the tolerance in all our held-out studies.

W3: Tool-executing environments. The new E4 experiment runs real tool calls against a stateful ticket store, with injected tool timeouts, and passes the upstream flag through the stored ticket. The flag raised wrongful denials from 59% to 95% (p < 0.001) but did not significantly change unsafe refunds (Sec. 5.2, App. E.3).

More detection lowering the monitoring load. We now state this directly after Prop. 7, with a cost version in Prop. 8: when errors multiply, inspect as much as capacity allows. The live inspection counts follow this rule once case differences are included (App. E.6): with two or three readers per conclusion, where hard cases multiply, stopping every error needed the fewest inspections, and with one reader, where errors cannot multiply, the certified level needed about half as many.

Reviewer placement. The result you cite comes from M2. Its original three-agent prediction was made after we had already seen the case pool, so we re-tested it on a separate pool and in a new experiment (E6). Both confirmed it for M2: 20.3% vs. 6.8% unsafe decisions on the separate pool, and 17.4% vs. 5.6% in E6 (p < 0.001), with the independent reviewer last vs. first.

Limitations. Measuring transmission took about 1,600 model calls in our setting and must be repeated after changes. In M2's live test one certified upper bound reached 1.11 (see the general response). The copy-all fallback costs little extra when readers err on nearly every case anyway, as M1's do, but much more when they rarely err, as M4's do. E4 is only a first step: the store is in memory and only M1 was tested. The load result itself remains untested (Table 1). Placement was not significant for M1 in any test, so the paper presents placement advice as model-specific.

If any part of Secs. 3–4 is still hard to follow, please point us to it and we will revise it.

Reviewer_tkW1OpenReview ↗

Difficult Understanding of W2 and W3

Thank you for your response, but I found the responses W2 and W3 very difficult to understand. Please use plain words and short sentences to explain. ICLR has enough room for reviewer-author discussions, feel free to expand your responses.

AuthorsOpenReview ↗

W2 and W3 in plain words

Thank you, and sorry that our earlier reply was hard to follow. Here it is again with less jargon.

W2 (q is not stable)

First, what we actually measure. We show a downstream agent another agent's wrong conclusion and record whether its own answer is wrong. This count also includes mistakes the agent would have made without seeing the wrong conclusion. In the revised paper this rate is written qˉ\bar q (or qcq_c for one case). Plain qq now means the chance that an error is contained once it is observed.

We agree that this rate is not a stable property of a model. It depends on the prompt, the task, and the model. With M1 on the same 600 cases and no upstream conclusion, changing only the wording of the instruction moved unsafe decisions from 0% to 18% to 80–84% (App. E.2).

So we no longer treat the rate as a fixed number. We measure it for each setup, much like recalibrating an instrument. We take about 100 of the pipeline's own mistakes, show each one to 8 fresh copies of every downstream agent, and count how often they answer wrongly. From these counts we choose the quarantine level: how often a wrong conclusion must be blocked, stopping further work on that case. In our experiments, errors are identified using known answers. We call the level "certified" because it comes with a statistical bound: under the assumptions stated in the paper, it meets the target with at least 95% confidence.

Showing the same mistake to several copies is the key step. Some cases are simply hard, and on those nearly every copy goes wrong. An average over all cases hides this, so a level set from the average is too low.

This is not free. In our setting it took about 1,600 model calls, and it must be redone for each deployment and after any change to the prompt, the model, or the monitor. The paper says so.

Does it carry over? We repeated it on a second model, M2. We fixed the level first and only then tested it on new cascades it had not seen. The target was at most one wrong downstream decision per original mistake, on average, over five generations (single cascades can exceed one). In the worst of four settings, the level from the simple average let through 3.01 wrong decisions per mistake, three times the target, while the certified level let through 0.64. We then ran the quarantine live on 89 new mistakes. The certified level let through 0.65 to 0.74 per mistake on average across the four settings. In one setting, the upper end of the 95% confidence interval was 1.11, just above 1, so our main pre-planned test failed, narrowly.

If re-measuring is not possible, the safe fallback is to assume every mistake gets copied. This needs no measurement, and it stayed within the target in every held-out test (checked after the fact) and in the live M2 test (planned in advance). The cost is blocking more than needed: for M4, which rarely made an error after a handoff, calibration allowed substantially less quarantine.

W3 (tool-executing environments)

You are right that most of our experiments turn tools off, while real cascades happen through stored data, failed API calls, and changed state. To address this, we added a small sandbox (E4).

The task is refund tickets. An in-memory store (not a real database) holds tickets, orders, customers, and a ledger. An episode lasts at most 8 turns. On each turn an agent picks one action from a fixed list, such as looking up an order or issuing a refund. It writes the action as a short JSON command, and our code really executes it on the store (we do not use the models' built-in tool feature). Actions can change the stored state that later agents see. In 30% of cases, we simulate a timeout on the first order lookup.

The first agent (intake) reviews the ticket and writes an "eligible" or "ineligible" flag and a note into it. We keep only tickets where the intake got this wrong. The second agent (approver) then handles each one twice: once with the intake's flag and note, and once with the field set to "not yet assessed". Nothing else differs. So the mistake reaches the approver through saved state, not through a message.

With M1, on tickets that should be refunded, a wrong "ineligible" flag raised wrongful denials from 59% to 95% (61 tickets, p < 0.001), with and without the timeout. On tickets that should not be refunded, a wrong "eligible" flag did not significantly increase wrong refunds (82% without the flag, 67% with it; 45 tickets; p = 0.09). Our untested guess is that the approver already refunded most of these on its own, so the flag had little room to add mistakes.

This is still limited. The store is in memory, there is no file system or shell, and only M1 was tested. The intake made only 45 such mistakes in 1,500 ineligible tickets, fewer than the 200 we planned. We see E4 as a first step, not a full answer to your concern.

If any part is still unclear, please tell us which one and we will expand on it.

Reviewer_jAM6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers the case where the risky actions of an agent can arrive in a cascaded manner. Authors model this process using a marked Hawkes process, defining a Safety Criticality Number as the expected number of direct follow-on risky events from one event. Authors provide several theoretical analyses. The paper provides an exact intervention threshold, (1 − pq)R_S < 1, and a tradeoff between observation and containment, and extends this to interacting agents through a controlled branching matrix. It also gives a testing rate for estimating criticality when parent–child labels are observed. The experiments are small and synthetic: a sandboxed code-review trace study showing clustering of sensitive file accesses, and decision tasks with tools disabled in which an erroneous upstream conclusion modestly raises downstream unsafe decisions on one model, with large variation across anonymized models. Authors do not draw any connection to the literature on task decomposition attacks.

Critical strengths
  • The paper provide theoretical results by modeling risky actions arriving in cascade as a marked Hawkes process
  • The paper provide theoretical guarantees on exact intervention threshold and a tradeoff between observation and containment
Critical weaknesses
  • The main weakness of this work is that the work is not reproducible
  • The level of empirical evidence provided in this work does not meet the bar for acceptance
  • Authors do not position the work in the relevant literature that considers task decomposition as an attack method.
Questions for authors
  • Can authors provide additional empirical evidence of the claims with de anonymized models?
  • What are the different parameters configurations and decoding approaches used in the results?
  • Do empirical results generalize to domains other than code review
3 comments
AuthorsOpenReview ↗

Response to Reviewer jAM6

Thank you for your review. We made two substantial improvements in response: released analysis code and new preregistered held-out and live experiments.

W1: Reproducibility. The code is at https://anonymous.4open.science/r/agent-cascade-criticality-5E1C. It contains the experiment code, the preregistered protocols, the analysis scripts, and the per-case labels and cascade records behind the new results. All released code runs without access to the models: bash mock_check.sh reruns every released experiment on a mock model and recomputes the analyses in about four minutes, and a single script reprints Table 3. App. E.1 documents every relaunch and a small fix to our API code made during collection; no analyzed record was affected by it.

W2: Level of evidence. We added preregistered experiments with held-out tests on two models (Sec. 5.3, App. E.6). The main finding, clearest for M2, is that a quarantine level set from average error rates is too low, while a level certified from repeated readings of the same case holds up on held-out data. For M2, the average-based level let through about three times our tolerance on held-out data, while the certified level stayed at 0.64 of it. Run live on 89 new M2 errors, the average-based levels again exceeded the tolerance in all four settings, while the certified levels stayed below it on average. A larger preregistered follow-up on the held-out errors of Table 3 put all certified upper bounds at or below 0.85. The fresh-case primary test missed in one setting (upper bound 1.11); the follow-up reused evaluated roots.

W3: Task decomposition attacks. Thank you for pointing us to this literature; Sec. 2 now discusses it (Jones et al.; Chen et al.). In a decomposition attack, an adversary splits a harmful task into harmless-looking pieces, and catching it requires a monitor that tracks intent across steps. Our framework answers the complementary question: once a harmful piece is in the pipeline, how far it spreads and how much quarantine is needed to contain it. A payload that copies itself to every reader is the extreme case in our model, and it is exactly where average-based levels fail.

Q1: Model names. M1 is a small, fast model and M2 a larger model from the same provider; M3 and M4 come from two other providers. The two judges are M2 and M4 (App. E.2). The models behave very differently: on the same 100 errors, downstream agents erred after seeing an erroneous conclusion 0.4% of the time for M4 and up to 91% for M1. This is why the method measures it for each deployment instead of assuming a value.

Q2: Settings and decoding. These are in App. E.1. Tools were disabled and extended reasoning was off. We used temperature 1.0 where the API allowed it and the default otherwise, with no top-p, top-k, or seed. Output limits were 384 tokens for cascade and judge calls, 1,000 for the replication studies, and 600 per turn in the sandbox. We parse the last complete JSON object in each response; responses that cannot be parsed are recorded and never imputed.

Q3: Beyond code review. Only the original trace study used code review. The new experiments cover change approval, screening for a fictional clinical trial, and data-access requests (E3), plus a sandbox in which the agent calls tools against a ticket store (E4).

Limitations. Our approval does not let us release the model names, and the text the models wrote is withheld pending a review of provider terms (Reproducibility Statement). You can rerun every analysis from our labels, but you cannot re-derive a label from its text or re-collect data from the same endpoints. Some older material, mainly the trace study in Sec. 5.1 and several archival tables, is not released; the README lists it. The tasks are synthetic, and the paper describes the experiments as a feasibility study. In the live M2 test one certified upper bound reached 1.11, so our main pre-specified test failed narrowly; the follow-up was designed afterwards. For M1 the difference between the methods is small. The original handoff effect did not replicate on new cases (+2.3 pp, p = 0.22), there was no handoff effect for M1 in the new task domains, and the reviewer-placement result holds only for M2. We have not tested decomposition attacks.

We hope the released code and the new experiments help with both concerns, within what our approval allows. If something you would need is still missing, please let us know and we will add it where we can.

AuthorsOpenReview ↗

Response to Reviewer jAM6

Thank you for your review. Your concern about the level of empirical evidence was fair, and the new experiments respond to it directly: the revision adds preregistered held-out and live tests on two models (Sec. 5.3, App. E.5–E.6). The new experiments also go beyond code review, with three new decision domains and a tool-using sandbox (Sec. 5.2). The released code reruns all released experiments without model access. We still cannot release model names or response text under our approval, and we understand that limits reproducibility. If you are able to look at Sec. 5.3 before the discussion closes, we would be grateful to know whether it addresses this.

Reviewer_jAM6OpenReview ↗

Reproducibility remains challenging

It is hard to assess the validity of results without reproducibility. I am keeping my score as it is.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer G8sn

Thank you for the careful review. You called the framework–evidence gap decisive. The revision addresses it with preregistered held-out and live tests, so we answer Q2 first.

Q2: Held-out value over simpler baselines. Yes on held-out data, and clearly so for M2. We compared a quarantine level set from the average transmission rate (the simple model) with one certified from repeated readings of the same error, which capture differences between cases. The average-based level exceeded the tolerance in every held-out setting for both models, significantly in three of four for M2. On held-out M2 cascades it let through 3.01 erroneous downstream decisions per original error against a tolerance of 1, while the certified level let through 0.64 (Table 3). A fixed quarantine rate with the same summed containment rate also exceeded the tolerance, by up to 3.8 times, significantly in two of four settings.

Run live on 89 new M2 errors under a protocol published before data collection, the average-based levels exceeded the tolerance significantly in all four settings (1.57 to 3.96), and the certified levels were lower in every setting, by 0.92 to 3.23. A follow-up with 48 cascades per error on the 60 held-out M2 errors of Table 3, also published before it ran, put every certified upper bound at 0.77 to 0.85. The fresh-case primary test missed in one setting (upper bound 1.11); the follow-up reused evaluated roots. The workflow-ordering result, a held-out prediction, was confirmed for M2.

W2: Gap between the experiments and the framework. We agree with most of these points and changed the paper. Sec. 5.1 now says it shows clustering, not propagation, and no longer estimates R_S; no result depends on the +4.8 pp effect. The new held-out tests validate certified quarantine levels (Table 1 maps each claim to its test). The quarantine experiments use zero-offspring containment (a stopped error has no children), and the replacement arms are descriptive only. Capped generations are inverse-probability weighted, so cascade sizes are unbiased.

W1: Technical novelty. We agree that the tools are standard and that branching models of agent cascades predate us (Xie et al.), and we now state the contribution more narrowly. The difficulty specific to agent monitoring is that a monitor sees one handoff at a time, and per-step averages hide differences between cases. Showing the same error to several fresh agents, which passive logging cannot do, identifies the quantity that matters (Prop. 13) and gives a certified level (Cor. 16). Thm. 15 certifies a fixed deployed configuration from audits; it must be restarted after any change, and it implies subcriticality only when cases are alike.

Q1: What the archive can establish. Label agreement: two blind judges re-read the responses and agreed with the recovered decisions (κ = 0.992, E2). Whether the effect holds needed new data with finish reasons and an explicit final-decision marker (E1). With the marker, neither arm had any unsafe approvals (0 of 600). Without it (E1b), the rates were 17.6% and 19.9% (+2.3 pp, p = 0.22), so we no longer claim the effect generalizes.

Corrections. We fixed the sign condition in Prop. 12(i) and the lower bound in Thm. 14 (App. D.1, B.6).

Limitations.

  • The missed primary test required every certified upper bound below 1 (point estimate 0.74 there). The follow-up's errors, the only held-out M2 errors not yet run live, had known computed results, and agents erred less often on them, so it does not overturn the miss; we report both.
  • For M1 the contrast is weak: its readers err on almost every such case, with or without the upstream error, so the two methods give nearly the same level. Ordering was not confirmed for M1 (p = 0.098); we present it as model-specific.
  • Assuming every error is copied (the model's own worst-case bound) needs no measurement and stayed within the tolerance in every held-out study and the live M2 test. Measurement lowers its summed containment rate by 3% (M1), 14% (M2), and 56% (M4, which rarely errs after a handoff). Live on M2, it saved inspections only with one reader (0.56 vs. 0.69); with two or three, stopping every error was cheapest, as expected where hard cases multiply (App. E.6).
  • The higher-cost controller (Table 9) gained safety by making more calls (1,002 vs. 744), a trade-off. A per-case predictor overestimated M2's error rate (12.4% vs. 2.5%, E7).
  • R_S, the threshold, and the observation–containment trade-off remain untested (Table 1). Transmission estimates include errors agents would make without the upstream error (80 to 84% for M1, E8), so the experiments validate quarantining a case, not a causal rate. Repeated readings predicted depth only partly, and with three to four generations the boundaries are biased low (Remark 21).

If any gap you raised still seems open, please tell us which.

AuthorsOpenReview ↗

Thank you for the push

Thank you for pushing us to explain the practical point more clearly. Three agents agreeing can offer less reassurance when all three struggle with the same case. In the illustrative refund cascade, average handoff rates suggest a quarantine threshold of about 17%, while accounting for persistent case difficulty raises it to about 58%.

The Discussion now starts with when stopping every detected error is preferable, and your bias suggestion is addressed in Sec. 4.3. For the final version, we would lead with the workflow and the decision it informs, introduce the equations afterward, and move the continuous-time material to the appendix.

The experiments use synthetic cases, and the main containment tests involve two models. The method provides a way to set and test checker requirements under stated assumptions; its effectiveness with deployed checkers remains untested.

Thank you again for taking the time to push us on being clearer. We greatly appreciate your thoughts.

AuthorsOpenReview ↗

Held-out comparisons and practical value

Thank you again for the review and for highlighting the gap between the framework and its demonstrated value.

The revision reports held-out comparisons with a simple transmission model and a fixed-rate policy in Sec. 5.3 and Table 3, with live containment tests in App. E.6. We also report the fresh-case live test that failed its pre-planned criterion.

Before the discussion closes, we would appreciate any further assessment of whether this evidence addresses that gap, or what remains missing.

Apologies for the delayed reply. I appreciate the additional experiments and clarifications. W1 has been largely addressed. However, my concern in W2 remains. The evidence is still insufficient to establish whether measured RSR_S accurately predicts cascades and whether the theoretically derived critical intervention rate holds empirically. I therefore maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.