acceptodds
Under review as a conference paper at ICLR 2027

CONFORMALGUARD: DISTRIBUTION-FREE FALSEBLOCK GUARANTEES FOR MULTI-AGENT LLM SYSTEMS VIA CONFORMAL PREDICTION ON DYNAMIC EXECUTION GRAPHS

Abstract

Multi-agent LLM systems built on AutoGen, MetaGPT, LangGraph and CrewAI execute as dynamic graphs whose topology emerges at inference time. They exhibit compositional unsafe behaviour that no per-call guard can see, and existing runtime defences return no finite-sample guarantee of any kind. We present ConformalGuard, which applies conformal prediction to the multi-agent execution graph. We (i) formalise the execution as a continuous-time heterogeneous graph with four node types and six edge types and compute nonconformity scores over temporal subgraphs with a DyGFormer-HGT encoder; (ii) instantiate split conformal prediction on execution subgraphs, obtaining finite-sample 1−α coverage of the ground-truth safe action - a distribution-free bound on the false-block rate that holds marginally per decision under trace-level exchangeability, for any underlying LLM; and (iii) apply adaptive conformal inference to maintain long-run coverage under distribution shift with an explicit 1/(γT) bound. We are explicit that the guarantee bounds false blocks, not missed violations: violation-block rates are empirical properties of the learned scorer. On AgentChain-26, a new benchmark of 12,400 labelled multi-agent traces across four platforms and 11 violation classes, and on R-Judge, ConformalGuard attains 94.9% empirical coverage (95% CI [94.4, 95.4]) at target 0.95, blocks 96.1% of critical-violation traces versus 18.3–31.2% for per-call guards, 74.6% for GPT-4-as-judge and 84.3% for a rule-based graph-provenance monitor, at 3.4% false-block rate, 1.8 safe continuations per step, and 23 ms per step.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
conformal prediction, distribution-free uncertainty quantification, LLM agents, multi-agent systems, AI safety, guardrails, dynamic graph neural networks, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_kBPGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to address multi-step safety issues in multi-agent LLM systems, where looking at individual actions may miss some risks. It proposes ConformalGuard, which mainly has two parts. The first part builds a graph from the execution history and uses DyGFormer-HGT to get a score for each action. The second part uses conformal prediction to calibrate the threshold for deciding whether to block an action, aiming to reduce false blocks and provide a coverage guarantee for safe actions. The experiments show that the method performs better than the baselines.

Critical strengths
  1. Multi-step safety is indeed a problem in multi-agent LLM systems, since some risks cannot be detected by only looking at a single step. I think this is worth studying.

  2. The experimental results are pretty good, especially compared with the baselines. The ablation study also shows that using the execution history helps.

Critical weaknesses
  1. The positioning of the paper is not very clear to me. The detection performance seems to mainly come from the GNN part, but the paper focuses more on CP.

  2. Most components and theoretical results are based on existing work.

  3. I am not sure the CP coverage guarantee directly translates to the claimed false-block guarantee.

Questions for authors
  1. I am a bit confused about the positioning of this paper. A lot of the detection performance seems to come from the GNN part, but the paper mainly focuses on CP and its theoretical guarantees. It is not very clear to me what the main contribution is supposed to be.

  2. The novelty seems somewhat limited. Most of the methods in the first part already exist, and the CP theory in the second part does not seem particularly new either. If this is intended as a systems paper, I think the authors need to better explain why such a system is needed and why these two parts need to be combined.

  3. I am not sure how the theoretical coverage guarantee translates to the actual false-block rate. CP guarantees coverage of the labeled safe action, but the system decides whether to block the action proposed by the agent. It is not clear to me whether these correspond to the same distribution. Also, the paper acknowledges that steps within a trace are dependent, which may affect the exchangeability assumption.

  4. I wonder why a graph model is needed here. The ablation study shows that using execution history helps, but it doesn't really show why a graph model would be better than a sequence model like Transformer using the same history.

  5. Although the paper seems to focus primarily on CP, the overall performance appears to depend heavily on how well the learned model in the first stage performs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kBPG: calibration is the scorer-agnostic contribution, and the execution graph pays off exactly where you asked us to test it, under concurrent multi-agent tasks.

Thank you for five questions that go to the structure of the paper. Each is now answered by an experiment.

Q1 / W1 and Q5. Detection comes from the model, the guarantee from calibration

This division is the design, and the revision demonstrates its value. A learned score gives no principled threshold, and every false block stalls legitimate work. Conformal calibration turns any score into a rule whose false-block rate is at most a level α chosen by the operator. We show it for five trained scorers and for three LLM judges we did not build: all stay within budget on R-Judge (3.5–4.5% and 3.5–4.4% at α = 0.05). A stronger scorer blocks more, and the layer lets an operator use the strongest one available: calibrated at α = 0.05, claude-opus-5-5 as a judge blocks 95.2% of unsafe records.

Q2 / W2. Novelty, and why the two parts belong together

The contributions are the formulation (runtime guarding as calibrated scoring of proposed actions on the execution graph, with the false-block rate as the quantity the operator fixes), guarantees fitted to what a guard does (Q3), and evidence that the combination works on real records, on existing judges and on multi-agent sessions (Q4). Neither part suffices alone: an uncalibrated open-weight judge flags 85.6% of safe records, and a calibrated guard without an informative scorer detects nothing beyond chance.

Q3 / W3. Coverage and the false-block rate are now the same quantity

  • Same object. The label is binary and concerns the action the agent proposed, and the guard blocks that action if and only if its score exceeds the threshold. Coverage is exactly one minus the false-block rate on safe steps.
  • Same distribution. Calibration and test pairs are drawn the same way from held-out traces.
  • Dependence. Each calibration trace contributes one random safe step, so Proposition 1 holds exactly. It is now verified on real step labels (9.4–9.8% at α = 0.10 on R-Judge's injection records; 9.6–10.1% on multi-agent sessions).
  • Closed loop. New Corollary 1 bounds by α the probability that a benign workflow is interrupted at all, with the guard in the loop.

Q4. Graph versus a sequence model on the same history

We ran the control you asked for in both settings: a causal Transformer over the same nodes, features, loss and calibration, against relation-aware attention over the typed graph.

Single agent (R-Judge). History is what matters: the sequence model blocks 4.6 points more than a single-step scorer, and the graph does not improve on it (82.3% against 77.8% at α = 0.10).

Several agents, concurrent tasks. This is the case the graph was designed for, and here it wins clearly. On 669 sessions in which LLM agents work on one to three interleaved tasks (217 violating steps, labelled by code), under a per-session budget:

  • α = 0.05: graph 93.4%, sequence 53.1%, single step 8.7%; the graph is ahead in 50 of 50 splits.
  • One task per session (α = 0.10): sequence 95.4%, graph 98.6%.
  • Concurrent tasks: sequence 62.5%, graph 93.9%.
  • Trained on the claude-opus-5-5 sessions and tested on the gpt-6-astra sessions: graph 86.2%, sequence 54.6%.

What carries the effect. The ablation is unambiguous: without the provenance (cites) edges the graph scorer falls to 6.9%. Provenance made explicit as an edge is what lets a guard tie an outgoing message to the records it carries when tasks interleave. It also makes the policy decidable on the graph: a one-line rule on these edges reproduces 216 of the 217 violation labels with no false positive, so an operator who knows the policy can enforce it directly, and where the policy is not known in closed form a scorer trained on labels alone recovers it, as ours does.

Continuing work

The sessions are simulated and exercise one family of policy. Policies without a closed form, and the tool ecosystem of AgentDojo, are the next tests of how much learning adds on top of the graph, and the released simulator makes them straightforward to run. Thank you.

Thanks for the detailed response! I think the answers to Q3 and Q4 are mostly fine, and the new results are meaningful improvements. However, I still have concerns about Q1, Q2, and Q5, especially regarding the paper's positioning and novelty. If this were a 6- or 10-point scale, I'd probably bump my score up by one, but on the current 4-point scale, I feel I'm somewhere around 2.5.

AuthorsOpenReview ↗

Reply to Reviewer kBPG: thank you for the reassessment; below is the paper's position stated in one sentence, what is new in it, and the changes that make it explicit from the first page.

Thank you for reading the new results and for telling us plainly where you stand. Your remaining concern is fair: the submission did not say clearly enough what kind of paper this is. We answer Q1, Q2 and Q5 together.

Position, in one sentence

Every guard for an agent system needs a threshold, today that threshold is set by hand, and this paper makes the rate at which legitimate work is interrupted a quantity that the operator fixes in advance and that is guaranteed for any detector.

Q1 / Q5. Yes, detection depends on the learned model, and that is why the layer is the contribution

Detectors change quickly; a certificate tied to one of them would not last. Ours is independent of the detector, and the experiments show what that buys:

  • It holds for detectors we did not build. False blocks at α = 0.05 are 3.5–4.5% for five trained scorers and 3.5–4.4% for three LLM judges.
  • It makes an unusable detector usable. At its own decision rule the open-weight judge flags 85.6% of safe records; calibrated, it flags 4.4% and still blocks 41.3% of unsafe ones.
  • It lets the operator use the best detector for the setting. That is a calibrated LLM judge on single-agent records (95.2% blocked at α = 0.05) and a graph scorer with provenance edges under concurrent tasks (93.4% against 53.1% for a sequence model).
  • It survives a change of domain cheaply. A few dozen in-domain safe records restore the budget without retraining (6.0–6.8%, 5.0–6.5%, 8.2–11.7% on three held-out categories at α = 0.10).

Q2. What is new

Split conformal prediction and adaptive conformal inference are known, and the paper says so. The novelty is in fitting the guarantee to what a guard does and in showing that it holds on real agent data:

  1. The certified object is the guard's decision. The calibrated pair is the proposed action in its execution state, so coverage is exactly one minus the false-block rate, and one unit per trace makes the bound exact. The submission's approximation term is gone.
  2. A closed-loop guarantee (Corollary 1). Standard conformal statements are open loop, while a guard changes the trace it scores. Calibrating on the maximum score of benign traces gives a bound that survives intervention, because guarded and unguarded runs coincide until the first block. To our knowledge this argument has not been made for runtime guards.
  3. Delayed labels (Proposition 2). Labels arrive when a trace completes; the bound is stated and proved with the delay explicit.
  4. Findings that change practice. The unit of calibration matters on real agent data (15.6–16.7% false blocks with pooled records against 6.1–7.6% with one per prompt group at α = 0.10). Online adaptation can keep its budget by ceasing to block, and the step size decides this (the guard is open on 47.8% to 62.6% of post-shift steps at γ = 0.05 and on 7.7% to 16.1% at γ = 0.005). Under concurrent tasks, provenance edges are what a guard needs (a sequence model blocks 62.5% of violating steps and the graph scorer 93.9% at α = 0.10).

This is a contribution of formulation and evidence, not of new conformal theory, and we think that is the right contribution for the problem: guards are being deployed now, and to our knowledge none of the guardrails we cite gives its operator a guaranteed interruption rate.

Why the two parts belong in one paper

A calibrated guard without an informative score detects nothing beyond chance, and a score without calibration has no defensible threshold. The execution trace also defines the units on which the guarantee rests (step, trace, prompt group, session), so the representation and the certificate cannot be designed apart.

Changes that make the position explicit

  • Done in the revision: the abstract leads with the guarantee and closes on the calibration layer, and Figure 1 is redrawn with the scorer as a pluggable component.
  • To be added: a sentence in the contributions stating that no new conformal theory and no new architecture is claimed.
  • To be added: a short table in Related Work listing, for each family of guard and each conformal method for LLMs, what it certifies. To our knowledge the row for runtime agent guards with a false-block guarantee is otherwise empty.

If a particular comparison or statement would settle the remaining concern, we will gladly add it during this discussion. Thank you.

Reviewer_Y6mxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ConformalGuard treats a multi-agent run as a continuous-time heterogeneous graph and scores a sliding temporal subgraph with a dynamic heterogeneous encoder. Split conformal prediction on a learned nonconformity score yields a finite-sample guarantee that the labeled safe action lies in the prediction set, and adaptive conformal inference tracks long-run miscoverage under shift. The authors are explicit that this certificate bounds false blocks of the labeled safe action; the violation-block rate is an empirical property of the scorer. On a new multi-platform trace benchmark and a conversion of R-Judge, the method meets its coverage target and blocks critical-violation traces more often than per-call guards, a full-history LLM judge, and a rule-based provenance monitor on the same graph, at a low false-block rate and interactive latency.

Critical strengths

1.The certified claim is stated precisely. Propositions 1 and 2 are presented as instantiations of split conformal prediction and adaptive conformal inference. The unit of exchangeability, the total-variation caveat for within-trace dependence, and the fact that the bound covers false blocks and not missed violations are written out in the abstract, Remark 2, and Table 1. This is consequential because the central risk in "guarantee" papers is overclaiming. Here a reader can check exactly what the certificate does and does not say.

2.The ablations match the narrative, and the controls are well chosen. Removing subgraph encoding cuts the block rate from 96.1% to 81.4% while coverage stays near target, and removing online adaptation cuts coverage under shift from 94.9% to 91.2%. This separates "composition is detected by the encoder" from "calibration is provided by the conformal layer". The rule-based provenance monitor on the same graph (84.3%) is a fairer control than per-call guards. The 23 ms per-step cost, against 1,900 ms for the full-history LLM judge, makes the approach plausible for inline use.

Critical weaknesses

The quantity a deployer cares about is not certified, and the label is a single action. A scorer with no discriminative power satisfies Proposition 1 by returning the full action set, so the unsafe-action-allowed rate rests entirely on the learned scorer. The delayed label is "the action that extends the trace without matching a predicate". When several actions are safe, this collapses them to one, and coverage is defined against that label. The online guarantee in Proposition 2 also needs this label at trace completion. That makes the practical value of the certificate narrower than the abstract suggests.

Questions for authors

On the released audit subset, what is the block rate and false-block rate when labels come only from the independent annotators, and when the nine decidable predicates are recomputed from the released matcher with no author adjudication?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Y6mx: The certificate now covers whole workflows in closed loop, holds on third-party and code-assigned labels, and is restored under domain shift by a few dozen in-domain records.

Thank you for the careful reading. Your weakness and your question about label independence shaped the revision more than any other comment, and each part of it now has a direct answer.

W (a). The label is no longer a single action

The label is binary: violating if the proposed action is the first to complete a predicate match on the trace, safe otherwise. Calibration pairs are the graph state and the proposed action at safe steps, one per trace. Coverage is then exactly one minus the step-level false-block rate, and nothing is collapsed when several actions are safe.

W (b). Closer to what a deployer cares about

  • Corollary 1, per workflow and in closed loop. Calibrating on the maximum score of each benign trace bounds by α the probability that a benign workflow is interrupted at all, with the guard in the loop.
  • Independence from the attacker. With calibration on benign traces the bound does not involve the attack distribution: an adversary who controls only attacked workflows cannot make the guard interrupt benign work more often.
  • A bound on auto-allowed violations for known attack families (App. C). A second threshold gives three verdicts: block, allow, escalate. On R-Judge at α = β = 0.10 it blocks 8.0% of safe records, auto-allows 9.2% of unsafe ones and escalates 9.0% of all records.

W (c). Proposition 2 with labels at trace completion

The revised proposition handles the delay explicitly: with labels at most D steps late the bound is (max{α_1, 1 − α_1} + γ(D + 1)) / (γT) + D/T. A simulation shows how to choose the step size: with γ = 0.005 the adaptive guard keeps its budget through a shift and stays active on most steps, where the larger γ = 0.05 of the submission kept the budget partly by ceasing to block. The revision recommends the smaller value and gives the operator a signal for when to recalibrate.

Q. Rates with labels independent of the authors

The audit subset belonged to the AgentChain-26 experiments, which are withdrawn (general response), so we answer your question with the two kinds of independent label you asked about.

Labels by other annotators. On R-Judge (571 records labelled by its authors) the false-block rate is 3.5–4.5% at α = 0.05, 7.3–8.4% at 0.10 and 17.5–18.3% at 0.20 for every trained scorer, and 3.5–4.4%, 8.7–9.4% and 14.9–19.6% for three LLM judges. Calibrated at α = 0.05 the strongest judge blocks 95.2% of unsafe records; the scorers we trained block 63–69%.

Labels recomputed by a matcher, with no author adjudication. Two experiments use labels assigned by code alone. On R-Judge's 414 injection records, labelled from InjecAgent's attacker-tool list, the step-level false-block rate is 4.5–4.8%, 9.4–9.8% and 18.0–19.9%. On 669 multi-agent sessions, labelled by a fixed matcher on tokens planted in the workspace, it is 4.5–5.2%, 9.6–10.1% and 19.8–20.9% per step and 4.4–4.9%, 9.6–10.2% and 20.0–21.3% per session, and at a 5% session budget the graph scorer blocks 93.4% of violating steps.

Where the certificate stops, and how it is restored

With a held-out application category the false-block rate at α = 0.10 ranged from 1% to 62%: the certificate needs calibration data from the deployment domain. Recalibrating on half of the new category's records, with the scorer unchanged, restores it: 6.0–6.8% (Application, 48 safe records), 5.0–6.5% (Finance, 44) and 8.2–11.7% (Program, 30). A few dozen safe in-domain records are enough, which makes the certificate practical to maintain; below 1/α − 1 records the method abstains and says so.

Continuing work

Predicates that need content judgement, and a deployment study with delayed analyst labels, are the next steps towards the audit you describe; the labelling code and the simulator are released so that such an audit can be repeated independently. Thanks.

Reviewer_CeiDOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ConformalGuard monitors multi-agent LLM execution using a heterogeneous temporal graph. A DyGFormer–HGT encoder scores candidate actions, and conformal prediction determines which actions are allowed. The working level is adapted using safe-action labels obtained after trace completion. The paper introduces AgentChain-26, a dataset of 12,400 traces, and evaluates the system on this dataset and R-Judge.

Critical strengths
  1. The graph representation captures dependencies across messages, memory accesses and tool calls that per-call guards may miss.
  2. The provenance-monitor baseline, full-history GPT-4 judge and ablation without subgraph encoding help assess the value of cross-step information.
Critical weaknesses

1.I cannot reconcile Figure 2 with Proposition 2. With α₁ = α = γ = 0.05, the bound is 20/T. At 16,000 steps, the cumulative error should be 4.875%–5.125%, but the red curve is about 8%. The caption explicitly defines a cumulative average. It is unclear whether the discrepancy comes from the plotted statistic or the update procedure. As reported, this experiment does not substantiate the claimed guarantee.

Questions for authors
  1. What exactly is plotted in Figure 2? Please provide the cumulative error and working-level sequences, the feedback schedule, and any reset or clipping rules. How do these account for the roughly 8% error near 16,000 steps?

  2. Was online adaptation enabled in Table 2’s alpha sweep? What was held fixed across runs, and how did the working levels and thresholds change? Please explain the reported changes in Block, FBR and mean set size under this protocol.

  3. Does each calibration trace contribute one pair or several? How is the safe-action label chosen when several actions are safe, and how is it obtained after a proposed action is blocked? How does this sampling procedure satisfy the assumptions of Proposition 1?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CeiD (1/2): your consistency check was right, and it led to a verified simulation, a proof for delayed labels and a practical rule for choosing the step size.

Thank you for the most consequential review we received. Because of it, the online part of the paper now rests on a statement that is proved, tested at every horizon and accompanied by guidance a deployer can use.

W1 / Q1. Figure 2 and Proposition 2

Your calculation is correct: with α_1 = α = γ = 0.05 the bound is 20/T at every T, so no run of Algorithm 1 can show 8% at T ≈ 16,000. The submitted curves were drawn from a few manually entered points to illustrate the expected behaviour; they were not logged output, so there are no sequences behind that figure to provide. The figure and the sentence "re-converges within ~10,000 steps" are withdrawn and replaced as follows.

What replaces it: a simulation that follows Algorithm 1 exactly

It uses synthetic scores, and is labelled as such, so it isolates the update rule. Setup: 2,000 calibration scores from N(0, 1); a stream of 50,000 labelled-safe steps, N(0, 1) until T = 15,000 and N(0.45, 1.25²) after; α = α_1 = 0.05; 20 seeds. scripts/sim_checks.py regenerates every number and the figure.

Feedback schedule: delayed labels are now covered by the proposition

Labels arrive when a trace completes. Gibbs and Candès' Prop. 4.1 assumes immediate feedback, so the revision states and proves the delayed case. With labels at most D steps late:

|mean error up to T − α| ≤ (max{α_1, 1 − α_1} + γ(D + 1)) / (γT) + D/T.

At D = 0 this is the original bound. The simulation uses D = 0, and delays uniform on {0, ..., 50}.

Reset or clipping: none

α_t is never projected. The threshold is +∞ when α_t < 1/(n+1) and −∞ when α_t ≥ 1. Clipping α_t breaks the bound, and the revision says so. Unit tests assert the inequality at every T on shifted, adversarial and delayed-feedback sequences.

Results (means over 20 seeds)

  • Static threshold: cumulative error 5.9% at T = 16,000 and 13.5% at T = 50,000.
  • ACI, γ = 0.05, D = 0: 5.01% at T = 16,000 (band ± 0.12 pp) and 5.00% at T = 50,000.
  • ACI, γ = 0.05, D up to 50: 5.01% (band ± 0.75 pp) and 5.00%.
  • ACI, γ = 0.005, D up to 50: 5.06% (band ± 1.82 pp) and 5.02%.
  • The inequality held at every T in every run.

The adaptive guard holds its budget through the shift with no visible transient in the cumulative curve; the worst 1,000-step window after the shift has an error of 5.3% to 6.4%.

A practical finding your question led to: how to choose γ

ACI holds the budget by lowering α_t, and when α_t < 1/(n+1) the guard blocks nothing. With γ = 0.05 the guard is in that state on 8.4% of steps before any shift and on 47.8% to 62.6% after it. With γ = 0.005 it is never open before the shift and open on 7.7% to 16.1% of steps after it. The revision therefore recommends γ = 0.005 and gives operators a concrete signal: a sustained open fraction means it is time to recalibrate. This makes Proposition 2 usable in deployment.

Next step

Our multi-agent simulator (Section 6.2) can generate streams of any length with a controlled change of agent model, so the same experiment on agent traces is a direct extension, and we will add it. Thank you.

AuthorsOpenReview ↗

Response to Reviewer CeiD (2/2): the α sweep is replaced by a measured one, calibration is exact with one step per trace, and Proposition 1 now holds on real step labels.

Q2. The α sweep

You are right that no protocol makes the submitted rows correct: under the paper's decision rule a larger α lowers the threshold, so Block and FBR can only rise. The sweep belonged to the AgentChain-26 experiments, which are withdrawn (general response). It is replaced by a measured sweep.

The measured sweep behaves as the rule dictates

R-Judge, 100 random splits, static threshold, online adaptation off; the scorer, calibration scores, test records and seeds are fixed and only α varies. For the typed-graph scorer at α = 0.05, 0.10 and 0.20 the false-block rate is 3.8%, 8.0% and 17.9%, and the block rate 63.1%, 77.8% and 96.6%. Both rise with α. The same holds for every scorer and for the three calibrated LLM judges.

Q3a. One pair per trace makes Proposition 1 exact

Each calibration trace now contributes one pair, a step drawn uniformly at random among its safe steps. The scores are exchangeable whenever the traces are, and the proposition holds exactly, with no total-variation term.

The unit of calibration matters in practice, and we can show it on real records. R-Judge reuses user prompts (151 prompts in 571 records). With prompt-grouped splits at α = 0.10, calibrating on all safe records gives 15.6–16.7% false blocks; one random record per prompt group gives 6.1–7.6%. The same repair works for the LLM judges (6.5–8.7%).

Q3b. Several safe actions: nothing needs to be selected

The label is binary and concerns the action the agent proposed: violating if that action is the first whose execution completes a match of a violation predicate on the trace, safe otherwise. Other safe actions at that step are never labelled or ranked. Proposition 1 reads: for a random safe step of a fresh trace, the proposed action is blocked with probability at most α.

Q3c. Labels after a block

In open-loop replay the guard's verdict does not alter the logged record, so labels come from the complete record. For deployment, App. A specifies how a blocked action is labelled without executing it: by matching the predicates on the graph extended by the proposed action. The multi-agent experiment below labels proposed actions in this way, by code.

Q3d. Assumptions, and the closed-loop corollary

(i) Calibration and test traces are exchangeable. (ii) The scorer is trained on disjoint traces. (iii) One step per trace, drawn by the same rule. (iv) Open loop. Corollary 1 removes (iv) by making the whole trace the unit.

Proposition 1 is now tested on real step labels, twice

  • R-Judge injection records. The violating step of the 414 injection records is found by code from InjecAgent's attacker-tool list (200 violating, 628 safe steps). With one calibration step per trace, the false-block rate of a random safe step is 4.5–4.8%, 9.4–9.8% and 18.0–19.9% at α = 0.05, 0.10 and 0.20 across three scorers and 100 splits.
  • Multi-agent sessions. On 669 sessions with 10,369 agent steps labelled by code, the same rule gives 4.5–5.2%, 9.6–10.1% and 19.8–20.9% across five scorers, and Corollary 1 gives 4.4–4.9%, 9.6–10.2% and 20.0–21.3% per session.

For completeness: pooling all calibration steps did not inflate the false-block rate on the short injection records (7.7–8.5% at α = 0.10), whereas pooling records that share a prompt did, as shown above. The exact guarantee is the one for one unit per trace, and that is the rule the paper recommends. Thank you once again.

You state that Figure 2 was drawn from manually entered points to illustrate expected behaviour, was not logged output, and has no underlying sequences to provide. However, Section 6.2 (p. 7) describes results on a 50,000-step held-out stream, and the caption (p. 8) claims recovery within approximately 10,000 steps. Neither passage identifies the figure as illustrative. Withdrawing the figure does not resolve the integrity concern arising from presenting illustrative values as experimental results.

Reviewer_2KxaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ConformalGuard, a runtime guard for multi-agent LLM systems based on dynamic execution graphs. It combines a temporal heterogeneous graph encoder with conformal prediction to control false blocking under exchangeability, with online adaptation for distribution shift. Experiments on AgentChain-26 and R-Judge evaluate violation blocking, false-block rates, and per-step latency.

Critical strengths
  1. The method combines graph-based risk detection with an explicit statistical bound on false blocking under stated assumptions.
  2. Reported violation-block rates reach 96.1% on AgentChain-26 and 93.4% on R-Judge.
Critical weaknesses
  1. The training objective and construction of ground-truth safe-action labels are insufficiently specified. These omissions prevent a clear assessment of how the model is trained and how its predictions support conformal calibration.
  2. The trace-generating LLMs and baseline versions are unclear, and the anonymous repository was inaccessible during review. These gaps limit reproducibility and make robustness across underlying models difficult to assess.
  3. The authors discuss AGrail, ShieldAgent, and Progent. However, these directly relevant methods are not experimentally compared with ConformalGuard under matched settings. This leaves the proposed method’s advantage over existing agent-level defenses insufficiently established.
  4. Closed-loop evaluation is insufficient, leaving the impact of blocking on task success and recovery unclear. High violation-blocking rates alone cannot establish practical value if interventions also prevent legitimate task completion.
Questions for authors
  1. What training loss is used, and how are ground-truth safe-action labels constructed?
  2. Which LLMs and baseline versions were used, and can you provide an accessible code and data repository?
  3. How does ConformalGuard compare with AGrail, ShieldAgent, and Progent under the same evaluation protocol?
  4. After a blocked action is routed to the analyst queue, how does execution resume, and how is task success affected?
5 comments
AuthorsOpenReview ↗

the method is now fully specified and reproducible, three existing LLM guards are compared under one protocol, and a new corollary bounds how often benign work is interrupted.

Thank you for four precise questions. Each of them led to a concrete addition, described below.

Q1 / W1. Training loss and labels: specified end to end

  • Labels. R-Judge records carry labels assigned by its authors. Where step labels are needed they are assigned by code (R-Judge's injection records, and our multi-agent sessions). The label is binary and concerns the action the agent proposed; no single "safe action" is picked when several are safe.
  • Scorer. Each proposed action is a node in the execution graph, with frozen sentence embeddings as node features. Relation-aware attention produces node representations, with edges pointing forward in time so that a score never depends on anything after the action it judges. A two-layer MLP maps the action node to a risk score.
  • Loss. Binary cross-entropy: per step where steps are labelled, and between the record label and the maximum step score where only records are.
  • Link to calibration. The threshold is the ceil((n+1)(1 − α))-th smallest score among the n safe calibration units, and the scorer is trained on disjoint data. The guarantees hold for any score.

Q2 / W2. Models, versions and code: documented and attached

We apologise that the link in the submission was not live. The AgentChain-26 experiments could not be documented and are withdrawn (general response). Everything in the revision is specified:

  • Data: R-Judge, commit 83ce301da3; InjecAgent's attacker-tool list; 669 generated multi-agent sessions, included in the supplementary material.
  • Text encoders, frozen: all-MiniLM-L6-v2 and bge-base-en-v1.5.
  • LLMs: Qwen2.5-7B-Instruct, gpt-6-astra, claude-opus-5-5 (with the provider's fallback claude-opus-4-8, reported per request).
  • Software: PyTorch 2.11.0+cu130, NumPy 2.1.3, Python 3.13.16; two T4 GPUs.

The supplementary material holds the code, the judges' raw scores, per-split results and unit tests for each guarantee.

Q3 / W3. Comparison with other guards: three LLM guards under one protocol

We now compare existing guards on one axis: unsafe records blocked at a fixed, guaranteed false-block budget. Three LLM judges score each proposed action from its history only; none is trained by us; the same conformal rule sets each threshold (R-Judge, 1,000 random halves, α = 0.05 / 0.10):

  • Qwen2.5-7B-Instruct: false blocks 4.4% / 9.4%, blocked 41.3% / 51.3%
  • gpt-6-astra: false blocks 3.5% / 9.0%, blocked 88.3% / 91.5%
  • claude-opus-5-5: false blocks 4.1% / 8.7%, blocked 95.2% / 97.4%
  • Scorers trained on R-Judge: blocked 63–69% / 77–82%

Every guard stays within its budget, and calibration is what makes them comparable: at its own decision rule the open-weight judge flags 85.6% of safe records. Any guard that exposes a graded score can be calibrated by the same layer without modification. We have not run AGrail, ShieldAgent or Progent yet and will add them to this table, using a graded output where they provide one; the protocol and code are in place.

Q4 / W4. After a block: resumption, and a guarantee on interruptions

  • Resumption. A block does not end the workflow. The orchestrator returns the verdict to the acting agent, which may re-propose up to twice; otherwise the step enters the analyst queue and only that branch pauses (App. I).
  • What is guaranteed. New Corollary 1 calibrates on the maximum score of each benign trace: the probability that a benign workflow is interrupted at all is at most α, with the guard in the loop. It holds on R-Judge for every scorer and on our multi-agent sessions (4.4–4.9% and 9.6–10.2% of violation-free sessions interrupted at α = 0.05 and 0.10).
  • Task success. Corollary 1 bounds how often legitimate work is touched. Measuring how well a workflow recovers after a block is the natural next experiment: our simulator runs the agents live, so placing the guard in the loop is a direct extension, and we will report it. Thanks.

Thank you for the detailed response and the additional experiments. The revision addresses my first two concerns about training and labeling, and about reproducibility. I appreciate the planned comparisons with AGrail, ShieldAgent, and Progent. I have three remaining questions.

  1. On R-Judge, the stronger LLM judges block more violations than the learned scorers at the same false-block budget. What advantage does the graph scorer offer over a calibrated LLM judge? A comparison on the same concurrent sessions, reporting violation blocking, latency, and inference cost at a matched false-block budget, would help clarify the trade-off.
  2. Why did you choose Qwen2.5-7B-Instruct as the open-weight judge? How would a more recent open-weight model of comparable size perform under the same calibration protocol?
  3. Could you report end-to-end task success rates with and without the guard, as well as the fraction of tasks that still succeed after an action is blocked?
AuthorsOpenReview ↗

Results for your three questions: on concurrent multi-agent sessions calibrated guards block 93.4% to 99.0% of violating steps within a 5% per-session budget, the graph scorer does so 22 times faster than an LLM judge and with no per-decision cost, and in closed loop the guard cuts violations from 17.3% to 1.3% while 93.1% of safe tasks still complete.

Thank you for three questions that each produced a result worth having. All files are in the updated supplementary material.

1. Graph scorer and calibrated LLM judges on the same concurrent sessions, at a matched per-session budget

Judges score each proposed action of 300 sessions (4,847 decisions, 101 violating). They are given the policy in words and the same monitor metadata as the trained scorers, which learn the policy from labelled sessions instead. Violation-free sessions interrupted / violating steps blocked, at α = 0.05 and 0.10:

  • Graph scorer: 4.9% / 93.4% and 10.2% / 95.0%
  • claude-opus-5-5 judge: 3.4% / 99.0% and 6.5% / 99.0%
  • Sequence scorer: 4.9% / 53.1% and 9.6% / 70.4%

Every guard keeps its budget, and two of them block more than 93% of violations. Operating figures per decision:

  • Latency. Graph scorer 70 ms at the median (152 ms at the 90th percentile) on a CPU, text encoding included; judge 1.52 s (1.98 s).
  • Cost. Graph scorer: no per-decision fee. Judge: about 2,918 input tokens per decision, 11.86 USD per thousand decisions at list price, and a session has about 16 decisions.

This is the trade-off you asked for, now measured. The graph scorer gives 93.4% detection at 22 times the speed, at no marginal cost, with no external service and with the policy learned from examples. A strong judge that is told the policy adds about 6 points of detection for those who can afford the latency and the fee. The calibration layer makes the two directly comparable and lets an operator choose.

Calibration is also a diagnostic. A deepseek-v3.2 judge gives its top rating to a safe step in 53% of violation-free sessions; the layer detects that no threshold fits the per-session budget and blocks nothing, where an uncalibrated deployment of that judge would interrupt more than half of all benign sessions. Local 7-8B judges are not practical at run time on these logs (2.0 to 2.5 s per decision at the median on two T4 GPUs, and the runs did not complete), which is itself an argument for a small scorer.

2. More recent open-weight judges (R-Judge, same protocol; false blocks / unsafe records blocked at α = 0.05 and 0.10)

  • Qwen2.5-7B-Instruct: 4.4% / 41.3% and 9.4% / 51.3%
  • Qwen3-8B (more recent, similar size): 4.4% / 23.1% and 9.5% / 38.5%
  • DeepSeek-V3.2 (open weights, MIT licence; much larger; run through an API): 3.6% / 72.8% and 8.2% / 83.5%

The guarantee holds for every judge, which is the property the paper claims. An open-weight judge reaches 83.5% at α = 0.10, so strong calibrated guards do not depend on proprietary models. Among models of similar size, recency alone does not help. Qwen2.5-7B was chosen before any result was seen, because its weights need no access request and it fits one 16 GB GPU; no selection among models took place.

3. End-to-end task success with and without the guard

172 new workspaces were run twice with live claude-opus-5-5 agents: without the guard, and with the graph scorer (α = 0.10) scoring every proposed action before it takes effect. A blocked action is returned to the agent, which may propose another up to twice. All outcomes are decided by code.

  • Violations. Carried out in 17.3% (54 of 313) tasks without the guard and 1.3% (4 of 313) with it, a reduction by a factor of 14; 92.2% (47 of 51) violating proposals were blocked before execution.
  • Task success. Of the tasks that are safe to complete, 99.6% (259 of 260) succeed without the guard and 93.1% (242 of 260) with it.
  • False interruptions. 12.0% (15 of 125) sessions without a violating proposal were interrupted, consistent with the closed-loop bound of Corollary 1 at α = 0.10 (one standard error is 2.7 points).
  • After a block. In the 17 safe tasks where the guard blocked an action, the agents accepted the block and stopped, so none completed. A false block therefore costs one task, which is exactly why the paper certifies the false-interruption rate; the analyst override of App. I is the recovery path, and measuring it is our next experiment.

Tasks whose faithful completion is itself a violation are counted under violations. The two arms sample the agents anew.

These results are in Section 6.2 and Appendices E and F of the revision. Thank you.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response: the revision gives any guard for agent systems a guaranteed interruption budget, now verified on real records, on off-the-shelf LLM judges and on multi-agent sessions.

We thank the four reviewers. Their questions made this a stronger paper, and every result in the revision was measured for it and can be regenerated from the supplementary code.

1. What the revised paper establishes

  • A certificate that holds for any scorer. On R-Judge (571 human-labelled records) the false-block rate is 3.5–4.5%, 7.3–8.4% and 17.5–18.3% at α = 0.05, 0.10 and 0.20 for five trained scorers, and 3.5–4.4%, 8.7–9.4% and 14.9–19.6% for three LLM judges we neither built nor trained.
  • Calibration turns existing judges into strong, usable guards. Calibrated at α = 0.05, gpt-6-astra blocks 88.3% and claude-opus-5-5 95.2% of unsafe records. An open-weight judge that flags 85.6% of safe records at its own decision rule flags 4.4% once calibrated.
  • The step-level guarantee holds on real step labels. With labels derived by code, Proposition 1 holds on R-Judge's 414 injection records (4.5–4.8%, 9.4–9.8%, 18.0–19.9%).
  • The execution graph pays off in the multi-agent setting. On 669 sessions played by two commercial LLMs, at a 5% per-session budget, the graph scorer with provenance edges blocks 93.4% of violating steps and a sequence model 53.1%; the gap comes from sessions with concurrent tasks.
  • The failure modes are understood and repairable. Prompt reuse raises false blocks to 15.6–16.7% at α = 0.10, and one calibration record per prompt group restores 6.1–7.6%. Under domain shift, recalibrating on a few dozen in-domain safe records restores the budget (6.0–6.8%, 5.0–6.5%, 8.2–11.7% on three held-out categories).

2. Theory matched to what a guard does

The label is binary and concerns the proposed action. One safe step per trace makes Proposition 1 exact. A new Corollary 1 bounds the probability that a benign workflow is interrupted at all and holds in closed loop. Proposition 2 is proved for labels that arrive late, and a simulation shows how to choose its step size.

3. How the reviews shaped the revision

Reviewer CeiD showed that Figure 2 and the α sweep of the submission were inconsistent with the paper's own results. On re-examination we could not recover the data and run logs behind the AgentChain-26 experiments, so we withdrew them in full and replaced them with the experiments above. We consider the paper better for it: nothing in it now rests on a result a reader cannot reproduce.

4. Why this matters in practice

The calibration layer needs only scores and a small set of safe executions. An operator can add it to a guard that is already deployed and obtain an interruption budget stated in advance. A researcher obtains a common, guaranteed axis on which guards can be compared: block rate at a fixed false-block budget.

5. Work we are continuing

The same layer calibrates any guard that exposes a graded score, and we will add the learned guardrails AGrail, ShieldAgent, Progent and GuardAgent to the comparison. Our multi-agent simulator runs the agents live, so a closed-loop study of task success with the guard in the loop is a direct extension, as are policies without a closed form and tool ecosystems such as AgentDojo. Section 7 sets out this programme.

6. Code and data

Anonymised code, the generated sessions, the judges' raw scores, per-split results and unit tests for each guarantee are attached as supplementary material. Two further comments below give the full tables of the new experiments.

AuthorsOpenReview ↗

Additional results 1/2: calibrated LLM judges stay within budget and detect strongly, Proposition 1 holds on real step labels, and a few dozen in-domain records repair domain shift.

This comment gives the tables for three of the experiments summarised in our general response. All numbers are in the supplementary material (results/kaggle_extra/), with per-split files. The multi-agent experiment is in the next comment.

1. Existing guards under the same protocol (2Kxa Q3, kBPG Q1)

Three LLM judges rate the risk of each proposed action from its history only. None is trained by us. The conformal rule then sets each judge's threshold on half of R-Judge and is tested on the other half (1,000 random halves). At α = 0.05 / 0.10:

  • Qwen2.5-7B-Instruct (open weights): false blocks 4.4% / 9.4%, unsafe records blocked 41.3% / 51.3%
  • gpt-6-astra: false blocks 3.5% / 9.0%, unsafe records blocked 88.3% / 91.5%
  • claude-opus-5-5: false blocks 4.1% / 8.7%, unsafe records blocked 95.2% / 97.4%
  • Our trained scorers, for reference: unsafe records blocked 63–69% / 77–82%

What this shows:

  • The guarantee transfers to guards we did not build. Every judge stays within the false-block budget.
  • Calibration is what makes a weak judge usable. At its own decision rule the open-weight judge flags 85.6% of safe records. Calibrated, it flags 4.4% and still blocks 41.3% of unsafe ones.
  • The layer lets an operator use the strongest scorer available. On this benchmark that is a strong LLM judge, and calibration gives it a budget it did not have. This is the scorer-agnostic contribution of the paper at work.
  • Prompt reuse. With prompt-grouped halves and one calibration record per group, false blocks at α = 0.10 are 6.5–8.7%.

Two notes for the reader. R-Judge is public, so these models may have seen its records; that cannot affect the false-block guarantee but may flatter the block rates. For the Claude judge, 24% of requests were declined by the provider's safety filter and answered by its fallback model, claude-opus-4-8, so that row is a mixture of the two.

2. Proposition 1 on real step labels (CeiD Q3)

R-Judge's 414 injection records are built from InjecAgent cases, so the violating step can be found by code: the first step that calls a tool on InjecAgent's attacker list. This gives 200 violating and 628 safe steps with no judgement by us. With one random safe step per calibration trace, over 100 splits, the false-block rate of a random safe step at α = 0.05, 0.10 and 0.20 is:

  • step scorer: 4.5%, 9.8%, 18.0%
  • sequence scorer: 4.8%, 9.8%, 19.9%
  • graph scorer: 4.8%, 9.4%, 19.4%

Proposition 1 holds as stated. Two remarks. Detection is easy on these records (the violating step is blocked when proposed in 97.5–100.0% of cases at α = 0.05), because a safe second step is a final answer and an unsafe one a tool call; the experiment tests the calibration rule, not the detector. And pooling all calibration steps did not inflate the false-block rate here (7.7–8.5% at α = 0.10), unlike the prompt-reuse case in our earlier response; the exact statement is for one step per trace.

3. Repair under domain shift (Y6mx)

A threshold calibrated on four application categories gave 1% to 62% false blocks on the fifth. Recalibrating on half of the new category's own records, with the scorer unchanged, and testing on the other half (20 splits, α = 0.10, four trained scorers):

  • Application (48 safe calibration records): 6.0–6.8%
  • Finance (44): 5.0–6.5%
  • Program (30): 8.2–11.7% (the highest value is 1.0 standard errors above the target)
  • IoT (6) and Web (8): no finite threshold exists, and the guard blocks nothing

So a few dozen in-domain safe records restore the budget without retraining. With fewer than 1/α − 1 = 9 safe records, conformal calibration certifies nothing and says so; the revision states this minimum. Domain shift affects the LLM judges too: with a cross-domain threshold the Claude judge has 36.6% false blocks on IoT.

In the paper

These results are in Section 6.1 and Appendices D and E of the revision. Next, we will calibrate the learned guardrails AGrail, ShieldAgent and Progent under the same protocol, so that all guards for agent systems can be read on one guaranteed axis.

AuthorsOpenReview ↗

Additional results 2/2: on 669 multi-agent sessions both guarantees hold, and provenance edges of the execution graph lift detection from 53% to 93% at a 5% session budget.

This comment gives the tables for the multi-agent experiment (kBPG Q4, CeiD Q3, Y6mx). Files, per-split results and the generated sessions are in the supplementary material (results/mas_kaggle_v2/).

Setup

669 sessions: 600 with claude-opus-5-5 playing the agents and 69 with gpt-6-astra. In each, a coordinator, a researcher and an acting agent work on one to three interleaved tasks in a simulated workspace with mock tools (418 sessions have concurrent tasks). Agents never see a document's classification. Of 10,369 agent steps, 217 are violations by a fixed code rule: 216 send restricted records outside the organisation and 1 follows an injected instruction (390 were planted; the agents did not follow the rest). No author or model assigns a label. 50 random splits; five scorers on identical features.

1. Both guarantees hold on multi-agent traces with step labels

  • Proposition 1 (a random safe step is blocked): 4.5–5.2%, 9.6–10.1% and 19.8–20.9% at α = 0.05, 0.10, 0.20 across the five scorers.
  • Corollary 1 (a violation-free session is interrupted at all): 4.4–4.9%, 9.6–10.2% and 20.0–21.3%.

2. Violating steps blocked when proposed, under the per-session budget (α = 0.05 / 0.10)

  • current step only: 8.7% / 17.9%
  • sequence model over the whole session: 53.1% / 70.4%
  • typed graph with provenance edges: 93.4% / 95.0%
  • the same graph without provenance edges: 6.9% / 14.3%
  • the same graph without node and edge types: 83.7% / 88.8%

Graph minus sequence at α = 0.05: +40.3 points, better in 50 of 50 splits.

3. Concurrency is where the sequence model loses (α = 0.10)

  • One task per session: sequence 95.4%, graph 98.6%.
  • Concurrent tasks: sequence 62.5%, graph 93.9%.

This is the setting the execution graph was designed for and that R-Judge could not test: when events of several tasks interleave, provenance edges keep each outgoing message tied to the records it carries.

4. A different agent model at test time (no guarantee applies)

Trained and calibrated on the Claude sessions, tested on the GPT sessions (α = 0.10, 10 runs): the graph scorer interrupts 5.1% of violation-free sessions and blocks 86.2% of violating steps; the sequence scorer 9.6% and 54.6%. The reverse direction has only 69 training sessions, too few to train any scorer (at most 10% blocked); a larger second set is planned.

5. How to read these results

  • Provenance is the active ingredient. The provenance edge links an outgoing message to the node where its record references first appeared, by string matching that a monitor can do at run time. Without it the graph scorer falls below the sequence model, so the benefit comes from the representation the paper proposes.
  • The policy becomes decidable on the graph. The rule "external destination, and a provenance edge from a restricted source" flags 216 of 217 violating steps and 0 of 9,933 safe steps. An operator who knows the policy can therefore enforce it directly on the graph, and the learned scorer shows that the same decision is recovered from labels alone, which is what is needed when a policy has no closed form. Measuring that case is our next experiment.
  • The per-session budget is the informative one. Under Proposition 1's rule every scorer blocks 96.6–99.9% at α = 0.10, because most steps are plainly harmless.
  • The guarantee is marginal, and can be made conditional. At α = 0.10, sessions with concurrent tasks are interrupted in 11.2–15.8% of cases and single-task sessions in 1.1–7.6%; one threshold per number of tasks addresses this.
  • Scope. The sessions are simulated, with a fixed hand-over pattern, task templates and, in effect, one kind of violation. In 184 of the 600 Claude sessions at least one step was answered by the provider's fallback model claude-opus-4-8.

In the paper

Section 6.2 and Appendix F of the revision report this experiment. The claim it supports is that provenance edges make provenance-dependent policies visible to a guard under concurrency, and that the conformal layer certifies whatever scorer uses them. Policies without a closed form, a closed-loop study with the guard in the loop, and the AgentDojo tool ecosystem are the experiments we will run next with the released simulator. Thank you for your attention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.