- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to address multi-step safety issues in multi-agent LLM systems, where looking at individual actions may miss some risks. It proposes ConformalGuard, which mainly has two parts. The first part builds a graph from the execution history and uses DyGFormer-HGT to get a score for each action. The second part uses conformal prediction to calibrate the threshold for deciding whether to block an action, aiming to reduce false blocks and provide a coverage guarantee for safe actions. The experiments show that the method performs better than the baselines.
Critical strengths
-
Multi-step safety is indeed a problem in multi-agent LLM systems, since some risks cannot be detected by only looking at a single step. I think this is worth studying.
-
The experimental results are pretty good, especially compared with the baselines. The ablation study also shows that using the execution history helps.
Critical weaknesses
-
The positioning of the paper is not very clear to me. The detection performance seems to mainly come from the GNN part, but the paper focuses more on CP.
-
Most components and theoretical results are based on existing work.
-
I am not sure the CP coverage guarantee directly translates to the claimed false-block guarantee.
Questions for authors
-
I am a bit confused about the positioning of this paper. A lot of the detection performance seems to come from the GNN part, but the paper mainly focuses on CP and its theoretical guarantees. It is not very clear to me what the main contribution is supposed to be.
-
The novelty seems somewhat limited. Most of the methods in the first part already exist, and the CP theory in the second part does not seem particularly new either. If this is intended as a systems paper, I think the authors need to better explain why such a system is needed and why these two parts need to be combined.
-
I am not sure how the theoretical coverage guarantee translates to the actual false-block rate. CP guarantees coverage of the labeled safe action, but the system decides whether to block the action proposed by the agent. It is not clear to me whether these correspond to the same distribution. Also, the paper acknowledges that steps within a trace are dependent, which may affect the exchangeability assumption.
-
I wonder why a graph model is needed here. The ablation study shows that using execution history helps, but it doesn't really show why a graph model would be better than a sequence model like Transformer using the same history.
-
Although the paper seems to focus primarily on CP, the overall performance appears to depend heavily on how well the learned model in the first stage performs.
3 comments
Thanks for the detailed response! I think the answers to Q3 and Q4 are mostly fine, and the new results are meaningful improvements. However, I still have concerns about Q1, Q2, and Q5, especially regarding the paper's positioning and novelty. If this were a 6- or 10-point scale, I'd probably bump my score up by one, but on the current 4-point scale, I feel I'm somewhere around 2.5.
Reply to Reviewer kBPG: thank you for the reassessment; below is the paper's position stated in one sentence, what is new in it, and the changes that make it explicit from the first page.
Thank you for reading the new results and for telling us plainly where you stand. Your remaining concern is fair: the submission did not say clearly enough what kind of paper this is. We answer Q1, Q2 and Q5 together.
Position, in one sentence
Every guard for an agent system needs a threshold, today that threshold is set by hand, and this paper makes the rate at which legitimate work is interrupted a quantity that the operator fixes in advance and that is guaranteed for any detector.
Q1 / Q5. Yes, detection depends on the learned model, and that is why the layer is the contribution
Detectors change quickly; a certificate tied to one of them would not last. Ours is independent of the detector, and the experiments show what that buys:
- It holds for detectors we did not build. False blocks at α = 0.05 are 3.5–4.5% for five trained scorers and 3.5–4.4% for three LLM judges.
- It makes an unusable detector usable. At its own decision rule the open-weight judge flags 85.6% of safe records; calibrated, it flags 4.4% and still blocks 41.3% of unsafe ones.
- It lets the operator use the best detector for the setting. That is a calibrated LLM judge on single-agent records (95.2% blocked at α = 0.05) and a graph scorer with provenance edges under concurrent tasks (93.4% against 53.1% for a sequence model).
- It survives a change of domain cheaply. A few dozen in-domain safe records restore the budget without retraining (6.0–6.8%, 5.0–6.5%, 8.2–11.7% on three held-out categories at α = 0.10).
Q2. What is new
Split conformal prediction and adaptive conformal inference are known, and the paper says so. The novelty is in fitting the guarantee to what a guard does and in showing that it holds on real agent data:
- The certified object is the guard's decision. The calibrated pair is the proposed action in its execution state, so coverage is exactly one minus the false-block rate, and one unit per trace makes the bound exact. The submission's approximation term is gone.
- A closed-loop guarantee (Corollary 1). Standard conformal statements are open loop, while a guard changes the trace it scores. Calibrating on the maximum score of benign traces gives a bound that survives intervention, because guarded and unguarded runs coincide until the first block. To our knowledge this argument has not been made for runtime guards.
- Delayed labels (Proposition 2). Labels arrive when a trace completes; the bound is stated and proved with the delay explicit.
- Findings that change practice. The unit of calibration matters on real agent data (15.6–16.7% false blocks with pooled records against 6.1–7.6% with one per prompt group at α = 0.10). Online adaptation can keep its budget by ceasing to block, and the step size decides this (the guard is open on 47.8% to 62.6% of post-shift steps at γ = 0.05 and on 7.7% to 16.1% at γ = 0.005). Under concurrent tasks, provenance edges are what a guard needs (a sequence model blocks 62.5% of violating steps and the graph scorer 93.9% at α = 0.10).
This is a contribution of formulation and evidence, not of new conformal theory, and we think that is the right contribution for the problem: guards are being deployed now, and to our knowledge none of the guardrails we cite gives its operator a guaranteed interruption rate.
Why the two parts belong in one paper
A calibrated guard without an informative score detects nothing beyond chance, and a score without calibration has no defensible threshold. The execution trace also defines the units on which the guarantee rests (step, trace, prompt group, session), so the representation and the certificate cannot be designed apart.
Changes that make the position explicit
- Done in the revision: the abstract leads with the guarantee and closes on the calibration layer, and Figure 1 is redrawn with the scorer as a pluggable component.
- To be added: a sentence in the contributions stating that no new conformal theory and no new architecture is claimed.
- To be added: a short table in Related Work listing, for each family of guard and each conformal method for LLMs, what it certifies. To our knowledge the row for runtime agent guards with a false-block guarantee is otherwise empty.
If a particular comparison or statement would settle the remaining concern, we will gladly add it during this discussion. Thank you.
Response to Reviewer kBPG: calibration is the scorer-agnostic contribution, and the execution graph pays off exactly where you asked us to test it, under concurrent multi-agent tasks.
Thank you for five questions that go to the structure of the paper. Each is now answered by an experiment.
Q1 / W1 and Q5. Detection comes from the model, the guarantee from calibration
This division is the design, and the revision demonstrates its value. A learned score gives no principled threshold, and every false block stalls legitimate work. Conformal calibration turns any score into a rule whose false-block rate is at most a level α chosen by the operator. We show it for five trained scorers and for three LLM judges we did not build: all stay within budget on R-Judge (3.5–4.5% and 3.5–4.4% at α = 0.05). A stronger scorer blocks more, and the layer lets an operator use the strongest one available: calibrated at α = 0.05,
claude-opus-5-5as a judge blocks 95.2% of unsafe records.Q2 / W2. Novelty, and why the two parts belong together
The contributions are the formulation (runtime guarding as calibrated scoring of proposed actions on the execution graph, with the false-block rate as the quantity the operator fixes), guarantees fitted to what a guard does (Q3), and evidence that the combination works on real records, on existing judges and on multi-agent sessions (Q4). Neither part suffices alone: an uncalibrated open-weight judge flags 85.6% of safe records, and a calibrated guard without an informative scorer detects nothing beyond chance.
Q3 / W3. Coverage and the false-block rate are now the same quantity
Q4. Graph versus a sequence model on the same history
We ran the control you asked for in both settings: a causal Transformer over the same nodes, features, loss and calibration, against relation-aware attention over the typed graph.
Single agent (R-Judge). History is what matters: the sequence model blocks 4.6 points more than a single-step scorer, and the graph does not improve on it (82.3% against 77.8% at α = 0.10).
Several agents, concurrent tasks. This is the case the graph was designed for, and here it wins clearly. On 669 sessions in which LLM agents work on one to three interleaved tasks (217 violating steps, labelled by code), under a per-session budget:
claude-opus-5-5sessions and tested on thegpt-6-astrasessions: graph 86.2%, sequence 54.6%.What carries the effect. The ablation is unambiguous: without the provenance (
cites) edges the graph scorer falls to 6.9%. Provenance made explicit as an edge is what lets a guard tie an outgoing message to the records it carries when tasks interleave. It also makes the policy decidable on the graph: a one-line rule on these edges reproduces 216 of the 217 violation labels with no false positive, so an operator who knows the policy can enforce it directly, and where the policy is not known in closed form a scorer trained on labels alone recovers it, as ours does.Continuing work
The sessions are simulated and exercise one family of policy. Policies without a closed form, and the tool ecosystem of AgentDojo, are the next tests of how much learning adds on top of the graph, and the released simulator makes them straightforward to run. Thank you.