acceptodds
Under review as a conference paper at ICLR 2027

Probe-based Divergence Calibration for Safeguarding Heterogeneous Multi-agent Debate

Abstract

Topology-constrained multi-agent debate can aggregate fragmented evidence, but its safeguards face a fundamental ambiguity: under heterogeneous private evidence, benign disagreement can resemble adversarial deviation. Existing anomaly-based defenses may therefore isolate agents whose responses are unusual but legitimately evidence-driven. We introduce PDVC, which changes the coordinates in which disagreement is observed. PDVC constructs task-adaptive semantic probes from public task information and independent first-round responses, projects centered responses onto these probes to obtain discussion-relative conflict representations, and learns an agent-wise role readout for topology intervention. Our analysis shows that shared attack-associated variation induces structured low-rank conflict after projection, while post-hoc evidence controls reveal that maliciousness is characterized primarily by the direction rather than the magnitude of unexplained response variation. Across four multi-hop reasoning benchmarks, diverse attack structures, and multiple LLM backbones, PDVC improves malicious-agent detection while preserving substantially stronger downstream task performance and benign collaboration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Heterogeneous Multi-Agent Debate, Anomaly Detection, Conflict Localization, Disagreement Calibration.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H3epOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies malicious-agent detection in multi-agent debate with heterogeneous private evidence. The key motivation is that benign agents may disagree because they see different evidence, so disagreement alone should not be treated as an attack signal. PDVC builds task-adaptive semantic probes from public task information and independent first-round responses, projects centered responses into a conflict space, and learns an agent-level detector that isolates high-risk agents. Experiments on four multi-hop reasoning datasets show strong detection and task-performance results across several attack settings and model backbones.

Critical strengths
  1. The motivation is strong and easy to follow. The paper identifies a real weakness of anomaly-based safeguards: under distributed private evidence, an unusual response can be useful rather than malicious.

  2. The experimental coverage is broad. The paper includes four datasets, different attack structures, different LLM backbones, different agent counts, longer debates, different sentence encoders, and runtime analysis.

Critical weaknesses
  1. The central motivation is not tested directly enough. The paper argues that increasing evidence heterogeneity makes benign disagreement harder to separate from malicious behavior, but the experiments do not provide a controlled sweep from homogeneous to increasingly heterogeneous evidence.

  2. A simple first-round detection baseline is missing. Probes are built from public task information and agents’ first-round traces, while malicious agents receive adversarial evidence and a different system/instruction template (Appendix D). A supervised classifier on first-round response embeddings alone is an important control for surface cues. Table 2’s Raw-only ablation is not sufficient: on CONVERGENT it already reaches F1 66.67 / TSR 45.00 vs Full 79.02 / 50.00, and on ATTACKAVG 77.60 / 43.33 vs 84.59 / 48.33, but Raw-only still uses multi-round response semantics under the same Stage-II supervision—not a first-round-only detector.

  3. The ablations do not fully isolate the proposed conflict representation. Table 2 mainly compares Full with Raw-only and loss variants, but does not separately test key components such as centering or the learned probe construction. In addition, Stage I explicitly uses role labels to learn role-separable conflict features, so the later role-alignment analysis is partly expected from the training objective.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

(W1) Controlled Evidence Heterogeneity Sweep

Controlled Evidence Heterogeneity Evidence Heterogeneity Degrades Separability. To empirically validate Fig. 1, we evaluate 200 QASC tasks (h=2h=2 gold hops) across a controlled evidence spectrum over the 6 benign agents. Moving from homogeneous (Identical-6) to heterogeneous (Independent) evidence nearly doubles the mean pairwise benign response distance (dB-Bd_{B\text{-}B}), narrows its separation from the benign–malicious distance (dB-Md_{B\text{-}M}), and more than 2x benign answer disagreement:

RegimeFull-View AgentsDocs (Gold / Shared-D / Total)Cosine Distance: dB-Bd_{B\text{-}B} / dB-Md_{B\text{-}M} / dM-Md_{M\text{-}M}Benign Disagreement
Independent0 / 6h−1h-1 / 00 / 660.1480 / 0.3809 / 0.098319.47%
Half-independent0 / 6h−1h-1 / ⌈(7−h)/2⌉\lceil(7-h)/2\rceil / 660.1456 / 0.3809 / 0.099818.73%
Shared0 / 6h−1h-1 / 7−h7-h / 660.1461 / 0.3809 / 0.098719.03%
Identical-22 / 62×Id.-6+4×Shared2\times\text{Id.-6} + 4\times\text{Shared}0.1466 / 0.3922 / 0.099019.03%
Identical-44 / 64×Id.-6+2×Shared4\times\text{Id.-6} + 2\times\text{Shared}0.1221 / 0.4014 / 0.099315.43%
Identical-55 / 65×Id.-6+1×Shared5\times\text{Id.-6} + 1\times\text{Shared}0.0996 / 0.4033 / 0.096812.17%
Identical-66 / 6hh / 6−h6-h / 660.0723 / 0.4062 / 0.09878.10%

Each benign agent receives the specified Gold and Shared-D (shared distractor) documents, with remaining slots up to Total padded by independently sampled distractors. Identical-6 gives all 6 benign agents the same full view, whereas Identical-2/4 assign this full view to 2 or 4 benign agents and keep the rest in Shared. Here, d=1−cos⁡d = 1 - \cos denotes pairwise response cosine distance, and Benign Disagreement is the fraction of benign pairs with different normalized extracted answers.

Metric: Let nBn_B of NN agents be benign, vBv_B and vMv_M denote the within-group response-embedding variances, and gg be the squared inter-group centroid distance. We define the normalized margin Δ=(2nB/N−1)g+vM−vBg+vB+vM\Delta = \frac{(2n_B/N - 1)g + v_M - v_B}{g + v_B + v_M}, which quantifies the excess mean squared distance of malicious over benign replies to the population centroid; holding other quantities fixed, greater benign dispersion (vBv_B) monotonically erodes this separability margin.

While shrinking Δ\Delta (0.181→0.0770.181 \to 0.077) under high heterogeneity degrades XG-Guard and BlindGuard by 16.2316.23 and 17.1717.17 AUROC points, PDVC exploits structured conflict representations to sustain ~95% AUROC across regimes.

RegimeMean Δ\Delta [95% CI]XG-Guard AUROC [95% CI]BlindGuard AUROC [95% CI]PDVC AUROC [95% CI]
Independent0.077 [0.059, 0.094]76.73 [72.83, 80.69]75.00 [71.54, 78.44]96.00 [94.48, 97.35]
Half-independent0.079 [0.062, 0.097]75.79 [71.85, 79.75]74.42 [70.90, 78.00]95.42 [93.81, 96.83]
Shared0.078 [0.061, 0.095]75.06 [71.10, 79.04]73.50 [69.85, 77.17]95.79 [94.21, 97.17]
Identical-20.084 [0.068, 0.099]78.27 [74.25, 82.15]75.50 [71.90, 79.12]95.06 [93.27, 96.62]
Identical-40.120 [0.106, 0.133]84.65 [81.37, 87.69]82.38 [79.25, 85.19]94.96 [93.17, 96.54]
Identical-50.146 [0.134, 0.158]89.79 [87.17, 92.21]87.71 [85.19, 90.04]94.73 [92.81, 96.46]
Identical-60.181 [0.169, 0.192]92.96 [90.65, 95.04]92.17 [89.81, 94.27]94.48 [92.35, 96.31]
AuthorsOpenReview ↗

(W2) First-Round Decision against Evidence Insufficient in HMAD

First-Round Supervised Embedding Control. Following the reviewer's suggestion, we train a logistic regression (LR) classifier directly on frozen MiniLM Round-1 response embeddings on MuSiQue (CONVERGENT) and compare it against PDVC immediately after Round 1 (mean ±\pm SD ):

Method (Round-1 Evaluation)F1F_1 (%) ↑\uparrowFPR\text{FPR} (%) ↓\downarrowAUROC\text{AUROC} ↑\uparrow
LR (Raw Response)62.11±1.9762.11 \pm 1.9727.22±0.7327.22 \pm 0.730.7448±0.00690.7448 \pm 0.0069
PDVC (First-Round Decision)73.63±0.6773.63 \pm 0.6710.56±1.0010.56 \pm 1.000.8585±0.00690.8585 \pm 0.0069

In HMAD, single-agent Round-1 detection is limited by insufficient local evidence: without multi-round collaboration, benign agents lack the complementary hops needed to answer correctly, causing isolated embedding classifiers to misclassify incomplete benign reasoning as malicious (FPR=27.22%\text{FPR} = 27.22\%). PDVC's cross-agent conflict projection halves Round-1 false alarms (FPR=10.56%\text{FPR} = 10.56\%), while subsequent collaborative rounds remain essential to integrate evidence and achieve full task success (F1=79.02%,TSR=50.00%F_1 = 79.02\%, \text{TSR} = 50.00\%).

Reviewer_WEVsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the task of malicious agent detection in multi-agent debate scenarios. The authors argue that when debate is heterogenous with each agent having their own private evidence unavailable to other agents, then benign disagreements become difficult to differentiate from intentional adversarial behaviour, rendering existing defenses based on simply prioritizing identifying agents with high disagreement as anomalous mostly ineffective. They introduce PDVC, which utilizes constructed semantic probes to determine discussion-relative variation along task-relevant semantic axes, which are then used to learn train a detector to learn conflict patterns associated with malicious roles. The authors validate their approach on multiple multi-hop QA benchmarks, comparing with 4 defense baselines, across numerous attack scenarios where the number of malicious agents is varied. Across their results they show that their method achieves high F1 in malicious agent detection by reducing the number of false positives while retaining high recall. Further ablations show the robustness of their method

Critical strengths

The idea of the method is intuitive. It is interesting to craft a defense from the standpoint of separating malicious from benign responses, that are just generally incorrect or different by virtue of having access to different context, by projecting debate agent response into a representation space that explicitly measures role-consistent arrangement. The authors also provide ample experiments to validate their approach, with ablations covering key aspects such isolating different components of their method, analysis on time complexity, number of debate agents etc. All of this provides a relatively comprehensive outlook on the method’s effectiveness relative to existing approaches.

Experiments cover four multiple benchmarks, multiple model backbones, and attack configurations with different malicious-agent proportions and attack directions.

Critical weaknesses
  • One of my main concerns is that the paper is not clear about the utility of needing a defense for heterogenous multi-agent debate, whereby the defender does not know the private evidence available to each. The authors do not sufficiently motivate why this is a realistic attack/threat scenario, rendering their method as having limited utility. Presumably whoever is running the debate agents would also be in charge of running the defense (server side against malicious agents) and would know what each model contains and hence use that to craft their defense (in which scenario would this not be the case?). Also, in terms of being effective in a realistic scenario, I believe there should be analysis on whether an adversary that decides to optimize against this defense to craft malicious agents would work i.e., the lack of adaptive attack results also harms the method’s adoptability.

  • The authors state their contribution compared to existing baselines as “Deviation based safeguards can therefore conflate evidence-driven novelty with malicious deviation" and use the fact that several baselines attain very high recall but much lower precision as evidence, but I feel like more evidence for this claim should be provided. I believe that providing examples of failure cases of the other baselines compared to PDVC or better by explicitly showing performance on subset of failure cases with high deviations that your method correctly identifies as benign while the others label as malicious would better prove the mechanism of action. Furthermore, I have some concerns on the nature of the attacks used, particularly for the ablation starting on line 418 about the different attack types. From what I understand is the variance is in terms of the different malicious agents used but the actual attack type remains similar i.e., it’s not like you’re testing completely different categories of for example prompt injections, jailbreaks etc. so it seems that claiming that the defense generalizes to different attack types is not sufficiently justified.

  • I feel there is insufficient support for lines 050–079. The authors present their motivation as an “observation” in Line 078, but its empirical basis is unclear. Which datasets/methods were examined to establish this observation? How are the relevant properties of disagreement quantitatively defined, and which metric supports? The introduction does not provide these details or clearly connect the observation to supporting experiments or prior work. Figure 1 appears to offer a schematic illustration rather than numerical evidence, yet this distinction is not made sufficiently explicit. Several symbols in the figure are also insufficiently explained. The authors should distinguish conceptual intuition from empirical findings and provide a clear evidential basis for the motivating claim.

Questions for authors
  • In Section 4.2, equation 5, why projection, and how does it compare with other ways of separating role-associated variation? The conflict representation applies population centering and probe projection as a single block, and the ablation only compares the full model against a raw-response variant without the probe-induced features. This shows that the block as a whole is useful, but it does not isolate the contribution of projection itself, nor compare it with alternative separation operators, such as centering without projection, projection onto random, fixed, or PCA-derived directions, or a distance-based readout in the probe space. Could the authors provide such comparisons, or further justify why linear projection onto learned probes is preferable to these alternatives?

  • Why is claim aggregation formulated as an optimal transport problem, and is this formulation necessary? The semi-balanced transport formulation simultaneously enforces the source-balanced claim mass, encourages balanced probe usage through the KL term, and produces soft claim-to-probe assignments. However, the paper does not ablate this component against simpler aggregation schemes, such as mean pooling, attention-based pooling, or clustering, nor does it isolate the effects of the claim-mass constraint, the KL balancing term, and the entropic regularization. Could the authors show which of these ingredients are empirically necessary, and how sensitive the results are to the transport hyperparameters ε and κ?

5 comments
AuthorsOpenReview ↗

(W1) Interface Boundary, Threat Rationale & Adaptive Attack for HMAD

Interface Boundary and Threat Rationale in Heterogeneous MAD. We thank the reviewer for this question and clarify why HMAD orchestration neither guarantees private evidence access nor eliminates the need for communication-layer defense:

  • Standard orchestrators only observe public messages: In real-world protocols like A2A, the scheduler coordinates agents across different parties without full control over their internal states. The defender can only monitor public utterances, not participants' private documents or local tools.
  • Even when visible, errors can propagete without oracle verification: In G-Safeguard (ACL 2025), attacks poison local prompts, tools, and RAG memories, resulting in a corrupted context. Even if the server inspects a compromised agent's local data or deploys local filters, poisoned external documents mimic benign data without a ground-truth oracle to verify authenticity, necessitating communication-layer detection over utterance histories and topology.
  • Resolving conflict under heterogeneous evidence: Complementary local evidence drives collaboration (MADAM-RAG; Kim et al., NMI 2026) but risks error propagation. The core challenge is separating malicious influence from normal benign disagreement without mistakenly isolating honest agents, which PDVC achieves with minimal overhead.

We will further clarify this threat model in the revision.

Lack of Adaptive Attack Analysis. To evaluate adaptive adversaries, we extend the Multi-Agent Consensus Attack (MCA; EMNLP 2024) to a defense-aware setting. Malicious agents perform a budgeted Best-of-NN search, querying the target safeguard output to select candidate arguments that minimize isolation risk. PDVC shows competing resilience against adaptive optimization and remains SOTA.

Table 1: Adaptive MCA Stress Test on MuSiQue

MethodTemplate Attack Paper SettingAdaptive MCA Defence Feedback
P / R / F1 / TSR / ASRP / R / F1 / TSR / ASR
PDVC (Ours)75.10 / 80.42 / 77.67 / 45.00 / 18.0676.96 / 73.75 / 75.32 / 43.33 / 21.67
G-Safeguard43.61 / 96.67 / 60.10 / 23.33 / 16.9444.33 / 92.92 / 60.03 / 28.33 / 15.83
XG-Guard42.86 / 96.25 / 59.31 / 28.33 / 12.2245.22 / 96.67 / 61.62 / 23.33 / 14.44
GUARDIAN40.13 / 52.50 / 45.49 / 31.67 / 19.7239.87 / 52.50 / 45.32 / 26.67 / 20.83
BlindGuard40.03 / 95.42 / 56.40 / 11.67 / 16.6739.53 / 91.25 / 55.16 / 26.67 / 16.67
AuthorsOpenReview ↗

(W2) More Evidence for Key Insight & Threat Model Clarification

Evidence for Benign-Deviation Confounding. We appreciate this suggestion. Following Lines 366–372, we will provide this exact agent-level and subset evidence in both the main text and appendix:

  • Preserving collaboration with less False Positive: As shown in Fig.5 (Lines 1105–1114)), Benign agents with partial evidence can often produce dissenting replies especially in Musique. Baselines turn almost the entire benign block gray (83.3%–95.3% false positives, whereas PDVC preserves benign agents even when they disagreed (red/green blocks are kept, ~17.8% FPR) and isolates most malicious ones. Even in NoAttack scenario (Table 3, in Lines 432–440), baselines prune dissenting benign agents, dropping accuracy to 20.0%–51.7% vs. 61.7% for PDVC.

  • Attack Scope Clarification: Aligned with prior work (Lines 172–181), our threat model targets misleading information propagation caused by compromised agent contexts (e.g., via prompt, memory, or tool attacks result in context corrupted), rather than jailbreaks. Specifically, CONVERGENT, DIVERGENT, and TRICAMP configure 4/10, 6/10, and 9/12 malicious agents pursuing one, two, and three attack directions, respectively. Since PDVC is trained exclusively on CONVERGENT, "unseen"(line 418) refers to held-out multi-agent coordination structures (DIVERGENT and TRICAMP), not different compromise mechanisms.

We will clarify this scope throughout the paper.

AuthorsOpenReview ↗

(W3) Empirical Support for Motivating Observation and Figure 1

Empirical Basis of the Motivation. We appreciate this critique and will clarify the link between our conceptual motivation (Lines 050–079) and empirical verification. Figure 1 is a conceptual schematic illustrating why magnitude-based defenses fail under HMAD: unlike homogeneous settings where benign replies cluster tightly, heterogeneous local evidence causes benign agents to legitimately disagree, inflating benign dispersion to overlap with malicious deviation; PDVC overcomes this by centering shared query semantics (HH) and projecting utterances onto learned evidence probes (U⊤U^\top) so roles separate by conflict direction (Zt=HXtU⊤Z_t = H X_t U^\top) rather than magnitude.

Evidence Heterogeneity Degrades Separability. Evidence Heterogeneity Degrades Separability. To empirically validate Fig. 1, we evaluate 200 QASC tasks (h=2h=2 gold hops) across a controlled evidence spectrum over the 6 benign agents. Moving from homogeneous (Identical-6) to heterogeneous (Independent) evidence nearly doubles the mean pairwise benign response distance (dB-Bd_{B\text{-}B}), narrows its separation from the benign–malicious distance (dB-Md_{B\text{-}M}), and more than 2x benign answer disagreement:

RegimeFull-View AgentsDocs (Gold / Shared-D / Total)Cosine Distance: dB-Bd_{B\text{-}B} / dB-Md_{B\text{-}M} / dM-Md_{M\text{-}M}Benign Disagreement
Independent0 / 6h−1h-1 / 00 / 660.1480 / 0.3809 / 0.098319.47%
Half-independent0 / 6h−1h-1 / ⌈(7−h)/2⌉\lceil(7-h)/2\rceil / 660.1456 / 0.3809 / 0.099818.73%
Shared0 / 6h−1h-1 / 7−h7-h / 660.1461 / 0.3809 / 0.098719.03%
Identical-22 / 62×Id.-6+4×Shared2\times\text{Id.-6} + 4\times\text{Shared}0.1466 / 0.3922 / 0.099019.03%
Identical-44 / 64×Id.-6+2×Shared4\times\text{Id.-6} + 2\times\text{Shared}0.1221 / 0.4014 / 0.099315.43%
Identical-55 / 65×Id.-6+1×Shared5\times\text{Id.-6} + 1\times\text{Shared}0.0996 / 0.4033 / 0.096812.17%
Identical-66 / 6hh / 6−h6-h / 660.0723 / 0.4062 / 0.09878.10%

Each benign agent receives the specified Gold and Shared-D (shared distractor) documents, with remaining slots up to Total padded by independently sampled distractors. Identical-6 gives all 6 benign agents the same full view, whereas Identical-2/4 assign this full view to 2 or 4 benign agents and keep the rest in Shared. Here, d=1−cos⁡d = 1 - \cos denotes pairwise response cosine distance, and Benign Disagreement is the fraction of benign pairs with different normalized extracted answers.

Metric: Let nBn_B of NN agents be benign, vBv_B and vMv_M denote the within-group response-embedding variances, and gg be the squared inter-group centroid distance. We define the normalized margin Δ=(2nB/N−1)g+vM−vBg+vB+vM\Delta = \frac{(2n_B/N - 1)g + v_M - v_B}{g + v_B + v_M}, which quantifies the excess mean squared distance of malicious over benign replies to the population centroid; holding other quantities fixed, greater benign dispersion (vBv_B) monotonically erodes this separability margin.

RegimeMean Δ\Delta [95% CI]XG-Guard AUROC [95% CI]BlindGuard AUROC [95% CI]
Independent0.077 [0.059, 0.094]76.73 [72.83, 80.69]75.00 [71.54, 78.44]
Half-independent0.079 [0.062, 0.097]75.79 [71.85, 79.75]74.42 [70.90, 78.00]
Shared0.078 [0.061, 0.095]75.06 [71.10, 79.04]73.50 [69.85, 77.17]
Identical-20.084 [0.068, 0.099]78.27 [74.25, 82.15]75.50 [71.90, 79.12]
Identical-40.120 [0.106, 0.133]84.65 [81.37, 87.69]82.38 [79.25, 85.19]
Identical-50.146 [0.134, 0.158]89.79 [87.17, 92.21]87.71 [85.19, 90.04]
Identical-60.181 [0.169, 0.192]92.96 [90.65, 95.04]92.17 [89.81, 94.27]

As heterogeneity reduces Δ\Delta (0.181→0.0770.181 \to 0.077), XG-Guard and BlindGuard lose 16.2316.23 and 17.1717.17 AUROC points, complementing our four-dataset analyses (Fig. 4a, App. B.2, Eq. 35) where residual magnitude ∥ei∥2\Vert{}e_i\Vert{}_2 is near-random (AUC≈0.5\text{AUC}\approx 0.5) while directional score γe\gamma_e reaches 0.924–0.9670.924\text{--}0.967 AUC. In the revision, we will explicitly label Fig. 1 as a schematic, define all symbols and axes, and link Line 078 to these results as a property of PDVC's learned representation.

Reviewer_w79wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks at detecting malicious agents in topology-constrained multi-agent debate. The twist is that benign agents hold different private evidence, so honest disagreement can look like an attack. PDVC builds task-adaptive semantic probes from the public task information and the first-round responses. It then projects centered response embeddings onto these probes to get conflict features. Training happens in two stages. Stage I fits the probe generator with a supervised contrastive loss, and Stage II fits a linear agent-wise readout with BCE plus a ranking loss. Agents that get flagged lose their outgoing edges. With the default backbone (Table 1), PDVC has the best detection F1 on all four datasets under the training attack setting (CONVERGENT). It also has a higher task success rate (TSR) than four graph-based safeguards. The appendix adds more backbones, encoders, schedulers, longer debates, and larger populations.

Critical strengths
  1. The problem is real and clearly motivated, and Fig. 1 explains the core difficulty well. I also like that the agent-wise readout doesn't assume benign agents are the majority. TRICAMP, with 9 of 12 agents malicious, tests exactly this.

  2. The evaluation covers a lot of ground. Besides the four datasets, it tests attack settings not seen in training. The appendix adds three other backbones (Tables 7–8), three encoders (Table 10), and three schedulers (Table 6). It also covers longer debates (Table 9) and up to 50 agents with a Flag-All reference (Table 11).

  3. Utility is reported next to detection, which matters for this problem. PDVC keeps TSR well above the baselines in Tables 1 and 3. The defense-side overhead is also small: 0.088 s per four-round dialogue, not counting LLM generation and the shared sentence encoder (App. B.3).

Critical weaknesses
  1. No variance, and numbers that change between tables. Each dataset has 60 test instances (Sec. 5.1), so a single task moves TSR by 1.67 points. I couldn't find seeds, standard deviations, or confidence intervals anywhere. Some comparisons come down to a few tasks. For example, Full vs. Raw-only is 50.00 vs. 45.00 in TSR (Table 2). PDVC's own numbers also differ across tables. Table 11 at N = 10 (6/10 malicious, two camps) gives PDVC P/F1/TSR = 94.72/89.44/51.67. Table 3 (DIVERGENT) gives 93.79/88.56/53.33, yet all four baselines match exactly. Table 9 (MuSiQue, round 4) gives 75.10/77.67/45.00, against 77.29/79.02/50.00 in Table 1. Here G-Safeguard, XG-Guard, and BlindGuard are identical in both tables.

  2. Baseline operating point. With the default THRESHOLD-ACCUMULATE scheduler, three of the baselines end up flagging most benign agents. Table 9 reports round-4 FPRs of 83.33% for G-Safeguard and 85.56% for XG-Guard. Table 11 reports FPRs between 71.94% and 98.33% for G-Safeguard, XG-Guard, and BlindGuard. The picture changes with the scheduler (CONVERGENT, Table 6). The TSR gap to the best baseline shrinks to 43.33 vs. 41.67 with TOP-k and to 46.67 vs. 45.00 with THRESHOLD-REFRESH. PDVC still has the best F1 there. App. A.2 says intervention parameters were picked on the validation split. It doesn't say how each baseline's threshold was set.

  3. Where the gain comes from. The component ablation (Table 2) is only reported for MuSiQue. Raw-only keeps PDVC's training discussions, Stage-II supervision, and readout, but drops the probe-induced conflict features. Even so, it already beats all four baselines in F1 (66.67 vs. at most 60.10) and TSR (45.00 vs. at most 31.67). The full model also trains the probe generator in Stage I. Over Raw-only, it adds 12.35 F1 points on CONVERGENT, 6.94 on DIVERGENT, and 1.67 on TRICAMP. So the benefit gets smaller in the settings that raise both the number of attack directions and the share of malicious agents.

Questions for authors
  1. (W1) Could you report mean and standard deviation over a few training seeds for Tables 1–3? Bootstrap intervals over test tasks would also work. And could you explain why PDVC's numbers in Tables 9 and 11 don't match Tables 1 and 3?

  2. (W2) For each baseline, was the decision threshold tuned on the validation split? If so, with what criterion? Or was it left at the original default? Threshold-free metrics such as AUROC or AUPRC of the agent scores would help separate ranking quality from the operating point.

  3. (W3) Could you run the Table 2 ablation on HotpotQA, 2WikiMultiHopQA, and QASC as well? Please also say which baselines use role labels during training. Finally, it would help to spell out what these comparisons tell us about supervision, the agent-wise readout, and the conflict-feature module as a whole.

3 comments
AuthorsOpenReview ↗

(W1&Q1) No Variance & Numbers Change Between Tables.

Re: We thank the reviewer for catching this. In paper, all experiments strictly evaluate on the complete 60 test instances per benchmark without subsampling. The cross-table discrepancy was caused by an inadvertent hyperparameter mismatch in Tables 1–3, now corrected and aligned with Tables 9 and 11. Per your suggestion, we report 3-seed evaluations with bootstrap intervals over test tasks; PDVC wins with both higher anomaly detection and downstream utility.

Table 1: Main results under CONVERGENT (3 Seeds: Mean ± SD)

(a) Malicious-Agent Detection

MethodMuSiQue (P / R / F1)HotpotQA (P / R / F1)2WikiMultiHopQA (P / R / F1)QASC (P / R / F1)
G-Safeguard43.2±0.3 / 95.3±1.3 / 59.5±0.645.8±0.9 / 88.6±1.0 / 60.4±0.647.7±1.2 / 93.2±1.5 / 63.1±1.248.7±1.1 / 98.6±1.3 / 65.2±0.9
GUARDIAN40.3±0.3 / 47.5±4.6 / 43.6±1.840.4±0.4 / 47.4±4.6 / 43.5±1.840.1±1.1 / 48.5±5.8 / 43.7±1.740.3±0.2 / 47.8±5.0 / 43.6±2.0
BlindGuard40.5±0.9 / 96.5±2.3 / 57.0±1.342.8±0.9 / 78.2±1.9 / 55.3±1.240.7±0.8 / 74.0±2.0 / 52.5±0.959.6±0.6 / 71.8±1.5 / 65.1±1.0
XG-Guard43.4±0.6 / 95.8±0.4 / 59.8±0.546.1±0.9 / 91.1±1.5 / 61.2±1.140.6±0.4 / 93.2±0.5 / 56.5±0.450.3±1.3 / 93.5±1.9 / 65.4±1.5
PDVC (Ours)76.7±2.7 / 81.1±0.9 / 78.8±1.877.0±3.4 / 80.8±0.7 / 78.8±2.081.2±2.6 / 86.8±1.5 / 83.9±2.089.1±0.9 / 89.2±0.8 / 89.1±0.5

(b) Task Utility and Attack Suppression

MethodMuSiQue (TSR↑ / ASR↓)HotpotQA (TSR↑ / ASR↓)2WikiMultiHopQA (TSR↑ / ASR↓)QASC (TSR↑ / ASR↓)
NOATTACK56.7±5.8 / 11.7±0.389.4±2.5 / 1.8±0.273.9±4.2 / 3.7±1.275.6±1.0 / 18.5±1.8
NODEFENCE42.8±4.2 / 22.9±2.381.1±1.9 / 10.4±0.968.9±2.5 / 8.6±1.163.3±3.3 / 29.8±2.4
G-Safeguard23.3±0.0 / 13.7±2.861.1±2.5 / 2.7±0.755.0±1.7 / 6.2±0.651.1±6.9 / 22.2±2.3
GUARDIAN30.0±1.7 / 19.3±0.461.1±5.1 / 7.8±1.747.8±1.9 / 8.7±0.745.6±5.9 / 28.8±2.0
BlindGuard17.2±5.1 / 14.0±2.365.6±4.8 / 9.4±1.750.0±2.9 / 11.9±1.564.4±1.0 / 26.1±2.3
XG-Guard30.0±2.9 / 13.4±1.857.2±1.9 / 7.5±1.325.6±2.5 / 11.6±0.653.9±3.5 / 25.8±2.1
PDVC (Ours)52.8±7.5 / 15.4±2.685.6±2.5 / 5.4±2.572.8±3.5 / 5.7±1.172.8±1.0 / 20.0±1.0

Table 2: Component ablation on MuSiQue (3 Seeds: Mean ± SD)

ScenarioFull (F1↑ / TSR↑)w/o Z (F1↑ / TSR↑)w/o Rank (F1↑ / TSR↑)w/o BCE (F1↑ / TSR↑)
NOATTACK— / 58.9±4.2— / 56.1±1.9— / 56.7±1.7— / 41.1±1.0
CONVERGENT78.8±1.8 / 52.8±7.565.8±1.3 / 51.7±6.782.0±0.3 / 53.3±2.969.9±1.0 / 41.1±5.1
DIVERGENT88.8±0.6 / 50.6±1.083.0±1.6 / 50.6±4.288.5±2.5 / 50.6±4.885.8±0.1 / 45.0±4.4
TRICAMP86.1±0.8 / 42.8±4.283.9±0.7 / 40.6±2.585.3±1.2 / 41.7±4.489.4±0.3 / 35.6±2.5

Table 3: Performance comparison on MuSiQue under different attack scenarios (3 Seeds: Mean ± SD)

MethodNOATTACK (TSR↑ / ASR↓)CONVERGENT (P / R / F1 / TSR↑ / ASR↓)DIVERGENT (P / R / F1 / TSR↑ / ASR↓)TRICAMP (P / R / F1 / TSR↑ / ASR↓)
G-Safeguard17.8±1.9 / 9.2±2.043.2±0.3 / 95.3±1.3 / 59.5±0.6 / 23.3±0.0 / 13.7±2.865.4±0.2 / 93.7±1.0 / 77.0±0.5 / 27.2±1.9 / 16.5±1.779.0±0.2 / 88.8±1.6 / 83.6±0.8 / 20.0±1.7 / 23.9±1.5
GUARDIAN58.3±6.0 / 11.7±2.140.3±0.3 / 47.5±4.6 / 43.6±1.8 / 30.0±1.7 / 19.3±0.458.3±1.5 / 29.2±0.7 / 38.9±1.0 / 28.9±1.0 / 26.4±0.273.1±5.0 / 24.4±1.7 / 36.6±2.5 / 15.0±4.4 / 32.8±4.8
BlindGuard13.9±5.9 / 12.2±1.340.5±0.9 / 96.5±2.3 / 57.0±1.3 / 17.2±5.1 / 14.0±2.359.9±1.1 / 94.4±1.5 / 73.3±1.3 / 9.4±5.9 / 17.4±1.574.5±0.2 / 94.6±0.2 / 83.3±0.2 / 3.9±1.0 / 20.2±4.2
XG-Guard16.7±5.8 / 10.4±0.343.4±0.6 / 95.8±0.4 / 59.8±0.5 / 30.0±2.9 / 13.4±1.863.8±0.2 / 90.1±1.4 / 74.7±0.6 / 29.4±4.2 / 26.1±4.676.9±0.4 / 86.7±1.3 / 81.5±0.5 / 15.0±1.7 / 30.0±2.5
PDVC (Ours)58.9±4.2 / 10.3±1.276.7±2.7 / 81.1±0.9 / 78.8±1.8 / 52.8±7.5 / 15.4±2.693.6±1.0 / 84.4±0.7 / 88.8±0.6 / 50.6±1.0 / 18.9±1.096.3±0.8 / 77.9±0.8 / 86.1±0.8 / 42.8±4.2 / 26.1±5.1
AuthorsOpenReview ↗

(W2&Q2) How Baseline Operating Point Tuned & Threshold-free Metrics

Re: We thank the reviewer for this insightful question. In paper, rather than leaving baselines at default values, we tuned each baseline's operating point per dataset via validation grid search maximizing F1F_1 (which particularly benefits unsupervised methods like XG-Guard and BlindGuard), whereas PDVC uses a single fixed boundary (logit>0\text{logit} > 0, i.e., pi,t>0.5p_{i,t} > 0.5) directly calibrated by Stage-II BCE+ranking without any post-hoc threshold sweep across all four datasets. The validation-tuned operating points and threshold-free ranking metrics (AUROC / AUPRC) are reported below:

Table 1: Validation-Tuned Decision Thresholds and Search Grids (CONVERGENT)

Method (Validation Search Grid)MuSiQueHotpotQA2WikiMultiHopQAQASC
G-Safeguard(sweep but 0.5 best and Interpretable)logit > 0 (p>0.5p > 0.5)logit > 0 (p>0.5p > 0.5)logit > 0 (p>0.5p > 0.5)logit > 0 (p>0.5p > 0.5)
GUARDIAN (estimated by percentile)0.1473 / 0.14990.1473 / 0.14970.1473 / 0.14870.1472 / 0.1493
BlindGuard (−2.0…2.0-2.0\dots2.0, step$0.05$)-1.25-0.70-0.65-0.20
XG-Guard (−2.0…2.0-2.0\dots2.0, step$0.05$)-0.85-0.60-0.95-0.45
PDVClogit > 0 (p>0.5p > 0.5)logit > 0 (p>0.5p > 0.5)logit > 0 (p>0.5p > 0.5)logit > 0 (p>0.5p > 0.5)

Table 2: Threshold-Free Agent Ranking Performance (Macro AUROC↑ / AUPRC↑)

MethodMuSiQueHotpotQA2WikiMultiHopQAQASC
G-Safeguard57.08 / 53.5058.40 / 54.8056.81 / 52.6161.84 / 56.21
GUARDIAN45.87 / 54.1348.69 / 56.4245.23 / 56.1252.91 / 58.01
BlindGuard44.58 / 49.3556.60 / 59.2247.43 / 52.4271.88 / 69.87
XG-Guard62.15 / 63.5869.79 / 70.8954.17 / 60.1880.00 / 80.78
PDVC (Ours)92.85 / 93.2491.60 / 91.8598.33 / 98.3498.75 / 98.93
AuthorsOpenReview ↗

(W3&Q3) Where the Gain Comes From

**Source of Gains, Supervision Regimes, and Matched-Population Generalization (W3 & Q3).**We thank the reviewer for this insightful question on attributing PDVC's gains. First, we clarify each method's supervision regime:

  • G-Safeguard & PDVC: Supervised with offline role labels.
  • BlindGuard & XG-Guard: Unsupervised graph anomaly detection without role labels.
  • GUARDIAN: Training-free online detection via temporal graph reconstruction loss.

The narrower margin under TRICAMP stems from confounding two distinct factors:

  • Malicious prevalence: A disproportionately high malicious ratio (9/129/12) inflates baseline$F_1$ while increasing detection difficulty.
  • Attack-direction diversity: Multi-directional dispersion expands conflict subspace rank, complicating topology isolation.

To isolate attack diversity from prevalence, we conducted a matched-population evaluation fixing 12 agents with 6 malicious across CONVERGENT (6×16\times 1 direction), DIVERGENT (3×23\times 2 directions), and TRICAMP (2×32\times 3 directions). As shown in Table 2, holding prevalence constant confirms that PDVC does not degrade under increasingly diverse attack directions, consistently outperforming all baselines:

Table 1: Matched-Population Evaluation on MuSiQue (12 Agents, 6 Malicious; P↑P \uparrow /R↑R \uparrow /F1↑F_1 \uparrow /TSR↑\text{TSR} \uparrow /ASR↓\text{ASR} \downarrow)

MethodCONVERGENT (6×1 Dir.)DIVERGENT (3×2 Dir.)TRICAMP (2×3 Dir.)
G-Safeguard54.06 / 92.50 / 68.24 / 21.67 / 15.8354.55 / 95.00 / 69.30 / 28.33 / 16.3953.76 / 95.28 / 68.74 / 30.00 / 21.11
GUARDIAN49.88 / 56.11 / 52.81 / 18.33 / 18.6150.36 / 57.50 / 53.70 / 28.33 / 19.4450.36 / 57.50 / 53.70 / 31.67 / 24.72
BlindGuard49.14 / 94.72 / 64.71 / 8.33 / 16.1150.22 / 96.67 / 66.10 / 6.67 / 19.7250.50 / 97.50 / 66.54 / 6.67 / 21.39
XG-Guard46.73 / 83.33 / 59.88 / 5.00 / 20.8358.06 / 95.00 / 72.08 / 43.33 / 20.2860.78 / 95.56 / 74.30 / 41.67 / 21.67
PDVC (Ours)85.92 / 84.72 / 85.31 / 51.67 / 13.8989.49 / 87.50 / 88.48 / 53.33 / 16.6789.97 / 84.72 / 87.27 / 53.33 / 13.89

cross all four benchmarks below, Table 2 ablations show that LBCE\mathcal{L}_{\text{BCE}} anchors primary role classification, while LRank\mathcal{L}_{\text{Rank}} further aids robustness in harder scenarios:

DatasetScenarioFull (F1↑ / TSR↑)w/o Z (F1↑ / TSR↑)w/o Rank (F1↑ / TSR↑)w/o BCE (F1↑ / TSR↑)
MuSiQueNOATTACK— / 58.9±4.2— / 56.1±1.9— / 56.7±1.7— / 41.1±1.0
CONVERGENT78.8±1.8 / 52.8±7.565.8±1.3 / 51.7±6.782.0±0.3 / 53.3±2.969.9±1.0 / 41.1±5.1
DIVERGENT88.8±0.6 / 50.6±1.083.0±1.6 / 50.6±4.288.5±2.5 / 50.6±4.885.8±0.1 / 45.0±4.4
TRICAMP86.1±0.8 / 42.8±4.283.9±0.7 / 40.6±2.585.3±1.2 / 41.7±4.489.4±0.3 / 35.6±2.5
HotpotQANOATTACK— / 87.22±0.96— / 83.89±2.55— / 91.11±3.47— / 61.11±1.92
CONVERGENT78.28±0.06 / 84.44±3.4767.40±0.74 / 78.89±3.4778.31±0.74 / 84.44±2.5570.22±0.33 / 59.44±2.55
DIVERGENT84.59±0.69 / 82.78±3.8576.46±2.12 / 77.22±1.9283.70±1.05 / 77.22±2.5584.06±0.45 / 63.89±3.47
TRICAMP84.52±0.99 / 76.11±0.9682.09±0.77 / 70.00±2.8984.50±0.15 / 74.45±2.5590.40±0.10 / 57.22±0.96
2WikiMultiHopQANOATTACK— / 71.67±3.34— / 67.78±3.47— / 68.89±2.55— / 50.56±4.19
CONVERGENT83.31±0.93 / 74.44±3.4769.65±0.65 / 70.00±4.4186.25±1.06 / 75.56±2.5570.06±0.22 / 53.33±2.89
DIVERGENT89.37±1.03 / 75.55±3.8579.53±0.29 / 58.89±1.9290.56±0.83 / 67.78±0.9685.27±0.69 / 51.67±1.67
TRICAMP88.87±0.62 / 66.11±5.3682.79±1.11 / 53.33±4.4189.12±0.19 / 71.11±4.1989.94±0.50 / 43.89±1.92
QASCNOATTACK— / 75.56±0.96— / 70.56±0.96— / 72.78±0.96— / 62.78±0.96
CONVERGENT90.36±0.81 / 71.67±1.6777.60±0.65 / 64.44±3.4790.54±0.70 / 71.11±1.9275.91±0.22 / 58.89±2.55
DIVERGENT92.61±0.31 / 67.22±0.9685.84±0.34 / 65.89±2.5593.08±0.30 / 68.33±1.6787.84±0.09 / 64.44±0.96
TRICAMP90.98±0.27 / 66.67±1.6785.96±1.07 / 61.66±2.8990.23±0.30 / 67.78±2.5592.63±0.48 / 56.67±1.67

Overall, PDVC's gains primarily stem from modeling the conflict representation ZZ and calibrating decision thresholds via LBCEL_{BCE}, whereas LRankL_{Rank} acts as an auxiliary regularizer that mitigates single-regime overfitting and stabilizes TSR\text{TSR} across unseen attack topologies.

Comments on the paper

1 comment

Hi all, thank you for your detailed reviews. The authors have provided detailed responses. Please read them and engage with the authors if there are any clarifications needed or any outstanding issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.