- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies malicious-agent detection in multi-agent debate with heterogeneous private evidence. The key motivation is that benign agents may disagree because they see different evidence, so disagreement alone should not be treated as an attack signal. PDVC builds task-adaptive semantic probes from public task information and independent first-round responses, projects centered responses into a conflict space, and learns an agent-level detector that isolates high-risk agents. Experiments on four multi-hop reasoning datasets show strong detection and task-performance results across several attack settings and model backbones.
Critical strengths
-
The motivation is strong and easy to follow. The paper identifies a real weakness of anomaly-based safeguards: under distributed private evidence, an unusual response can be useful rather than malicious.
-
The experimental coverage is broad. The paper includes four datasets, different attack structures, different LLM backbones, different agent counts, longer debates, different sentence encoders, and runtime analysis.
Critical weaknesses
-
The central motivation is not tested directly enough. The paper argues that increasing evidence heterogeneity makes benign disagreement harder to separate from malicious behavior, but the experiments do not provide a controlled sweep from homogeneous to increasingly heterogeneous evidence.
-
A simple first-round detection baseline is missing. Probes are built from public task information and agents’ first-round traces, while malicious agents receive adversarial evidence and a different system/instruction template (Appendix D). A supervised classifier on first-round response embeddings alone is an important control for surface cues. Table 2’s Raw-only ablation is not sufficient: on CONVERGENT it already reaches F1 66.67 / TSR 45.00 vs Full 79.02 / 50.00, and on ATTACKAVG 77.60 / 43.33 vs 84.59 / 48.33, but Raw-only still uses multi-round response semantics under the same Stage-II supervision—not a first-round-only detector.
-
The ablations do not fully isolate the proposed conflict representation. Table 2 mainly compares Full with Raw-only and loss variants, but does not separately test key components such as centering or the learned probe construction. In addition, Stage I explicitly uses role labels to learn role-separable conflict features, so the later role-alignment analysis is partly expected from the training objective.
Questions for authors
See Weaknesses.
2 comments
(W2) First-Round Decision against Evidence Insufficient in HMAD
First-Round Supervised Embedding Control. Following the reviewer's suggestion, we train a logistic regression (LR) classifier directly on frozen MiniLM Round-1 response embeddings on MuSiQue (CONVERGENT) and compare it against PDVC immediately after Round 1 (mean SD ):
| Method (Round-1 Evaluation) | (%) | (%) | |
|---|---|---|---|
| LR (Raw Response) | |||
| PDVC (First-Round Decision) |
In HMAD, single-agent Round-1 detection is limited by insufficient local evidence: without multi-round collaboration, benign agents lack the complementary hops needed to answer correctly, causing isolated embedding classifiers to misclassify incomplete benign reasoning as malicious (). PDVC's cross-agent conflict projection halves Round-1 false alarms (), while subsequent collaborative rounds remain essential to integrate evidence and achieve full task success ().
(W1) Controlled Evidence Heterogeneity Sweep
Controlled Evidence Heterogeneity Evidence Heterogeneity Degrades Separability. To empirically validate Fig. 1, we evaluate 200 QASC tasks (h=2 gold hops) across a controlled evidence spectrum over the 6 benign agents. Moving from homogeneous (Identical-6) to heterogeneous (Independent) evidence nearly doubles the mean pairwise benign response distance (dB-B), narrows its separation from the benign–malicious distance (dB-M), and more than 2x benign answer disagreement:
Each benign agent receives the specified Gold and Shared-D (shared distractor) documents, with remaining slots up to Total padded by independently sampled distractors. Identical-6 gives all 6 benign agents the same full view, whereas Identical-2/4 assign this full view to 2 or 4 benign agents and keep the rest in Shared. Here, d=1−cos denotes pairwise response cosine distance, and Benign Disagreement is the fraction of benign pairs with different normalized extracted answers.
Metric: Let nB of N agents be benign, vB and vM denote the within-group response-embedding variances, and g be the squared inter-group centroid distance. We define the normalized margin Δ=g+vB+vM(2nB/N−1)g+vM−vB, which quantifies the excess mean squared distance of malicious over benign replies to the population centroid; holding other quantities fixed, greater benign dispersion (vB) monotonically erodes this separability margin.
While shrinking Δ (0.181→0.077) under high heterogeneity degrades XG-Guard and BlindGuard by 16.23 and 17.17 AUROC points, PDVC exploits structured conflict representations to sustain ~95% AUROC across regimes.