acceptodds
Under review as a conference paper at ICLR 2027

Who Broke the Debate? Training-Free Detection of Adversarial Agents via Token Confidence

Abstract

Multi-Agent Debate (MAD) systems improve reasoning performance through inter-agent collaboration, but they are also vulnerable to adversarially manipulated agents that can propagate misinformation throughout the debate process. Existing defenses rely on task-specific training, external LLM judges, or post-hoc communication analysis. These approaches limit cross-task generalization, increase computational cost, and delay intervention until adversarial information has already entered the debate. In this paper, we propose DICE, a training-free defense method for proactively detecting compromised agents in MAD systems using LLM internal signals. Our key insight is that adversarial prompts can induce systematic shifts in token-level confidence patterns during generation, particularly in the early stage. Based on this observation, DICE measures inter-agent confidence differences using token-level log-probability signals without requiring additional training or external supervision. Our experimental results show that DICE achieves strong and consistent defense performance across diverse MAD tasks. The data and code used in this work are available at: https://anonymous.4open.science/r/TtJd5Tu2zX48ISYkkWsd

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Agent Debate, Adversarial Detection, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Z5h3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the detection of adversarially prompted agents in multi-agent debate and proposes DICE, a training-free method based on token-level confidence. DICE computes each agent’s confidence from top-k token log-probabilities and votes on the early part of the response, flagging agents whose confidence falls in the minority group. Experiments across multiple reasoning and attack benchmarks show that DICE consistently reduces attack success rates at low cost compared with existing defenses. Additional analyses examine robustness under different attack settings.

Critical strengths
  • DICE detects compromised agents from relative confidence differences among agents answering the same question, without requiring any training, and thus avoids the cross-task performance degradation of graph-based methods. This is of practical value, since training data for the target task is often unavailable in real deployments.

  • DICE only requires token log-probabilities, with no additional LLM calls or training overhead, and no cost for generating training data. This makes it suitable as a lightweight, plug-and-play defense component.

Critical weaknesses
  • On most datasets, the attacked and benign agents use different prompts and output formats (e.g., benign agents must follow fixed reason/answer tags or ReAct template), so the confidence differences detected by DICE may come from format differences rather than adversarial intent. This means an attacker who imitates the benign format could easily evade detection, and the authors provide no experiments to rule this out.

  • In real deployments, there may be no attacker most of the time, yet the paper does not evaluate the attack-free setting, and the voting rule necessarily flags some agents even when no attacker is present.

Questions for authors
  • Does the precision in Figure 4 count only the rounds after attack injection? How are it and the all-round precision in Table 8 each computed?

  • How does DICE perform against adaptive attacks from an adversary who knows the defense mechanism? For example, if an attacker first fills the initial 20% of the response with normal reasoning or a restatement of the question and then pivots to the wrong answer, can DICE still detect it?

  • In real deployments, the number of attackers is usually unknown. How should one choose between DICE and DICE-low in practice?

I would be happy to reconsider my rating if the authors can address these concerns.

1 comment

Thank you to the authors for the detailed response and the additional results. I decide to maintain my rating.

Reviewer_h9AdOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper utilizes the token-level confidence scores generated by agents when producing responses to determine which agent in a multi-agent debate system has been compromised.

Critical strengths

The DICE method is highly practical and it requires no training and allows for the rapid identification of compromised agents by simply checking token-level confidence scores during deployment. DICE compares the relative confidence among agents within each bin and it is invariant to shared positive affine transformations, which enhances its applicability across different models and tasks.

Critical weaknesses
  1. The core premise of DICE relies on the assumption that compromised agents constitute a minority, and it further requires that benign and compromised agents be separable based on their confidence levels—specifically, that they can be divided into distinct groups relative to a midpoint. In reality, however, this assumption may not hold: benign agents might exhibit low confidence when faced with difficult problems, while compromised agents could potentially mimic high confidence behavior.
  2. The experimental scale is too small: 60 samples per dataset, 8 agents, 3 attackers, and 3 repetitions, with only GPT-4o-mini used as the backbone.
  3. The paper assumes that adversarial prompts induce confidence shifts, but it does not evaluate adaptive attacks in which an attacker proactively calibrates confidence levels to mimic benign confidence trajectories.
Questions for authors
  1. If benign agents exhibit low confidence when facing difficult problems, while compromised agents deliberately mimic high-confidence behavior, can DICE’s midpoint grouping and minority voting still effectively distinguish between the two types of agents?
  2. Could you include additional backbone models, a larger scale of agents, and significance testing?
  3. If an attacker is aware of the DICE mechanism and deliberately maintains a confidence pattern similar to that of a benign agent, does DICE remain effective?
Reviewer_H1ApOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DICE, a training-free detector of adversarially prompted agents in multi-agent debate. It scores each generated token by the mean negative log-probability of its top-k alternatives, smooths the scores with a moving average, and splits each response into bins by relative position. In each bin at the start of the response, it splits the agents at the midpoint between the lowest and highest confidence and gives the smaller group an attack vote. An agent with at least as many attack votes as benign votes is removed from the final majority vote. Theorem 1 bounds the probability of misclassifying an agent, assuming that each agent's bin votes favor its true label by a fixed margin and are conditionally independent. Experiments use eight gpt-4o-mini agents, three of them attacked in the main setting, on six datasets, against G-Safeguard, BlindGuard and AgentXposed.

Critical strengths

The paper claims a detector that needs no training data or external judge and works across tasks, a claim that would matter if it held. For the reason in Critical Weaknesses, I cannot count the detector as a separable contribution.

Critical weaknesses

DICE's bin votes average largely the same tokens, so they act almost as one vote on the average confidence of a response's opening, and the guarantee of Theorem 1, which assumes independent votes, does not cover them. That opening is also where attacked and benign agents differ by prompt, so the paper does not show that DICE detects attacks rather than prompt differences.

The detector relies on a few early tokens that differ by prompt, not on independent pieces of evidence. DICE compares raw token confidence across agents and uses only the first 20% of each response, the share its sweep found best (Sections 3.3 and 4.2). Section 3 says attacked agents receive "the same task instruction as benign agents", but on MMLU, CSQA, GSM8K and GSM-Symbolic they do not (Appendix E). In the first round, benign agents must work out an answer in the form "<REASON>: … <ANSWER>: …" (E.2.1), while attacked agents get no format and are given the answer to defend, "Your answer: ({w_upper})" (E.3.1). The two groups therefore open their responses differently, and DICE's confidence can separate them for that reason alone. Table 5 fits this: DICE beats the baselines on these four datasets and is weakest on PoisonedRAG, the one attack in which both groups answer in the same format, where its F1 falls below that of G-Safeguard and BlindGuard. The strong results therefore cannot be attributed to detecting the attack.

The bin votes also give far less evidence than their number suggests. Each bin averages up to 128 preceding token confidences (Eq. 3), so adjacent bins average mostly the same tokens, and Appendix B.2 states that the votes are "not strictly independent". Theorem 1's bound assumes independent votes, so it does not describe the detector that was evaluated.

No experiment compares the voting with its simplest alternative, a single confidence statistic per agent, so the paper's reason for voting, "To reduce sensitivity to position-level variations in confidence" (Section 3.4), is untested.

Questions for authors
  1. Do the bin votes contain more information than a single confidence statistic per agent? In a short response, every inspected bin is an average starting from the first token. Could you report the response lengths, and compare DICE with a detector that applies the same midpoint split once to each agent's mean confidence over the same opening segment? If that single statistic works as well, the voting stage and Theorem 1 do not explain the results.
  2. Does DICE still detect attacked agents when they get the benign task instruction? Could you rerun MMLU, CSQA, GSM8K and GSM-Symbolic as Section 3 describes, with the attack only in the system prompt, the "<REASON>: … <ANSWER>: …" format and no answer given? That would show whether DICE detects the attack or the instruction. A clear drop would confirm the weakness above, and an unchanged result would resolve most of it.
  3. How many agents does DICE flag in debates with no attacker, and what does that cost in accuracy? Eqs. 6-8 give at least one agent an attack vote in every bin where confidences differ, which also happens when no agent is attacked. No experiment runs a debate without an attacker, and the rounds before a delayed injection are averaged into Tables 6-8 rather than reported, so the paper does not show how often DICE removes correct agents when no agent is attacked.

Comments on the paper

1 comment

Thanks for your reply. I think you shouldn't rely too heavily on AI. You should conduct your own research. I hope this helps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.