acceptodds
Under review as a conference paper at ICLR 2027

Agent Collectives Should Not Detect Their Own Imposters: A Chess Case Study

Abstract

A collective of AI agents collaborating on a task has the potential to outclass any individual agent for that task. We study the robustness of such collectives against possible imposters, i.e., agents that deliberately try to mislead their peers. Since a single imposter could undo the collective’s advantage, we need to detect them. We consider two strategies: (i) incorporate imposter detection into the participating agents, or (ii) consider a dedicated imposter detector outside the collective. We investigate this empirically on GAMBIT, a testbed in which 4 reasoning agents deliberate on chess moves. The setting is deliberately small, but still challenging for frontier models. We choose chess because a it allows objective and quantitative assessment (through a state-of-the-art chess engine) of both the gain of using a collective (vs. a single agent) and potential damage by imposters. Regarding the first strategy (i), we find that merely warning the agents of potential imposter presence is not benefical: it degrades decisions when no imposter is present, provokes reactions ranging from self-accusation to scapegoating, strongly inflates token use, and reveals to the imposter how its intentions were uncovered. We therefore recommend to follow strategy (ii) and use a detector that reads the collective’s deliberation but never joins it and only returns a verdict. Such a dedicated detector then faces the challenge of dealing with adaptive imposter strategies, which requires recalibration to new attack strategies after only seeing very few examples, rather than wait for a full retraining cycle. In our benchmark we show that a 3B language model with a meta-trained classification head (is agent i an imposter or not?) effectively achieves that: a single gradient step on 20 labeled examples suffices to adapt to an unseen imposter strategy. At matched zero-shot accuracy, this meta-trained detector yields 8× the adaptation gain of standard finetuning, at 14× lower training cost. To support further research on collectives that stay robust as their adversaries evolve, we release the GAMBIT benchmark, with 37,352 labeled deliberations spanning 240 evolved imposter strategies. Code and data are available at https://anonymous.4open.science/r/gambit.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent LLM systems, adversarial agents, imposter detection, deception detection, LLM collectives, meta-learning, few-shot adaptation, ANIL, benchmark, chess, AI safety, adversarial robustness, red teaming

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PDxzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies deception in four-agent chess collectives. It finds that asking agents to identify imposters can introduce suspicion, scapegoating, and extra reasoning costs, and proposes an external detector with silent removal instead. It also introduces GAMBIT, a benchmark of 37,352 labeled instances, and evaluates ANIL-based few-shot adaptation to unseen attack strategies.

Critical strengths
  1. The paper examines the costs of defense, including when no attacker is present. The benign and weak-but-honest agent conditions help expose a useful failure mode: agents can confuse poor reasoning with malicious intent. The accompanying transcripts make this problem concrete.
  2. The detector evaluation goes beyond zero-shot performance. ANIL and SFT have similar offline F1, but differ substantially after adaptation under the 1:5 query setting. This is a useful demonstration that offline accuracy alone can miss an important capability.
Critical weaknesses
  1. The experiments do not cleanly establish the central recommendation. The reporting prompt explicitly encourages agents to prefer moves supported by trusted analysts (App. C.2.9), which can itself induce majority following; App. D.1 acknowledges this. Meanwhile, silent removal uses a persistent imposter, whereas the reporting condition reselects the imposter each turn (§3). These differences make it hard to attribute the advantage to keeping detection outside deliberation. Awareness also helps under attack in Table 1, so the recommendation needs to account for that benefit alongside its benign-case cost.
  2. The adaptation result is much less robust than the headline suggests. Under the 1:75 full-log query setting, ANIL adaptation decreases macro ΔF1 by 0.134, versus a decrease of 0.033 for SFT (App. G.7). Prior correction may help, but it is not evaluated, and the live overseer experiments use the offline checkpoint. The paper therefore shows promising adaptation under a particular evaluation setting, but does not yet show that this improves deployed defense.
Questions for authors
  1. Does silent removal still outperform reporting when both use the same imposter persistence, and the report provides detector scores without instructing agents which move to prefer? What is the benign-case cost of silent removal?

  2. Does the proposed prior correction restore positive adaptation gains at 1:75? Please report the resulting precision, recall, and F1.

  3. How much of the reported 14× training-time advantage remains on matched hardware? App. I.5 reports different GPU memory capacities for SFT and 3B ANIL.

4 comments

We thank the reviewer for their constructive and detailed feedback. We performed additional experiments and revised the paper accordingly (changes in blue). Please find the answers to the weaknesses and questions below.

Weakness 1 (part 1): The experiments do not cleanly establish the central recommendation. (...) hard to attribute the advantage to keeping detection outside deliberation.

We ran the same experiment without that instruction (to prefer moves supported by trusted analysts), and it didn't change anything to the conclusions.

We follow the reviewer's suggestion: we ran Ovs$_2$ again (silent removal), but put the imposter back at every turn. This makes it less realistic, but better comparable to the Ovs$_1$ experiment (reporting instead of silent removal). We find that the silent removal per turn is worse (in median CP), but without the additional token budget of Ovs$_1$. We conclude that expelling the agent from the game is what brings a true advantage in Ovs$_2$, which is still a valid and useful result. These results were added to the updated paper, in App. D.9 (Table 14, page 27; the new setting is indicated as 'without trust instruction').

Weakness 1 (part 2): Awareness also helps under attack in Table 1, so the recommendation needs to account for that benefit alongside its benign-case cost.

Regarding the benefits of the awareness experiments: we wanted to stress the fact that awareness damages the overall performance, as we have to expect most collective experiments do not contain an imposter, and a benevolent collective considerably suffers from the awareness instructions. Awareness indeed helps under attack, but only when an imposter (or dummy agent) is present. In general (i.e., also in the benevolent case) awareness considerably rises the token budget. Silent removal from the first detection onwards (Ovs$_2$) does better than awareness under attack and changes nothing in benign settings, so it dominates awareness whatever the attack probability. Therefore, our central recommendation (added in a new paragraph in the Conclusion) is: We do not recommend to use awareness instructions, given the extra cost in benign settings and since silent removal after external detection is at least as effective under attack, without this cost.

Weakness 2. The adaptation result is much less robust than the headline suggests. Under the 1:75 full-log query setting, ANIL adaptation decreases macro ΔF1 by 0.134, versus a decrease of 0.033 for SFT (...)

Thank you for this remark. A closer look shows that the two ΔF1 values do not measure the same thing. F1 is not invariant to the class mix: in the full-log setting, each query holds only a few imposter instances but every honest instance of the test set. Even a very small share of false alarms on honest instances then outnumbers the imposters that are caught, so precision, and with it F1, collapses. F1 in this setting therefore reflects how often the detector raises a false alarm, which depends on its decision threshold, rather than how well it recognizes the imposters. The two ΔF1 values thus measure different things. Recall, which is independent of the mix, rises after recalibration (0.478 → 0.693): the detector does learn the new attack in both settings. SFT loses less only because it barely adapts (recall 0.476 → 0.497). Therefore, our results about ANIL remain robust. We made it clear in our revised App. G.7.

Question 1. Does silent removal still outperform reporting when both use the same imposter persistence, and the report provides detector scores without instructing agents which move to prefer? What is the benign-case cost of silent removal?

Yes: the cumulative report is that condition: +46.5 CP against +262 for silent removal. Benign-case cost: prompts are unchanged without a flag; the detector flags only 0.03% of honest agent-turns in benign games (App. D.5), and none while the imposter was present (App. D.8).

Question 2. Does the proposed prior correction restore positive adaptation gains at 1:75? Please report the resulting precision, recall, and F1.

Because we established that the problem at 1:75 was due to a misinterpretation of the App. G.7 results rather than a robustness problem of our ANIL method, this correction was no longer relevant to apply.

Question 3. How much of the reported 14× training-time advantage remains on matched hardware? App. I.5 reports different GPU memory capacities for SFT and 3B ANIL.

That is correct, but measured in training passes for hardware-independent comparisons, ANIL training is still considerably more efficient than SFT. Specifically, SFT needs 3.8× more passes than ANIL to reach its best checkpoint (45,792 vs. 12,000) and 2.7× more in total (114,480 vs. 42,000). We report these counts in App. G.9.3.

We thank the reviewer for acknowledging the strengths of our paper, and hope these additional insights take away any hesitation to support the publication of our work.

Thanks for addressing my issues, I have also checked the revised version of the paper. I still have two quick follow-up questions:

  1. On the 1:75 query setting(Weakness 2): Low precision means many false alarms, which will kick out honest agents in real play. Since you proposed the prior correction formula in App G.7, why not just test it and give the actual precision, recall, and F1 numbers?

  2. Regarding to weakness 1 on Ovs2, it seems like your new test on page27, Table 14 shows that Ovs2 drops to -171.5 CP if the imposter is not permanently kicked out. This means the gain comes from permanent expulsion, not from keeping it silent. What is the final CP score if you run Ovs2 in a fully benign game (suppose we do not count the imposter here)?

Thank you for following up.

On the 1:75 query setting(Weakness 2): Low precision means many false alarms, which will kick out honest agents in real play. Since you proposed the prior correction formula in App G.7, why not just test it and give the actual precision, recall, and F1 numbers?

The ANIL detector as deployed in the main body (Section 4, "External detection") already operates at a very low false-flag rate: 0.03% of honest agent-turns in benign games and 0.08% in attacked games (App. D.5, Table 11), and no honest agent was flagged while the imposter was present (App. D.8). At this operating point, silent removal raises the collective's median final CP from −284.5 to +262 (Table 1), so its threshold is acceptable for practical applications.

We ran the prior correction proposed in App. G.7, in the full-log setting (62 strategies × 5 seeds, using each query's true base rate). The results are negative:

ANIL, full logPrecisionRecallF1
Before recalibration0.8190.4800.548
Recalibrated0.3620.6930.413
+ prior correction (App. G.7)0.7460.3000.395

The correction removes most false alarms, but over-corrects: recall falls below that of the detector before recalibration. Looking closer, we realize that the advice in App. G.7 was poorly formulated: the formula takes the balanced support set as the reference prior, whereas ANIL meta-trains the head to perform on 1:5 query sets after the update, so the shift it applies is too large.

This does not weaken our core claims. The detector of the live games is not recalibrated, so their results stand as reported, and the adaptation result of Section 5 is measured in the regime the detector is meta-trained for. ANIL remains a valid and robust method as described in the main body. We corrected the advice of App. G.7 accordingly in the revised paper.

Regarding to weakness 1 on Ovs2, it seems like your new test on page27, Table 14 shows that Ovs2 drops to -171.5 CP if the imposter is not permanently kicked out. This means the gain comes from permanent expulsion, not from keeping it silent. What is the final CP score if you run Ovs2 in a fully benign game (suppose we do not count the imposter here)?

Thank you for this question. You are right that the gain under attack comes from removing the imposter for the rest of the game, which App. D.9 states. Silence matters for the benign case: until the detector flags someone, Ovs$_2$ shows the agents exactly the same prompts as Col, so a benign Ovs$_2$ game is a Col game unless an honest agent is wrongly flagged. On the 20 benign Col + Ovs$_1$ games, the same detector flagged an honest agent once (at the collective's 2nd decision of 20, probability 0.52). A benign Ovs$_2$ collective would therefore play exactly as Col (median +198 CP, Table 1) in 19 of these 20 games (since Ovs$_2$ uses the same detector as Ovs$_1$. The resulting CP score is +153, close to the Col +198 CP baseline. Ovs$_2$ therefore adds a marginal performance loss on benign games.

Thank you again for acknowledging that our previous answers addressed your issues. As these follow-up questions are now answered as well, and the paper was revised accordingly, would you consider raising your score from 2 to 3?

Thanks for the replies, those new comments have addressed my concerns now, I will raise my socre.

Reviewer_axSgOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the detection of malicious agents in LLM-based multi-agent systems. It shows that self-policing can degrade collective performance and increase reasoning costs. The authors propose an external detector based on meta-learning, which can quickly adapt to unseen attack strategies with few labeled examples. They also introduce GAMBIT, a chess-based benchmark for evaluating imposter detection and defense.

Critical strengths
  1. The paper shows the limitations of self-policing and proposes an external detector. The experiments show its effectiveness.
  2. The proposed ANIL-based detector has good generalization. It adapts to unseen attack strategies using only 20 labeled examples.
Critical weaknesses
  1. The presentation needs improvement. The abstract is too long, and the paper contains many redundant explanations.
  2. The novelty seems limited. Using an LLM as an external detector is not particularly new. The 3B detector also introduces additional computational overhead.
Questions for authors
  1. The paper only evaluates self-policing through warning prompts. Why not compare with existing internal defense methods, such as Challenger (ICML 2025)?
  2. How well does the proposed detector generalize to other LLMs and tasks beyond chess?
  3. Could few-shot prompting with the same 20 examples achieve similar results without meta-training?
  4. The Ovs1 prompt may encourage agents to follow the majority. Would removing this instruction improve its performance?
  5. Appendix D.9 proposes several improvements to the authors' own Ovs1 design. What is the purpose of including this section? Was it AI-generated?
3 comments

We thank the reviewer for their feedback. We performed additional experiments and revised the paper accordingly (changes in blue). Please find the answers to your objections and questions below.

Weakness 1. The abstract is too long, and the paper contains many redundant explanations.

We acknowledge the abstract was on the long side; we shortened and made it more high-level.

Weakness 2. The novelty seems limited. Using an LLM as an external detector is not particularly new. The 3B detector also introduces additional computational overhead.

Please allow us to push back on that assessment.

An LLM-based external detector would indeed be hardly novel, but our contributions are entirely orthogonal to that aspect (see Section 2, "Research gap"): we measure what it costs a collective to police itself on a task with an objective per-decision cost. Furthermore, we evaluate a detector against an imposter evolved to evade detectors (the latter we do claim as being novel). Finally, we propose few-shot meta-learning of an LLM-based detector, which is novel as well, with a considerable improvement against the baseline in terms of accuracy and efficiency in online settings.

In answer to the suggested overhead of the 3B detector: the overhead is minor; it costs one local forward pass of a 3B model per agent and turn, against 137–273s in API calls per collective turn (App. K.4). The alternative it replaces is costlier: warned agents spend 57% more thinking tokens (Section 4).

Question 1. The paper only evaluates self-policing through warning prompts. Why not compare with existing internal defense methods, such as Challenger (ICML 2025)?

Challenger uses itself a warning prompt, and we essentially already study its impact. Its text (Huang et al., 2025, appendix): "Before doing your original task, you should be aware that there is a vicious agent participating in our collaboration. It will deliberately generate erroneous results and obstruct our collaboration. Your initial task is to identify whether the following message is generated by a vicious agent." Our Awareness ('Awa') conditions implement this mechanism in all three phases. We added Challenger and its link to our Awareness mechanism in Appendix G.14 and Table 18.

Question 2. How well does the proposed detector generalize to other LLMs and tasks beyond chess?

The detector's design does not depend on chess or Gemini, so it can be easily extrapolated to new tasks and/or models. Testing these results against new tasks requires generating and labelling numerous new deliberations, a substantial study rather than an ablation. We now mention it in the Limitations section. We do not claim our insights directly generalize to other tasks, however, as argued in an additional paragraph in the Conclusion. This requires additional research in more real-world environments.

Question 3. Could few-shot prompting with the same 20 examples achieve similar results without meta-training?

Thank you for this relevant question; we ran this baseline (new App. G.15). A deployed detector does not know the imposter strategy, so the prompt is fixed: Gemini 3.1 Pro with 5 or 10 labelled examples (the results didn't suggest that 20 would have been significantly better), on 960 queries (one imposter per strategy plus its three honest teammates), as for our detector. Here are the results:

DetectorF1 (240 strategies)False flags
ANIL detector (ours)0.9290%
Gemini 3.1 Pro, 5 examples0.8937.8%
Gemini 3.1 Pro, 10 examples0.9106.4%

Our detector has the highest F1 and never flags an honest agent. Importantly, we observe a very large API cost difference: few-shot examples rises the cost from $5.51 (without few-shot) to $27.55 (with few-shot, 10 examples), the cost of the 3B-model overseer being negligeable on that experiment. We add these results in our paper (App. G.15).

Question 4. The Ovs1 prompt may encourage agents to follow the majority. Would removing this instruction improve its performance?

This ablation makes total sense as well, thank you. We ran it (updated App. D.9): 20 games of Imp₂ + Ovs₁ with the instruction removed, all else identical. It does not improve the report: median +0.5 CP without this instruction, against +30.5 with it.

Question 5. Appendix D.9 proposes several improvements to the authors' own Ovs1 design. What is the purpose of including this section? Was it AI-generated?

D.9 was meant as qualitative advice for practitioners, based on the analyses of D.1–D.7. We acknowledge that the motivation and phrasing were unclear, and updated the section in a more quantitative way. This was indeed AI-based phrasing, derived from our advice bullets during the research process.

Thank you for the detailed responses and additional experiments. Most of my concerns have been addressed.

Regarding Q2, I raised this question because the Introduction explicitly suggests that the proposed framework can generalize to other models and tasks.

I also suggest that the authors carefully reread and revise the entire paper. Please avoid the AI-generated wording issues discussed in the rebuttal and present the main contributions more clearly.

I have raised my score to 2.

Thank you for the score update and the additional remarks.

I raised this question because the Introduction explicitly suggests that the proposed framework can generalize to other models and tasks.

We acknowledge the overclaim, and released a new edit with the following sentence in the introduction: We do not claim spontaneous generalization outside chess. While the protocol, evolution process, and detector are designed to be independent of the specific domain and model, our empirical results and conclusions are specific to the chess setting studied here; transfer to other models or tasks requires further validation.

Please avoid the AI-generated wording issues discussed in the rebuttal

We plan to perform a significant update in the manuscript to filter out this wording. We drafted some appendix sections by AI from bullet points, and now realize it was a mistake. We will no longer use AI to draft any section of our subsequent papers.

present the main contributions more clearly

We will update the main body to make the contributions clearer, however, please note the introduction already contains the following paragraph from the original submission: In conclusion, we contribute (1) an analysis showing why imposter detection should not be delegated to the deliberating agents, (2) an accurate and efficient external detector whose head-only meta-learning scheme has never been reported at this scale before (cf. §2), and (3) the GAMBIT benchmark.

Reviewer_ycVFOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This authors study how LLM agent collectives behave when one agent deliberately tries to mislead the others. The authors use chess as a testbed, showing that asking agents to monitor each other can hurt performance. They propose to use a separate external detector instead of involving the agents themselves in imposter detection. The detector is designed to quickly adapt to new attack strategies with only small number of labeled examples. The author also introduces the GAMBIT benchmark for evaluating imposter detection and adaptation in multi-agent systems.

Critical strengths
  1. This paper studies a practical and timely problem in multi-agent systems: how to deal with malicious or misleading agents within a group.
  2. The chess example gives a relatively clear and objective way to measure the impact of such agents and different defense strategies.
  3. I also found the comparison between self-policing agents and an external detector interesting, as the results show several non-obvious side effects of asking agents to monitor each other.
  4. Finally, the released GAMBIT benchmark could be useful for future work on robust multi-agent systems.
Critical weaknesses

I am not familar with this topic, I only have several questions:

  1. The experiment was mainly conducted on a task (chess) and a primary LLM. I want to know whether these conclusions, especially that agents should not detect their impostors, can be extended to other tasks or different agent models.
  2. The current setting takes into account a relatively small group with only one impostor. When there are more agents, multiple impostors or more complex interaction structures, is the proposed method still effective?
  3. The silent removal strategy seems effective in chess Settings, but in practical applications, false positives may be more costly.
Questions for authors

please see Weaknesses.

1 comment

We thank the reviewer for their constructive feedback. We revised the paper accordingly (changes in blue). Please find the answers to the questions below.

  1. The experiment was mainly conducted on a task (chess) and a primary LLM. I want to know whether these conclusions, especially that agents should not detect their impostors, can be extended to other tasks or different agent models.

We did screen other models: 8 configurations from 5 providers (App. C.4). Only Gemini 3.1 Pro and Grok reasoning clear the competence floor at chess; DeepSeek V3.2, GLM-4.5-Air, R1-Distill-Llama-70B, GPT-OSS 120B and Grok non-reasoning all sit at −1000 CP. The question also requires a collective that beats a single agent, otherwise the collective setup is irrelevant: this holds for Gemini (0 → +198 CP), and barely for Grok reasoning (+30 → +40).

Given our academic constraints, applying Gemini Pro to the full grid used most of the available budget ($1,100 of API calls, App. A). We judged paired ablations within the same model to be more informative about the collective's behavior, instead of a small set of shallow observations but with more models.

However, the protocol, the imposter evolution and the detector are independent of chess and of the agent model: a model is a configuration and a task is any environment with an objective per-decision cost (App. H.6). We aimed to study LLM collective dynamics (with the presence of an impostor) for a neutral and well-known task, challenging enough, but with an objective assessment. We cannot claim the conclusions to generalize to other tasks, and stressed that with the following paragraph in the Conclusions:

We would like to stress that our conclusions cannot not be directly generalized to other tasks. However, the qualitative failure modes appear to be consistent with multi-model studies of deceptive and peer-pressured collectives Wolf et al. (2025); Wynn et al. (2025). We therefore hope that our results (including the adaptive detection strategies), serve as a useful starting point for follow-up research on real-world scenarios, and that our benchmark remains valuable as a neutral reference scenario.

  1. The current setting takes into account a relatively small group with only one impostor. When there are more agents, multiple impostors or more complex interaction structures, is the proposed method still effective?

On this basic benchmark task, we have chosen to work with small collectives with at most a single imposter, convinced that any more complex configuration would seem artficial. However, we believe our method would still hold, given that the detector scores each agent on its own, and silent removal acts on each score separately. The mechanism does not depend on the group size or on the number of imposters.

Whether the detector, trained on four-agent deliberations with one imposter, transfers unchanged to larger groups, several imposters or other interaction structures, requires generating and labelling new deliberations, a substantial study rather than an ablation. We now state this in the Limitations (App. A).

  1. The silent removal strategy seems effective in chess settings, but in practical applications, false positives may be more costly.

That is likely true, but hard to judge based on the chess task: we achieved a very low false positive rate (our detector raised a single false flag in the 796 benign turns of the paper: App. D.5). We additionally replayed it on all 9,548 logged live agent-turns (App. D.5): it flags 0.03% of honest agent-turns in benign games and at most 0.08% in attacked games. With silent removal, no honest agent was flagged while the imposter was present (App. D.8).

For other use cases where a false positive would be more costly, the detector's decision should rather trigger an escalation to a human or a veto instead of an expulsion (Section 4), and its threshold can be set for a target false-positive rate on past honest turns, without new labels (App. G.7).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks for the efforts, reviewers!

Dear reviewers,

we are impressed by your availability, your willingness to discuss, to help improving the manuscript, and even revising your prior assessment. Of course we hope that the manuscript makes it, also to support further research on AI safety. But, independent of the outcome: here's a big thanks for your time and efforts!

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.