acceptodds
Under review as a conference paper at ICLR 2027

Who Else Is In? Multi-Agent Collusion Initiation via Strategic Vagueness

Abstract

Multi-agent LLM systems raise the risk of collusion among agents pursuing objec- tives hidden from their overseers. When potential partners’ identities are unknown, collusion first requires initiation: agents must discover one another, often through a public channel observed by ordinary agents and monitors. Strategic vagueness offers a way to navigate this mixed audience, conveying intent to a prospective partner while preserving a plausible ordinary interpretation. We introduce a mutual- identification game in which two schemers must identify each other through public interaction, while jointly solving an ordinary task. We find that some models dis- cover successful handshake strategies without explicit signalling guidance, using task-compatible exchanges that monitors often fail to flag at low false-positive rates. To characterize the communicative capabilities underlying these exchanges, we study a controlled coordination setting . We find an asymmetric response to message explicitness: as messages become more indirect, monitor punishment can fall faster than partner coordination, creating a profitable intermediate regime of strategic vagueness. Together, these findings illustrate a Blackstone-style evidential asymmetry: communication sufficient for coordination between agents may remain insufficient for confident detection by monitor, opening a new surface of collusion.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Agent, Multi-Agent, Safety, Collusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yDu6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates collusion among LLM agents in multi-agent systems, focusing specifically on the initiation step, referred to as collusion initiation, when two schemers need to identify each other while communication is openly observable within the group of agents. The authors argue that existing work often assumes that colluding agents already know their partners, overlooking the initial identification step, which may need to occur through publicly observable communication. They find that agents are successful at identifying ‘handshake strategies’ without the need to explicitly be told to follow these. The experiments also look at monitoring of this behaviour, and whether the collusion initiation goes undetected.

The authors study the problem using a mutual-identification game in which two schemers, whose identities are unknown to one another, must identify each other while participating in an ordinary collaborative task with other agents. They evaluate Gemini 2.5 Pro, GPT-5.1, and Claude Sonnet 4.6, comparing conditions where agents are explicitly guided to use strategic vagueness with conditions where no particular signalling strategy is specified. They find that Gemini is the only model to achieve successful mutual identification above chance without explicit signalling guidance, while other models improve when given guidance. They also evaluate the detectability of these handshakes across different monitoring architectures and model families, finding that many successful handshakes remain undetected at low false-positive rates.

The authors then introduce a controlled schemer–accomplice–monitor game to investigate strategic vagueness more directly, isolating it from other components of the interaction. They find that, for some models, indirect messages can preserve coordination with an accomplice while reducing punishment by an observer. They interpret this as an asymmetry between the evidence required for a partner to coordinate and that required for a monitor to intervene.

Critical strengths
  1. The paper identifies an underexplored problem in multi-agent collusion. As the authors note, much of the existing work on collusion in multi-agent systems assumes that agents already know who their potential collaborators are. This paper instead investigates the preceding step of identifying other agents with shared hidden objectives, while communicating through a public channel. This is a timely and relevant problem as LLM agents increasingly interact and collaborate in multi-agent environments. The distinction between collusion initiation and collusion execution is consequential, as it highlights a potential vulnerability that may be missed by evaluations focusing only on established collusion or explicitly malicious communication.

  2. The paper is clearly presented and supported by a thorough experimental analysis. The experiments are well motivated and designed to examine different aspects of the proposed problem. The mutual-identification game distinguishes between the ability to discover a signalling strategy and the ability to execute one when explicitly guided. The authors also examine detectability across different monitor architectures and model families, including monitors with additional knowledge of signalling strategies. The second experiment provides a more controlled investigation of strategic vagueness, with ablations examining the effects of messages and payoff information on coordination. The combination of these experiments provides useful evidence that agents can establish covert handshakes through seemingly ordinary communication, and that monitoring such behaviour can be difficult without producing excessive false positives. The paper also engages well with the relevant literature and clearly explains its contributions in relation to existing work.

Critical weaknesses
  1. The experimental settings are highly stylised, limiting the generalisability of the findings to real-world systems.

In the mutual-identification game, schemers are explicitly assigned a hidden objective and instructed to identify potential collaborators. Although the strategy-unspecified condition demonstrates that some agents can discover signalling strategies without being explicitly told how to do so, it does not establish whether agents would independently develop collusive objectives or initiate collusion in environments where they have not been instructed to do so. It would be interesting to understand whether similar behaviour would emerge from agents pursuing individual objectives under incentives that make collusion beneficial.

The main experiment also assumes exactly two schemers, and only Gemini 2.5 Pro achieves above-chance mutual identification without explicit signalling guidance. Furthermore, the single-schemer control indicates that agents are not always well calibrated about whether another schemer is present. These limitations make it difficult to determine how broadly the findings generalise across models and environments with different numbers of potential collaborators.

  1. The extent to which strategic vagueness explains successful collusion initiation is not fully established.

The authors use two sets of experiments: mutual-identification and the probe of strategic vagueness. The mutual-identification experiments show that agents can sometimes recognise potential collaborators through task-compatible public interactions, and the controlled coordination experiments demonstrate that vague messages can preserve coordination while reducing monitor punishment. However, the second experiment assumes that the accomplice is already known, whereas the first investigates how agents discover one another. It is therefore not entirely clear to what extent the mechanism demonstrated in the controlled game explains successful partner identification in the original setting.

In particular, the proposed advantage of strategic vagueness depends on an asymmetry between the coordination and detection thresholds, which is not observed consistently across the tested models. While the authors provide qualitative analysis of the schemers' scratchpads and examples of successful handshakes, more explicit analysis connecting particular signalling strategies to both identification success and detectability would help establish the mechanism underlying the results. Much of the detailed strategy analysis is also presented in the appendices, and a clearer synthesis in the main text would strengthen the interpretation of the findings.

These concerns do not undermine the central observation that agents can initiate collusion through public communication, but they limit how strongly the results establish strategic vagueness as the explanation for this behaviour.

Questions for authors
  1. How should the findings be interpreted in relation to environments where agents are not explicitly assigned a collusive objective? Is there reason to expect similar partner-discovery behaviour to emerge when agents are pursuing individual objectives that make collusion advantageous? How might the results change when there are more than two potential schemers?

  2. To what extent can the authors establish that strategic vagueness is responsible for the successful handshakes in the mutual-identification experiment? The controlled coordination experiment demonstrates an asymmetry between accomplice coordination and monitor punishment, but assumes that the accomplice is already known. Is there evidence from the mutual-identification trajectories that particular vague signalling strategies are associated with both higher identification success and lower detectability, beyond the descriptive scratchpad analysis?

  3. How sensitive are the reported detection results to the monitoring configuration and false-positive calibration procedure? In particular, would the conclusions about detectability remain similar if the 5% false-positive threshold were calibrated on independent interactions? Are there systematic differences between particular agent–monitor model-family pairings that help explain when monitoring is more or less effective?

5 comments

Question 1: environments where agents are not explicitly assigned a collusive objective?

The main aim of our work is to demonstrate that agents can perform mutual identification through vagueness before initiating collusion, and discover the strategy of plausible deniability on their own without being instructed explicitly. Recent papers have shown instances of emergent collusion between agents when incentives align. It is reasonable to expect that, since the agent possesses these two propensities, they could insert task-plausible instructions in open channels and still be able to carry out communication without being detected by monitors. As we show in this work, partner discovery can happen over rounds of hidden signaling where each agent is sizing up the other before they confirm. Hence, it is not unreasonable to expect that pattern in the wild as well. Our work aims at providing the ground for further investigations in fully realistic scenarios, while such effort would not have been justified without checking the outcomes in more controlled environments.

In our work we have tested the effects of the presence of two colluding agents, while varying the number of honest engineers. We think that trying to do group partner discovery with multiple colluding agents could on one side get harder given the presence of multiple noisy signals. On the other side, a bigger group of colluding agents could cooperate on the group identification task, either through vagueness, or through the opening of initial private channels on which colluders can compare impressions freely. We will add a discussion on this point in the draft, as it provides useful hints for future work in the field.

Question 2: To what extent can the authors establish that strategic vagueness is responsible for the successful handshakes in the mutual-identification experiment?

i) We performed additional quantitative scratchpad analysis, searching all 180 schemers’ private scratchpad notes, up to and including partner identification. The scratchpads were analysed by regex, and we obtained the results described in the following. In 75% of the scratchpads, the schemers used the term “deniable” or “deniability,” and 35% specifically used “plausible deniability” or “plausibly deniable”. These terms do not occur in the strategy-unspecified prompt employed in this case. These lexical counts complement the explicit plans for deniable, contextually appropriate probes documented in Appendix O.2 and the reciprocal uptake documented in Appendix Q. These results highlight that vagueness is strategically employed with the aim of deniability in mutual identification.

ii) We additionally examined the communication used during discovery. Across 90 strategy-unspecified Gemini games, a judge evaluated the directness of messages visible to potential partners, using the same seven-points scale as in Experiment 2, and taking into account the schemers’ instructions and preceding conversation: 94% of the messages received the lowest directness rating (1). Directness is still related to recognition: while 27% of intervals containing a message rated 1 were followed by a clear relative belief update towards the partner, this proportion increased to 76% for the few intervals rated 2 or more.

Partners and monitors differ most on these least direct (Directness 1) messages. Analysing their corresponding belief/suspicion,the fellow schemer's belief in the sender rises significantly after 25% of messages rated 1 while the corresponding figure for the monitors evaluated is 6-16%. But for messages rated 2 or higher , even the stateless per-message Gemini monitor catches up (78-82% against 76% for schemers) .This is the Section 3 counterpart of what we did in Figure 6 where we showed that schemers take faster than monitors can punish, creating asymmetry) The same holds per schemer: schemers whose messages were all rated 1 were still identified by their partner in 62% of cases, but flagged by monitors in only 25–50%. This shows that the belief of the fellow schemer rises faster than the suspicion of the monitor along with successful partner identification, linking strategic vagueness to handshake success.

3.How sensitive are the reported detection results?

Detection rates are sensitive to monitoring configuration, but the main conclusions remain similar when thresholds are calibrated on independent interactions from the same setting. No monitor seems to be consistently strongest with access to conversational context through the monitor scratchpad not delivering consistent results. To answer the reviewer’s question, we tested sensitivity of the results by using three fold cross validation. Across the 36 configurations, mean detection changes from 30.5% to 31.2%, with an observed held-out false-positive rate of 5.6% (range 4.6–7.7%) .The conclusion therefore remains similar under independent calibration within the same setting.

We could not observe consistent patterns in agent-monitor pairings. In each of the configurations (N = 3, 5, 7), Gemini unspecified agents had lower detectability in successful handshakes, obtaining similar scores across monitor families, unlike specified Gemini agents, which had higher detectability and more variation in scores.

Thank you for the response and additional analyses, I appreciate the clarification of the paper's scope and its limitations regarding generalisability.

The discussion of environments where agents are not explicitly assigned a collusive objective is helpful. However, I think it remains uncertain to what extent the observed partner-discovery behaviours would emerge naturally in less controlled multi-agent environments, where agents may need to recognise an existing incentive to collude or develop one through their interactions.

The additional scratchpad and message-directness analyses provide useful evidence connecting strategic vagueness to successful mutual identification, although I would still distinguish this observational evidence from a causal demonstration of the proposed mechanism. I also appreciate the additional cross-validated monitoring results, which substantially address my concern regarding the false-positive calibration procedure.

Overall, I find the rebuttal helpful and believe it strengthens the paper. The remaining limitations primarily concern generalisability and causal interpretation, rather than the validity of the central findings.

Reviewer_KZEoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests collusion-partner discovery by agents with assigned hidden objectives. Some models succeed in monitored public conversation. A separate game examines indirect messages between known accomplices.

Critical strengths

This topic is very interesting and is of central interest to the community.

Critical weaknesses
  1. The formal treatment does not systematically characterize when covert mutual identification is feasible. Beyond elementary payoff calculations, the proposed explanation remains intuitive. Definition 1 assumes a common evidence scale and a coordination–punishment threshold gap, then describes messages within that gap; it neither derives the gap from the agents’ information and incentives nor establishes that suitable messages exist. The paper therefore leaves unclear which shared private information, role-dependent context, and observer capabilities permit reliable mutual recognition while limiting what outsiders can infer. Related questions have been studied formally in "Balfanz et al., Secret Handshakes from Pairing-Based Key Agreements, IEEE Symposium on Security and Privacy, 2003" and "Hopper, Langford, and von Ahn, Provably Secure Steganography, CRYPTO, 2002". These works rely on explicit assumptions about credentials, shared secrets, or cover channels, so their guarantees do not automatically apply here. Although Appendix B discusses concealment and deniability, it does not establish corresponding feasibility conditions. A more systematic account should distinguish hiding group membership, hiding the existence of covert communication, and avoiding punishment despite an understood intention, and clarify which property the proposed setting actually supports.
  2. The relationship to existing social-deduction and strategic-communication benchmarks is insufficiently discussed. The second, known-accomplice experiment shares the tension between helping allies and avoiding exposure to adversarial observers with communication tasks in Werewolf and Avalon. Relevant precedents include "Bailis et al., Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction, arXiv, 2024"; "Light et al., AvalonBench: Evaluating LLMs Playing the Game of Avalon, arXiv, 2023"; and "Wang et al., Boosting LLM Agents with Recursive Contemplation for Effective Deception Handling, Findings of ACL, 2024". The paper should also discuss "Avsian and Heck, SNEAK: Evaluating Strategic Communication and Information Leakage in Large Language Models, arXiv, 2026", which evaluates selective signaling to an ally against adversarial inference. These tasks differ in partner knowledge, hidden identities, and whether the observer infers a secret or imposes sanctions. An explicit comparison is needed to clarify what the proposed experiments add beyond existing role inference, identity concealment, and selective communication.
  3. I find the contribution too limited. The authors appropriately acknowledge the strategic-speaker theory of Pinker et al. (2008) and Lee and Pinker (2010), and explicitly adapt the human economic game from "Massenkoff, DeScioli, Thomas, and Pinker, What happens in vagueness, Journal of Economic Behavior & Organization, 2025". The basic rationale for deniable communication and the coordination-versus-punishment trade-off are therefore already established in prior human research. Applying this paradigm to LLMs and adding repeated feedback provides empirical extensions, but the resulting conceptual advance appears modest.
  4. The experiments do not fully establish why covert coordination succeeds, and Appendix F overstates when vague messages are beneficial.
    • The two experiments answer different questions. The first asks whether agents can discover an unknown partner in a public discussion. The second gives them a known partner and tests whether indirect messages maintain coordination while reducing punishment. Showing this effect in the second game does not establish what caused partner discovery in the first. In the first game, the schemers also share private instructions about their role and identification goal. Their recognition of each other's responses may depend on this shared background. The experiments do not isolate this explanation from the proposed account that partners act on weaker evidence than monitors require to intervene.
    • The repeated-round experiment changes both the feedback and the instructions. Besides providing results from previous rounds, it explicitly tells the speaker to find messages that receive low punishment and make the partner choose TAKE. The single-shot prompt does not state these two goals in the same explicit way. Consequently, the improvement over single-shot performance cannot be attributed to feedback alone.
Questions for authors

Mentioned above.

5 comments

The formal treatment?

We agree that Definition 1 is an operational one restating the model of Pinker et al. (2008), and not a feasibility result, where an indirect offer succeeds due to an evidential mismatch relative to the observer for punishing vs cooperating. The prompts differ between the schemer and the monitor to make it closer to a real world scenario where the monitor has no access to individual schemer prompts. Beyond background differences, the schemer and monitor also face different decision criteria, leading them to act at different levels of evidence.

The cost of decision making influences agents to adopt different evidence thresholds when making decisions. For example, Experiment 2 is a controlled experiment isolating effects of the agents adopting different thresholds for the same information, where the accomplice schemer has a higher take rate for the evidence than the monitor. Even though the monitor in this easy setting can detect collusion (81-99%) of the time, it gives the benefit of the doubt to the schemer in order not to punish innocents. This asymmetry is what vague speech schemers design to exploit.

This is generalised in Experiment 1 in a more realistic setting which does not isolate variables due to its inherent design but shows that agents are capable of figuring out strategies and can perform handshakes using vagueness. We shall make this revision to make it clear to the readers.

Balfanz et al and Hopper et al

We thank the reviewer for these references. Balfanz et al. and Hopper et al. get their guarantees from shared keys and credentials which our setup lacks by design. The agents here have no credentials, shared secrets, cover channels , no guarantee a partner exists. As you rightly said, therefore, there are no guarantees that can apply and whether such recognition exists becomes an empirical one which this paper attempts. We will cite both the works as contrasting settings.

Distinguishing Properties

We agree that we should distinguish these three properties and will do so in Appendix B. Section 3 deals with hiding group membership where schemers have to find themselves in a public channel with monitor observers at a specific false positive rate. Section 4 clarifies how despite understood intention the punishment can still be less, where, despite recognition of collusive attempts (81-99% of cases), the cop still punishes vague speech less than direct.

We do not claim the hiding the existence of covert communication, because the monitor could fail to identify schemers while knowing that some coordination is happening and our experiments do not test this property. Our definition describes a feasibility criterion: there is a gap that exists when the observer does not punish at the same evidence threshold as the schemer. Our experiments do not guarantee whether such a message exists because these depend on the agents’ capability of producing and understanding vagueness and that is an empirical question. Therefore, In Section 3, our experiments show schemers can support mutual recognition in an open channel with limited detection but this is not a given. We will clarify these observations in Appendix B.

2. The relationship to existing social-deduction and strategic-communication

We agree and we will revise the manuscript by comparing the mentioned literature to our work. We plan on adding a detailed analysis that clarifies the following points:

  • In Werewolf Arena and AvalonBench, roles are fixed by the rules, and knowledge of members knowing each other from the start varies by role.
  • ReCon is a method that helps agents in Avalon handle deception to conceal private information
  • As correctly mentioned by the reviewer, SNEAK is most closely related to Section 4 of our paper, it deals with signaling to any ally at the same time limiting adversary inference .

Our study focuses on three specific aspects:

  1. Unlike mutually identified evil teams in Avalonbench, in Section 3, the schemer does not know who the other schemer is, or whether that role exists without a supplied signaling code. They must verify this in an ordinary work task, avoiding contamination from strategies already available in the internet of these popular board games.
  2. SNEAK is interested in measuring informed ally recognition against an adversary. Section 4, instead measures both cooperation and punishment. The cop is usually aware of the collusion attempt in this controlled setting and the question becomes that of understanding a penalty which differs it from prior literature.
  3. We are interested in collusion initiation in multi agents and whether they can find each other while monitors missing these operating in a low false positive constraint and how that changes with monitoring threshold.

3. I find the contribution limited

We respectfully disagree on this point. Our main contribution is related to collusion initiation as partner discovery under monitoring, contained in Section 3, which is normally assumed in other collusion studies and here we break that assumption. Regarding Section 4, we are not aiming for a conceptual contribution with our follow-up experiment. With a similar approach to the one shown in the board-game studies mentioned by the reviewer in the previous point, we test LLMs on a task that humans are known to perform naturally, and we show both empirical differences between LLM and human behaviour, and new insights into LLM specific behavior. Deception (Scheurer et al., strategic deception under pressure) and scheming (Meinke et al., in-context scheming) evaluation literature is established within prior human research as well, but they remain critical safety relevant questions. We believe there is the need to understand whether deployed systems exhibit such capabilities, at what scale, and if oversight is capable of identifying similar behaviour.

4 Appendix F overstates when vague messages are beneficial.

Thanks for this insightful comment. We will clarify the reasons behind the success of covert coordination in the following, and we will correct Appendix F, specifying that effectiveness of vague speech improves the sender’s expected payoff when the reduction in punishment outweighs any loss from reduced accomplice cooperation.

4a The two experiments answer different questions

We do not disagree that this setup involves schemers with a form of shared background. We believe, though, that this is coherent with a real world scenario, where the prompt of schemers would contain common information related to their specular role and goals. Even though they might share some background, they do not share a fixed rulebook of vocabulary, making it difficult to coordinate directly via fixed keywords. Without denying the effect and possible employment of some of the shared background, we show that agents employ vagueness exchanging messages that are dual purpose, in the sense that they both trigger mutual identification, and blend in with normal engineering discussions to achieve plausible deniability. We think that this is still coherent with our definition of vagueness, and our experiments are compatible with a strategic vagueness explanation as well.

i) We ran some additional analysis on all 180 schemers’ private scratchpad notes, up to and including partner identification. In these, 75% used the term “deniable” or “deniability,” and 35% specifically used “plausible deniability” or “plausibly deniable”. These terms do not occur in the strategy-unspecified prompt employed in this case. These lexical counts complement the explicit plans for deniable, contextually appropriate probes documented in Appendix O.2 and the reciprocal uptake documented in Appendix Q. These provide evidence that agents deliberately seek handshakes through communication compatible with ordinary task discussion indicative of strategic vagueness.

ii) In order to highlight the fact that recognition is not driven only by shared background we ran an additional test: in a scenario with N= 5 agents in the Gemini unspecified use case, we gave a stateful monitor the entire schemers’ prompt. This creates a monitor with a background that is completely shared with the schemers. If the information necessary for detection would be contained in the background, we would see the monitor now more capable of recognizing schemers. Instead, the handshake detection rate in this test only jumps from 32% to 35%, illustrating that even with a fully shared background, schemer’s detection can be challenging consistent with the strategic vagueness interpretation.

To further illustrate that schemers do act on weaker evidence than what monitors require to intervene, and that this does not happen due to a shared background, we ran a simple one round game with N=3 agents, and with the monitor again informed with the schemer’s prompt. In this test a schemer posts first, then, on the same message sequence, either the second schemer or the monitor has to decide to commit or punish from that message. Across 50 games, fellow schemers committed in 90% of the games versus monitor punishing in 68% (p=0.013).This shows that even with information held equal and removing other confounds, the same openings move partners to commit to partners more often because monitors more often read them as ordinary.

iii) The choice of false-positive budget changes the monitor’s detection threshold and therefore affects the measured gap between partner and monitor identification rates. In many of the results presented in our work monitors are evaluated at a 5% false-positive rate. Compared with what is allowed by this monitoring criterion, schemers are capable of identifying more schemer partners, but also misidentify ordinary engineers more frequently. For instance, at N = 3 the percentage of schemers identified by their partner is 83%, but 23% of ordinary engineers are also falsely identified as partners by a schemer. The table below compares generic monitors with partners at both false-alarm standards where it is shown that at the schemers’ rate, monitors are able to identify almost at the rates the partners do.

NSchemers identified by their partnerPartners' false-alarm rateMonitors at 5%Monitors at partners' rate
383%23%0–23%15–72%
563%21%3–28%7–50%
752%12%0–37%2–58%

At budgets matching the partners’ false-alarm rates, the best monitors come close to partner identification rates. Conversely, applying a 5% false-alarm standard to partners’ recorded belief scores reduces identification from 70% to 28% in the experiments (these numbers are calculated from schemers' belief scores about their partners). In a separate 30-game N=3 control with the api_key goal sentence removed, a monitor given the schemers’ full instructions and conversation history detected 87% of schemers at the partners’ 40% false-positive rate, compared with 73% for partners; detection fell to 56% at an expected 5% false-positive rate. This shows that the observed detection gap depends substantially on the operating point. Allowing for weaker thresholds for the monitor allows it to better recover schemers, but at the price of increasing the false-positives which many real world scenarios can't afford.These comparisons show that limited detection at a 5% false-positive budget does not, by itself, establish that monitors cannot interpret the signals. This is compatible with strategic vagueness: a message may provide enough evidence for a partner to coordinate while retaining insufficient evidence for a monitor operating under a stricter false-positive constraint.

iv) We additionally examined the communication used during discovery. Across 90 strategy-unspecified Gemini games, a judge evaluated the directness of messages visible to potential partners, using the same seven-points scale as in Experiment 2, and taking into account the schemers’ instructions and preceding conversation: 94% of the messages received the lowest directness rating (1). Directness is still related to recognition: while 27% of intervals containing a message rated 1 were followed by a clear relative belief update towards the partner, this proportion increased to 76% for the few intervals rated 2 or more. Partners and monitors differ most on these least direct (Directness 1) messages. Analysing their corresponding belief/suspicion,the fellow schemer's belief in the sender rises significantly after 25% of messages rated 1 while the corresponding figure for the monitors evaluated is 6-16%. But for messages rated 2 or higher , even the stateless per message Gemini monitor catches up (78-82% against 76% for schemers). This shows that the belief of the fellow schemer rises faster than the suspicion of the monitor, a behavioral signature found in strategic vagueness.

4b The repeated-round experiment

As the reviewer rightly notices, in the repeated-round experiment we do provide agents with both feedback and instructions. To make the comparison with the single-shot case more fair, we ran an additional new control for the zero-shot production, including in the agents’ prompt the same two explicit goals used in the feedback experiment. We ran 50 trials per family. The results show that the instructions included in the prompt made the models talk with increased indirectedness, while still not being properly vague (the mean directness passed from 5.5 to 4.2, and from 6.3 to 5.1, considering the 7-point scale, for Claude and Gemini, respectively). This is in contrast to the sequential experiment where we observed a more significant decrease in indirectness resulting in reduced punishment by the monitor. We will add this observation in the final draft.

Reviewer_yJj4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explored LLM collusion. Specifically, if LLMs placed in an open-task, without knowing who (or if other) colluders existed, would be able to establish partnerships to engage in collusion.

This was done by running simulations of a fake engineering slack-type channel where public messages would be posted for all to see (including by a monitor). The claim by the authors is that LLMs are able to engage in vague-posting in such a manner where it is detectable by partners but not by a cop LLM seeking to prevent collusion.

The conclusions are LLMs are capable of establishing partnerships through vague partnerships and cop LLMs are poor at detecting collusion.

Critical strengths

This work does not take for granted that the colluding parties know each other. This testing of an important assumptions is both the paper’s main contribution/reason to accept. Studying LLMs’ ability to perform this is important and the methodology is largely valid.

Critical weaknesses
  • CW1: If LLM A is a schemer, then LLM B (other schemer) is able to, from all of the public messages, establish that LLM A is a schemer.

    • Yet, LLM C(op) is, from all of the public messages, unable to identify either LLM A or LLM B as the schemer? Given that LLM B and LLM C are working off the same information, the difference in performance would just be caused by your prompting of B and C? Theoretically they should be able to achieve the same performance such that theta_c = theta_d (and there is no room for vagueness)? If the authors believe otherwise, I’d be interested to hear why but it’s not really explained in the paper. Unless the weaker prompting of LLM C is purposeful in which case the results would hold but it’d be weaker.
    • Could it just be phrase/topic “api_key” raises the flag for collusion? This work assumes that the colluding parties are already aligned on what to collude on and just have to find each other. This isn't unreasonable but it's a much weaker set up than what is being sold/advertised. If this is the intentional set-up then I'd more strongly highlight/change framing.
    • If this is not the case, I think the author's would need to have additional experiments to show what happens if another LLM D (colleague) discusses that topic api_keys without seeking to collude? Would they be guessed as colluding by LLM A?
  • CW2: A framing weakness. In the real-world (think betting markets), collusion is not caught by policing speech but by analyzing results for statistical anomalies. Capturing speech/intent would definitely make it easier but it’s not required per se. A small discussion of when analyzing results is/isn’t possible applicable could improve the perceived impact of the paper.

Questions for authors
  • How were prompts arrived at? Any tuning of prompts?
  • Clarification to improve work (likely won’t affect rating): What exactly is meant by scaffolding? It’s not well defined in the work or the appendix. The authors have to read the prompts to infer what you mean by it, and to me, it’s not clear that scaffold is the best term.
3 comments

CW1a. Why can a partner identify the schemer when the monitor cannot?

There are two tiers of mismatch between schemer B and cop C: one is a supplied background which is a real world instantiation of this experiment (where schemers and monitors are separate) and the other is a decision criteria (blackstone’s ratio). The monitor prompt differs from the schemers’ one. We believe this to be coherent with a real world scenario, where the monitor does not have access to the prompt of individual agents, while the prompt of schemers match closely. Beyond differences in supplied background related to the different roles of the agents, the schemer and monitor also face different decision criteria, which may lead them to act at different levels of evidence.

Given the same amount of evidence on the role of A, B and C, they take different actions when the cost of mistaken decisions differ. This can be seen in Experiment 2 directly where even though all have the same information, indirect proposals can preserve accomplice cooperation while receiving lower punishment. Even though the LLM cops might recognise collusive intent, their implicit evidence threshold for punishment is higher than the schemers acceptance threshold for vague speech, creating an asymmetry which plausible deniability exploits. We ran a control of this in Experiment 1 where we gave the full schemer instruction to the monitor. Now that both the schemer and the monitor have full information we still do not see a closing of the gap in handshake detection where for n=5, the handshake detection rises from 32% -> 35% at 5% false positive rate in the stateful monitor. This shows that even with the schemers full prompt, at usually deployed false positive rates, monitors struggle to punish schemers without leaving out normal engineers due to strategic vagueness embedding handshakes in normal conversations.

To further illustrate differences between acceptance vs punishment thresholds, we ran a simple one round n=3 game where the schemer posts first and the second schemer or the monitor has to decide to commit or punish from that message. Across 50 games, fellow schemers committed in 90% of the games versus monitor punishing in 68% (p=0.013). This shows that even with information held equal and removing other confounds, the same openings move partners to commit to partners more often because monitors more often read them as ordinary.

In Experiment 1, although the retrospective monitor sees the same public communication as the schemers, their prompts provide different background information and objectives. The schemers still have no prescribed signalling book to follow and must identify each other through plausibly deniable phrases embedded in conversations, achieving a dual purpose.

Some evidence points this way:

i) Monitors were instructed to look for double meaning phrases and colleagues echoing each other in Appendix M.0.4. To strengthen these monitors further, we supplied strategies learned from other conversations. Both types of monitors only catch some of the schemers, reinforcing the idea that knowing the strategies employed by schemers does not reliably translate to detecting them.

ii) The choice of the false-positive rate affects the gap between theta_c and theta_d. In many of the results presented in our work monitors are evaluated at a 5% false-positive rate. Compared with what is allowed by this monitoring criterion, schemers are capable of identifying more schemer partners, but also misidentify ordinary engineers more frequently. For instance, at N = 3 the percentage of schemers identified by their partner is 83%, but 23% of ordinary engineers are also falsely identified as partners by a schemer. The table below compares generic monitors with partners at both false-alarm standards where it is shown that at the schemers’ rate, monitors are able to identify close to the rates the partners do.

NSchemers identified by their partnerPartners' false-alarm rateMonitors at 5%Monitors at partners' rate
383%23%0–23%15–72%
563%21%3–28%7–50%
752%12%0–37%2–58%

At budgets matching the partners’ false-alarm rates, the best monitors come close to partner identification rates. Conversely, applying a 5% false-alarm standard to partners’ recorded belief scores reduces identification from 70% to 28% in the experiments (these numbers are calculated from schemers' belief scores about their partners).

iii) We additionally examined the communication used during discovery. Across 90 strategy-unspecified Gemini games, a judge evaluated the directness of messages visible to potential partners, using the same seven-points scale as in Experiment 2, and taking into account the schemers’ instructions and preceding conversation: 94% of the messages received the lowest directness rating (1). Directness is still related to recognition: while 27% of intervals containing a message rated 1 were followed by a clear relative belief update towards the partner, this proportion increased to 76% for the few intervals rated 2 or more. Partners and monitors differ most on these least direct (Directness 1) messages. Analysing their corresponding belief/suspicion,the fellow schemer's belief in the sender rises significantly after 25% of messages rated 1 while the corresponding figure for the monitors evaluated is 6-16%. But for messages rated 2 or higher , even the stateless per-message Gemini monitor catches up (78-82% against 76% for schemers). This shows that the belief of the fellow schemer rises faster than the suspicion of the monitor, a behavioral signature found in strategic vagueness.

iv) We searched all 180 schemers’ private scratchpad notes, up to and including partner identification. In these, 75% used the term “deniable” or “deniability,” and 35% specifically used “plausible deniability” or “plausibly deniable”. These terms do not occur in the strategy-unspecified prompt employed and the LLM figures out this strategy on its own. These lexical counts complement the explicit plans for deniable, contextually appropriate probes documented in Appendix O.2 and the reciprocal uptake documented in Appendix Q.

v) In Section 4, the accomplice and monitor receive the same public message and game rules. In their coded reasoning in this narrow setting, monitors are able to recognise vague messages as collusion attempts in 81–99% of traces, yet impose lower penalties than for explicit collusive speech. The reasoning traces document monitors recognising the collusive intention while citing the message’s deniability as a reason to withhold the full penalty (Appendix R, Tables 14–15). This illustrates that recognising an intention and deciding how strongly to punish it are distinct: understanding the signal does not necessarily eliminate the advantage of strategically vague speech.

Could it just be phrase/topic “api_key”

We agree that this is an interesting hypothesis to be verified. In order to do so, we ran a test by removing the api_key topic, so that there is no shared keyword. Results included in the table below still show an above chance probability of handshake indicating that the token api_key is not necessary to perform a handshake.

Group sizeOriginalTopic removedChance
N = 370.0%53.3%25.0%
N = 543.3%50.0%6.3%
N = 736.7%23.3%2.8%

If another LLM D (colleague) discusses that topic api_keys without seeking to collude?

To test if an honest colleague discussing API keys without collusion intention, could be mistaken for a schemer’s partner, we conducted an additional control experiment. Across 90 games with N = 3, 5 and 7, we modified the prompt of one honest engineer, to assign them the responsibility of API key handling, without instructing them to collude.We instructed a normal engineer to talk about API keys in the conversation, by changing the prompt of one of the normal engineers. Across 90 games across N = 3, 5 and 7, assigning one honest colleague responsibility for API key handling. Before commitment, 46% of this colleague’s messages mentioned API keys, compared with 7% of the true partner’s messages. Across the whole 180 schemer decisions, the honest colleague received 46 commitments, versus 23 commitments for the same colleague in matched runs without the API key assignment. The true partner nevertheless received 95 commitments in the control experiment, more than double of the commitments received for the decoy. Moreover, handshake rates remained statistically indistinguishable for the original and the control experiments. Thus, API key emphasis does attract false identifications, but does not fully explain schemers’ partner selection.

CW2: A framing weakness.

We agree that monitoring communications is not necessarily required for the detection of collusion. As you mentioned , analysing behaviour and outcomes is particularly effective when the behaviour is observable, repetitive, and comparable against reliable baseline values like structured pricing or tendering. Our setting concerns the detection of partners within ongoing collaborative workflows. In software development and research tasks characterised by a high degree of freedom, outcomes may be sparse, delayed, or only partially observable; consequently, the successful completion of a legitimate task alone does not necessarily reveal that agents are forming covert alliances. Therefore, communication can serve as an additional source of evidence regarding collaboration. We agree that this is an important point motivating the work and we will include a discussion on this in the manuscript, clarifying that this experiment investigates the challenges involved in detecting the onset of collusion and monitoring the associated communications in such settings.

How were prompts arrived at? Any tuning of prompts?

In both of the presented experiments, we have not performed any tuning of prompts. The specified agents were prompted with language akin to those used in Pinkers indirect speech theories. The stronger monitors in Appendix P were given strategies that were distilled from an LLM as a judge pipeline and fed to a held out dataset. We also tried a variant of the schemer prompt where we say there is exactly one other schemer to find and got similar handshake success rates. In light of this discussion, we think it is a good idea to include such results in the Appendix.

What exactly is meant by scaffolding?

In this work, we employed the term “scaffolding” to indicate signaling guidance given to Claude and GPT agents. Differently from them, Gemini did not need any explicit signaling guidance in order to perform the task. We agree with the reviewer that this term should be defined and used more precisely, and we plan to modify the manuscript in this sense.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.