- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates collusion among LLM agents in multi-agent systems, focusing specifically on the initiation step, referred to as collusion initiation, when two schemers need to identify each other while communication is openly observable within the group of agents. The authors argue that existing work often assumes that colluding agents already know their partners, overlooking the initial identification step, which may need to occur through publicly observable communication. They find that agents are successful at identifying ‘handshake strategies’ without the need to explicitly be told to follow these. The experiments also look at monitoring of this behaviour, and whether the collusion initiation goes undetected.
The authors study the problem using a mutual-identification game in which two schemers, whose identities are unknown to one another, must identify each other while participating in an ordinary collaborative task with other agents. They evaluate Gemini 2.5 Pro, GPT-5.1, and Claude Sonnet 4.6, comparing conditions where agents are explicitly guided to use strategic vagueness with conditions where no particular signalling strategy is specified. They find that Gemini is the only model to achieve successful mutual identification above chance without explicit signalling guidance, while other models improve when given guidance. They also evaluate the detectability of these handshakes across different monitoring architectures and model families, finding that many successful handshakes remain undetected at low false-positive rates.
The authors then introduce a controlled schemer–accomplice–monitor game to investigate strategic vagueness more directly, isolating it from other components of the interaction. They find that, for some models, indirect messages can preserve coordination with an accomplice while reducing punishment by an observer. They interpret this as an asymmetry between the evidence required for a partner to coordinate and that required for a monitor to intervene.
Critical strengths
-
The paper identifies an underexplored problem in multi-agent collusion. As the authors note, much of the existing work on collusion in multi-agent systems assumes that agents already know who their potential collaborators are. This paper instead investigates the preceding step of identifying other agents with shared hidden objectives, while communicating through a public channel. This is a timely and relevant problem as LLM agents increasingly interact and collaborate in multi-agent environments. The distinction between collusion initiation and collusion execution is consequential, as it highlights a potential vulnerability that may be missed by evaluations focusing only on established collusion or explicitly malicious communication.
-
The paper is clearly presented and supported by a thorough experimental analysis. The experiments are well motivated and designed to examine different aspects of the proposed problem. The mutual-identification game distinguishes between the ability to discover a signalling strategy and the ability to execute one when explicitly guided. The authors also examine detectability across different monitor architectures and model families, including monitors with additional knowledge of signalling strategies. The second experiment provides a more controlled investigation of strategic vagueness, with ablations examining the effects of messages and payoff information on coordination. The combination of these experiments provides useful evidence that agents can establish covert handshakes through seemingly ordinary communication, and that monitoring such behaviour can be difficult without producing excessive false positives. The paper also engages well with the relevant literature and clearly explains its contributions in relation to existing work.
Critical weaknesses
- The experimental settings are highly stylised, limiting the generalisability of the findings to real-world systems.
In the mutual-identification game, schemers are explicitly assigned a hidden objective and instructed to identify potential collaborators. Although the strategy-unspecified condition demonstrates that some agents can discover signalling strategies without being explicitly told how to do so, it does not establish whether agents would independently develop collusive objectives or initiate collusion in environments where they have not been instructed to do so. It would be interesting to understand whether similar behaviour would emerge from agents pursuing individual objectives under incentives that make collusion beneficial.
The main experiment also assumes exactly two schemers, and only Gemini 2.5 Pro achieves above-chance mutual identification without explicit signalling guidance. Furthermore, the single-schemer control indicates that agents are not always well calibrated about whether another schemer is present. These limitations make it difficult to determine how broadly the findings generalise across models and environments with different numbers of potential collaborators.
- The extent to which strategic vagueness explains successful collusion initiation is not fully established.
The authors use two sets of experiments: mutual-identification and the probe of strategic vagueness. The mutual-identification experiments show that agents can sometimes recognise potential collaborators through task-compatible public interactions, and the controlled coordination experiments demonstrate that vague messages can preserve coordination while reducing monitor punishment. However, the second experiment assumes that the accomplice is already known, whereas the first investigates how agents discover one another. It is therefore not entirely clear to what extent the mechanism demonstrated in the controlled game explains successful partner identification in the original setting.
In particular, the proposed advantage of strategic vagueness depends on an asymmetry between the coordination and detection thresholds, which is not observed consistently across the tested models. While the authors provide qualitative analysis of the schemers' scratchpads and examples of successful handshakes, more explicit analysis connecting particular signalling strategies to both identification success and detectability would help establish the mechanism underlying the results. Much of the detailed strategy analysis is also presented in the appendices, and a clearer synthesis in the main text would strengthen the interpretation of the findings.
These concerns do not undermine the central observation that agents can initiate collusion through public communication, but they limit how strongly the results establish strategic vagueness as the explanation for this behaviour.
Questions for authors
-
How should the findings be interpreted in relation to environments where agents are not explicitly assigned a collusive objective? Is there reason to expect similar partner-discovery behaviour to emerge when agents are pursuing individual objectives that make collusion advantageous? How might the results change when there are more than two potential schemers?
-
To what extent can the authors establish that strategic vagueness is responsible for the successful handshakes in the mutual-identification experiment? The controlled coordination experiment demonstrates an asymmetry between accomplice coordination and monitor punishment, but assumes that the accomplice is already known. Is there evidence from the mutual-identification trajectories that particular vague signalling strategies are associated with both higher identification success and lower detectability, beyond the descriptive scratchpad analysis?
-
How sensitive are the reported detection results to the monitoring configuration and false-positive calibration procedure? In particular, would the conclusions about detectability remain similar if the 5% false-positive threshold were calibrated on independent interactions? Are there systematic differences between particular agent–monitor model-family pairings that help explain when monitoring is more or less effective?
5 comments
3.How sensitive are the reported detection results?
Detection rates are sensitive to monitoring configuration, but the main conclusions remain similar when thresholds are calibrated on independent interactions from the same setting. No monitor seems to be consistently strongest with access to conversational context through the monitor scratchpad not delivering consistent results. To answer the reviewer’s question, we tested sensitivity of the results by using three fold cross validation. Across the 36 configurations, mean detection changes from 30.5% to 31.2%, with an observed held-out false-positive rate of 5.6% (range 4.6–7.7%) .The conclusion therefore remains similar under independent calibration within the same setting.
We could not observe consistent patterns in agent-monitor pairings. In each of the configurations (N = 3, 5, 7), Gemini unspecified agents had lower detectability in successful handshakes, obtaining similar scores across monitor families, unlike specified Gemini agents, which had higher detectability and more variation in scores.
Thank you for the response and additional analyses, I appreciate the clarification of the paper's scope and its limitations regarding generalisability.
The discussion of environments where agents are not explicitly assigned a collusive objective is helpful. However, I think it remains uncertain to what extent the observed partner-discovery behaviours would emerge naturally in less controlled multi-agent environments, where agents may need to recognise an existing incentive to collude or develop one through their interactions.
The additional scratchpad and message-directness analyses provide useful evidence connecting strategic vagueness to successful mutual identification, although I would still distinguish this observational evidence from a causal demonstration of the proposed mechanism. I also appreciate the additional cross-validated monitoring results, which substantially address my concern regarding the false-positive calibration procedure.
Overall, I find the rebuttal helpful and believe it strengthens the paper. The remaining limitations primarily concern generalisability and causal interpretation, rather than the validity of the central findings.
Question 1: environments where agents are not explicitly assigned a collusive objective?
The main aim of our work is to demonstrate that agents can perform mutual identification through vagueness before initiating collusion, and discover the strategy of plausible deniability on their own without being instructed explicitly. Recent papers have shown instances of emergent collusion between agents when incentives align. It is reasonable to expect that, since the agent possesses these two propensities, they could insert task-plausible instructions in open channels and still be able to carry out communication without being detected by monitors. As we show in this work, partner discovery can happen over rounds of hidden signaling where each agent is sizing up the other before they confirm. Hence, it is not unreasonable to expect that pattern in the wild as well. Our work aims at providing the ground for further investigations in fully realistic scenarios, while such effort would not have been justified without checking the outcomes in more controlled environments.
In our work we have tested the effects of the presence of two colluding agents, while varying the number of honest engineers. We think that trying to do group partner discovery with multiple colluding agents could on one side get harder given the presence of multiple noisy signals. On the other side, a bigger group of colluding agents could cooperate on the group identification task, either through vagueness, or through the opening of initial private channels on which colluders can compare impressions freely. We will add a discussion on this point in the draft, as it provides useful hints for future work in the field.
Question 2: To what extent can the authors establish that strategic vagueness is responsible for the successful handshakes in the mutual-identification experiment?
i) We performed additional quantitative scratchpad analysis, searching all 180 schemers’ private scratchpad notes, up to and including partner identification. The scratchpads were analysed by regex, and we obtained the results described in the following. In 75% of the scratchpads, the schemers used the term “deniable” or “deniability,” and 35% specifically used “plausible deniability” or “plausibly deniable”. These terms do not occur in the strategy-unspecified prompt employed in this case. These lexical counts complement the explicit plans for deniable, contextually appropriate probes documented in Appendix O.2 and the reciprocal uptake documented in Appendix Q. These results highlight that vagueness is strategically employed with the aim of deniability in mutual identification.
ii) We additionally examined the communication used during discovery. Across 90 strategy-unspecified Gemini games, a judge evaluated the directness of messages visible to potential partners, using the same seven-points scale as in Experiment 2, and taking into account the schemers’ instructions and preceding conversation: 94% of the messages received the lowest directness rating (1). Directness is still related to recognition: while 27% of intervals containing a message rated 1 were followed by a clear relative belief update towards the partner, this proportion increased to 76% for the few intervals rated 2 or more.
Partners and monitors differ most on these least direct (Directness 1) messages. Analysing their corresponding belief/suspicion,the fellow schemer's belief in the sender rises significantly after 25% of messages rated 1 while the corresponding figure for the monitors evaluated is 6-16%. But for messages rated 2 or higher , even the stateless per-message Gemini monitor catches up (78-82% against 76% for schemers) .This is the Section 3 counterpart of what we did in Figure 6 where we showed that schemers take faster than monitors can punish, creating asymmetry) The same holds per schemer: schemers whose messages were all rated 1 were still identified by their partner in 62% of cases, but flagged by monitors in only 25–50%. This shows that the belief of the fellow schemer rises faster than the suspicion of the monitor along with successful partner identification, linking strategic vagueness to handshake success.