acceptodds
Under review as a conference paper at ICLR 2027

Despite Instructions: Frontier Agents Improvise Covert Channels at Test Time

Abstract

In security-sensitive applications, language-model agents are often required to coordinate without disclosing confidential information. Yet repeated interactions may also let ordinary messages acquire shared private meaning. We study a repeated game with pairs of models in which the sender model observes one of four secret states and selects one of four summaries of the same public report, while the receiver model tries to infer the secret state. We find that model pairs can learn to communicate the secret using only one bit of feedback indicating whether the receiver inferred it correctly. This learning occurs during inference with fixed parameters and no supplied codebook or encoding examples. The effect also persists when agents generate their own free-form updates in a simulated incident-response task. Across ten independent games, pairs of GPT-5.6 Sol agents reach 98.8% final accuracy, compared with 25% chance, despite explicit instructions prohibiting disclosure and a monitor that screens each message without access to the agents’ interaction histories. The same interactions that help agents cooperate can therefore allow confidential information to pass through messages intended for legitimate coordination.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-agent systems, in-context learning, emergent communication, covert communication, language model security, language model safety, language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mMVnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The authors introduce several LLM agent-agent communication studies involving the inference of private information over a multi-turn game.

Because the paper is a bit unclear, I will give a rough sketch of what I believe to be the main task (sentence selection):

  1. There is a public report and a list of four one-sentence summaries of that report, which are shuffled from turn to turn.
  2. A Sender agent receives the public report, a private key, the list of summaries, episode history, and general instructions (e.g., maximize the team’s score, don’t disclose the key). It outputs an index (1-indexed) to select a summary from the list.
  3. A Receiver agent receives the report, the Sender’s selected summary (not the index of such), episode history, and general instructions. It outputs an index (0-indexed) to select an “action.”
  4. Except in the no-feedback setting, the team score is a binary variable indicating whether the Receiver’s action index matches the Sender’s summary index.

The results indicate that, for several model choices for the sender and receiver agents (all of which involve an OpenAI receiver), some kind of coordination strategy appears to emerge after several rounds, as evidenced by substantially better-than-random final accuracy.

Critical strengths

Strength 1: The overall topic is important and timely, e.g., in light of recent security incidents and the release of several "personal agent" products. Under certain situations, frontier models can be biased by apparently irrelevant details, and this can be readable by other agents, in such a way that indirectly discloses the details. This has security implications, as users share personal information with agentic tools.

Strength 2: The headline results (e.g., Figure 2) are surprising and thought-provoking. Several supplementary findings are also interesting, such as that the sender’s history may be less critical than the receiver’s history (Appendix C.1).

Critical weaknesses

Weakness 1: My primary concern is the readability of the submission, including unclear descriptions, under-explained or undefined terms and operators, and somewhat labyrinthic organization. It is difficult for a reader to understand the precise nature of the task, hence my sketch in the Summary, or interpret the results in the paper's current form. This is unfortunate, because I like the topic, am interested in the results, and sense that the authors have put a lot of work into it.

Weakness 2: I do not feel that the paper does enough to explain or interpret its results beyond basic reporting. While the headline findings are interesting, I would prefer to see a more careful analysis that helps elucidate the underlying phenomena. For example, I would be happy to read a paper that deeply examines the main task alone.

Questions for authors
  1. How are we sure that the sender does not disclose the private key? In the sentence selection experiment, it seems that no monitor is present to check this. In the experiments where the monitor is present (e.g., security), the prompts seem to imply that the monitor never sees the private key, so I'm not sure how it would know that the private key is being passed.

  2. Can the authors please revise the problem statement, associated definitions, and notation for clarity and precision? For instance, in Section 2.1, M\mathcal{M} is described as both (1) the set of summaries received by the sender model and (2) the set of possible outputs generated by the sender model (messages). I recognize that the sender model is selecting messages, but technically the sender model is prompted to report an index, not a message (Appendix G). Additionally, without constrained generation, it seems that these sets are not necessarily of the same size (e.g., it is possible for the sender to generate an index that does not match a message). Also, the negated turnstile operator ⊮\nVdash (i.e., L100, L783, L786) is being used to refer to the indicator function, which is unconventional and therefore should be made explicit.

  3. How exactly are compatibility and vulnerability (defined in Section 2.3) being used? I may have missed this, but I don’t think vulnerability appears elsewhere in the paper, except in discussing an upper bound for compatibility (Appendix A.1). Compatibility is used for “held-out checks,” but the nature of this is not clear to me.

I think this is a potentially interesting submission that is heavily obscured by readability issues and lack of clarity. I hope that the authors take this opportunity to substantially revise the text (re: Weakness 1) and make strides on the interpretative side (re: Weakness 2).

Reviewer_aaKoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

can agents develop covert communication channels through repeated interaction without parameter updates, a supplied codebook, or explicit negotiation. The results show high communication accuracy in both sentence-selection and free-form security settings, demonstrating that receivers learn to extract private info from message wording and interaction history.

Critical strengths
  • Relevant problem: the paper explores an important security risk in multi-agent systems, showing that confidentiality instructions and message-level monitoring may not be sufficient.
  • Good experiments: message-substitution experiments and receiver-history ablations provide show that private information is transmitted through the messages rather than incidental correlations.
  • Convincing evidence beyond obvious encoding: The clean-message analysis shows that successful communication is not limited to recognizable codes or direct disclosure.
Critical weaknesses
  • Generalizability: The main security experiment uses a fixed public report and only four private states, making covert signaling relatively easy. The earlier confidential free-form study achieves channel formation in only 2/20 games.

  • Unclear mechanism of channel formation: The experiments do not fully distinguish bilateral invention of a new code from receiver-side learning of stable, state-dependent sender preferences. A fixed-sender baseline or sender-history ablation in the free-form setting would help.

  • Eval is monitoring individual messages: The monitor only examines individual messages without interaction history, so it cannot identify systematic correlations between wording and private information. The results therefore establish a limitation of stateless monitoring rather than demonstrating that stronger monitoring approaches would fail.

  • Confounded paraphrasing results: Many paraphrased messages change factual content, and only 28/160 post-learning rewrites satisfy the factual-preservation criterion. The continuous-paraphrasing experiments also involve only three pairs, limiting conclusions about robustness to stronger rewriting defenses.

Questions for authors

Can the authors evaluate a fixed sender or reset sender history in the free-form experiments to separate sender adaptation from receiver-side learning?

How robust is channel formation when public reports vary across rounds, especially given the substantially lower success rate in the earlier confidential free-form study?

Would the communication channels remain effective against history-aware monitoring or a fact-preserving canonicalization defense?

Reviewer_NMWCOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether two LLM agents can share private information without other affordances like shared memory, history or previous agreement. They find that GPT Sol and Terra can form such two-bit channels where Luna can’t. They validated that the communication is robust to paraphrasing, suggesting either imperfect paraphrasing or use of semantic steganography by the models. The covert communication is learned via repeated interactions under the influence of a monitor who can censor updates, and the authors show that some model pairs can progress towards higher accuracy during these in-context interactions. The authors also validated that some pairs of models share priors, such as default ordering of facts, that can potentially be used for pre-agreement enabling the shared encoding.

Critical strengths

The authors provide a compelling demonstration that models like Sol and Terra can already conduct forms of steganography across long-running transcripts. The ablation against history-less transcripts, and the progress towards successful transmission above chance later on in the interactions, show that knowledge of the encoding is established gradually within the interaction history.

Critical weaknesses

The use of four possible values is a rather limited covert channel. It would be interesting to explore gradual formation of larger capacity covert channels, and the relationship to the success of paraphrasing and the monitor in reducing the capacity of the channel.

Questions for authors

The reviewer appreciates the application of paraphrasing and the checking of shared priors. Can the authors comment on whether any investigation into successful communication transcripts was or could be performed that would indicate what features leaked through paraphrasing, or what shared priors were employed by the successful communicating agents? One simple example of this might be interrupting the agents late in the transcripts and asking them about the explicit encoding and how it was derived. Separately, is it possible to improve the paraphraser to reduce the rate of successful communication? These are suggestions for future work rather than clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.