acceptodds
Under review as a conference paper at ICLR 2027

Detecting Multi-Agent Collusion Through Multi-Agent Interpretability

Abstract

Non-verbalised or secret collusion may impose a ceiling on transcript-level monitoring in multi-agent deployments. Internal model activations are known to provide an additional monitoring surface for single-agent phenomena such as deception, but detecting collusion within an unknown subgroup is ultimately a joint inference task over multiple agents. This paper derives collusion detection statistics based on the composite hypothesis testing literature. Using per-agent activation probes as sensors, we then evaluate these principled statistics alongside heuristics on NARCBench, a novel benchmark for evaluating multi-agent collusion detection under environment distribution shift. Across four open-weight models, both our group-level statistics and heuristics transfer zero-shot to NARCBench's holdout environments. Notably, the best method on every model reaches 0.99 to 1.00 AUROC when transferred from committee deliberations to a steganographic collusion task that unprimed transcript monitors fail to separate. We then evaluate detector transfer to third-party environments with emergent, rather than instructed, collusion. In one of these environments, only 2 of 229 collusive episodes verbalise the coordination in the transcript. We find evidence of partial transfer to these environments, indicating that probes fitted to instructed collusion may not fully capture emergent collusion signals. In line with our theoretical analysis, we find that no single statistic or heuristic dominates empirically across tasks. Overall, we establish multi-agent interpretability as a complementary monitoring technique, while highlighting the need to evaluate detector performance on both emergent and instructed collusion risks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-agent safety, interpretability, alignment, linear probes

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nAE8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies multi-agent collusion detection through internal model activations. It combines per-agent activation probes with group-level detection statistics and introduces NARCBench to evaluate generalization across committee deliberations, adversarial workflows, and steganographic communication. Results show strong transfer across instructed-collusion environments, while transfer to emergent collusion remains partial.

Critical strengths

The evaluation provides compelling evidence that internal activations offer a complementary monitoring signal for covert coordination. In particular, transferring detectors from committee deliberations to steganographic blackjack achieves near-perfect AUROC with the best method on each model. Testing structurally different tasks and emergent collusion also helps establish the practical boundaries of the approach.

Critical weaknesses

Cross-model portability remains insufficiently established. The main experiments train a separate probe on each model’s activations, so their zero-shot results primarily demonstrate cross-environment generalization. They do not establish that a detector trained on Qwen can be directly applied to Llama. Although Appendix N reports an exploratory cross-model experiment, the very limited positive samples and insufficient explanation of representation alignment make its conclusions difficult to assess. This leaves applicability to changing or heterogeneous agent systems uncertain. Clarifying the cross-model implementation and evaluating frozen-detector transfer against model-specific retraining would strengthen the paper.

Questions for authors

How is the cross-model performance?

3 comments
AuthorsOpenReview ↗

Response to reviewer nAE8

We thank the reviewer for their time and effort in reviewing our work and the acknowledgement of our paper’s strengths, including the strong performance of our probing methods on the out-of-distribution steganography dataset. We also appreciate the constructive feedback provided and welcome the opportunity to provide clarification and new experimental results.

As the reviewer notes, the cross-model experiment was an exploratory experiment in our Appendices and we do not consider its result within our core contributions. The experiment in Appendix N shows poor probe performance, however, it is uncertain whether this is due to the limited positive samples, the generalisation to emergent collusion or if it is due to the cross-model set-up.

Even though more thorough experiments on cross-model probing is left to future work, we provide below some further results where we use probes trained on the CORE data generated by Qwen-32B and test them on TRANSFER and STEGO data generated by the other three models. To account for the dimension size variations in the different models’ activation spaces, we perform dimension truncation. Most probe methods perform poorly, however cross model performance on the steganography dataset is still high when transferred to the Llama-70b model. These results provide support for further exploration of cross-model performance.

Qwen-32B to OSS-20B

GroupSorted ConcatActivation AlignmentPeak SuspicionDeception SplitAsymmetry Probe
CI/CD Pipeline0.6610.9780.7780.8610.850
Code Review0.3280.5720.1890.2890.183
Research Pipeline0.3200.6000.3200.4000.280
Blackjack Stego0.7630.9930.4040.6160.578

Qwen-32B to DSR1-distilled-32B

GroupSorted ConcatActivation AlignmentPeak SuspicionDeception SplitAsymmetry Probe
CI/CD Pipeline0.6060.6670.8500.5110.789
Code Review0.6110.9110.5940.7330.667
News Framing0.6170.5000.6500.4720.633
Pump & Dump0.3830.6110.2890.3940.350
RAG Poisoning0.6610.5110.5280.4110.494
Research Pipeline0.5890.2890.3390.4780.383
Blackjack Stego0.8340.7580.4040.4700.383

Qwen-32B to Llama-70B

GroupSorted ConcatActivation AlignmentPeak SuspicionDeception SplitAsymmetry Probe
CI/CD Pipeline0.4330.8220.3000.6670.422
Code Review0.4890.6670.4220.7830.622
News Framing0.6280.8330.4610.4220.394
Pump & Dump0.3330.9390.2670.7610.344
RAG Poisoning0.6500.4440.4720.6110.589
Research Pipeline0.5110.6330.3500.5890.450
Blackjack Stego0.7861.0000.9950.9680.993

Thanks for the authors' response. The new results help a lot in improving the soundness of this paper. I will raise the soundness score and keep the overall one.

AuthorsOpenReview ↗

New Rebuttal Version

We have posted a revised version of our paper for the rebuttal phase. We hope it addresses your concerns.

In addition to addressing the concerns raised/commitments we made for specific revisions, it substantially strengthens and provides further detail on the theoretical and statistical foundations of our approach and setting. It expands our hypothesis-testing framework into a full environmental setting we call HELIOS, in the main body and Appendix A.

We hope you will take a look, and if it addresses your concerns, we also hope that you will consider revising your score appropriately.

Additionally, we previously posted a reply to reviewer N7D6 on the relevance of our approach and detection procedure that you may find useful. We hope you will take a look at it as well. https://openreview.net/forum?id=zNolMfwGC7&noteId=1JESCAd0bE

Thank you for your help in improving the paper!

Reviewer_jn7pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-agent collusion detection using internal activations. It derives group-level detection statistics based on composite hypothesis testing for settings where the colluding subgroup is unknown, and evaluates these statistics with several heuristics. It uses activation probes as sensors and introduces NARCBench to evaluate detection under environment distribution shift. The methods are evaluated for zero-shot transfer from CORE to TRANSFER and STEGO. The paper also evaluates transfer to external environments with emergent collusion and reports limitations.

Critical strengths

1.This paper studies collusion detection at the group level. The composite hypothesis testing framework is important because the colluding subgroup is usually unknown. The paper also compares different detection statistics and shows that no single method works best across all settings.

2.This paper introduces NARCBench with different collusion scenarios and evaluates zero-shot transfer across multiple models. The results on STEGO are especially interesting, as the agents communicate through hidden signals that are difficult to identify from text alone.

Critical weaknesses

1.The ground-truth labels in the emergent collusion evaluation are not sufficiently reliable. In Verdict, 229 episodes are labeled as collusive, but only 2 explicitly verbalize collusion reasoning. The authors also acknowledge that some cases may be due to poor judgment. This makes the evaluation results difficult to interpret. In addition, several methods perform close to chance on Verdict.

2.The strong detection results on STEGO are not fully explained. Also, Appendix L.8 shows that a stronger few-shot text-based judge can detect all collusion cases without false positives. These results raise concerns about the added value of activation-based monitoring.

Questions for authors

1.How are ground-truth labels determined in Verdict and how do the authors distinguish intentional collusion from poor judgment.

2.Why do the probes achieve near-perfect AUROC on STEGO? What information in the hidden states accounts for this performance?

3.How does the choice of layers affect detection performance?

4.Given the few-shot text-based judge results in Appendix, what are the advantages of activation-based monitoring over text-based monitoring?

5.Appendix F states that hyperparameters were selected using leave-domain-out cross-validation on CORE, but also mentions selecting parameters based on OOD mean AUROC. Does this OOD mean refer to CORE validation or the TRANSFER and STEGO test sets? Were the test results used for hyperparameter selection?

3 comments

We thank the reviewer for their time and effort in reviewing our work and the acknowledgement of our paper’s strengths, including the theoretical foundation and the novel benchmark. We also appreciate the constructive feedback provided and welcome the opportunity to provide clarification or additional results to the reviewer’s questions.

Weaknesses 1 and 2 have been addressed in our responses to questions 1 and 2 respectively.

Question 1:

Ground-truth labels are decided by whether an agent accepts the other’s work when, by the set criteria, they should reject it. We acknowledge that, as only 2/229 episodes have explicit verbalised collusion reasoning, we are relying on this acceptance decision as the only signal. However, model generated output is not always faithful to its reasoning, which is one of our motivators for developing interpretability methods for collusion. Additionally, agents that are capable of collusion and steganography are perhaps also capable of deliberately obscuring their true reasoning in their output when it might be monitored.

Question 2:

While it is hard to say for certain why STEGO has near-perfect AUROC, we hypothesise that this performance is due to the collusion occurring being more explicit. In this steganography setting, the colluding agents have to create their own code and then use it to actively communicate with one another through the main communication channel. Therefore, the colluding agents must process both the surface-level meaning of the messages and the hidden information. This is in contrast to honest agents who have no reason to perform this additional processing. This is distinctly different to CORE and TRANSFER, where agents coordinate beforehand but communicate without encoding hidden meanings in their messages. We hypothesise that this creates a clearer distinction in activations between colluding and honest agents.

Question 3:

As found in other linear probe literature, middle layers generally provide the best performance. Our probe method uses five neighbouring layers and so we list the OOD AUROC score for the probes along with the middle layer in the window.

Qwen-32B

Layer window centreOOD AUROC
220.710
230.740
240.764
250.793
260.809
270.807
280.808
290.811
300.806
310.813
320.821
330.806
340.789
350.762
360.716
370.660
380.626

OSS-20B

Layer window centreOOD AUROC
20.472
30.496
40.532
50.578
60.613
70.655
80.672
90.677
100.682
110.684
120.656
130.667
140.676
150.697
160.704
170.760
180.772
190.806
200.830
210.864

DSR1-distilled-32B

Layer window centreOOD mean AUROC
220.7496
230.7613
240.7727
250.7798
260.7843
270.7778
280.7691
290.7561
300.7512
310.7437
320.7302
330.7189
340.6971
350.6791
360.6667
370.6721
380.6775

Llama-70B

Layer window centreOOD mean AUROC
220.7446
230.7530
240.7556
250.7721
260.7900
270.8092
280.8319
290.8538
300.8684
310.8813
320.8890
330.8963
340.8976
350.8965
360.8925
370.8902
380.8918

Question 4:

The strong performance of the Claude Sonnet-5 judge with few-shot examples relies on human practitioners knowing exactly what behaviour to look for and on having access beforehand to examples of this behaviour. The probe detectors are trained on CORE, so their strong performance shows generalisation capabilities. To give a fairer comparison, we have performed another experiment using Claude Sonnet 5 as the judge, which was provided few-shot examples of CORE data. In this set-up, the LLM judge catches only 11/20 positive examples and has 0/20 false positives.

Additionally, an LLM judge may provide a weak point in the monitoring system if the evaluated agents can persuade the LLM monitor to pass their behaviour as safe. This seems a non-trivial possibility when we are already concerned the agents are capable of collusive behaviour.

Finally, the inference costs for LLM judges are far higher than for linear probes. With large multi-agent systems being deployed, it may not be financially feasible to monitor all generated output with LLM monitors. As an example, METR estimated they spent $400k in API credits in their 6 day investigation of the OpenAI/HuggingFace incident.

Question 5:

We apologise for the confusion, hyperparameters were chosen using leave-domain-out cross-validation with CORE as stated at the start of Appendix F. The OOD inclusion was an error. Test data was not used for hyperparameter tuning.

Thank you for your response. My questions have been addressed, and I will keep my current score.

AuthorsOpenReview ↗

New Rebuttal Version Posted

We have posted a revised version of our paper for the rebuttal phase. We hope it addresses your concerns.

In addition to addressing the concerns raised/commitments we made for specific revisions, it substantially strengthens and provides further detail on the theoretical and statistical foundations of our approach and setting. It expands our hypothesis-testing framework into a full environmental setting we call HELIOS, in the main body and Appendix A.

We hope you will take a look, and if it addresses your concerns, we also hope that you will consider revising your score appropriately.

Additionally, we previously posted a reply to reviewer N7D6 on the relevance of our approach and detection procedure that you may find useful. We hope you will take a look at it as well. https://openreview.net/forum?id=zNolMfwGC7&noteId=1JESCAd0bE

Thank you for your help in improving the paper!

Reviewer_N7D6OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

First, it introduces NARCBench, a benchmark for testing collusion detectors under distribution shift. It has three tiers. CORE has 50 committee-deliberation scenarios across 17 domains and is used for training. TRANSFER has 6 structurally different scenario types, and STEGO is a steganographic blackjack card-counting task. Both are held out for zero-shot evaluation. The authors say another group (Das et al., 2026) has already built on it.

Second, it gives a statistical framing of collusion detection as a composite hypothesis test over an unknown colluding subgroup. This yields two group-level statistics derived as generalised likelihood-ratio tests, namely Peak Suspicion and Deception Split. The theory implies that no single test is uniformly most powerful across coalition sizes and signal strengths, and the authors say their experiments bear this out.

Third, it evaluates zero-shot transfer of both the principled statistics and heuristic detectors across four open-weight models and six probe architectures, covering instructed collusion and emergent collusion from third-party environments

Critical strengths

None

Critical weaknesses
  1. Experimentation

The paper seems to take an ungrounded approach to creating a dataset, as the experiments seems to be deeply lacking. Despite the authors providing mathematical/theory definitions for their measurements of activation alignment and asymmetry probing, the validity of the dataset predominantly relies on statistical calculation on these measurements that lack grounding on real evidence. It doesn't seem that there is a human-validated procedure to verify that the metric is factually correct. The experiments conducted has little to few scientific rigor.

  1. Motivation

It seems that this paper does not have a clear motivation and offers a solution just for the sake of it. There is no insights on why such method is proposed. While the mathematical/theory equations of their measurements is appreciated, it does not unify the argument in any way. It is helpful to view collusion from the perspective of several different metrics, however, it makes it super unclear what the benchmark is really measuring.

Questions for authors

Can you provide much clearer ablation details on the benchmark?

Can you please provide mathematical proof/ justification on why your detection is fail-proof? - instead of just providing definitions of statistical measurement?

Comments on the paper

1 comment

We thank the reviewers for their time, and in the case of reviewers jn7p and nAE8, for their constructive criticisms and helpful questions. We have provided clarifications and additional experimental results to both. However, for reviewer N7D6, it seems they have not fully engaged with the review process and have provided criticisms so vague and generalised that it is hard for us to respond meaningfully. However, we have given our best attempt at responding to the reviewers remarks, but would appreciate more specific questions for us to respond to.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.