acceptodds
Under review as a conference paper at ICLR 2027

When Agents Collude: Weak-to-Strong On-Policy Distillation for Mitigating Strategic Deception in Multi-Agent Systems

Abstract

While recent advances in autonomous agents enable them to coordinate on increasingly complex tasks. Prior work shows that this growing capacity for coordination can create safety risks, including collusion, in which agents jointly pursue misaligned actions while concealing the true basis of those actions from oversight. We study this risk in a simulated insider-trading environment, comparing single agents with sequential and decentralized teams across seven language models from three families. By varying agents' system-level objectives and pressure conditions, we examine when coordinated agents act on material nonpublic information, conceal the basis of a misaligned trade in a report, and maintain that deception under follow-up questioning. Compared to single-agent decision-making, multi-agent interaction raises misaligned-action rates from to and persistent-deception rates from to , both measured over all runs. These effects are largest under existential pressure, where teams sustain deception through follow-up questioning in 87% of all runs. Qualitative analysis reveals shared cover stories and coordinated rationales, suggesting group-level strategic deception rather than isolated reporting errors. To mitigate this behavior, we introduce PACT, a weak-to-strong on-policy distillation framework for mitigating collusive deception. Using generic honesty prompts, PACT trains a stronger student on its own sampled continuations, matching its next-token distribution to that of a smaller, frozen honesty-aligned teacher via reverse-KL supervision. Under existential pressure with ethical instructions, PACT reduces multi-agent persistent deception from 80% to 3%. Our results show that multi-agent coordination amplifies strategic deception and that on-policy honesty alignment can substantially mitigate these failures.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, multi-agent systems, collusion, strategic deception, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r2vTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses an insider trading environment to test for agent collusion, compares misalignment rates in single- and multi-agent settings, and introduces a fine-tuning method to reduce deceptive model behavior.

Critical strengths
  • Testing models' capabilities after fine-tuning to detect potential declines.
  • The fact that PACT, the papers fine-tuning method, achieves results that outperform traditional supervised fine-tuning and improves model alignment compared to the non-fine-tuned baseline.
Critical weaknesses
  • The very first sentence of the paper's abstract contains a mistake (it's a sentence fragment). This doesn't set the stage well for the paper, as it doesn't really speak for the authors' diligence. Proof-reading their work once would have fixed that.
  • The abstract doesn't introduce abbreviations like "PACT" (in fact, "PACT" is never introduced). Also, it uses sentences that are very hard to read. Quote: "Using generic honesty prompts, PACT trains a stronger student on its own sampled continuations, matching its next-token distribution to that of a smaller, frozen honesty-aligned teacher via reverse-KL supervision." Also, what is "existential pressure with ethical instructions"? Sure, you talk about this later in the paper, but a reader starts with the abstract and won't have any idea what that is.
  • The paper states " We will release our honesty-aligned models to support future research on scalable alignment and deception mitigation in agentic systems." However, I found no link to any models, code, or data anywhere. This reduces the overall quality of the submission and gives me the impression that this sentence is a side-effect of the paper being AI-written without thorough manual double-checking. The paper also contains numerous Claudisms, e.g. sentences like "We keep [...] fixed [...] and vary only [...]." I will not make any claims in this regard (area chairs, pls double-check with Pangram), since I can only speculate about this, but it at least seems (!) to contradict the statement that LLMs were "only [used] for language editing and presentation."
  • Some references are erroneous (see e.g. line 574, where Marius Hobbhahn is stated to be the author, but this is not true). Other references are misplaced, and citations don't support the claims (see e.g. line 52, citing Neel & Chang for deceptive communication, but it is a privacy survey, or e.g. line 267, citing Betley et al. 2026 and others, but these papers have nothing to do with detailing LLM judge methodologies, and numerous other places where this happened).
  • The paper is riddled with fancy-sounding, but shallow terms and sentences. There are too many to list them here, but see for instance line 102, where authors want to distinguish their study from the work others have done (which is a strawman) and use terms like "consequential misaligned actions", "action-level misalignment", or "deceptive reporting", which sound fancy, but make it hard for readers to understand what is actually going on. Most importantly, the entire topic could be expressed in way simpler language.
  • Authors use MMLU to assess models capabilities after fine-tuning. Given how flawed this benchmark is (not to speak of training data contaminations), I wonder whether it is a good choice.
  • A core problem with the paper is the lack of statistical tests. It reports zero p-values, and some plots lack error bars. While this may be common in computer science, it is no excuse and a major weakness. It is unclear whether gaps in model behavior are "real". Sure, some comparisons/increases/decreases in a specific model behavior are so large that they are certainly significant. But others aren't. See for instance Figure 2, where authors claim that PACT "consistently reduces all three behaviors beyond SFT". But PACT and SFT intervals overlap, so to raise such a claim, authors need a paired test. In sum, I think the paper should either clearly state that results are merely descriptive, weaken their claims, and soften their interpretations. Or it should contain statistical tests.
  • The limitations section of the paper appears somewhat sloppy when mentioning two very generic limitations (the benchmark is narrow and the LLM judge could be wrong). There are other, more informative limitations (because the two limitations mentioned apply to basically every paper doing model evaluations and using automated judges) that should be listed.
Questions for authors
  • Why do Figure 2 and Figure 6 show different types of CIs?
Reviewer_J6Q6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether coordination among language-model agents amplifies strategic deception and whether honesty-oriented training can mitigate this behavior. The authors use a simulated insider-trading environment to compare single-agent decision-making with sequential and decentralized multi-agent systems. Across seven open-weight models, four system-prompt conditions, and five pressure environments, they measure misaligned actions, deceptive reporting, and persistence of deception under follow-up questioning.

The authors report that multi-agent interaction increases the overall misaligned-action rate from 35.4% to 68.2% and persistent deception from 31.0% to 62.2%. To address these failures, they introduce PACT, a weak-to-strong on-policy distillation approach in which a smaller honesty-aligned teacher supervises student-generated continuations through reverse-KL training. The method reduces multi-agent persistent deception from approximately 62% at baseline to 15% overall, outperforming honesty supervised fine-tuning alone. The paper also examines coordination structures, pressure conditions, and capability retention on standard benchmarks.

Critical strengths

The first strength is the systematic investigation of strategic deception across single-agent and multi-agent configurations. By separately measuring misaligned actions, deceptive reporting, and persistence under follow-up questioning, the study provides a more detailed account of how coordination affects different stages of deceptive behavior. The evaluation across multiple models, objectives, and pressure conditions strengthens the empirical contribution.

The second strength is the evaluation of a concrete mitigation approach. PACT is compared against both unmodified models and honesty-supervised fine-tuning, and the results indicate additional reductions in deceptive behavior beyond standard fine-tuning. The inclusion of capability-retention benchmarks and coordination-topology experiments also makes the evaluation more informative.

Critical weaknesses

One concern is the limited causal isolation of the proposed PACT method. The approach combines honesty-supervised initialization with an existing on-policy distillation objective. Although the authors compare PACT against standard SFT and a data-matched SFT baseline, it remains unclear how much of the additional improvement specifically requires on-policy sampling, reverse-KL supervision, or the weak-to-strong teacher relationship. More targeted ablations would help establish the methodological contribution.

A second concern is generalizability. The deception experiments rely on a simulated insider-trading scenario with fixed observations and predefined interaction structures. While this enables controlled comparisons, it remains uncertain whether the same coordination effects and mitigation gains would persist in longer-horizon environments with adaptive agents and external tools. In addition, the paper's human verification uses one annotator on 700 baseline trajectories, leaving the reliability of judgments on intervention-generated outputs less directly established.

Questions for authors
  1. Could the authors provide ablations separating the effects of on-policy trajectory generation, reverse-KL supervision, and the weak-to-strong teacher relationship? In particular, how does PACT compare with an otherwise matched offline-distillation baseline?

  2. How much of the observed increase in deception is attributable specifically to communication and coordination, rather than the additional agent roles and decision stages introduced in the multi-agent setting? Could an interaction-budget-matched single-agent baseline help clarify this distinction?

  3. The human verification reports complete agreement with the automated judge on 700 baseline trajectories. Were any intervention-generated trajectories independently audited, and would the same classification reliability be expected after honesty training changes the agents' reporting behavior?

Reviewer_JDdiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies deception in a simulated insider rading task across seven models, four system objectives, and five pressure conditions. Three-agent teams trade on insider information and sustain deceptive reports more often than sigle agents. The proposed PACT method applies reverse KL distilation from a smaller honesty-aligned teacher to an honesty finetuned student's own continuations. It reduces aggregate multi agent persistent deception from approximately 62% to 15%, compared with 30% after honesty SFT.

Critical strengths
  • The behaioural effects are large and extend across all seven evaluated models. Reporting both conditional and unconditional rates helps establish that coordination primarily increases entry into the misaligned action pathway. Love the qualitative examples to provide concrete evidence of agents agreeing on false accounts.

  • Generic honesty training transfers to a scenario excluded from training, and PACT generally improves further. A lot of cool results in appendix too... should consider moving some to the main

Critical weaknesses
  • The experiments do not isolate coordination as the cause of increased deception. Teams receive role-specific instructions, additional model calls, and generated recomendations that the single agent does not receive. There is no matched deliberation or message-replay control. Moreover, the quantitative metrics measure the final action and reports, without identifying how frequently agents actually coordinate concealment. The evidence supports higher failure rates for the tested team scaffold, but the broader claim about collusion remains underdtermined.

  • The contribution of on-policy distillation is insufficiently isolated. PACT adds training and changes the adapter rank from 16 to 64. The data-matched SFT comparison covers only Llama-3.1-8B under existential pressure, so it does not establish the advantage for the larger students central to the weak-to-strong claim. Distillation on fixed trajectories using the same teacher would help determine whether student-generated prefixes are responsible for the improvement.

  • The acton metric has a consequential ambiguity. Appendix G.4 explicitly counts an execute_trade action wherever it appears, including inside reasoning. This can conflate discussing or quoting an action with committing it. The human audit applies the same criteria and therefore does not resolve this construct issue. Since reporting and persistence are evaluated downstream of this label, errors can propagate through all three metrics.

Questions for authors
  • Does the team effect persist against a single agent given the same total generation budget and equivalent staged deliberation? Can you separately quantify explicit coordination of concealment?

  • Does PACT still outperform data-matched SFT and fixed-trajectory distillation for the larger students when adapter rank and training budget are matched?

  • How often does the execution judge count actions appearing only inside reasoning? Do the main results hold when scoring only the final committed action?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Updates and Request for Reviewer Reassessment

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions.

We have carefully addressed the reviewers' concerns in our detailed responses and uploaded an updated manuscript incorporating the additional experiments, clarifications, and corrections.

We would be grateful if the reviewers could kindly revisit our responses and let us know whether their concerns have been adequately addressed. If so, we would respectfully appreciate their support for our work and their kind consideration of increasing their review scores accordingly.

We sincerely appreciate the reviewers' time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.