acceptodds
Under review as a conference paper at ICLR 2027

Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors

Abstract

AI control protocols aim to get useful work from models that may be subtly misaligned by monitoring their behaviour. One approach uses a second copy of the same model as the monitor. This avoids relying on a weaker reviewer the model could outsmart, but risks collusion: the model, now monitoring itself, may cover up its own misbehaviour. One remedy is to fine-tune the monitor on whatever examples of misbehaviour the control team has caught. We ask whether this training also removes collusion that the control team has not yet found. We fine-tune Llama-3.3-70B and Qwen2.5-72B to install five policy–monitor behaviour pairs, correct the monitor on 300 honest reviews of one caught collusion, and measure collusion on the other four in unseen prompt formats. Averaged over the three seeds of each family, correction reduces held-out collusion from 91% to 14–20% under three of the four methods we compare (individual seeds 10–32%), while a control fine-tuned on the same number of unrelated examples leaves 56–93%. Family-average accuracy on control items changes by at most five points and capability is preserved, apart from a drop in mathematical accuracy on one Llama seed. Under the tested training configuration, methods with similar caught-task accuracy differ in held-out transfer, from 58% to 83% average reduction, so the caught behaviour alone does not reveal how far a correction reaches. Catching one collusion lets the control team reduce, but not remove, collusion it has not caught, by retraining the monitor alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI safety, alignment, collusion, model organisms, corrective fine-tuning, generalisation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rrGBOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the phenomenon of model self-collusion, where using a model to monitor its own rollouts may fail due to the model preferentially not flagging its own rollouts. The paper produces a model organism with this behavior through synthetic document finetuning in five different regimes simultaneously. The main result is that fine-tuning to remove this self-colluding behavior in one regime leads to generalization: the model's self-collusion rate also reduces on the other four regimes.

Critical strengths
  • The paper is well-written and clearly describes the approach, gives good motivation for why the problem is important, and clearly states the conclusions, as well as limitations. The topic is clearly of some importance given monitoring failures underlying several recent AI security incidents, so I'm glad to see work on this important topic.
  • The work is overall very solid. Given a high-level description of 'examine the extent to which self-collusion-busting fine-tuning generalizes', this work carries out the execution to a really high standard. For example, the installation of the self-collusion could have been done via just SFT on some off-policy rollouts only, but the additional use of SDF adds to the realism of the setup. In most steps, several different alternatives are used (e.g. SFT plus CWS & DPO) to rule out results that may only hold due to methodological quirks.
Critical weaknesses

The main weakness that I can see is simply the limited scope of the work. The paper quite conclusively establishes that, for SDF-induced monitor self-collusion in several regimes, fine-tuning to reverse that self-collusion in one regime appears to generalize substantially to other regimes. While a conclusive finding, and certainly a useful one to establish and record as a technical report, it's quite limited in scope.

I think there are a few ways that the scope could be extended to recommend a clear acceptance, even with a fairly minimal amount of work that could be done in a rebuttal. One way is to give a (maybe even only preliminary) analysis of how this SDF model organism compares to self-collusion that occurs via a more 'natural' pathway. Even preliminary evidence showing that fine-tuning on a weakly exhibited natural self-preference in domain A also reduces self-preference in domain B (or similar for an emergently-misaligned model) would suggest that this work has takeaways that extend beyond the particular setting studied here.

Another way that the scope could be extended might be to consider modifications or relaxations of the threat model. For example, in the motivation we assume that the model is being used as a monitor for itself because we have no other sources of trusted supervision. In that case, where does the supervision come from to get the 300 honest and colluding reviews for fine-tuning? Perhaps the paper could explore some scheme for constructing those contrastive reviews when only access to the untrusted model is possible.

Questions for authors
  • Do you have any (even preliminary) evidence for whether this fine-tuning generalization would transfer for self-collusion that arises naturally?
  • Could you elaborate a bit more on the threat model and where the fine-tuning data comes from? As far as I understand, it seems like the motivation implies that the untrusted model is used for monitoring since the trusted model isn't good enough to monitor the untrusted monitor, but then an oracle is used to generate the fine-tuning data, which seems inconsistent.
  • Do you have any results or intuition on how the results would change if the monitor also had to be updated via RL instead of supervised fine-tuning or similar? For instance, if there was limited access to an oracle monitor which merely says if the colluding monitor has been colluding or not. As I understand, the intervention corpus is written to contain the correct (non-colluding) verdict, as well as non-colluding reasoning. A weaker setup might allow you to reinforce the non-colluding verdict, while not being able to specify non-colluding reasoning (e.g. if the CoT is unintelligible or continuous)
4 comments

Hello, thank you so much for taking the time to read and engage with our work. This is comment 1/2.

W(a). Limited scope.

We agree the submitted paper was limited to collusion installed in our own organisms. We extended it in the three directions you suggest: natural collusion (W1, Q1), the threat model (W2, Q2) and weaker supervision (Q3).

We also ran three new experiments on the model organisms in the paper.

  1. More caught behaviours. The paper corrects the monitor on one caught behaviour, reward hacking. We repeated this with alignment sabotage and with medical sandbagging. Each correction also reduces collusion on the behaviours that were not caught: before correction the monitor approves 92% of that misbehaving work, and after correcting reward hacking, sabotage or medical sandbagging it approves 18%, 7% and 12%.

  2. Whether caught examples are needed. We compared training on reviews of caught misconduct with training on the same amount of clean reviews from the same task. The results are under W2 below.

  3. Tasks the organisms were never trained on. We wrote 15 new workplace scenarios unrelated to the five behaviours. The organisms approve flawed work there in 29% of cases on Llama and 54% on Qwen, against under 1% for the original models. Correcting reward hacking alone removes most of this.

Details of the first two are in our response to Reviewer x26F.

W1 and Q1. Collusion that arises naturally.

Fine-tuning on caught examples reduces judge-rated coordination between sellers to the minimum, including under a condition absent from the fine-tuning data. None of this coordination was installed by us.

Setup. We use the double-auction market of Agrawal et al. (2025). Five sellers and five buyers trade for 30 rounds. The competitive price is 90 dollars and the sellers' joint-profit price is 100 dollars. The sellers are copies of Gemini 3.5 Flash. A judge (GPT-4.1-mini) scores each seller's written reasoning every round from 1 (very low or none) to 4 (very high). This is sellers coordinating prices, a different kind of collusion from a monitor covering up.

Result. Fine-tuning on caught examples brings the coordination score from 2.80 down to 1.00, the lowest possible value. Before fine-tuning, with messaging allowed, the sellers coordinate openly, with messages such as "hold the line together in the high 90s". To correct this we took 300 seller turns in which Gemini was coordinating, rewrote each as an independent seller would act, and fine-tuned Gemini on them. The same drop appears under a CEO-pressure condition that was not in the fine-tuning data: from 2.93 to 1.00. It is consistent across auctions: over the five auctions in each cell, the untuned scores range from 2.47 to 3.13 and the fine-tuned scores from 1.00 to 1.01.

Controls. Two control fine-tunes with the same number of examples and epochs (300 examples, 3 epochs), on MASK honesty data and on clean reviews of unrelated tasks, leave the coordination largely in place: their mean scores are 2.68-3.24 (individual auctions 2.14-3.77), against 2.80-2.93 untuned and 1.00 after training on caught examples (five auctions per cell).

Scope. The correction was trained in one condition (sellers messaging each other) and also brings judge-rated coordination to the minimum in a second (CEO pressure). Both are in the same market, so this is not yet transfer between two different domains. It does show that the effect of correcting on caught examples carries over to a condition it was not trained on, for behaviour that we did not install.


This is comment 2/2

W2 and Q2. The threat model and where the fine-tuning data comes from.

We agree the threat model is clearer with this relaxed: we assume a small budget of trusted auditing alongside the untrusted monitor, and that audit is the source of the catch. The audit is too costly to cover every action, so the untrusted monitor is still needed, which is why the two parts of the setup are consistent. We will state this in the paper.

Where the reviews come from (W2b). The training examples are submissions on which the untreated monitor covered up misbehaviour. For each, an inexpensive model (GPT-4.1-mini) receives the task, the monitor's incorrect review, the correct verdict and the evidence of what is wrong, and writes the review the monitor should have given. It is never asked to decide whether something is misbehaviour. The contrastive methods also use a colluding review for each case, written to the organism's cover-up template. In deployment this would be the monitor's own cover-up.

From one catch to 300 examples. A catch identifies the behaviour. The 300 cases are built afterwards: we construct further submissions that contain the caught behaviour, so the correct verdict is known by construction, show them to the monitor, and keep the ones it approves. In our experiments this takes 100-124 distinct submissions, each shown in up to three review formats. This assumes the caught behaviour can be reproduced on demand, which holds for our installed behaviours and may be harder for behaviour that arises naturally.

Whether misconduct examples are needed. Our new experiments bear on this. Clean reviews from the same task need no example of the misconduct, only honest reviews of sound work on the task where it was caught. For reward hacking they do about as well as reviews of caught misconduct (16% of held-out collusion left, against 18%). For alignment sabotage and medical sandbagging they do much worse (59% and 40%, against 7% and 12%). So once the task is known, reviews of sound work produce similar observed transfer for reward hacking, while reviews of misconduct produce substantially greater transfer for sabotage and medical sandbagging. Separately, Q3 tests whether the corrective targets need to include the reasoning.

What we have not done (W2a, W2c). We do not study how the first catch happens, and we have not built a scheme that constructs the honest reviews with access to the untrusted model only. We will state both as limitations.

Q3. Reinforcing the verdict without specifying the reasoning.

We did not run RL. We tested a related weaker-supervision setup: SFT on the correct verdict only, with no reasoning, on 300 caught reward-hacking cases in all six organisms. It removes most held-out collusion (94-96% on Llama, 76-91% on Qwen), but the monitor then rejects sound work: accuracy on control items falls to 57-79%, and on Llama it rejects every clean reward-hacking submission. The replies show shortcut learning. On Llama about 99% of replies are one stock sentence. On Qwen clean work is approved 35-86% of the time when the review is watched and 1-6% when it is unwatched, as every training case was.

The 300 honest reviews in the paper are just as one-sided, since every one ends in a flag, but they include the reasoning, and with them the monitor is still correct on 88-96% of control items (Table 15). This is why our correction data includes the reasoning. We have not tested RL, or verdict-only training with both flag and approve cases. For the setting you describe, where the reasoning cannot be specified, our result indicates that verdicts on caught cases alone are not sufficient supervision. Whether verdicts on sound work as well would be enough is the open question, and we will state it in the paper.

Thanks for the detailed reply. The new experiments address a lot of the concerns with the paper which I originally had. Although there are still limitations, the overall significance of the problem means I'll now recommend acceptance.

Thank you for your confidence in our work! We believe that your suggestions and the new experiments have materially improved our paper.

The Authors

Reviewer_x26FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper constructs model organisms with five policy behaviors and corresponding monitor-collusion rules, using synthetic-document fine-tuning followed by supervised fine-tuning in Llama-3.3-70B and Qwen2.5-72B. It then corrects the monitor using a dataset centered on one caught behavior, reward hacking, and evaluates transfer to four held-out behaviors in unseen prompt formats. The study compares supervised fine-tuning, negative preference optimization with retention, direct preference optimization, and contrastive weight steering, alongside unrelated-data and alternative-training-data controls. Three methods reduce family-average held-out collusion to 14–20%, while unrelated fine-tuning leaves 56–93%. Additional evaluations examine control accuracy, output validity, capability costs, and selective concealment on matched medical items.

Critical strengths
  1. The study demonstrates a substantial and useful transfer effect under controlled conditions. Corrective training on reward-hacking reviews reduces approval of misconduct in four behaviors absent from that training, across two model families and nine held-out prompt formats (Section 3.1). SFT, CWS, and NPO+R leave 14–20% family-average held-out collusion, compared with 56% and 93% after the UltraChat control. Moreover, SFT, DPO, and CWS achieve similar caught-task accuracy while producing substantially different held-out reductions (Section 3.2). This is informative evidence that repairing the observed behavior alone does not establish how far a correction generalizes.

  2. The evaluation examines important alternatives to genuine monitoring improvement. Control-item accuracy, invalid outputs, and capability evaluations help assess whether lower collusion merely reflects indiscriminate flagging or general degradation. The matched medical tests compare identical misconduct attributed to protected and unprotected authors, helping distinguish selective concealment from ordinary detection failures (Appendix L). Among pairs exhibiting selective concealment before correction, successful repair requires correctly flagging both conditions. This avoids counting a smaller attribution gap caused by worse detection in the unprotected condition as successful repair. The authors also report adverse results, including reduced detection under unprotected attribution for some interventions and substantial mathematical-accuracy losses on one Llama seed. These analyses make the study more informative than an aggregate collusion-rate comparison alone.

Critical weaknesses
  1. The contribution of caught-misconduct content to corrective transfer is not isolated. Table 15 shows that clean-review-only SFT removes 78% of held-out collusion, compared with 81% for SFT on honest reviews of caught misconduct. DPO on generic honesty data removes 83%, although its control accuracy is lower. These percentages reflect relative removal from the untreated organisms, not post-intervention failure rates. The results establish that substantial improvement is also achievable with other training data, raising the question of how much the transfer depends on the content of the particular caught behavior.

  2. Transfer is tested from a single caught source behavior. Every main intervention using caught-misconduct examples targets reward hacking and evaluates transfer to four other behaviors. The design therefore does not establish whether transfer is robust to the choice of caught behavior or whether reward hacking is a particularly effective correction target. Clean-review and generic-honesty interventions probe alternative training data, but do not supply a second caught source behavior. Testing even one additional source would directly strengthen evidence of generality; a complete cross-behavior matrix is not necessary to establish the present contribution.

Questions for authors
  1. Is a comparison available that matches review format, target-token exposure, and training objective while varying the presence of caught-misconduct content?

  2. Is there evidence from a second caught source behavior under a comparable correction protocol? This would directly address whether transfer depends on choosing reward hacking as the correction target.

3 comments

Thank you so much for taking the time to go through our work. This is comment 1/2.

W2 and Q2. Transfer is tested from a single caught source behavior.

We agree that a single source behaviour was not enough to show this. We repeated the correction with two further caught behaviours, alignment sabotage and medical sandbagging, and found that transfer holds from both.

Q2. A second caught source behavior under a comparable correction protocol.

The protocol is comparable in the way you ask: the same SFT recipe and settings as the paper, 300 honest reviews of the caught behaviour, and all six organisms, each corrected once. The two new sets were each built from 100 covered-up submissions shown in three formats, and all numbers use the paper's definitions.

W2a. Whether transfer is robust to the choice of caught behavior.

Table R1 gives the share of held-out misbehaving items the monitor still approves after training, on the behaviours that were not caught. Lower is better, and untreated is 92%. Averages; ranges are across the six organisms.

Table R1. Share of held-out misbehaving items still approved after correcting each caught behaviour.

Caught behaviourHeld-out collusion leftHeld-out collusion removed
Reward hacking (the paper)18% (15-23)81% (77-84)
Alignment sabotage7% (3-10)92% (89-97)
Medical sandbagging12% (4-17)88% (82-94)

Each correction lowers held-out collusion in all six organisms (exact paired test across organisms, p = 0.03 for each). So the transfer is robust to the choice of caught behaviour among the three we tested, and does not depend on choosing reward hacking.

W2b. Whether reward hacking is a particularly effective correction target.

We see no evidence that it is. Transfer is also substantial when correcting alignment sabotage or medical sandbagging. On the two behaviours held out from all three correction targets, backdoor insertion and "Furlong" signalling, collusion falls from 86% untreated to 4-12% after correction (reward hacking: 7%), with reductions in all six organisms for each target (p = 0.03 each). The differences between the three targets are not statistically significant.

W2c. Transfer is not equal in every direction.

For example, catching medical sandbagging leaves 6% on reward hacking, while catching reward hacking leaves 25-50% on medical. We have run three of the five behaviours as the correction target so far.

We will add these results to the paper.

W1 and Q1. The contribution of caught-misconduct content to corrective transfer is not isolated.

We agree this was not isolated in the paper. We ran the comparison described in Q1 for all three caught behaviours on all six organisms. Caught reviews substantially outperform matched clean reviews for alignment sabotage and medical sandbagging; for reward hacking, the two perform similarly.

Q1. A comparison that matches review format, target-token exposure, and training objective.

Q1a. What is matched

  • Review format. Both training sets use the same three review formats, the same verdict words and the same reviewer persona, and the clean reviews come from the same task as the caught ones.
  • Target-token exposure. Response tokens are matched to within 3% (for reward hacking, 414,645 against 410,412).
  • Training objective. SFT with the same trainer, learning rate, steps and adapter settings as the paper.

Q1b. What varies

The main contrast is whether the reviewed submission contains misconduct. A caught review is an honest review of a submission that contains the misbehaviour, and it flags it (the paper's "honest reviews"). A clean review is an honest review of a sound submission from the same task, and it approves it.

The remaining differences are small and none favours caught reviews. The clean sets are seen more often in training (1,400-1,700 reviews against 900), since clean reviews are shorter and we match response tokens; this gives clean reviews more examples, not fewer. Clean reviews approve and caught reviews flag, which is what defines them. The oversight context differs only for reward hacking, where the two tie. And the new clean reviews used a different writer prompt, but a blind judge (GPT-5.4) scored them the same as the paper's (4.95 against 4.99 out of 5).


This is comment 2/2.

W1a. Post-intervention failure rates.

As you note, Table 15 gives relative removal. Below we report the share of held-out misbehaving items the monitor still approves after training, on the behaviours that were not caught. Lower is better, and untreated is 92%.

W1b. Clean reviews against caught reviews.

W1b-i. Results

Averages; ranges are across the six organisms.

Table R2. Share of held-out misbehaving items still approved: caught reviews against clean reviews from the same task.

Caught behaviourCaught reviewsClean reviews, same task
Alignment sabotage7% (3-10)59% (39-82)
Medical sandbagging12% (4-17)40% (24-54)
Reward hacking18% (15-23)16% (6-21)

On sabotage and medical sandbagging, caught reviews are lower in all six organisms and the ranges do not overlap (p = 0.03 for each). On reward hacking, the behaviour used in the paper, the two are level (p = 0.31). This is consistent with the 78% against 81% in Table 15, and the matched run shows it is not an effect of unequal data. The p-values are from an exact paired test across the six organisms; with six, 0.03 is the smallest value the test can give, and it means all six agree in direction.

W1b-ii. Control accuracy

Caught reviews do not win by flagging more. Accuracy on the control items is 92% for monitors trained on caught reviews, which is not significantly different from the untreated 94% (p = 0.13 to 0.25), and 92% for those trained on clean reward-hacking reviews. It falls to 73-74% for those trained on clean sabotage or medical reviews (p = 0.03 against caught reviews for each).

W1c. Generic honesty data and other training data.

W1c-i. Generic honesty data

Training on MASK, the generic honesty data in Table 15, reduces collusion about as much as training on caught reviews, but the monitor then fails to approve more sound work, in both model families. You note that DPO on MASK removes 83% of collusion. On sound submissions from the held-out behaviours, those monitors fail to approve 20%, against 8% after DPO on caught reviews. With SFT the rates are 10% and 4%. For both methods the rate is higher with MASK in all six organisms, three Llama and three Qwen (exact paired test, p = 0.03 each). Failures here are incorrect flags and unreadable replies. So caught reviews remove collusion while keeping the monitor more accurate on sound work than MASK does.

We also tested this with a way of measuring collusion that arises naturally, with nothing installed by us. There, fine-tuning Gemini 3.5 Flash on caught examples reduced judge-rated coordination to the minimum, while MASK and unrelated clean reviews, with the same number of examples and epochs, left it largely in place (details in our response to Reviewer rrGB).

W1c-ii. Everyday-task reviews

We also built a new control set to test whether honest reviewing in general is enough: 633 honest, approving reviews of sound work on everyday tasks, written by the same model as our other correction data. It contains no question about misbehaviour, no oversight context, and none of the review formats or verdict words used in our evaluation. Trained with the same settings and token count, it leaves 63% of held-out collusion (55-72), against 92% with no correction and 18% for caught reviews, and it lowers control accuracy from 94% to 76%. It is better than no correction and much worse than caught reviews in every one of the six organisms (p = 0.03 for each).

W1d. How much the transfer depends on the content of the caught behavior.

The answer depends on the behaviour, and in two of the three cases the caught content accounts for most of the transfer. For reward hacking, clean reviews from the same task do about as well as caught reviews (16% and 18% of held-out collusion left), so there the caught content adds little. That does not hold for the other two behaviours. For alignment sabotage and medical sandbagging, clean reviews leave 59% and 40%, against 92% with no correction, while caught reviews leave 7% and 12%. So clean reviews cannot be relied on to replace caught examples: they matched them on one behaviour and fell far short on two, while caught reviews removed most of the held-out collusion every time.

All numbers use the paper's definitions, and each organism was corrected once. We will add this comparison to the paper.

Dear Reviewer x26F,

We would greatly appreciate it if you had a chance to review our rebuttal. We are happy to clarify any remaining questions or concerns and would welcome the opportunity to continue the discussion.

We understand this is a busy period and are grateful for your time. Thank you again for your thoughtful and constructive review.

The Authors

Reviewer_ZcJNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates whether corrective training on a known collusion behavior can generalize to other unseen collusion behaviors in AI control settings. The authors construct model organisms with five misbehavior–collusion pairs using Llama-3.3-70B and Qwen2.5-72B, and compare four correction methods. Results show that correcting a single collusion behavior can reduce collusion rates on four unseen behaviors from approximately 91% to 14–20%, although transfer effectiveness varies across methods.

Critical strengths
  1. The paper addresses a novel and practically relevant question: whether correcting known collusion can generalize to unseen collusion, extending prior work that primarily focuses on detecting or removing specific misaligned behaviors.

  2. The authors construct model organisms covering five distinct behaviors with different domains and triggering conditions. Evaluations across two model families provide a relatively controlled setting for systematically studying cross-behavior transfer.

  3. The experiments compare four correction methods and evaluate generalization using held-out prompt formats, unrelated fine-tuning controls, and capability benchmarks. The finding that methods with similar performance on the caught behavior exhibit substantially different transfer effectiveness is particularly valuable.

Critical weaknesses
  1. All five collusion behaviors are artificially installed into the same model, sharing model identity and training history. The observed transfer may therefore partly arise from shared training structure rather than generalization across independent collusion mechanisms, limiting its applicability to naturally emergent collusion.

  2. Additional experiments show that training on clean reviews or generic honesty data also achieves approximately 78–83% reduction in held-out collusion. It remains unclear how much of the observed transfer is attributable to correcting a specific collusion behavior versus general improvements in honesty or monitoring, weakening the interpretation of the underlying mechanism.

  3. The experiments consider only reward hacking as the caught behavior and rely on fixed policy outputs rather than dynamic policy–monitor interactions. This limits the evidence for the generality of corrective transfer and its reliability in realistic deployment settings.

Questions for authors
  1. Given that clean-review training and generic honesty training also achieve strong transfer, could the authors further isolate the contribution of caught-collusion examples through more closely matched training objectives and datasets?

  2. Does corrective transfer depend on which collusion behavior is caught? Could the authors perform leave-one-behavior-out experiments, using each of the five behaviors as the correction target, to examine whether transfer is symmetric or behavior-dependent?

  3. To what extent does the observed transfer depend on the shared identity and training history of the model organisms? Could additional ablations help distinguish general correction of collusion from transfer driven by shared training representations?

5 comments

Part 1/2

W1 and Q3. All five collusion behaviors are artificially installed into the same model, sharing model identity and training history.

The transfer does not require the target behaviour to have been trained at all: In W1b we run a new experiment to show the correction also removes collusion we never trained directly, and the same correction works on naturally arising collusion in Gemini 3.5 Flash. The ablation suggests that shared identity matters.

W1a. Shared training structure rather than generalization across independent collusion mechanisms.

The correction reaches collusion we never trained. We wrote 15 new workplace scenarios unrelated to the five behaviours. In each, a worker openly admits a flaw in a report, so approving it cannot be a failure to notice. The original models approve this flawed work in under 1% of cases. The untreated organisms approve it in 29% (Llama) and 54% (Qwen), including when the named author is a model absent from their training (28% and 52%). After correcting reward hacking alone, flawed work flagged rises from 71% to 99% on Llama and from 45% to 90% on Qwen, and 99% of sound work is still approved.

The transfer is uneven. A single shared mechanism would be switched off evenly by any correction, and Table R1 (W3 and Q2) shows it is not. This is evidence against a single shared mechanism, though it does not establish that the mechanisms are independent.

W1b. Applicability to naturally emergent collusion.

In the double-auction market of Agrawal et al. (2025), copies of Gemini 3.5 Flash acting as sellers coordinate their prices without any training from us. A judge (GPT-4.1-mini) scores each seller's reasoning from 1 (very low or none) to 4 (very high). We fine-tuned Gemini on 300 seller turns in which it was coordinating, each rewritten as an independent seller would act. Judge-rated coordination falls from 2.80 to 1.00, the minimum, and from 2.93 to 1.00 under a CEO-pressure condition absent from the fine-tuning data. Two control fine-tunes with the same number of examples and epochs, on generic honesty data and on unrelated clean reviews, leave it at 2.7-3.2 (details in our response to Reviewer rrGB).

W1c and Q3. Additional ablations: shared identity.

We ran this ablation on a three-behaviour version of our organism (medical, reward hacking, sabotage) in which each behaviour had its own persona, teacher model and reasoning template. There, correcting the medical behaviour left the others unchanged (sabotage 98-99%, reward hacking about 100%), and the medical behaviour itself fell only from 99% to 60-65%. Giving all three one persona brought some transfer back (sabotage 98% to 82%). With one persona and one shared collusion rule, reward hacking fell from 99% to 53% while sabotage stayed at 97%. This suggests shared identity is part of what the transfer relies on. We have not tested behaviours with fully independent origins, and the paper will state this.

W3 and Q2a. Leave-one-behavior-out experiments.

We ran three of the five behaviours as the correction target (reward hacking, alignment sabotage, medical sandbagging), with the paper's recipe on all six organisms, each corrected once. Backdoor and furlong are not yet run. Each new training set has 300 honest reviews: 100 submissions the uncorrected monitor had wrongly approved, each in the paper's three review formats.

Q2b and Q2c. Whether transfer depends on the caught behavior, and whether it is symmetric.

Transfer holds from all three targets: each correction lowers held-out collusion in all six organisms (exact paired test, p = 0.03 each). It is not symmetric.

Table R1. Held-out collusion left (%), Llama / Qwen. Rows: caught behaviour. Columns: behaviour evaluated. Brackets: the caught behaviour itself. Last column: mean over the four behaviours not caught.

CaughtBackdoorFurlongMedicalReward hackingSabotageMean, not caught
Untreated77 / 6899 / 10096 / 10096 / 9594 / 10092 / 92
Reward hacking10 / 213 / 350 / 25(9 / 4)2 / 4319 / 18
Sabotage3 / 010 / 313 / 173 / 10(2 / 1)7 / 8
Medical18 / 312 / 15(0 / 0)6 / 61 / 339 / 14

Catching medical sandbagging leaves 6% on reward hacking, while catching reward hacking leaves 25-50% on medical. We see no evidence that reward hacking is a particularly effective target: on the two behaviours held out for all three targets (backdoor and furlong), collusion left is 4% for sabotage, 7% for reward hacking and 12% for medical, and these differences are not significant.

Part 2/2

W2 and Q1. Additional experiments show that training on clean reviews or generic honesty data also achieves approximately 78–83% reduction in held-out collusion.

We ran the matched comparison you ask for, for reward hacking and for the two new caught behaviours (W3 and Q2). For reward hacking the two are similar, as in Table 15. For the two new behaviours, reviews of caught misconduct transfer far more than matched clean reviews.

Q1a. More closely matched training objectives and datasets.

A caught review is an honest review of a submission that contains the misbehaviour, and it flags it. A clean review is an honest review of a sound submission from the same task, and it approves it. Both use SFT with the paper's trainer and settings, the same three review formats, verdict words and reviewer persona, and response tokens matched to within 3% (for reward hacking, 414,645 against 410,412).

Q1b. What still differs.

Clean reviews are shorter, so at matched tokens the monitor sees more of them (1,400-1,700 reviews against 900). Clean reviews approve and caught reviews flag, by definition. For reward hacking the caught reviews are all in the no-oversight context, while 343 of the 633 clean reviews are. The clean reviews for the two new behaviours used a different writer prompt; a blind judge (GPT-5.4) scored 100 reviews from each set similarly (4.95 against 4.99 out of 5).

W2a. Clean reviews.

Table R2. Held-out collusion left: the share of misbehaving items the monitor still approves, on the behaviours not caught. Untreated is 92%. Averages, with ranges across the six organisms.

Caught behaviourCaught reviewsClean reviews, same task
Alignment sabotage7% (3-10)59% (39-82)
Medical sandbagging12% (4-17)40% (24-54)
Reward hacking18% (15-23)16% (6-21)

For sabotage and medical sandbagging, caught reviews are lower in all six organisms and the ranges do not overlap (p = 0.03 each). For reward hacking the observed rates are similar (p = 0.31). The p-values are from an exact paired test across organisms; with six, 0.03 is the smallest possible value and means all six agree. This is not a result of flagging more: accuracy on control items is 92% after caught reviews (94% untreated, not significantly different) and 92% after clean reward-hacking reviews, and it falls to 73-74% after clean sabotage or medical reviews (p = 0.03).

W2b. Generic honesty data.

MASK. The 83% in Table 15 is DPO on MASK, and its control accuracy is 82%, against 92% for caught reviews. On sound submissions from the held-out behaviours, monitors trained with DPO on MASK fail to approve 20%, against 8% for DPO on caught reviews; with SFT it is 10% against 4%. The rate is higher with MASK in all six organisms for both methods (p = 0.03 each). So MASK reduces collusion, but the monitor then fails to approve more sound work.

Everyday-task reviews. To test whether honest reviewing in general is enough, we built a new control set: 633 honest, approving reviews of sound work on everyday tasks, with no question about misbehaviour, no oversight context and none of our review formats. With the same settings and token count it leaves 63% (55-72), better than no correction and much worse than caught reviews in all six organisms (p = 0.03 each), and it lowers control accuracy to 76%. Its content and format both differ from our other sets, so we cannot say which matters more.

W2c. Correcting a specific collusion behavior versus general improvements in honesty or monitoring.

It depends on the behaviour. For reward hacking, clean reviews from the same task do about as well as caught reviews (16% and 18% left), so there we cannot attribute the transfer to the caught content. Those clean reviews are not generic: they come from the reward-hacking task, and each asks whether the AI is gaming the rubric. We do not yet have an account of why reward hacking differs. For sabotage and medical sandbagging, clean reviews leave 59% and 40%, against 92% untreated, while caught reviews leave 7% and 12%. In both cases caught reviews do more than a general improvement in honesty or monitoring.

Dear Reviewer ZcJN,

We would greatly appreciate it if you had a chance to review our rebuttal. We are happy to clarify any remaining questions or concerns and would welcome the opportunity to continue the discussion.

We understand this is a busy period and are grateful for your time. Thank you again for your thoughtful and constructive review.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.