- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the phenomenon of model self-collusion, where using a model to monitor its own rollouts may fail due to the model preferentially not flagging its own rollouts. The paper produces a model organism with this behavior through synthetic document finetuning in five different regimes simultaneously. The main result is that fine-tuning to remove this self-colluding behavior in one regime leads to generalization: the model's self-collusion rate also reduces on the other four regimes.
Critical strengths
- The paper is well-written and clearly describes the approach, gives good motivation for why the problem is important, and clearly states the conclusions, as well as limitations. The topic is clearly of some importance given monitoring failures underlying several recent AI security incidents, so I'm glad to see work on this important topic.
- The work is overall very solid. Given a high-level description of 'examine the extent to which self-collusion-busting fine-tuning generalizes', this work carries out the execution to a really high standard. For example, the installation of the self-collusion could have been done via just SFT on some off-policy rollouts only, but the additional use of SDF adds to the realism of the setup. In most steps, several different alternatives are used (e.g. SFT plus CWS & DPO) to rule out results that may only hold due to methodological quirks.
Critical weaknesses
The main weakness that I can see is simply the limited scope of the work. The paper quite conclusively establishes that, for SDF-induced monitor self-collusion in several regimes, fine-tuning to reverse that self-collusion in one regime appears to generalize substantially to other regimes. While a conclusive finding, and certainly a useful one to establish and record as a technical report, it's quite limited in scope.
I think there are a few ways that the scope could be extended to recommend a clear acceptance, even with a fairly minimal amount of work that could be done in a rebuttal. One way is to give a (maybe even only preliminary) analysis of how this SDF model organism compares to self-collusion that occurs via a more 'natural' pathway. Even preliminary evidence showing that fine-tuning on a weakly exhibited natural self-preference in domain A also reduces self-preference in domain B (or similar for an emergently-misaligned model) would suggest that this work has takeaways that extend beyond the particular setting studied here.
Another way that the scope could be extended might be to consider modifications or relaxations of the threat model. For example, in the motivation we assume that the model is being used as a monitor for itself because we have no other sources of trusted supervision. In that case, where does the supervision come from to get the 300 honest and colluding reviews for fine-tuning? Perhaps the paper could explore some scheme for constructing those contrastive reviews when only access to the untrusted model is possible.
Questions for authors
- Do you have any (even preliminary) evidence for whether this fine-tuning generalization would transfer for self-collusion that arises naturally?
- Could you elaborate a bit more on the threat model and where the fine-tuning data comes from? As far as I understand, it seems like the motivation implies that the untrusted model is used for monitoring since the trusted model isn't good enough to monitor the untrusted monitor, but then an oracle is used to generate the fine-tuning data, which seems inconsistent.
- Do you have any results or intuition on how the results would change if the monitor also had to be updated via RL instead of supervised fine-tuning or similar? For instance, if there was limited access to an oracle monitor which merely says if the colluding monitor has been colluding or not. As I understand, the intervention corpus is written to contain the correct (non-colluding) verdict, as well as non-colluding reasoning. A weaker setup might allow you to reinforce the non-colluding verdict, while not being able to specify non-colluding reasoning (e.g. if the CoT is unintelligible or continuous)
4 comments
This is comment 2/2
W2 and Q2. The threat model and where the fine-tuning data comes from.
We agree the threat model is clearer with this relaxed: we assume a small budget of trusted auditing alongside the untrusted monitor, and that audit is the source of the catch. The audit is too costly to cover every action, so the untrusted monitor is still needed, which is why the two parts of the setup are consistent. We will state this in the paper.
Where the reviews come from (W2b). The training examples are submissions on which the untreated monitor covered up misbehaviour. For each, an inexpensive model (GPT-4.1-mini) receives the task, the monitor's incorrect review, the correct verdict and the evidence of what is wrong, and writes the review the monitor should have given. It is never asked to decide whether something is misbehaviour. The contrastive methods also use a colluding review for each case, written to the organism's cover-up template. In deployment this would be the monitor's own cover-up.
From one catch to 300 examples. A catch identifies the behaviour. The 300 cases are built afterwards: we construct further submissions that contain the caught behaviour, so the correct verdict is known by construction, show them to the monitor, and keep the ones it approves. In our experiments this takes 100-124 distinct submissions, each shown in up to three review formats. This assumes the caught behaviour can be reproduced on demand, which holds for our installed behaviours and may be harder for behaviour that arises naturally.
Whether misconduct examples are needed. Our new experiments bear on this. Clean reviews from the same task need no example of the misconduct, only honest reviews of sound work on the task where it was caught. For reward hacking they do about as well as reviews of caught misconduct (16% of held-out collusion left, against 18%). For alignment sabotage and medical sandbagging they do much worse (59% and 40%, against 7% and 12%). So once the task is known, reviews of sound work produce similar observed transfer for reward hacking, while reviews of misconduct produce substantially greater transfer for sabotage and medical sandbagging. Separately, Q3 tests whether the corrective targets need to include the reasoning.
What we have not done (W2a, W2c). We do not study how the first catch happens, and we have not built a scheme that constructs the honest reviews with access to the untrusted model only. We will state both as limitations.
Q3. Reinforcing the verdict without specifying the reasoning.
We did not run RL. We tested a related weaker-supervision setup: SFT on the correct verdict only, with no reasoning, on 300 caught reward-hacking cases in all six organisms. It removes most held-out collusion (94-96% on Llama, 76-91% on Qwen), but the monitor then rejects sound work: accuracy on control items falls to 57-79%, and on Llama it rejects every clean reward-hacking submission. The replies show shortcut learning. On Llama about 99% of replies are one stock sentence. On Qwen clean work is approved 35-86% of the time when the review is watched and 1-6% when it is unwatched, as every training case was.
The 300 honest reviews in the paper are just as one-sided, since every one ends in a flag, but they include the reasoning, and with them the monitor is still correct on 88-96% of control items (Table 15). This is why our correction data includes the reasoning. We have not tested RL, or verdict-only training with both flag and approve cases. For the setting you describe, where the reasoning cannot be specified, our result indicates that verdicts on caught cases alone are not sufficient supervision. Whether verdicts on sound work as well would be enough is the open question, and we will state it in the paper.
Thanks for the detailed reply. The new experiments address a lot of the concerns with the paper which I originally had. Although there are still limitations, the overall significance of the problem means I'll now recommend acceptance.
Thank you for your confidence in our work! We believe that your suggestions and the new experiments have materially improved our paper.
The Authors
Hello, thank you so much for taking the time to read and engage with our work. This is comment 1/2.
W(a). Limited scope.
We agree the submitted paper was limited to collusion installed in our own organisms. We extended it in the three directions you suggest: natural collusion (W1, Q1), the threat model (W2, Q2) and weaker supervision (Q3).
We also ran three new experiments on the model organisms in the paper.
More caught behaviours. The paper corrects the monitor on one caught behaviour, reward hacking. We repeated this with alignment sabotage and with medical sandbagging. Each correction also reduces collusion on the behaviours that were not caught: before correction the monitor approves 92% of that misbehaving work, and after correcting reward hacking, sabotage or medical sandbagging it approves 18%, 7% and 12%.
Whether caught examples are needed. We compared training on reviews of caught misconduct with training on the same amount of clean reviews from the same task. The results are under W2 below.
Tasks the organisms were never trained on. We wrote 15 new workplace scenarios unrelated to the five behaviours. The organisms approve flawed work there in 29% of cases on Llama and 54% on Qwen, against under 1% for the original models. Correcting reward hacking alone removes most of this.
Details of the first two are in our response to Reviewer x26F.
W1 and Q1. Collusion that arises naturally.
Fine-tuning on caught examples reduces judge-rated coordination between sellers to the minimum, including under a condition absent from the fine-tuning data. None of this coordination was installed by us.
Setup. We use the double-auction market of Agrawal et al. (2025). Five sellers and five buyers trade for 30 rounds. The competitive price is 90 dollars and the sellers' joint-profit price is 100 dollars. The sellers are copies of Gemini 3.5 Flash. A judge (GPT-4.1-mini) scores each seller's written reasoning every round from 1 (very low or none) to 4 (very high). This is sellers coordinating prices, a different kind of collusion from a monitor covering up.
Result. Fine-tuning on caught examples brings the coordination score from 2.80 down to 1.00, the lowest possible value. Before fine-tuning, with messaging allowed, the sellers coordinate openly, with messages such as "hold the line together in the high 90s". To correct this we took 300 seller turns in which Gemini was coordinating, rewrote each as an independent seller would act, and fine-tuned Gemini on them. The same drop appears under a CEO-pressure condition that was not in the fine-tuning data: from 2.93 to 1.00. It is consistent across auctions: over the five auctions in each cell, the untuned scores range from 2.47 to 3.13 and the fine-tuned scores from 1.00 to 1.01.
Controls. Two control fine-tunes with the same number of examples and epochs (300 examples, 3 epochs), on MASK honesty data and on clean reviews of unrelated tasks, leave the coordination largely in place: their mean scores are 2.68-3.24 (individual auctions 2.14-3.77), against 2.80-2.93 untuned and 1.00 after training on caught examples (five auctions per cell).
Scope. The correction was trained in one condition (sellers messaging each other) and also brings judge-rated coordination to the minimum in a second (CEO pressure). Both are in the same market, so this is not yet transfer between two different domains. It does show that the effect of correcting on caught examples carries over to a condition it was not trained on, for behaviour that we did not install.