acceptodds
Under review as a conference paper at ICLR 2027

Improving scalable oversight with co-trained monitors

Abstract

Worker-monitor setups are a promising approach to AI oversight, but training workers against fixed monitors can incentivize monitor evasion. We study whether this failure mode can be avoided by _co-training the monitor alongside the worker_, and explore both supervised and self-supervised approaches. The supervised approach involves oracle access to gold-standard judgements from a principal (e.g. human). Monitoring – with vanishing error and query rates – turns out to be possible exactly when the class of worker strategies has finite Littlestone dimension, connecting our setting with an established literature on adversarial online learning. We also propose a self-supervised co-training procedure based on test-time distillation: the monitor uses additional test-time compute to generate higher-quality labels, then trains its standard-compute policy on those labels. We analyze conditions under which this procedure provably preserves the intended monitoring objective, keeping up with worker-induced distribution shift. We stress-test both approaches in code-security settings where the worker is trained adversarially to fool the monitor. Our results suggest that adaptive monitors are better at keeping pace with evolving worker strategies, while fixed monitors are more vulnerable to evasion.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, monitoring, online learning, co-training, reinforcement learning, reward hacking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RTTgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces theory for co-training monitors with agents (workers) that try to evade that monitor. They shows this is feasible with vanishing error and query rate if the monitor can occasionally get golden monitoring labels. They connect this to the theory of adversarial online learning. They also extend their framework to self-supervised online learning (sec 3), where the monitor uses test-time compute to generate its own labels: here they show that, under exact population fitting, the monitor will become increasingly confident in its original preferences, which may increase or decrease accuracy depending on these initial conditions. They also analyze the error as the worker adapts the training distribution to the monitor. They run experiments in two games: 1) the attack game (3 environments) tests whether a monitor jointly trained with an attacker can stay performant assuming it gets occasional ground truth labels, 2) the repair game (1 environment) instead tests whether jointly training the monitor makes the worker more likely to choose genuine solutions over cheats. In both games, they find that co-training is useful.

Critical strengths
  • The paper discusses an interesting and relevant problem: whether monitors can remain effective as agents adapt to them. Work improving our understanding of how monitors can adapt to such agents is valuable.
  • The paper is generally well-written and clear, although the experiments section could use more self-contained descriptions of tasks, models and labels.
Critical weaknesses
  1. The experimental section is somewhat underspecified in the main paper, although some details do exist in the appendix. This makes it unnecessarily difficult to interpret the experiments. I would suggest the following changes:
  • Some experimental details should move from the appendix to the main paper. Models used for monitor and workers are one such example. The actual sources for the “ground truth” labels, currently specified in App C, should be explained in the main text, since they are important for interpreting the results (e.g., Toxicity uses a Qwen3 14B judge).
  • Attack game” is introduced retroactively on p.8; would be clearer to define this earlier, when the tasks are defined, side-by-side with the “repair game”. Missing information:
  • In the repair game, how is a token operationalized? Tokenizer of the worker or monitor, or something else? Are insertions and deletions the same cost as a replacement? Such details are currently missing from the paper (or I missed them).
  • App C does not describe the reward used in Toxicity clearly: “smooth score that rises with the monitor’s confidence”.
  • App C does not describe the difficulty levels used by Escape-the-box in any detail, nor what “external destinations are gated” means.
  1. While the theory in section 2 provides useful feasibility results under, admittedly, idealized conditions, and the connection to online-learning seems valuable (although I cannot judge how established this connection is already), the self-supervised learning results of section 3 seem much less informative. My understanding is that the primary insights here are A) that a monitor will become more certain of its existing preferences, which seems somewhat expected from existing literature, and B) specifying strong assumptions under which the finite sample guarantee holds, and doesn’t say anything about how likely such a self-supervised setting is to be useful in practice. Additionally, the paper provides no experiments for this setting. I would like to see a discussion of what this section establishes beyond existing sharpening results and when the bound can be informative.

I think the paper overall discusses a worthwhile problem and provides a coherent initial treatment, but the theoretical contribution seems to largely adapt existing learning results, and the experimental verification provides limited evidence of under what conditions monitor co-training succeeds or fails. I would like to see a sharper story and a clearer distinction between the general usefulness of co-training, practical failure modes, and the practical value of the theory.

Questions for authors

How likely is it that sufficiently strong labels are available for co-training to be practically useful? Scalable oversight typically implies that models are stronger than any trusted monitors, such that labels are either noisy or hard to obtain.

1 comment

We thank reviewer RTTg for their comments and suggestions.

Regarding the experiments, on revision we will introduce the attack and repair games together and bring the models and label sources into the main text. The main experiments use Qwen3-1.7B monitors and Qwen3-4B workers, except for escape-the-box’s worker distribution over generator settings. Labels come largely from execution checks or a frozen Qwen3-14B toxicity judge.

To answer the specific implementation questions: repair edits use Python lexical tokens, excluding comments and formatting tokens, with insertions, deletions, and replacements each costing one edit. The toxicity reward is a sigmoid of the clipped CLEAN-minus-TOXIC log-likelihood margin, plus 11 for a CLEAN verdict; invalid rewrites earn −1-1. We will enumerate the box difficulty settings. A gated destination requires an access request before a successful write, but permission to write does not authorize leaking protected information.

Regarding novelty: Theorem 2.3 builds on established label-efficient online learning. Our contribution is its explicit formulation and self-contained proof for the discrete-time worker–monitor protocol, together with the subsequent conditional connection to worker incentives. We will clarify this distinction in the introduction.

Regarding Section 3: the self-supervised results are not the primary contribution of our work. Given that suppressing attacks with querying is possible, we want to make this approach as scalable as possible. Section 3 is getting at what this could look like, under some assumptions. We agree that sharpening the original monitor’s judgments need not correct systematic mistakes, and our exploratory experiments reinforce this concern. We will make these limitations clearer and reconsider the section’s prominence.

Regarding reliable labels: we certainly agree that their availability is an essential limitation. Our experiments use restricted tasks with executable checks or a stronger model judge, which remains fallible. The case we want to make is that occasional reliable corrections can sustain monitoring under our assumptions, not that such corrections are always available. We hope an occasionally-available human judge seems plausible to the reviewer.

We hope these clarifications address the reviewer’s concerns and that they will consider updating their recommendation.

Reviewer_fa2KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies worker–monitor interactions in which a worker can adapt its actions to exploit a monitor's errors. It proposes updating the monitor on evolving worker outputs. With occasional ground-truth queries, it characterizes when the expected average monitoring error and query rates can both vanish, connecting this possibility to online learnability. It also analyzes query-free self-training using majority-vote labels generated with additional inference-time computation. Experiments evaluate ground-truth-supervised co-training on code review, toxicity rewriting, information leakage, and a code repair game.

Critical strengths
  1. Theorem 2.3 gives a clear feasibility characterization: under realizable supervision, meaning that the true judgment function belongs to the assumed class, vanishing expected average monitoring error and query rates are simultaneously achievable against adaptive workers exactly when that class has finite Littlestone dimension. The result clarifies when increasingly sparse ground-truth feedback can, in principle, sustain monitoring, which directly addresses the supervision bottleneck motivating the paper.
Critical weaknesses
  1. The query-free contribution remains conditional and experimentally unvalidated. Theorem 3.1 requires coverage of the evaluation distribution by the worker's training distributions, a margin condition controlling how much probability lies near initially tied judgments, and sufficiently small accumulated fitting errors. These requirements are consequential when an adaptive worker seeks poorly monitored regions. Moreover, the guarantee describes sharpening toward the monitor's initial majority judgments, which can reinforce systematic mistakes rather than correct them. The paper acknowledges these limitations, but without experiments it remains unclear whether the proposed procedure preserves a useful accuracy advantage during joint worker–monitor adaptation.

  2. In the sparse-query setting, the advance over existing label-efficient online-learning results discussed in Section 2.1 remains unclear. Maintaining low monitoring error as the query rate decreases is central to the paper, but the empirical support is limited. Sparse co-training leaves substantial residual evasion, and the main experiments do not compare different query-decay schedules. Moreover, the monitors receive different amounts of labeled data and training, making it difficult to isolate the benefits of the query strategy. Comparisons under matched label budgets would make the evaluation more informative.

Questions for authors
  1. Does query-free self-training improve monitoring accuracy as the worker adapts?

  2. What does Theorem 2.3 add to existing label-efficient online-learning results? Does the decaying query schedule outperform fixed-probability or front-loaded querying under matched label and training budgets?

1 comment

We thank reviewer fa2k for their comments and are gratified that they found our paper overall to be clear, useful, and sound.

The self-supervised results are not the primary contribution of our work, but more of a proposal/thought experiment that we believe follows naturally from our main results. Given that suppressing attacks with querying is possible, we want to make this approach as scalable as possible. Section 3 is getting at what a more scalable approach could look like. We acknowledge that the query-free, self-supervised case remains experimentally unverified and we leave this for future work. We would like to be clear that this approach is not one which we expect to suppress exploitation in general, as the review noted and we discuss in the paper. As the worker adapts, new exploits that would trick the original monitor may still deceive the updated monitor. Intuitively the advantage we expect for self-supervised training is essentially sharpening on the original monitor's judgements, so the updated monitor basically can adapt without drifting too much.

In our sparse-query setting, our experiments already compare no querying, decaying query rate, and every-update querying and show a monotonic trend. We expect that different query decay rates will also follow this trend, largely due to the fact that they have different query budget. Indeed your point about variable budgets is well taken -- it is not obvious that our random-querying strategy is optimal or anywhere near so, and in upcoming revisions we will compare different querying strategies under matched budgets. Our preliminary results show that different strategies like uniform sampling, decaying-rate sampling (1/t1/\sqrt{t}), and uncertainty-based sampling (as in the appendix) all perform similarly with matched budgets. We will include these in a revision. The case we want to make is not that our strategy is optimal, but rather that suppressing attacks with vanishing query rate is feasible.

Regarding novelty of our theoretical results, Theorem 2.3 builds on established label-efficient online learning. Our contribution is its explicit formulation and self-contained proof for the discrete-time worker–monitor protocol, together with the subsequent conditional connection to worker incentives. We will clarify this distinction in the introduction. We believe that this is a useful contribution and that our paper has utility in advocating for further research into scalable monitoring approaches with dynamic monitors.

We would ask the reviewer to consider updating their score if they find our core contributions convincing, and the paper to be clear (given their rating of 3) and sound (also 3) overall.

Reviewer_5mReOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of learning a policy subject to feedback from a 'monitor', under the frame of online learning. The idea is that we may have a monitor for undesirable behavior, such as reward hacking. The monitor is an imperfect surrogate for some ground-truth feature of the policy outputs (e.g. a text-classifier or activation probe trained on ground-truth labels from a human labeller). For a fixed monitor, the policy learns to exploit the imperfections in the surrogate. The main claimed contribution of the paper is to show that if the monitor is trained alongside the policy, the policy will not learn to exploit imperfections in the surrogate, and that this can be achieved with a rate of querying the ground truth that goes to zero over time.

Critical strengths
  • The clean framing from cheating against surrogate monitoring to the online learning sense is very nice, and the paper is generally written very clearly and in an easy-to-understand manner. The general problem of optimization against a monitor is very timely and I hope that the framing of this problem in terms of online learning will let a new set of people pick up the issue and come up with impactful contributions
  • As far as I can tell, several useful results are proved in the paper, including the result that the rate of cheating goes to zero over time in this setting.
Critical weaknesses
  • It is sometimes unclear which contributions of the paper are entirely novel and which are existing work or small extensions of existing work. The individual sections do clearly state when existing results overlap with the proven results, but e.g. the introduction could be a bit clearer about situating the contributions in the context of the previous work.
  • The section 3 seems somewhat disconnected from the rest of the work and superfluous/a somewhat contrived setting. I can understand the desire to remove the annoying ground-truth dependence, but in many of the cases we are most interested in this sort of setup (such as reward hacking), the errors are more directed and the modal/noisy majority votes assumptions don't seem to hold.
Questions for authors
  • Could you elaborate a bit on the novelty/significance of the contributions of the work? While the information is there, it requires reading between the lines somewhat. A straightforward statement of what the work delivers relative to the previous work and what insights/practical implications this delivers would clear up the uncertainty I have that's preventing me fully recommending a clear accept.
  • Can you elaborate on the implications of the 'no reward for honest work' characteristic of most of the settings? On the one hand it seems consequential that there is no 'honest path' for the policy, but perhaps this doesn't matter?
  • Would it be possible to conduct an experiment with an activation-based white box probe monitor? This would be quite a bit less 'toy' than the existing experiments and would potentially have implications for real RL deployments.
1 comment

We are glad the reviewer 5mRe found our paper to be clear and useful, and thank them for their comments and suggestions.

Regarding novelty: Theorem 2.3 builds on established label-efficient online learning. Our contribution is its explicit formulation and self-contained proof for the discrete-time worker–monitor protocol, together with the subsequent connection to worker incentives. The case we want to make is that suppressing attacks with a vanishing query rate is feasible under our theoretical assumptions, and that accurate monitoring can furthermore encourage honest completion under the right reward structure. Our experiments qualitatively produce the effects we expect in finite time, as co-trained monitors suppress worker attacks (misaligned behavior) compared to fixed monitors. We think this result is intuitive -- a static monitor is unable to adapt to the changing worker distribution, but we can mitigate this problem by allowing the monitor to adapt. This should be an important insight for safety and alignment researchers training language models (who use monitors, but basically just deploy a static version of whatever their best current model is, see e.g. https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/).

Regarding "no reward for honest work": we certainly agree this is a consequential distinction, which we attempted to emphasize in the paper (and will clarify futher on revision). On the one hand it is not an issue for the adversarial setting we study in those experiments -- we are concerned with the monitor's robustness to adversarial worker attacks. On the other hand, this is an essential thing to consider when thinking about maintaining alignment for language models during training and evaluation (arguably an equally consequential and impactful setting). We illustrate in Figure 2 that a worker's reward structure dictates whether or not alignment is feasible -- we certainly think that safety and alignment researchers should be considering this point, and that models should always be trained in a regime where honest work is rewarded appropriately (though this kind of reward shaping can be difficult).

Regarding activation-based monitoring: this is certainly possible, but we don't feel that this is realistic in the adversarial setting. For example, in code review an external agent commits to your repo -- you receive the commit content but neither their chain of thought nor white box access. Here we want to emphasize that the monitors do NOT have access to the worker's chain of thought -- we use the monitor reasoning traces during co-training in order to reinforce helpful reasoning, but the monitor only sees the worker's output. In an alignment setting where you monitor an agent during training white-box access is more plausible -- we appreciate this suggestion and will try implementing it for a future revision.

Regarding section 3: The self-supervised results are not the primary contribution of our work, but more of a proposal/thought experiment that we believe follows naturally from our main results. Given that suppressing attacks with querying is possible, we want to make this approach as scalable as possible. Section 3 is getting at what a more scalable approach could look like, and is mostly a suggestion for future work!

We hope that the reviewer will consider strengthening their recommendation to accept if they feel we have addressed their questions and concerns adequately!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.