- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces theory for co-training monitors with agents (workers) that try to evade that monitor. They shows this is feasible with vanishing error and query rate if the monitor can occasionally get golden monitoring labels. They connect this to the theory of adversarial online learning. They also extend their framework to self-supervised online learning (sec 3), where the monitor uses test-time compute to generate its own labels: here they show that, under exact population fitting, the monitor will become increasingly confident in its original preferences, which may increase or decrease accuracy depending on these initial conditions. They also analyze the error as the worker adapts the training distribution to the monitor. They run experiments in two games: 1) the attack game (3 environments) tests whether a monitor jointly trained with an attacker can stay performant assuming it gets occasional ground truth labels, 2) the repair game (1 environment) instead tests whether jointly training the monitor makes the worker more likely to choose genuine solutions over cheats. In both games, they find that co-training is useful.
Critical strengths
- The paper discusses an interesting and relevant problem: whether monitors can remain effective as agents adapt to them. Work improving our understanding of how monitors can adapt to such agents is valuable.
- The paper is generally well-written and clear, although the experiments section could use more self-contained descriptions of tasks, models and labels.
Critical weaknesses
- The experimental section is somewhat underspecified in the main paper, although some details do exist in the appendix. This makes it unnecessarily difficult to interpret the experiments. I would suggest the following changes:
- Some experimental details should move from the appendix to the main paper. Models used for monitor and workers are one such example. The actual sources for the “ground truth” labels, currently specified in App C, should be explained in the main text, since they are important for interpreting the results (e.g., Toxicity uses a Qwen3 14B judge).
- Attack game” is introduced retroactively on p.8; would be clearer to define this earlier, when the tasks are defined, side-by-side with the “repair game”. Missing information:
- In the repair game, how is a token operationalized? Tokenizer of the worker or monitor, or something else? Are insertions and deletions the same cost as a replacement? Such details are currently missing from the paper (or I missed them).
- App C does not describe the reward used in Toxicity clearly: “smooth score that rises with the monitor’s confidence”.
- App C does not describe the difficulty levels used by Escape-the-box in any detail, nor what “external destinations are gated” means.
- While the theory in section 2 provides useful feasibility results under, admittedly, idealized conditions, and the connection to online-learning seems valuable (although I cannot judge how established this connection is already), the self-supervised learning results of section 3 seem much less informative. My understanding is that the primary insights here are A) that a monitor will become more certain of its existing preferences, which seems somewhat expected from existing literature, and B) specifying strong assumptions under which the finite sample guarantee holds, and doesn’t say anything about how likely such a self-supervised setting is to be useful in practice. Additionally, the paper provides no experiments for this setting. I would like to see a discussion of what this section establishes beyond existing sharpening results and when the bound can be informative.
I think the paper overall discusses a worthwhile problem and provides a coherent initial treatment, but the theoretical contribution seems to largely adapt existing learning results, and the experimental verification provides limited evidence of under what conditions monitor co-training succeeds or fails. I would like to see a sharper story and a clearer distinction between the general usefulness of co-training, practical failure modes, and the practical value of the theory.
Questions for authors
How likely is it that sufficiently strong labels are available for co-training to be practically useful? Scalable oversight typically implies that models are stronger than any trusted monitors, such that labels are either noisy or hard to obtain.
We thank reviewer RTTg for their comments and suggestions.
Regarding the experiments, on revision we will introduce the attack and repair games together and bring the models and label sources into the main text. The main experiments use Qwen3-1.7B monitors and Qwen3-4B workers, except for escape-the-box’s worker distribution over generator settings. Labels come largely from execution checks or a frozen Qwen3-14B toxicity judge.
To answer the specific implementation questions: repair edits use Python lexical tokens, excluding comments and formatting tokens, with insertions, deletions, and replacements each costing one edit. The toxicity reward is a sigmoid of the clipped CLEAN-minus-TOXIC log-likelihood margin, plus 1 for a CLEAN verdict; invalid rewrites earn −1. We will enumerate the box difficulty settings. A gated destination requires an access request before a successful write, but permission to write does not authorize leaking protected information.
Regarding novelty: Theorem 2.3 builds on established label-efficient online learning. Our contribution is its explicit formulation and self-contained proof for the discrete-time worker–monitor protocol, together with the subsequent conditional connection to worker incentives. We will clarify this distinction in the introduction.
Regarding Section 3: the self-supervised results are not the primary contribution of our work. Given that suppressing attacks with querying is possible, we want to make this approach as scalable as possible. Section 3 is getting at what this could look like, under some assumptions. We agree that sharpening the original monitor’s judgments need not correct systematic mistakes, and our exploratory experiments reinforce this concern. We will make these limitations clearer and reconsider the section’s prominence.
Regarding reliable labels: we certainly agree that their availability is an essential limitation. Our experiments use restricted tasks with executable checks or a stronger model judge, which remains fallible. The case we want to make is that occasional reliable corrections can sustain monitoring under our assumptions, not that such corrections are always available. We hope an occasionally-available human judge seems plausible to the reviewer.
We hope these clarifications address the reviewer’s concerns and that they will consider updating their recommendation.