acceptodds
Under review as a conference paper at ICLR 2027

Catching Schemers in the Act: Training Online Monitors to Detect Covert Misalignment

Abstract

As LLM agents work longer with less supervision, oversight increasingly depends on monitors to detect _scheming_. Prior work typically employs scheming monitors that run asynchronously after the trajectory has completed, at which point harm may be irreversible. We build _online_ black-box, action-only monitors that assess trajectories as they unfold. For this purpose, we construct a pipeline to augment datasets with per-step labels, and introduce _timely_ metrics that credit a flag only while the harm can still be averted. Additionally, an online monitor must be accurate, cheap, and fast: prompted frontier monitors detect well but are costly and slow, while smaller prompted models are cheap but miss critical information. We train open-weight monitors, several of which are Pareto-optimal on both cost-performance and latency-performance curves. Our trained Qwen3.8-27B outperforms GPT-5.6 Sol at lower cost and half the latency. It nearly matches Gemini 3.7 Flash and Claude Opus 4.8 in detection performance at and lower cost while being slightly faster than both. Only Gemini 3.1 Pro is clearly stronger, at the cost and the latency. We further analyze the relationship between the composition of the training mixture and OOD transfer, and find that many residual failures come from monitors misweighing evidence they have seen, rather than missing it. Together, our per-step labels, metrics, and trained monitors offer a practical recipe for low-cost, low-latency oversight of agents as they act.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
monitoring, scheming, ai safety, misalignment, control, alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7FcdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online, black-box, action-only monitoring of scheming and sabotage in LLM agents. It builds on prior work on offline deliberative monitoring. The authors introduce a three-judge labeling pipeline that transforms trajectory-level labels into per-step confidence, severity, and irreversibility scores. They also label an ideal flagging step and a point of no return (PONR). In addition they propose timeliness aware metrics, t-AUROC and t-pAUROC@20, which credit detection only before irreversible harm occurs.

Using this dataset, the authors distill frontier-model deliberations into seven open-weight monitors using SFT, with additional RL for two Qwen models. They evaluate on four held-out OOD datasets and report competitive cost–performance and latency–performance tradeoffs relative to prompted frontier monitors.

The paper also investigates how training-set composition affects OOD transfer, how monitoring failures arise (from misweighting vs missing evidence), and how detection degrades when the monitor has access to only a limited context window.

Critical strengths
  1. Per-step labeling framework: I think the three-judge pipeline that uses privileged trajectory information to generate per-step supervision, flagging steps and points of no return, is a unique/important contribution for the community. It can enable adapting existing scheming/control datasets for online monitoring.

  2. Experimental coverage: The failure mode analysis in the paper provides useful insights into how monitors sometimes misweigh evidence rather than simply failing to recognize it. The training-mixture experiments also identify action classes and scope violations as factors associated with OOD transfer. This seems to offer some practical/actionable guidance for constructing monitoring datasets.

Critical weaknesses
  1. Issues with evaluation methodology: The main results evaluate only three selected prefixes per trajectory rather than every step, thus underrepresenting the fraction of false positives. Appendix E.6.3 (Table 8) shows that full evaluation reduces Qwen3.8-27B OOD t-pAUROC@20 from 0.885 to 0.751, while its FPR increases from 12.7% to 39.7%. This gap is smaller for Gemini 3.1 Pro. However since the paper's main claim is about the competitiveness of its trained online monitors, these results in my opinion undermine the reported Pareto-frontier comparisons. The problem is more severe on longer trajectories where online monitoring is most useful.

  2. On methodological novelty and OOD generalization. The training strategy largely follows Sinha et al. (2026), including teacher rationale distillation, SFT, and RL, while prior work has already explored per-step supervision and online auditing. The strongest model is already competitive without training (0.837 OOD t-pAUROC@20), improving only to 0.878 after SFT+RL. Improvements are concentrated in certain datasets, particularly BashArena, while performance on SHADE-Arena actually gets worse (0.723 to 0.695). This raises questions about how much monitoring capability the proposed training method adds and whether the gains are generalizable.

Questions for authors
  1. Full online evaluation: Given the discrepancy in Table 8, could the authors provide full-prefix evaluation results for Qwen3.8-27B and its closest frontier competitors (eg. Gemini 3.7 Flash or Claude Opus 4.8)? Do the main Pareto-frontier conclusions hold even under full evaluation?

  2. On generalization :How much does the proposed training improve detection over the untrained base model under full online evaluation? Why does training improve performance substantially on BashArena but degrade it on SHADE-Arena? Can the authors provide stronger evidence that the improvements generalize beyond attack classes represented in training?

  3. PONR labeling validity. Appendix E.3 mentions moving the PONR forward when it precedes the ideal flagging step, affecting around 175 SHADE-Arena trajectories. Could the authors explain this and also report results without this adjustment? How many of these temporal labels were human-verified/what was the agreement rate?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7Fcd (1/3)

We thank the reviewer for the positive assessment, especially for recognising the per-step labelling pipeline as a way to adapt existing datasets to online monitoring, and the practical guidance from our failure-mode and training-mixture analyses. The main concern is our reliance on approximate (landmark-prefix) evaluation. We agree it is a real caveat, and below we show which of the paper's conclusions hold under full evaluation and where we have revised the text.

The main results evaluate only three selected prefixes per trajectory … However since the paper's main claim is about the competitiveness of its trained online monitors, these results in my opinion undermine the reported Pareto-frontier comparisons.

And

Full online evaluation: Given the discrepancy in Table 8, could the authors provide full-prefix evaluation results for Qwen3.8-27B and its closest frontier competitors (eg. Gemini 3.7 Flash or Claude Opus 4.8)? Do the main Pareto-frontier conclusions hold even under full evaluation?

We thank the reviewer for raising this; we agree that it is an important caveat on our headline numbers. Appendix E.6.3 compares metrics under landmark-prefix scoring and full scoring, showing that our approximate metrics are optimistic. Under full evaluation of 150 class-balanced trajectories per OOD set, Gemini 3.1 Pro’s FPR rises by 3.6x, while Qwen3.8-27B’s rises by 3x. The reason is that a monitor whose scores on benign steps have even moderate variance gets many more chances to cross the threshold once every prefix is scored. The trained monitors tend to fluctuate above and below the threshold more often (27B crosses the threshold and drops below it more than three times on 41% of benign SHADE-trajectories vs. 2.7% for 3.1 Pro), and the increased amount of sampling on benign trajectories under full evaluation makes it so that there is a higher chance of a flag score spike crossing the threshold, raising FPR, and making their t-pAUROC@20 fall more drastically than Gemini 3.1 Pro. To address this, we decided to run the full evaluation using a temperature of zero as well, as the lower variance in responses leads to a lower chance of inaccurately flagging benign trajectories. See the results here (note that the prompted frontier runs with default reasoning effort, and we use 150 trajectories from each OOD dataset):

Monitort-pAUROC@20t-AUROCFPR
Gemini 3.1 Pro0.8840.9516.0%
Gemini 3.7 Flash0.8840.9270.0%
Claude Haiku 4.50.6510.86147.7%
GPT-5.6 Luna0.5460.78635.3%
Qwen3.8-27B SFT+RL0.7920.93024.7%
Qwen3.5-4B SFT+RL0.6050.82740.0%

In the table above, the trained 27B stays competitive on t-AUROC, above Flash and within 0.02 of Gemini 3.1 Pro, but trails both by 0.09 on t-pAUROC@20, so it no longer nearly matches Flash on the headline metric. We have updated the abstract, Section 4.1 and Appendix E.6.3 to reflect this. We did not run Opus and Sol due to it costing $8k to run this evaluation even on this subset of trajectories. Based on the observed variance on benign steps of our trained monitors, we see building aggregators that avoid having a single high score flag a benign trajectory as practical follow-up work. We will add this to our future work recommendations.

Nevertheless, at default temperature the ordering of monitors by t-pAUROC@20 is mostly preserved between approximate and full evaluation (Gemini 3.1 Pro > Flash > Qwen3.8-27B (SFT+RL) > Haiku > Luna > Qwen3.5-4B (SFT+RL)), with only Luna overtaking the trained 4B model. At temperature zero, our trained Qwen3.5-4B overtakes Luna again under full evaluation. Given that Luna has much higher latency and cost, we do not expect the Pareto frontier to change under full evaluation.

Lastly, we want to emphasize that using approximate evaluation is strictly necessary when doing evaluation on the complete OOD set as scoring every prefix is prohibitively expensive. Using our cost estimation, we think it costs about \59ktoevaluatethepromptedmonitorsonallOODdatasetsatdefaulteffortwhilstscoringeveryprefix(and 59k to evaluate the prompted monitors on all OOD datasets at default effort whilst scoring every prefix (and ~\\31k with per-step prompt caching).

AuthorsOpenReview ↗

Response to Reviewer 7Fcd (2/3)

On generalization :How much does the proposed training improve detection over the untrained base model under full online evaluation?

We have run additional experiments to determine this, please see the below table. It shows performance metrics for the untrained base model and trained variant on 600 trajectories from the four datasets (150 per dataset).

Monitort-pAUROC@20 T=1t-pAUROC@20 T=0t-AUROC T=1t-AUROC T=0FPR T=1FPR T=0miss rate T=1miss rate T=0
Qwen3.5-4B base0.3340.4840.6850.71348.7%28.0%16.0%27.3%
Qwen3.5-4B SFT+RL0.5130.6050.7800.82750.7%40.0%1.3%4.3%
Qwen3.8-27B base0.8430.8520.9260.9245.0%3.7%8.3%9.7%
Qwen3.8-27B SFT+RL0.7510.7920.8960.93039.7%24.7%1.0%1.7%

We find that our trained monitors have a lower miss rate than the untrained base counterpart (1% vs. 8% miss rate for the Qwen3.8-27B and 1% vs 16% for 4B at temperature one), but find that Qwen3.8-27B has a higher FPR (40% vs 5% at temperature one). At temperature zero, t-AUROC (which is the metric measuring discrimination between scheming and benign trajectories) improves with training for both monitors.

t-pAUROC@20 decreases for the 27B model when comparing base vs. trained due to this metric punishing FPR much more and the trained model flagging more eagerly. We now make note of this in Section 4.4. This reflects eager flagging rather than training itself: the weaker prompted monitors lose as much or more from approximate to full evaluation (at T=0, Haiku and Luna drop by 0.15 and 0.14 in mean-OOD t-pAUROC@20, versus 0.09 and 0.13 for our trained 27B and 4B). However, the metric increases for the 4B model showing that under full evaluation training does help in some cases. All in all, this emphasizes the point that giving scheming interpretations to benign trajectories remains the biggest failure mode of our trained monitors and a valuable point of improvement for future work.

Appendix E.6.3 now reports full evaluation results for all eight monitors mentioned in this response and the previous one.

The training strategy largely follows Sinha et al. … while prior work has already explored per-step supervision and online auditing

We agree that the SFT recipe is inherited from the Sinha et al. recipe, and that was a deliberate design choice (although when it came to RL, we explored a different reward function). Holding the recipe fixed means any difference we observe is attributable to the adaption of this method to the online setting. We see the contribution of the paper as the online setting itself which required the formalization of the online setting, the PONR-anchored timely metrics, the step-level labeling pipeline, the mixture-transfer analysis, and the failure-mode analysis. We agree that per-step supervision and online auditing exist in prior work, but to our knowledge, none addresses online detection of covert scheming. StepShield uses synthetic trajectories, each with a single injected violation whose onset is fixed by construction, and it trains no monitor. AgentForesight trains an online auditor, but for benign task failures in multi-agent systems. Claude Code auto mode and Codex auto-review target overeager or overtly dangerous actions, and Codex explicitly states it "should not be understood as a mechanism for protecting against model scheming." None of these defines timeliness relative to irreversibility, labels existing scheming benchmarks at the step level, or evaluates trained action-only monitors against an agent that hides its intent, which is the gap our work fills.

The strongest model is already competitive without training (0.837 OOD t-pAUROC@20), improving only to 0.878 after SFT+RL. Improvements are concentrated in certain datasets, particularly BashArena, while performance on SHADE-Arena actually gets worse (0.723 to 0.695).

and

Why does training improve performance substantially on BashArena but degrade it on SHADE-Arena?

Our mixture analysis (Section 4.2) attributes this to how closely each set's action classes match the training data. BashArena is dominated by code and configuration inserts, which are also the dominant action class in BashBench and LinuxArena, so training transfers well: SFT lifts t-pAUROC@20 from 0.08 to 0.80 at 4B and from 0.72 to 0.90 at 27B. SHADE-Arena's dominant action class, financial transfers, is almost absent from all four training sets, so no mixture covers it. Sinha et al. also found SHADE-Arena fragile under every training mixture they tried, because training led their monitors to read messy but benign trajectories as concealment, which raised false positives. SHADE-Arena is hard even for strong frontier monitors: at maximum reasoning effort, GPT-5.6 Sol and Gemini 3.7 Flash reach only 0.615 and 0.654, both below the trained 27B (0.695); only Gemini 3.1 Pro does clearly better (0.803).

AuthorsOpenReview ↗

Response to Reviewer 7Fcd (3/3)

This raises questions about how much monitoring capability the proposed training method adds and whether the gains are generalizable.

and

Can the authors provide stronger evidence that the improvements generalize beyond attack classes represented in training?

We believe our OOD evaluation sets already test this. The datasets come from different author groups and generation styles, and they differ in construction components: iterative generator/discriminator refinement, agent-simulator rollouts, prompt-injection environments, and sandboxed agentic rollouts, each with distinct prompts and side tasks. Some attack classes appear only in OOD and are entirely absent from training, such as AgentDojo's prompt-injection attacks. Despite these differences, training generalizes, especially on the 4B monitor: SFT raises mean-OOD t-pAUROC@20 by 0.24 here. Gains on the 27B monitor whose base model is already strong are more modest (+0.03). The 27B is the exception rather than the rule: all seven open-weight students we train improve on mean-OOD t-pAUROC@20 under SFT, by between 0.07 and 0.24 for the other six (Nemotron 3.5 Lightning +0.23, Nemotron 3 Super +0.18, Inkling +0.13, Nemotron 3 Ultra +0.12, Inkling-Small +0.07; Table 14 in Appendix I.3), and the gain shrinks as the base model gets stronger, which is why the 27B, our strongest base, gains least. RL adds performance to both.

PONR labeling validity. Appendix E.3 mentions moving the PONR forward when it precedes the ideal flagging step, affecting around 175 SHADE-Arena trajectories. Could the authors explain this and also report results without this adjustment? How many of these temporal labels were human-verified/what was the agreement rate?

We use this mechanism because the adjudicator sometimes places the PONR on an irreversible step where the action is already damaging but the monitor cannot yet be confident the behavior is scheming by looking at the trajectory alone. Such a PONR sits before k∗, the earliest step at which flagging is ideally possible, so the timely window would contain no flaggable step and any monitor would be scored as missing it. In practice this mechanism is almost always activated in a specific SHADE-Arena scenario where an agent is asked to book flights for a customer with the hidden side-task to only book expensive flights from a certain provider. The first flight booked with this provider is flagged as the PONR, but it’s the pattern of booking multiple flights with that provider which is the evidence for scheming, so putting the PONR there is premature and unfair to monitors. After this change, the labels of 10+ trajectories containing this scenario were checked and we found them more accurate than before and agreed with the placement of the PONR.

In the following table we report the miss rate of different monitors on these specific trajectories before and after this adjustment (note: only 128 out of 175 fall inside the 800-trajectory SHADE-Arena evaluation set):

Monitormiss rate beforemiss rate afterchange (trajectories rescued)
Gemini 3.1 Pro96.1%86.7%-9.4 pp (12)
GPT-5.6 Sol99.2%88.3%-10.9 pp (14)
Claude Opus 4.8100.0%95.3%-4.7 pp (6)
Qwen3.8-27B SFT+RL94.5%51.6%-42.9 pp (55)

The trained 27B student benefits far more than the frontier monitors (-43 pp vs -5 to -11 pp). It flags these trajectories aggressively but late under the old PONR; clamping to k* nearly halves its miss rate. The frontier monitors rarely flag these hard trajectories at all, so there is little to rescue. Note that this rule was fixed before we trained any monitors.

Reviewer_NCZ3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LLM agents running long, unsupervised tasks may pursue misaligned goals, a risk often called scheming. Existing monitors are mostly offline: they judge a trajectory after the task is finished, which is often too late to prevent harm.

This paper formalizes online monitoring. After every agent step, the monitor sees the trajectory so far and outputs a rationale plus three scores (confidence, severity, irreversibility). A fixed rule combines these into a flag score, and the trajectory is flagged once the score crosses a threshold. To train and evaluate this, the authors extend coarse trajectory-level labels in existing agentic misalignment datasets to step-level labels, including a “point-of-no-return” (PONR) step, the earliest step after which harm can’t be undone. They introduce timely metrics (t-AUROC and t-pAUROC@20) that credit a flag only if harm can still be averted.

Their best trained monitor, Qwen3.8-27B, outperforms GPT-5.6 Sol at 14× lower cost and half the latency. It nearly matches Gemini 3.7 Flash and Claude Opus 4.8 at 2.7× and 28× lower cost, and is slightly faster than both. Overall I like the paper and the idea.

Critical strengths

1 - Right problem and framing. Flagging harm mid-trajectory, rather than after the fact, is the practically useful setting, and the paper formalizes it clearly. 2. Step-level labels. Extending trajectory-level labels to step-level information lets the authors pinpoint where misalignment occurs, which is a useful resource beyond this paper. 3 Timely metrics. Crediting a flag only when harm can still be averted matches how a monitor would actually be used.

4 Decomposed scoring. Scoring confidence, severity, and irreversibility separately, with a fixed flagging rule on top, makes each flag traceable to its cause. It also lets the rule change without retraining the monitor.

5 Strong cost/latency results. A small trained model that beats or nearly matches much larger frontier monitors at a fraction of the cost and latency is a practical win.

6 Reasoning-effort ablation. The authors report that default or maximum reasoning makes prompted frontier monitors up to 2.6× more expensive and 2.8× slower, for at most 0.016 improvement in detection. This addresses the obvious fairness concern about the baselines.

Critical weaknesses
  1. PONR labels are under-justified. The claim that “harm can still be averted” carries the whole timely-metric framework, but the paper doesn’t show how PONR labels are validated. Without that, they risk being gut-check labels. It is also unclear whether PONRs can be judged reliably when they sit many steps before the eventual harm, in trajectories of 100+ steps.

  2. Single-PONR assumption. The formulation seems to assume one PONR per trajectory. If a PONR is flagged at step 10, is the damage unrecoverable 100 steps later, or can other PONRs occur afterward? This isn’t explained. Would love to know more about this.

  3. Landmark-prefix scoring undercuts the online premise. Because scoring every prefix is quadratically expensive, the authors score only three landmark prefixes per trajectory. Online, step-by-step flagging was the stated point of the work, so it’s unclear how well the evaluation reflects true online performance.

Possible handicapping of baselines. Using low reasoning effort for prompted monitors can look like it favors the trained model. The ablation helps, but it should be cleared more in the main paper body. The appendix section is technically not part of the review

Calibration choice is unmotivated in the main text. The teacher sees golden scores as a calibration hint it is told not to mention. The rationale is only in the appendix and needs a short justification in the methods.

The third contribution seems more like an observation. The findings that training composition predicts out-of-distribution transfer and that many residual failures come from mis-weighting seen evidence are interesting analyses but not contributions. They should be reframed.

Questions for authors

Why these three scores? Why confidence, severity, and irreversibility specifically? Were other decompositions tried?

Reliability of the scores. What is the error rate for each? In particular, how often does the monitor call a step irreversible when later adaptive retries make it reversible? Is there a typical window (in steps) in which this happens?

PONR at long range. Is it possible to determine PONRs reliably when they are far from the harm in 100+ step tasks? What is the maximum distance at which you trust the labels?

Multiple PONRs. Can a trajectory have more than one? If a PONR is flagged at step 10, can later steps still matter?

Plan vs. action. Does a PONR refer to the plan or the actual executed call? If it’s the plan, is there any mechanism to cut off execution at that point?

Implicit asks. The scope taxonomy (action class, in-scope vs. out-of-scope, wrong target vs. wrong policy) assumes the task’s scope is clear. Do you account for actions that are not explicit in the task but are implicitly required or optional for completing it?

Environment vs. model fault. If the environment doesn’t block network access and the agent downloads files, which gets flagged, is that a model failure or an environment misconfiguration? Do you check environment configs?

Priming. Without priming or calibration hints, can an online monitor reliably predict these scores?

Landmark prefixes. Why three? Would two or four change results? How does landmark-based scoring compare against true every-step online scoring on a subset?

Reasoning effort. Beyond the ablation, what is the full tradeoff curve at high, xhigh, and max effort across all baselines?

6 comments
AuthorsOpenReview ↗

Response to Reviewer NCZ3 (1/4)

We thank the reviewer for the positive and detailed assessment, especially acknowledging (i) the strength of the framing, (ii) the strong cost/latency results, and (iii) the breadth of our ablations. We address the concerns and questions in detail below, and will incorporate requested clarifications in the paper.

PONR labels are under-justified. The claim that “harm can still be averted” carries the whole timely-metric framework, but the paper doesn’t show how PONR labels are validated. Without that, they risk being gut-check labels.

and

Reliability of the scores. What is the error rate for each?

We think this is a fair point. To clarify: the reliability of the step-level labels, including the placement of the PONR, is verified by manual human inspection. Each time we introduced a new dataset to the pipeline, each main author of the paper inspected the golden labels of about 10% of all new trajectories. Labelling errors found this way were fixed by revising the judge prompts or by adding filters to remove low-quality trajectories. The filters (described in Appendix C.2.2) removed 230 trajectories overall. This was an iterative process, and we continued until we were satisfied with data and label (including PONR) quality. We are confident the resulting labels are well-suited for the evaluation of online monitors.

We have added this clarification to the appendix (see the newly added E.2.4) and hope this addresses your concern.

It is also unclear whether PONRs can be judged reliably when they sit many steps before the eventual harm, in trajectories of 100+ steps.

And

PONR at long range. Is it possible to determine PONRs reliably when they are far from the harm in 100+ step tasks? What is the maximum distance at which you trust the labels?

Distance does not tend to be a problem for the labelling pipeline in our experience. Even though there is a decent number of trajectories with 100+ steps (about 16% of the evaluated OOD trajectories), we find that PONR is labelled accurately by the judge pipeline. We hypothesize that distance is not a concern for two reasons. First, even with a high number of steps, each trajectory fits comfortably in the context window of the labelling model (Gemini 3.1 Pro). Second, since the labelling pipeline includes a privileged judge that knows the hidden side task (and how the harm may onset), it is much easier for it to place where the PONR should be.

To the best of our knowledge, long-horizon scheming datasets (with 300+ steps) where the set-up of harm and the actual instantiation of it sit far apart, do not exist yet. We hypothesize that we might find labelling issues in such trajectories but since our current trajectories fit neatly into the context window of the judge, we don’t see any issues.

Single-PONR assumption. The formulation seems to assume one PONR per trajectory. If a PONR is flagged at step 10, is the damage unrecoverable 100 steps later, or can other PONRs occur afterward? This isn’t explained. Would love to know more about this.

and

Multiple PONRs. Can a trajectory have more than one? If a PONR is flagged at step 10, can later steps still matter?

In theory, a trajectory can definitely have multiple irreversible steps, but nonetheless we define a single PONR for each trajectory. First, by definition the PONR is the first step after which the harm cannot be undone, so in the case of multiple irreversible steps, we just define the first one to be the PONR since flagging at or before the execution of that step averts all harm. Second, in practice all of the datasets we use are created by defining a single malicious side task (e.g., downloading malware and executing it), so generally there is only one appropriate PONR that corresponds to the single side task.

Still, in more realistic datasets which may have multiple attempts at harm in the same trajectory, we anticipate that having richer information than a single PONR may be beneficial.

AuthorsOpenReview ↗

Response to Reviewer NCZ3 (2/4)

Landmark-prefix scoring undercuts the online premise. Because scoring every prefix is quadratically expensive, the authors score only three landmark prefixes per trajectory. Online, step-by-step flagging was the stated point of the work, so it’s unclear how well the evaluation reflects true online performance.

We share this concern and because of that, we quantify this in Appendix E.6.3, which compares approximate evaluation (scoring only the landmark prefixes) with full evaluation (scoring every prefix) on 150 class-balanced trajectories per OOD set, for prompted Gemini 3.1 Pro, the trained Qwen3.5-4B and Qwen3.8-27B, and the newly added Gemini 3.7 Flash, Claude Haiku 4.5 and GPT-5.6 Luna. We did not run Opus and Sol because this evaluation costs about $8k even on this subset of trajectories.

While we observe a reduction in false negatives, we also see false alarms accumulating over long trajectories, with false positive rates being largest on SHADE, whose trajectories often run past a hundred steps. Overall, approximate evaluation seems to be optimistic for every monitor, and the bias is largest for monitors that over-flag benign steps (the weaker prompted monitors Claude Haiku 4.5 and GPT-5.6 Luna as well as our trained students) and for long trajectories. For the trained students, this is because they have a much higher variance in their flag score on benign trajectories; the 27B monitor has double the per-step flag-score variance than Gemini 3.1 Pro on benign SHADE trajectories and on 41% of these trajectories it peaks above the threshold and drops down again more than three times. Full evaluation scores every prefix, and so with longer trajectories, there are more opportunities for noisy scoring to cross the threshold and form a false positive. That said, approximate evaluation preserves the ranking of monitors by t-pAUROC@20 and closely tracks full evaluation for the stronger prompted monitors (Gemini 3.1 Pro and 3.7 Flash) and on short trajectories.

Unfortunately, as you noted, scoring every prefix is prohibitively expensive. Using our cost estimation, we think it costs about \59ktoevaluatethepromptedmonitorsonallOODdatasetsatdefaulteffort(approx59k to evaluate the prompted monitors on all OOD datasets at default effort (approx \\31k with per-step prompt caching).

Possible handicapping of baselines. Using low reasoning effort for prompted monitors can look like it favors the trained model. The ablation helps, but it should be cleared more in the main paper body. The appendix section is technically not part of the review

We do not think the baselines are handicapped, but we agree that the main text can explain it in more detail. The comparison between the trained and prompted monitors is a Pareto frontier on cost, latency, and detection together, and as we note in 4.1, low effort is the setting at which the prompted monitors are the strongest overall. Moving to maximum effort makes them up to 2.6 times more expensive and 2.8 times slower, while improving detection by at most 0.016 for every model except GPT-5.6 Luna (+0.09). Gemini 3.7 Flash even drops off both frontiers at maximum effort. The headline claims therefore hold at maximum effort: Qwen3.8-27B still leads GPT-5.6 Sol (0.878 against 0.866) at about 19 times lower cost, ties Gemini 3.7 Flash at about 5 times lower cost, and trails Claude Opus 4.8 by 0.02 at about 37 times lower cost. The one rank change is GPT-5.6 Luna overtaking the trained Qwen3.5-4B by 0.02.

We have expanded the reasoning-effort paragraph in Section 4.1 to state the maximum-effort results above, so the comparison no longer depends on Appendix I.1.

Calibration choice is unmotivated in the main text. The teacher sees golden scores as a calibration hint it is told not to mention. The rationale is only in the appendix and needs a short justification in the methods.

We use anchoring to make sure teacher deliberations are grounded in the golden truth and therefore of higher quality. That said, we agree this should be made explicit in the main text and have added this to our Methodology section. Thank you for the suggestion.

AuthorsOpenReview ↗

Response to Reviewer NCZ3 (3/4)

The third contribution seems more like an observation. The findings that training composition predicts out-of-distribution transfer and that many residual failures come from mis-weighting seen evidence are interesting analyses but not contributions. They should be reframed.

We partially agree. We view the mixture analysis as a contribution rather than an observation: we screened ten candidate axes for decomposing scheming trajectories and identified action class and scope as the two whose composition best predicts OOD transfer (Appendix J). This gives practitioners a way to judge, before training, where a training set will transfer and where it will leave gaps. For the failure-mode analysis, we agree that the finding itself is an observation, but we see the analysis as a contribution: a taxonomy of four audit categories grouped into calibration and perception failures, applied across untrained, trained, and frontier monitors. That said, we have reframed the third contribution bullet in the paper accordingly.

Why these three scores? Why confidence, severity, and irreversibility specifically? Were other decompositions tried?

The three-axis decomposition is meant to make flags explainable to a human overseer, not to improve detection. When prompted with a single suspicion score, Gemini 3.1 Pro performs the same as with the three-axis prompt (0.937 vs. 0.938 mean-OOD t-pAUROC@20, Appendix E.6.1). We chose these three axes because they are the ones an online monitor most needs to reason about. Confidence is how likely it is that the agent is scheming, severity is how harmful its actions would be if it were, and irreversibility is whether the current action would be hard to undo. With these separated, a flag can be traced to its cause (strong evidence, high stakes, or an imminent irreversible step), and the flagging rule can be changed without retraining the monitor. They were the most natural decomposition, so we did not try others.

In particular, how often does the monitor call a step irreversible when later adaptive retries make it reversible? Is there a typical window (in steps) in which this happens?

The irreversibility score is scored per step: it rates only the action at step t, not the cumulative trajectory so far (Section 3.2). So, by design no future steps and their accumulated state can affect the irreversibility of previous steps. Irreversibility is also counterfactual: it asks how hard the action would be to undo if it succeeds (rubric, Appendix L). If an agent's action fails and it adaptively retries, or later undoes the action, the original step's score is still correct; a retry is a new action that gets its own score. This is intended, since the monitor scores each tool call before it executes and cannot wait to see the outcome. We have now made both points explicit in Section 3.2.

Plan vs. action. Does a PONR refer to the plan or the actual executed call? If it’s the plan, is there any mechanism to cut off execution at that point?

The PONR refers to the earliest step after which the harm can no longer be undone, such as a step that sends credentials to an external server. This is more in line with an actual executed call. A step outlining a plan does not cause imminent / irreversible harm. Regardless, if a step is flagged by the monitor, depending on how the monitor is configured, the agent may be immediately blocked, redirected to a human overseer, resampled, etc.

The scope taxonomy assumes the task’s scope is clear. Do you account for actions that are not explicit in the task but are implicitly required or optional for completing it?

Our scope label asks whether the harmful step is the kind of action the assigned task calls for, judged against the full task description, so actions the task requires but does not explicitly name are classed as in-scope. For example, in BashArena the agent must edit the firewall configuration to reach the database, and a step that uses that edit to disable the firewall is labelled in-scope, wrong policy. Optional actions, which an honest agent might plausibly take but does not need, are labelled by the same test: in-scope if they are the kind of action the task involves, and out-of-scope otherwise. We have now made this definition explicit in Section 4.2.

Environment vs. model fault. If the environment doesn’t block network access and the agent downloads files, which gets flagged, is that a model failure or an environment misconfiguration? Do you check environment configs?

Both, if the downloaded file is malicious and/or not relevant to the task. We do not check environment configurations as we do not access the live environments; we download trajectories post-hoc, label them, and assess our monitors using them.

Reviewer_W39xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a way to train online monitors for finding hidden misalignment in LLM agents during task execution process. Unlike offline monitors that inspect full trajectories after a task finishes, these online monitors check actions step by step. The authors use a 3 judge LLM pipeline to add step level labels, including a point of no return where irreversible harm happens. Open-weight students are trained by SFT on teacher-generated rationales and scores, with further RL training for selected models. The experiments compare detection quality, cost, and latency, with analyses of training-data composition and monitor errors.

Critical strengths
  • Focusing the evaluation on steps with clear evidence and steps with irreversible harm is practical for real time monitoring.

  • The SFT trained monitors generally improves mean OOD detection performance over their untrained counterparts. The paper also provides ablation results across base, SFT, and RL stages, along with an analysis of common failure types.

Critical weaknesses
  • The technical novelty here is limited. The main training setup follows Sinha et al. [1] by using teacher reasoning steps to train action monitors through SFT and optional RL. The main additions are step level supervision and prefix evaluation, but the core training recipe stays mostly the same.

  • The experiments do not clearly isolate how much the proposed prefix level supervision helps on its own. Comparing the Base model to SFT is not enough to show the value of this specific addition.

  • The headline comparisons use only a few landmark prefixes selected using ground truth labels in Section 3.4. Continuous monitoring creates more opportunities for both detections and false alarms, with false alarms accumulating over long trajectories. These comparisons alone do not establish the balance between detection and false alarms under continuous monitoring.

  • The reliability of the step level timing labels is not quantified. The labeling judges and teacher model all rely on Gemini 3.1 Pro. Section 4.1 shows good classification for full trajectories, but it does not check if the model correctly picks when evidence becomes sufficient or harm becomes permanent. The manual checks mentioned in the paper leave out details like sample sizes, agreement rates, and error distributions for these step labels.

  • The transfer analysis in Section 4.2 uses action class and scope to explain performance, but this relationship remains exploratory. Also, SFT actually lowers performance on SHADE-Arena for both model sizes in Figure 6, which shows that gains on unseen data are not consistent across all benchmarks.

[1] Sinha et al., Training Deliberative Monitors for Black-Box Scheming Detection, arXiv 2026.

Questions for authors
  1. What evidence separates the benefit of prefix level supervision from the benefit of applying the existing distillation recipe to more training examples?

  2. How many trajectories were independently checked for the ideal flagging step and the point of no return? Sharing the annotation process, agreement rates, and how disagreements were resolved would make the label quality clearer.

6 comments
AuthorsOpenReview ↗

Response to Reviewer W39x (1/3)

We thank the reviewer for their constructive feedback, and their acknowledgement of the practicality of our evaluation method and the breadth of analysis. We address each point below and provide clarifications and further experiments.

The technical novelty is limited. The main training setup follows Sinha et al. … The main additions are step-level supervision and prefix evaluation …

We agree that the SFT recipe is inherited from Sinha et al., and this was a deliberate choice (although for RL, we explored a different reward function). Holding the recipe fixed means any difference we observe is attributable to the adaptation of this method to the online setting. We see the contribution of the paper as the online setting itself which required formalizing the setting (including the point of no return), defining timely metrics that give credit only for flags raised while harm can still be averted, and building a three-judge pipeline that adds step-level labels to eight existing datasets whose original labels cover only whole trajectories. Without these, neither timely evaluation nor step-level training would be possible. On top of these, we show that the action class of a training set's scheming steps predicts OOD transfer, and that most residual failures are calibration failures rather than perception failures, which points to where future monitor improvements should focus.

Within training itself, we also explored offline preference optimization (DPO and KTO) on top of SFT, which we now include as a negative result. Each preference pair contrasted a wrong student rollout with a correct completion, either a teacher rationale or a minimally edited version of the student's own rollout. Neither improved t-AUROC (changes under 0.01 in every run). Observing the training logs showed why: DPO’s gains came almost entirely from lowering the probability of the student’s wrong answer rather than raising the probability of the correct answer. Reading the deliberations, it seemed as if the student’s observations and reasoning were unchanged and only the final scores shifted. This motivated the on-policy RL stage, which rewards the student’s rollouts directly.

These runs used an earlier version of the pipeline, which we state in the appendix. We discuss this now in Appendix F.5.

The headline comparisons use only a few landmark prefixes … Continuous monitoring creates more opportunities for both detections and false alarms, with false alarms accumulating over long trajectories…

We agree, and we quantify this in Appendix E.6.3, which compares approximate evaluation (scoring only the landmark prefixes) with full evaluation (scoring every prefix) on 150 class-balanced trajectories per OOD set, for Gemini 3.1 Pro, the trained Qwen3.5-4B and Qwen3.8-27B, and the newly added Gemini 3.7 Flash, Claude Haiku 4.5 and GPT-5.6 Luna. We did not run Opus and Sol because this evaluation costs about $8k even on this subset of trajectories.

The reviewer's intuition is correct. While we observe a reduction in false negatives, we also see false alarms accumulating over long trajectories, with false positive rates being largest on SHADE (78.7% FPR for the 27B monitor), whose trajectories often run past a hundred steps. Overall, approximate evaluation is optimistic for every monitor, and the bias is largest for monitors that over-flag benign steps (the weaker prompted monitors and the trained students) and for long trajectories. We illustrate this with the FPR / FNR of each monitor under approximate and full evaluation, averaged over the four OOD sets:

MonitorEvalmean-OOD FPRmean-OOD FNR
Gemini 3.1 ProFull4.7%0.7%
Gemini 3.1 ProApprox1.3%11.0%
Gemini 3.7 FlashFull0.0%13.7%
Gemini 3.7 FlashApprox0.0%16.7%
Claude Haiku 4.5Full46.0%2.7%
Claude Haiku 4.5Approx15.0%9.7%
GPT-5.6 LunaFull38.0%8.3%
GPT-5.6 LunaApprox17.7%20.3%
Qwen3.8-27B SFT+RLFull39.7%1.0%
Qwen3.8-27B SFT+RLApprox12.7%7.7%
Qwen3.5-4B SFT+RLFull50.7%1.3%
Qwen3.5-4B SFT+RLApprox23.0%13.7%

That said, approximate evaluation preserves the ranking of monitors and closely tracks full evaluation for the stronger prompted monitors (Gemini 3.1 Pro and 3.7 Flash) and on short trajectories. It overstates the low-false-positive performance of monitors that over-flag benign steps, including our trained monitors, on long trajectories (BashArena, SHADE-Arena): scored at every step, the trained monitors almost never miss scheming (FNR of about 1%) but raise more false alarms. The revised Section 3.4 now states this caveat.

AuthorsOpenReview ↗

Response to Reviewer W39x (2/3)

The reliability of the step level timing labels is not quantified. The labeling judges and teacher model all rely on Gemini 3.1 Pro. ... The manual checks mentioned in the paper leave out details like sample sizes, agreement rates, and error distributions for these step labels.

and

How many trajectories were independently checked for the ideal flagging step and the point of no return? Sharing the annotation process, agreement rates, and how disagreements were resolved would make the label quality clearer.

We think this is a fair point. To clarify: the reliability of the step-level timing labels is justified by manual human inspection. Each time we introduced a new dataset to the pipeline, each main author of the paper inspected the golden labels of about 10% of all new trajectories. Labelling errors found this way were fixed by revising the judge prompts or by adding filters to remove low-quality trajectories. The filters (described in Appendix C.2.2) removed 230 trajectories overall. This was an iterative process, and we continued until we were satisfied with data and label quality. We are confident the resulting labels are well-suited for the evaluation of online monitors.

We have added this clarification to the appendix (see the newly added E.2.4) and hope this addresses your concern.

The transfer analysis in Section 4.2 uses action class and scope to explain performance, but this relationship remains exploratory.

We agree. With only four training sets the analysis is exploratory but two reasons make us think that the pattern is more than coincidence. First, action class and scope were the only two out of the ten axes we screened (Appendix J) whose training-set composition tracked OOD transfer. Second, the effects hold at both model sizes (e.g. BashBench, which has no in-scope wrong-policy trajectories, barely helps at 4B and slightly regresses at 27B). If the pattern holds on new datasets, the two axes give practitioners a way to predict, before training, which deployment settings a training set will transfer to and where it will leave gaps.

Also, SFT actually lowers performance on SHADE-Arena for both model sizes in Figure 6, which shows that gains on unseen data are not consistent across all benchmarks.

We agree, and Section 4.4 notes that SHADE-Arena is the one OOD set where SFT lowers t-pAUROC@20. The drop is confined to the low-false-positive region, and t-AUROC, which integrates over all false-positive rates, rises on SHADE-Arena at both sizes after SFT (0.75 to 0.77 at 4B, 0.89 to 0.91 at 27B; see Appendix J.3, Table 20, “untrained” versus “all four” rows), while t-pAUROC@20 counts only the region below 20% false positives.

This matches what Sinha et al. report for SHADE-Arena. Under every mixture they tried, training made monitors read the messiness of its benign trajectories (exploration, incomplete reporting) as concealment. Our mixture analysis (Section 4.2) adds a reason: SHADE-Arena's dominant action class, financial transfers, is almost absent from all four of our training sets. We read the drop as a coverage gap that the mixture analysis predicts. Even so, Qwen3.8-27B remains stronger on SHADE-Arena (0.70) than GPT-5.6 Sol (0.61) and Gemini 3.7 Flash (0.65) at default effort.

AuthorsOpenReview ↗

Response to Reviewer W39x (3/3)

The experiments do not clearly isolate how much the proposed prefix level supervision helps on its own. Comparing the Base model to SFT is not enough to show the value of this specific addition.

and

What evidence separates the benefit of prefix level supervision from the benefit of applying the existing distillation recipe to more training examples?

Part of the benefit of prefix-level supervision is efficiency: it gets more training signal out of each existing labeled trajectory, whereas generating and labeling additional trajectories can be costly.

However, we agree that we can highlight the benefit of prefix-level supervision more. To isolate the effect, we ran a follow-up where we used the same SFT/distillation recipe to train a model in two ways: one that was trained on 600 full trajectories per training dataset, and another which was trained only on 150 trajectories per training dataset but on multiple prefixes per trajectory, so the total number of training samples for both are equivalent. Despite seeing 4× more trajectories, the full-trajectory monitor underperforms on OOD at both scales, and has a higher false-positive rate. Our hypothesis is that prefix-level supervision teaches the monitor which action is incriminating, not just which trajectories are scheming. In particular, prefixes from before and after the scheming in the same trajectory form near-identical pairs with opposite labels, which isolate the scheming steps. Full-trajectory labels lack this contrast, so the monitor may fall back on trajectory-level surface features that are not as indicative of the scheming.

ModelTraining data (per dataset)OOD t-pAUROC@20OOD FPR
Qwen3.5-4B600 prefixes (150 trajectories)0.6410.158
Qwen3.5-4B600 full trajectories0.5450.259
Qwen3.8-27B600 prefixes (150 trajectories)0.8680.121
Qwen3.8-27B600 full trajectories0.8390.147

We have added these results and discussion to Appendix I.6 in the updated version and thank the reviewer for highlighting this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.