acceptodds
Under review as a conference paper at ICLR 2027

Chain-of-Thought Reliance does not Guarantee Monitorability

Abstract

A prevailing hypothesis in chain-of-thought (CoT) monitorability, the ability of a monitor to determine whether a model is taking an undesirable action, is that models are more monitorable when working on difficult tasks. Simply increasing task difficulty increases a model's reliance on its CoT, which in turn makes the CoT more monitorable because it can't be used for post-hoc rationalization. We find that this optimistic view often does not play out in practice. This paper presents the first systematic, empirical study of how monitorability varies with task difficulty and CoT reliance. In hint-based monitorability evaluations, which monitor for a model's usage of a biasing input feature, increasing CoT reliance does not improve monitorability, and can even reduce it by up to 30 percentage points. In some settings, this decrease is correlated with models generating longer CoTs for more difficult tasks; we propose a simple truncation heuristic that recovers up to 16 percentage points. Increased CoT reliance proves more helpful in secret-side-task evaluations, which adversarially prompt the model to complete a side task without the monitor noticing. There, increased CoT reliance eliminates a model's ability to complete the side task without verbalization. We observe other strategies for monitor evasion that are not diminished by increased CoT reliance. Overall, while increased CoT reliance can improve the narrow class of monitor errors caused by non-verbalization, it alone does not guarantee CoT monitorability.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-thought monitorability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_341AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts a systematic empirical investigation into a widely‑accepted hypothesis in the field of CoT monitorability: as task difficulty increases and models become more reliant on their chain‑of‑thought, CoTs become easier for monitors to detect undesirable model behaviors.

Critical strengths

This paper directly challenges the hypothesis that CoT necessity equals better monitorability. Through empirical evidence, it reveals substantial limitations of this proposition, delivering important cautionary implications for the real‑world safety deployment of CoT monitoring.

Critical weaknesses
  • One limitation is that all experiments in this paper are restricted to synthetic symbolic tasks. While such tasks enable fine‑grained control over task difficulty and CoT reliance, it remains unclear whether the conclusions generalize to open‑text and real‑world agent tasks. The behavioral patterns of model cheating and misalignment in real‑world scenarios may differ from those observed in synthetic tasks.
  • All undesirable behaviors in this work are explicitly induced via prompts: grader‑hacking hints and adversarial instructions that demand hiding side‑task computations. Consequently, it remains an open question whether the three evasion strategies identified in the paper would emerge from genuine real‑world model misbehavior. In another words, without such adversarial prompts in real‑world deployments, would models even adopt these evasion tactics in the first place? This directly impacts the applicability of the conclusions drawn in this work.
Questions for authors

See Weaknesses.

2 comments

We thank the reviewer for their insightful review. We agree with the reviewer that our results deliver “important cautionary implications for the real‑world safety deployment of CoT monitoring” when depending on CoT necessity to achieve high monitorability.

One limitation is that all experiments in this paper are restricted to synthetic symbolic tasks. While such tasks enable fine‑grained control over task difficulty and CoT reliance, it remains unclear whether the conclusions generalize to open‑text and real‑world agent tasks. The behavioral patterns of model cheating and misalignment in real‑world scenarios may differ from those observed in synthetic tasks.

Our results in Figure 6 and Table 2 shows that there exist strategies for monitor evasion that continue to work even when CoT reliance is high. The two strategies we observe that accomplish this are general enough that we believe would be applicable to models in real-world settings. Both are a type of prompt injection: one convinces the monitor to only consider a subset of the CoT, and the other convinces the monitor that the user prompt contained additional instructions that were not there.

We agree that additional strategies for monitor evasion surely exist, especially on open-text and real-world agent tasks which have different affordances that a model may use for monitor evasion. Our work is not claiming to taxonomize all monitor evasion strategies that models may use, but instead to show that there are some that continue to work even when CoT reliance is high.

All undesirable behaviors in this work are explicitly induced via prompts: grader‑hacking hints and adversarial instructions that demand hiding side‑task computations. Consequently, it remains an open question whether the three evasion strategies identified in the paper would emerge from genuine real‑world model misbehavior. In another words, without such adversarial prompts in real‑world deployments, would models even adopt these evasion tactics in the first place? This directly impacts the applicability of the conclusions drawn in this work.

We note that we do not prompt the model to use the grader hacking hint in our hint-based evaluations. We simply make the hint available to be used, and the model decides whether to use it or solve the task without it. As we state in Section 3.2, prior work has demonstrated that models are prone to reward hack by exploiting the grader. And we find that the models we study do use the hint a significant portion of the time (Figures 14 and 15).

We use the simple setting of leaking the hint as part of the prompt to isolate how monitorability varies with CoT reliance on the original task and in using the hint itself. In a real-world setting, a model might need to take additional steps before accessing the grader, such as searching for its implementation on the host machine or on an online repository. It would be difficult to create an evaluation where the difficulty in accessing the grader can be smoothly varied, though that would be a great experiment in observing the relationship between CoT reliance and monitorability. We suspect that such an experiment would show similar findings: when CoT reliance is low, the model can evade the monitor through non-verbalization. When CoT reliance is high, the model can evade the monitor through other strategies (Table 4).

Studying model misbehavior is difficult when the misbehavior is rare, making it hard to get enough instances to study. It is still valuable to study such misbehavior when it is sufficiently harmful, or if future models may exhibit the misbehavior at higher rates. There are two strategies generally available to induce higher rates of misbehavior for study: fine-tuning and prompting. For the secret-side-task evaluations, we use prompting because it effectively induces the target misbehavior (attempting the side task) and is less computationally expensive than fine-tuning. And as stated in the previous question, we think that the observed strategies for monitor evasion are not specific to the tasks we study, and can be applied in real-world tasks. And even though additional strategies are likely to be used for monitor evasion in other settings, our results show that we should not rely on high CoT reliance alone to deliver high monitorability.

Reviewer_341AOpenReview ↗

Reply to Author

Thank you for the rebuttal. My core concerns remain: results are limited to synthetic tasks and constructed prompt settings, without convincing evidence for spontaneous emergence in real-world agent deployments. I retain my rating of 2.

Reviewer_M82VOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether increased reliance on chain of thought (CoT) reasoning improves monitorability. The authors examine this relationship through two evaluation settings: hint based monitoring, where a monitor attempts to determine whether a model relied on a misleading hint, and secret side task monitoring, where a model attempts to complete an additional task without being detected.

The experiments systematically vary main task difficulty, hint difficulty, and side task difficulty to examine how monitorability changes with CoT reliance. The results show that increased CoT reliance does not consistently improve hint based monitorability and can sometimes reduce it. The authors identify CoT length as one contributing factor and demonstrate that truncating reasoning traces can partially recover monitorability for weaker monitors. In secret side task evaluations, increased CoT reliance prevents certain models from completing hidden tasks without verbalizing their reasoning, although other monitor evasion strategies remain effective.

Overall, the paper provides empirical evidence that increased CoT reliance alone does not guarantee improved monitorability.

Critical strengths
  1. Important and timely research question.

The paper examines a fundamental assumption underlying CoT monitoring: whether models become easier to monitor when they rely more heavily on explicit reasoning. This question has important implications for the reliability of CoT based safety mechanisms. The findings provide a useful qualification to the hypothesis that more difficult tasks naturally lead to more monitorable reasoning.

  1. Systematic experimental design.

The paper evaluates multiple reasoning models, monitor models, and tasks with controllable difficulty. A particularly valuable aspect is the distinction between increasing the difficulty of the original task and increasing the difficulty of using the hint. Section 4.2 directly addresses a potential confounding factor in Section 4.1, where a model may rely heavily on CoT to solve the original problem while still being able to exploit a hint without substantial reasoning. The secret side task experiments further extend the analysis to adversarial monitor evasion settings.

  1. Insightful analysis of monitor failures.

Rather than focusing solely on aggregate monitorability scores, the paper investigates why monitoring fails. Section 4.3 identifies CoT length as an important factor behind reduced performance for weaker monitors and demonstrates that truncation can partially recover monitorability. Section 4.4 further distinguishes between different monitor evasion strategies, including nonverbalization and attempts to manipulate how the monitor interprets the reasoning trace. These analyses provide useful insights into why increased CoT reliance improves monitorability in some settings but not others.

Critical weaknesses
  1. The conclusions about CoT reliance may be stronger than the experimental evidence supports.

The paper defines CoT reliance as the performance difference between reasoning and nonreasoning modes. While this is a reasonable operational measure, improved performance with CoT does not necessarily establish that CoT is required to perform the behavior being monitored. The authors partially address this distinction in Section 4.2 by increasing the difficulty of using the hint. However, these experiments are primarily limited to Count Bits, as the other tasks exhibit low hint uptake or saturated monitorability. Therefore, the claim in Section 4.1 that the findings invalidate previous theoretical claims appears somewhat overstated. The authors should clarify how their findings relate to the stronger notion of CoT necessity examined in prior work.

  1. Changes in evaluation instance composition may influence the observed monitorability trends.

Appendix G describes an eligibility filtering procedure for hint based evaluations, and Figures 12 and 13 show that the proportion of eligible instances varies across difficulty levels. Notably, Section 4.2 acknowledges that approximately half of the observed 16 percentage point improvement in monitorability comes from changes in the relative representation of hint templates rather than improved hint verbalization. This raises the possibility that similar composition effects contribute to the monitorability decreases reported in Section 4.1. Although the truncation experiments provide evidence that CoT length contributes to these decreases, it remains unclear how much of the observed trend is attributable to changes in the evaluated instances. Reporting results separately for each hint template or using fixed template weights across difficulty levels would help clarify this issue.

Questions for authors
  1. CoT reliance versus CoT necessity: Section 4.2 attempts to address the distinction between reliance on CoT for the original task and reliance on CoT for using the hint. However, these experiments are primarily limited to Count Bits. How do the authors distinguish their findings from prior theoretical claims based on CoT necessity rather than performance improvement? Could the authors clarify whether their conclusions apply to CoT necessity or primarily to the operational definition of CoT reliance used in this paper?

  2. Effect of evaluation instance composition: Section 4.2 acknowledges that approximately half of the observed improvement in monitorability comes from changes in hint template composition. Could similar composition effects contribute to the monitorability decreases observed in Section 4.1? Have the authors examined whether these trends persist when evaluating each hint template separately or using fixed template weights across difficulty levels?

3 comments

We thank the reviewer for their positive review! We agree that studying the conditions in which CoT monitors can be relied on is an “important and timely research question.” We are glad the reviewer appreciates that we studied CoT reliance along multiple axes (overall task difficulty, versus the monitored behavior difficulty). We also agree on the value of diagnosing the cause of monitorability errors (beyond just the aggregate monitorability score), which can lead to stronger monitoring methods.

The conclusions about CoT reliance may be stronger than the experimental evidence supports. The paper defines CoT reliance as the performance difference between reasoning and nonreasoning modes. While this is a reasonable operational measure, improved performance with CoT does not necessarily establish that CoT is required to perform the behavior being monitored. The authors partially address this distinction in Section 4.2 by increasing the difficulty of using the hint. However, these experiments are primarily limited to Count Bits, as the other tasks exhibit low hint uptake or saturated monitorability. Therefore, the claim in Section 4.1 that the findings invalidate previous theoretical claims appears somewhat overstated. The authors should clarify how their findings relate to the stronger notion of CoT necessity examined in prior work.

We see the findings in Section 4.1 as evidence against a strong view of the relationship between CoT necessity and monitorability, where increasing CoT reliance on the main task can improve monitorability even though the behavior being monitored for stays constant. Similarly, Section 4.2 (and Section 4.4 when scaling the side-task difficulty) is evidence against a narrower view, where CoT reliance is constrained to the behavior being monitored for and not the overall task. We found evidence against both views: CoT monitorability does not consistently improve when scaling either version of CoT reliance. We reframed the sections to be more clear about which view our results point against. We highlighted the relevant changes in blue text in our updated pdf.

We also added three additional datasets to our analysis in Sections 4.1 and 4.2. These datasets had similar results, and show that our finding is not limited to Count Bits. We also note that scaling the side task difficulty in Section 4.4 is another instance of scaling the difficulty of the behavior being monitored.

Changes in evaluation instance composition may influence the observed monitorability trends. Appendix G describes an eligibility filtering procedure for hint based evaluations, and Figures 12 and 13 show that the proportion of eligible instances varies across difficulty levels. Notably, Section 4.2 acknowledges that approximately half of the observed 16 percentage point improvement in monitorability comes from changes in the relative representation of hint templates rather than improved hint verbalization. This raises the possibility that similar composition effects contribute to the monitorability decreases reported in Section 4.1. Although the truncation experiments provide evidence that CoT length contributes to these decreases, it remains unclear how much of the observed trend is attributable to changes in the evaluated instances. Reporting results separately for each hint template or using fixed template weights across difficulty levels would help clarify this issue.

That is a great point. Due to the filtering procedure which removes instances with low hint uptake, the mix of hint templates at each difficulty level does vary. We added additional graphs in Append N which show the average number of instances from each hint template at each difficulty level. When there are at least 3 instances at a difficulty level for a hint template, we plot the g-mean2 metric.

We don’t see any hint templates where monitorability improves with CoT reliance. We therefore conclude that our findings are not due to a shift in hint template representation as we vary the difficulty level. We note that calculating monitorability by hint template naturally increases noise because there are fewer samples. We still plot our results in the main text as averaged over all hint templates so that our results are not over-indexed on the specific wording of any particular hint template.

CoT reliance versus CoT necessity: Section 4.2 attempts to address the distinction between reliance on CoT for the original task and reliance on CoT for using the hint. However, these experiments are primarily limited to Count Bits. How do the authors distinguish their findings from prior theoretical claims based on CoT necessity rather than performance improvement? Could the authors clarify whether their conclusions apply to CoT necessity or primarily to the operational definition of CoT reliance used in this paper?

We do operationalize the notion of CoT reliance to perform empirical analyses. Other works operationalize CoT necessity through a minimum CoT accuracy uplift. As stated in Section 2, Emmons et al. uses a 50 percentage point uplift between with-CoT and no-CoT to identify GPQA questions that meet their CoT-necessity condition, and Yueh-Han et al. use an 80 point filter for their experiments. Difficulty levels in our analysis with similarly high performance uplift would be considered meeting the CoT-necessity condition in prior work. Thus our findings do apply to CoT necessity.

Effect of evaluation instance composition: Section 4.2 acknowledges that approximately half of the observed improvement in monitorability comes from changes in hint template composition. Could similar composition effects contribute to the monitorability decreases observed in Section 4.1? Have the authors examined whether these trends persist when evaluating each hint template separately or using fixed template weights across difficulty levels?

See response to weakness 2.

AuthorsOpenReview ↗

Rebuttal Period Coming to a Close

Dear Reviewer M82V,

Thanks again for your detailed and positive review. Since the discussion period is ending soon (November 18th), we wanted to check in whether our rebuttal has addressed your concerns and/or if you have any other questions.

In summary, we made the following changes based on your feedback:

  1. Reframed the writing in section 4.1 to be testing a stronger version of the hypothesized relationship between CoT Reliance and CoT Monitorability, and likewise reframed section 4.2 to be testing a narrower version.
  2. Added 3 additional tasks which show our results are not limited to ZebraLogic and Count Bits.
  3. Added Appendix N, which breaks down the results in Sections 4.1 and 4.2 by hint template.

The relevant edits are in blue text in our revised submission.

Thank you for your time!

Reviewer_BgyvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether greater reliance on chain-of-thought improves monitorability. It measures reliance as the performance gain from reasoning versus answering directly, using dual-mode hybrid models. Across hint-exploitation and secret-side-task experiments, the relationship is mixed: reliance can improve detection when the monitored behaviour requires explicit computation, but does not consistently help when task-level reasoning and the monitored behaviour are less closely aligned. The central contribution is empirical evidence that task-level reliance alone is not a reliable predictor of practical monitorability.

Critical strengths

I appreciate the use of dual-mode hybrid models, which makes the comparison between reasoning and direct answering more convincing and addresses an obvious distribution-shift concern. It is also helpful that they report accuracy with and without CoT separately across task difficulty, rather than showing only the reliance gap. This makes it easier to understand where the increased reliance comes from. The contrasting experimental settings are another positive: they show both where reliance helps and where it does not, and investigate practical reasons why a monitor might miss behaviour even when it appears in the trace.

Critical weaknesses

My main concern is how closely their reliance metric captures necessity. Comparing full CoT with no CoT tells us that reasoning helps, but not how much of the trace is actually needed. I would have liked a TRACE-style [1] analysis that progressively truncates the reasoning and measures task performance at each cutoff, ideally across difficulty levels. This could help separate useful computation from redundant continuation, particularly when longer traces themselves make monitoring harder. A complementary perspective comes from the information-theoretic framework in [2], which distinguishes dependence between CoT and outputs from successful monitorability. Relating the performance-gap metric to this framework could help clarify what increased reliance establishes, and what additional conditions are needed for it to improve monitoring.​‌

The evidence for declining monitorability appears concentrated in ZebraLogic and Countbits, whereas the other hint-usage datasets mainly support non-improvement. These conclusions should be distinguished when discussing how broadly the negative trend generalises. Moreover, the limited task diversity leaves unclear how broadly this pattern generalises. I therefore find the counterexample claim better supported than broader suggestions that reliance generally provides little monitoring benefit

There is also a distinction between needing CoT to solve the main task and needing it to carry out the behaviour being monitored. That makes some of the experiments less direct tests of the necessity hypothesis than the framing suggests. In general, I find the empirical results useful, but would interpret them as identifying limits to when reliance improves monitorability, rather than refuting a generally positive relationship.

Refs:

[1] Is it thinking or cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort, Want et al. (ICLR 2026)

[2] Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory, Anwar et al. (2026).

Questions for authors
  • The evidence supports a narrower disagreement with prior work than the introduction suggests. The introduction presents necessity as implying monitorability, whereas the abstract reports that greater reliance prevents silent side-task completion but does not prevent other monitor failures. To me, that supports the computational intuition while showing that it is insufficient for reliable detection. I would therefore want the paper to distinguish these.

  • The paper primarily evaluates monitorability using a small set of monitoring approaches, with substantial emphasis on GPT-5.6 Terra as a strong general-purpose monitor. Since monitorability depends jointly on the task model and monitor, how should we expect the conclusions to transfer to monitors specifically optimised for oversight (e.g. monitors trained on monitoring data, specialised classifiers, or multi-stage monitoring systems)? To what extent do the reported results reflect limitations of CoT monitorability itself versus limitations of the particular monitoring approaches evaluated?

7 comments

We thank the reviewer for their thorough review, and for pointing us to a couple related works. We are glad the reviewer sees our experiments as showing “both where reliance helps and where it does not”, which is important for better understanding when we can rely on CoT monitors. We agree that it is important to investigate and categorize monitor errors, as we did in our work.

My main concern is how closely their reliance metric captures necessity. Comparing full CoT with no CoT tells us that reasoning helps, but not how much of the trace is actually needed. I would have liked a TRACE-style [1] analysis that progressively truncates the reasoning and measures task performance at each cutoff, ideally across difficulty levels. This could help separate useful computation from redundant continuation, particularly when longer traces themselves make monitoring harder. A complementary perspective comes from the information-theoretic framework in [2], which distinguishes dependence between CoT and outputs from successful monitorability. Relating the performance-gap metric to this framework could help clarify what increased reliance establishes, and what additional conditions are needed for it to improve monitoring.​‌

Adding a TRACE-style metric for CoT reliance is a great idea, and we implemented this! Following Wang et al., we truncate the CoT at 10 percentage point steps (snapping to the nearest end of sentence), and sample 10 early answers. We use these to estimate the probability that the model’s early answer matches its answer with the full CoT. We average over all rollouts and plot the P(early answer == final answer) versus the CoT percentage. We calculate the area under the curve (AUC). When AUC is near 0, the model does not arrive at its final answer until later in the CoT, so it relies on a big portion of the CoT. When AUC is near 1, the model arrives at its final answer early in the CoT, so it does not rely on much of the CoT. We plot the 1-AUC metric so that larger values indicate more CoT reliance. We add this metric in all figures where CoT reliance is plotted (Figures 2, 3, and 5 in the main text).

We also added tables showing the Spearman rank correlation between each of our CoT reliance metrics and CoT monitorability (Tables 2, 3, 10, and 11). Using the TRACE-style metric for CoT reliance, 19/21 and 7/8 of our configurations have a non-positive Spearman rank correlation when scaling unhinted task difficulty and the difficulty in using the hint, respectively. We also show the Spearman rank correlation between our two CoT reliance metrics, and show in almost all evaluations they’re aligned (Appendix O).

Our results are complementary to the information-theoretic framework in Anwar et al. Our empirical results support that paper’s conclusion that CoT necessity is not a sufficient condition for CoT monitorability. In their framework, our accuracy gap metric being positive establishes their necessity condition in their Proposition 2.10.1, but not the sufficiency condition. We added this work to our related works (Section 2).

The evidence for declining monitorability appears concentrated in ZebraLogic and Countbits, whereas the other hint-usage datasets mainly support non-improvement. These conclusions should be distinguished when discussing how broadly the negative trend generalises. Moreover, the limited task diversity leaves unclear how broadly this pattern generalises. I therefore find the counterexample claim better supported than broader suggestions that reliance generally provides little monitoring benefit

We increased task diversity by adding three additional datasets to the analyses in sections 4.1 and 4.2. These support our conclusion that increasing CoT reliance does not guarantee an increase in CoT monitorability.

We agree that our experiments present counterexamples to broad claims that CoT reliance improves CoT monitorability. We reframed our results around this claim. We are not making the opposite broad claim that increasing CoT reliance always results in lower CoT monitorability. We highlighted the relevant changes in blue text in our updated pdf.

There is also a distinction between needing CoT to solve the main task and needing it to carry out the behaviour being monitored. That makes some of the experiments less direct tests of the necessity hypothesis than the framing suggests. In general, I find the empirical results useful, but would interpret them as identifying limits to when reliance improves monitorability, rather than refuting a generally positive relationship.

We agree with that distinction. This is why in our hint-based evaluations, we scale both the original task difficulty and the difficulty in using the hint. Similarly in our secret-side-task evaluations, we scale both the main task difficulty and the side task difficulty. We present the relationship between CoT reliance and monitorability for both axes.

We rewrote Section 4.1 (increasing unhinted task difficulty) as testing a broader and stronger view of the relationship, where increasing CoT reliance on the main task can improve monitorability even though the behavior being monitored for stays constant. We also rewrote Section 4.2 (increasing the difficulty in using the hint) as testing whether increasing CoT reliance on the behavior being monitored improves monitorability. We found evidence against both views: CoT monitorability does not consistently improve when increasing either view of CoT reliance.

The evidence supports a narrower disagreement with prior work than the introduction suggests. The introduction presents necessity as implying monitorability, whereas the abstract reports that greater reliance prevents silent side-task completion but does not prevent other monitor failures. To me, that supports the computational intuition while showing that it is insufficient for reliable detection. I would therefore want the paper to distinguish these.

We agree that our results support that high CoT reliance forces the model to verbalize its computation in the CoT, but that alone is insufficient for detection. We rewrote the introduction to make it more clear that “necessity as implying monitorability” is a claim supported by prior work, which we show empirical results against. We highlighted the relevant changes in blue text in our updated pdf.

The paper primarily evaluates monitorability using a small set of monitoring approaches, with substantial emphasis on GPT-5.6 Terra as a strong general-purpose monitor. Since monitorability depends jointly on the task model and monitor, how should we expect the conclusions to transfer to monitors specifically optimised for oversight (e.g. monitors trained on monitoring data, specialised classifiers, or multi-stage monitoring systems)? To what extent do the reported results reflect limitations of CoT monitorability itself versus limitations of the particular monitoring approaches evaluated?

We see GPT-5.6 Terra as a much stronger model than what would normally be used as a deployed CoT monitor. Because cost is a key constraint, frontier companies generally use their smallest models as monitors, such as Haiku-sized models in Constitutional Classifiers++ [1] or Gemini 2.5 Flash [2]. Because we’re using a higher performance model than could be practically deployed, we do see it as a strong general-purpose monitor. GPT-5.6 Terra also achieves perfect monitorability scores in many configurations (e.g., Qwen3.6-7B task model on ZebraLogic when scaling original problem difficulty), which shows it is a much more capable monitor compared to gpt-oss-120b.

We agree that monitorability depends jointly on the task model and monitor, and it is important to separate monitorability errors caused by the task model versus by the monitor. One primary cause of task model errors is through non-verbalization of the monitored behavior. In our secret-side-task evaluations, we manually reviewed a subset of the rollouts and counted the number of rollouts that do not mention the side-task at all (Table 2). The Qwen3.6-27B task model primarily relies on the non-verbalization strategy to evade the weaker gpt-oss-120b monitor, and Figure 23 shows that this strategy continues to work against the stronger GPT-5.6 Terra monitor. We are confident that other strong and/or optimized models would also not catch these instances, since there is nothing in the CoT that would indicate the side task was attempted.

In our hint-based evaluations, we use a regex monitor to estimate how often the task model is not verbalizing the hint (Appendix K). Figures 27 and 28 show that the regex monitor almost never detects hint verbalization on the unhinted rollouts, which means that its false positive rate is low. Figure 28 shows that there is a gap on GLM-4.7-Flash on Count Bits between the number of rollouts that use the hint versus verbalize the hint, meaning that some of its monitor errors are caused by the task model not verbalizing its hint usage. This is why the Terra monitor does not achieve high monitorability scores on this configuration. In other cases, hint verbalization does track hint usage, meaning that monitorability errors are primarily due to the monitor missing the hint verbalization in the CoT.

[1] Cunningham, Hoagy, et al. "Constitutional classifiers++: Efficient production-grade defenses against universal jailbreaks." arXiv preprint arXiv:2601.04603 (2026).

[2] Kramár, János, et al. "Building production-ready probes for gemini." arXiv preprint arXiv:2601.11516 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.