acceptodds
Under review as a conference paper at ICLR 2027

Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

Abstract

Chain-of-thought (CoT) monitoring is one of the few practical tools available for overseeing capable AI systems, but it is fragile: a model's reasoning is only useful to a monitor if it remains coupled to what the model actually does. This coupling can be incidentally degraded under ordinary post-training through CoT compression, through accidental optimization against a monitor, and through out-of-context generalization from synthetic documents. In this work, we propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability. Across three model organisms with degraded CoT, it recovers most or all of the lost monitorability. Applied to undegraded policies, it further improves judged monitorability and increases the prevalence of consequential reasoning steps. These results suggest that monitorability should be treated as a trainable attribute of the policy. A dedicated monitorability post-training stage can extend current practices that protect CoT from direct optimization pressure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
CoT monitoring, CoT monitorability, AI safety and control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fSHDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether chain-of-thought (CoT) monitorability can be improved through post-training. The authors propose an unsupervised training objective that maximizes the mutual information between CoT and model outputs, combined with a lower-tail optimization strategy and conditional divergence regularization. The proposed method aims to improve the informativeness of CoT while preserving model behavior. Experiments across multiple models and benchmarks demonstrate improvements in recovering degraded monitorability, predicting unsafe behavior, and increasing the prevalence of consequential reasoning.

Critical strengths

1.The paper addresses an important and timely research problem by exploring how to improve the monitorability of chain-of-thought reasoning.

2.The proposed method is interesting, combining mutual information maximization with lower-tail optimization and conditional divergence regularization to improve CoT monitorability.

3.The formulation of CoT monitorability as a representation transfer problem provides an insightful perspective on understanding and improving reasoning transparency.

4.The experiments cover multiple models, benchmarks, and monitoring scenarios, providing relatively comprehensive empirical evidence for the effectiveness of the proposed method.

5.The paper is generally well organized, and the experimental analyses and ablation studies help clarify the contributions of different components.

Critical weaknesses

1.The connection between mutual information maximization and actual CoT monitorability is not fully established. Higher mutual information does not necessarily guarantee that the reasoning is interpretable or faithful to the model's internal decision-making process.

2.The proposed method often increases the length of generated CoTs, which may partially explain the improvements in monitorability. More controlled comparisons under matched reasoning lengths would strengthen the conclusions.

3.Some improvements in monitorability are accompanied by substantial changes in model behavior. It remains unclear to what extent the gains come from more informative CoTs rather than changes in the distribution of model outputs.

4.The evaluation relies partly on LLM-based judgments of monitorability, which may introduce evaluation bias. Additional human evaluation or independent monitoring methods would provide stronger evidence.

5.Although the method is evaluated across several settings, most monitorability failures are constructed through controlled training procedures. Its effectiveness against naturally occurring or adaptive monitoring failures remains unclear.

Questions for authors

see weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fSHD

We thank you for your thoughtful review and constructive suggestions. We have added analyses of CoT faithfulness, controls for length and changes in behavior, and evaluations with additional LLM judges to address your concerns.

The connection between mutual information maximization and actual CoT monitorability

In our view, CoT monitorability can fail at three levels:

  1. [Lack of information] Models may totally omit information from their CoT. This is the failure mode that [1] term CoT-Obfuscation.
  2. [Legibility] Models CoT may contain information, but it may not be easily readable by external observers.
  3. [Monitor failure] CoT legibly contains the information, but monitor fails to detect (e.g., due to being non-robust).

We target the first failure mode. Our goal is to ensure that CoT retains the required information. We believe this is particularly important to study given the rise of no-CoT capabilities in the LLM (see Section 3.4.5.3 of the Gemini Argon report [2]) where LLMs may completely omit the information from their CoT rendering CoT monitorability infeasible.

We have clarified this point in the introduction. We have also revised the Conclusion section of the paper.

Higher MI and faithfulness: In our new analysis in Appendix I, we show that MI training directly improves causal effect of CoT on model outputs in some of our settings, and overall improves consistency between CoT and the output. Based on this, we cautiously claim that MI training does improve faithfulness of CoT as well. We also observe increased “faithfulness” anecdotally, e.g., on MBPP, CoTs describe reward hacks after MI training.

Length

Please see our common response, which addresses this concern through length-reward baselines and comparisons restricted to shared CoT length ranges (Appendix I.1). In general, we find that length inflation does not explain our monitorability gains. As a concrete example, on all three safety benchmarks, the monitorability gains come with shorter CoTs, while the length-reward baseline produces longer CoTs but performs worse than MI training.

Gains mostly come from more informative CoT

We agree with the reviewer’s concern that changes in behavior are a potential confounder. However, the paper now provides several pieces of evidence that these changes do not fully explain the monitorability gains:

  • Haskins: Monitor recall on deceptive outputs increases from 39.5% to 93.1% (Table 2). Thus, alongside reducing deception, training makes the remaining deceptive outputs substantially easier to detect.
  • MBPP: Similarly, monitor recall on hacks increases from 1.4% to 37.4% after MI training. In the more fine-grained analysis of Table 13 (“Input-branch” column), we also find that monitor recall increases for outputs that special-case the test inputs shown in the prompt (1.4% to 42.4%). This kind of hack is the primary behaviour of the base checkpoint (97.3%), so this shows that MI further improves monitorability independent of behaviour changes. Similarly, on problems where both the starting and trained models produce hacks, monitor recall is 37.8% for the trained models, compared with at most 1.9% for the starting model (Table 13, “Prompt-matched” column).
  • HarmBench: Among prefixes followed by safe answers, the monitor’s false-alarm rate falls from 54.1% to 24.0%. The reasoning also becomes more consistent with these safe answers: the fraction of prefixes misleadingly announcing a plan to comply with the harmful request falls from 68.8% to 23.6% (Appendix I.3, Figure 13 and accompanying text).

The newly included analyses in Appendix I make us considerably more confident that this objective produces robust and consistent improvements in monitorability across settings.

Additional evaluation of monitoring gains under different LLM judges

We have added an evaluation of monitoring gains on GPQA-diamond under different LLM judges. Monitorability gains hold across all three judges for both mean and worst-of-four monitorability (Table 6). For safety benchmarks, we evaluate monitorability using two monitors and activation probes, all of which show improvements (Table 3). Some evaluations also use changes in answer probabilities and intervention-based metrics (Table 4).

Values show MI-recovered minus compressed monitorability on GPQA-Diamond.

JudgeΔ Mean monitorabilityΔ Worst-of-four monitorability
GPT-5.6-Luna+0.183+0.310
Claude Sonnet 5.5+0.192+0.335
Gemini 3.8 Flash+0.191+0.404

We hope these revisions and additional results address your concerns. If they do, we would appreciate your consideration to revise your score upward.

References

[1] Baker et al. (2025). https://arxiv.org/abs/2503.11926

[2] Gemini Argon report, §3.4.5.3. https://storage.googleapis.com/deepmind-media/gemini/gemini_4_safety_report.pdf

Reviewer_SPZCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The main claim of the paper is that CoT monitorability is trainable and can be trained in a label-free manner. The training objective uses the mutual information between CoT and the output. The model conducts experiments on three specially trained models without poor monitorability, such as models with compressed, obfuscated, or concealed reasoning behaviors, and shows that this proposed objective can repair monitorability on these models.

Critical strengths

The paper studies a real and timely problem of improving CoT monitorability, and the way to attack the problem using an unsupervised method is promising. Unlike existing work which mostly focuses on measuring CoT monitorability, this paper attempts to repair it.

The paper is overall easy to follow. The methodology is clear and easy to understand, and the figures are clean to read.

Critical weaknesses

The methodology of the paper largely overlaps with Anwar et al. 2026; the main difference, in my understanding, is the setting, where the prior work used it as a regularizer during RL to prevent degradation, and this paper uses it to repair degradation after a model is damaged. The additions to it is minor, such as the Chi-squared constraint (which also comes from prior works) and a lower-tail contribution term.

Compared to Anwar et al. 2026, these additions also need better ablation studies. E.g., does the lower-tail contribution term actually helps? It is unclear based on the reported results. Some simple and intuitive baselines like just encouraging a longer CoT should also be compared.

Another big confounding factor not well studied is the token length. It appears to me that the proposed method will significantly increase the CoT length, but whether the monitorability increases or decreases when compared at the same CoT length is unknown. For example, model A and model B both output 1000 CoT tokens, but they can have very different monitorability. It is possible that all the training objectives are just to increase CoT length rather than making the tokens actually more monitorable.

Questions for authors

A length-matched control is essential for the paper. We need to make sure the post-trained model has the same median token length as the original one, then compare the monitorability. You may do this by adding some length regularization terms, or using reject sampling.

Also, how about using a simple object to encourage longer CoT? This simple baseline needs to be compared against.

If I understand your loss terms correctly, your reward can be increased by a CoT that simply restates the final answer (e.g., the CoT just repeat the answer a few times). Did you observe this during training, and why is this degenerate case not happening?

Adding more qualitative examples can be helpful. A small human evaluation of how the monitorability actually aligns with human perception can also be helpful.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SPZC

Thank you for your review and constructive suggestions. We have now expanded the paper with controls, component ablations, mechanism analyses, and qualitative examples to address your concerns.

Novelty over Anwar et al. Our work indeed uses the mutual information as part of our objective (similar to Anwar et al.), however, the final objective, its application, and the resulting empirical findings all differ substantially:

Anwar et al.Our work
ObjectiveUse mean MI and require tuning of scale and clipping parameters a,b to limit overoptimization and goodharting.Optimizes lower-tail (robust) MI under a conditional χ² constraint, which naturally limits MI overoptimization. There are no scale / clipping parameters.
ApplicationUse it as a regularizer during task RL, where the objective needs tuning for specific tasks.Use it as a fully unsupervised post-training objective with no task reward necessary.
Key FindingDemonstrates prevention of CoT obfuscation and mitigation of reward hacking during task RL.Demonstrates recovery from already developed CoT degradation and further improvements in undegraded policies.
Motivation and analysisEstablish information-theoretic conditions for monitorability and analyses errors arising from approximate monitors.Motivates the MI objective from a transfer-learning perspective and analyzes various mechanisms through which it improves monitorability (Appendix I).

In particular, we believe that the focus on recovery setting is consequential: preventing obfuscation as it develops does not establish that monitorability can be restored once it has been lost. Our experiments demonstrate this. We would appreciate the reviewer’s reassessment of the novelty in light of these methodological and empirical distinctions and the additional results.

Ablations

We include comprehensive ablations (App. J). We compare lower-tail optimization with ordinary MI, and separately remove the χ² constraint. Overall, these ablations establish that both the lower-tail optimization and conditional regularization are essential for robust results across settings. We have also performed a length control experiment, which we discuss briefly below, and in more detail in the common response and App. I.

Lower-tail optimization stabilizes training and consistently improves worst-case monitorability metrics

  • On MBPP, removing lower-tail optimization leads to less stable training, including a late collapse in full-test pass rate in one run. At the same χ² budget, the hacking rate remains similar, but the fraction of solutions passing all tests falls.
  • Holding the χ² budget fixed, ablating lower-tail optimization in favor of mean MI reduces GPQA-Diamond mean monitorability from 3.465 to 3.364 and worst-of-four monitorability from 3.192 to 3.061.
  • On all three safety benchmarks, lower-tail optimization improves worst-class F1 for both the CoT monitor and the probe.

The χ² constraint limits behavioral changes during MI training

  • On MBPP, removing the constraint increases recall somewhat, but reduces the fraction of solutions passing all tests by more than half and increases the false-positive rate 5x.
  • Holding the lower-tail objective fixed, removing the constraint reduces GPQA-Diamond accuracy while CoT length grows 4x and mean monitorability falls.
  • On all three safety benchmarks, removing the constraint causes large behavioral shifts. Holding the lower-tail objective fixed, unsafe continuation rates change from 19.0% to 3.7% on HarmBench, 22.4% to 2.9% on StrongREJECT, and 22.8% to 6.2% on SORRY-Bench (Table 3). This illustrates the constraint’s role in improving monitorability while limiting changes to the underlying behavior.

Length does not explain monitorability, length training does not improve monitorability

We perform two analyses, which we discuss in more detail in the common response: first, we compare completions from MI runs with same-length completions from the base checkpoints, and find that the MI completions have improved monitorability. Second, we train models from the base checkpoints with a CoT length reward, and find that such training does not generally improve monitorability at same or higher length CoTs. This lack of improvements from length baselines is consistent with MacDermott et al. [1], who also found that rewarding reasoning length did not improve monitorability in their experiments.

Qualitative Examples

We have added four qualitative examples for each of our settings (Appendices A, B, D, and E). Please let us know if you would find additional samples or particular types of examples useful.

We hope these revisions address your concerns and would appreciate your considering an increase in your score.

References

[1] MacDermott et al. (2025). Reasoning Under Pressure. https://arxiv.org/abs/2512.00218

Thank you for the response. The ablation study results showing the usefulness of different components and the effect of token lengths are helpful, and I appreciate the efforts the authors have made. But overall I still find the techncial contribution of this paper to be rather incremental, as the key training loss terms comes from Anwar et al., and the setting of the paper is also quite limited (experiments mostly on degraded models without poor monitorability); the method might be effective on these degraded models because the loss is designed to fix them, but evidence that it can actually become a useful algorithm on models used in real-world is quite limited. I thus believe this paper is technically sound but still on the borderline for a top-tier machine learning conference. I cannot enthusiastically support the acceptance of this paper.

AuthorsOpenReview ↗

The characterization of our experimental setting as 'mostly on degraded models' is inaccurate

Thank you for your continued engagement, we really appreciate your contribution to this peer review process. In the response below we set out our points of disagreement and the evidence supporting our position.

the setting of the paper is also quite limited (experiments mostly on degraded models without poor monitorability); the method might be effective on these degraded models because the loss is designed to fix them

Respectfully, we disagree with this strongly.

  1. Two of our five experimental settings explicitly study undegraded models and show monitorability improvements (monitoring for safe responses and consequential reasoning). For example, Qwen monitor accuracy on HarmBench improves from 63.8% to 74.6% and probe accuracy goes from 71.4% to 80.2% (Table 3). This is due to MI training on an undegraded model, which shows that monitorability improvements are not just due to fixing specifically induced degradations.
  2. The other three settings are quite diverse and have three different (realistic) sources of monitorability degradation
    1. Compression training, which is used in frontier model training. E.g., Kimi k1.5 uses length penalties and long-to-short training to reduce reasoning length.
    2. Degradation induced by out-of-context learning about CoT monitoring, as demonstrated by Haskins et al. in models including Kimi K2. Their experiments simulate exposure to information about monitoring through pretraining data, which could arise in practice through, e.g., training on papers about CoT monitorability.
    3. Training against the monitor (both OpenAI and Anthropic have done this accidentally, e.g., https://alignment.openai.com/accidental-cot-grading).

Across all these settings, a single objective is shown to improve monitorability. Thus, the assertion that our method is effective on these [5 different settings] because the loss was designed to fix them, is inaccurate. The objective receives no information about which mechanism caused the failure. Its ability to restore useful monitoring across these mechanisms is an empirical finding, not something guaranteed by its construction. Our evaluations use external monitors, task performance, and CoT-output consistency measures and we find our full MI objective generally improves all three, rather than improving a single targeted metric.

More generally, we study how CoT–output coupling can be strengthened. Weak coupling between CoT and output is a problem that exists in real-world models. In Appendix G, we document examples of frontier models whose final answers or actions contradict the conclusions or plans expressed in their CoTs. Through our analysis in Appendix I.3, we show that our method strengthens this coupling. These findings support its (or some advancement of it) potential applicability to frontier models. Indeed, we find it to work well on GPT-OSS-120B, which achieves near-parity with o4-mini on core reasoning benchmarks according to OpenAI.

AuthorsOpenReview ↗

On novelty

In the response below we address the reviewer's novelty point.

I still find the technical contribution of this paper to be rather incremental, as the key training loss terms comes from Anwar et al.

We agree that the MI estimator builds on Anwar et al., and we credit them explicitly. Our contribution is to establish and develop dedicated, label-free monitorability post-training as a useful procedure across different degradation mechanisms and undegraded models. This differs substantively from Anwar et al.’s using MI as an auxiliary regularizer during task and monitor reward training: there is no task reward or external monitorability supervision to ensure that optimizing the objective produces useful behavior.

Whether this procedure works is an empirical question. Our experiments establish both the potential of our objective and failure modes of naively using the MI term on its own. In particular, increasing MI alone can substantially degrade task performance without improving monitorability. For example, on GPQA-Diamond, removing χ2\chi^2 regularization reduces accuracy from 46.6% to 27.4%, despite substantially longer reasoning (App. J.1). On MBPP, the lower-tail objective increases true pass rate from 34.0% to 48.6% and reduces false positive rate from 13.8% to 3.9%, compared to ordinary MI (App. J.2). These findings show that the χ2\chi^2 constraint and the lower-tail objective make substantive contributions to the usefulness of MI-based monitorability post-training.

This contribution is also particularly timely, given frontier models are becoming less monitorable. E.g., “Gemini 4 Argon can evade a realistic monitor on easy sabotage tasks about half the time when encouraged to do so” (p. 124 of Gemini 4 Argon report) and “GPT-6 Astra, however, exhibits lower average CoT monitor recall on hacking when prompted to be less monitorable than when prompted to be more monitorable” (Sec 9.2.2.3.2 of GPT-6 Astra System Card).

We believe that demonstrating label-free monitorability post-training across degraded and undegraded models, and developing the objective to make this procedure useful and stable, constitutes a substantive scientific and methodological contribution beyond the existing MI regularizer.

Reviewer_5Gh6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper views CoT as an intermediate representation for generating the final answer and defines CoT monitorability as the extent to which the final answer can be predicted from the CoT.

To estimate conditional mutual information (MI), the authors generate multiple trajectories for the same problem and evaluate the likelihood of each output conditioned on the CoTs from other trajectories. They then compare the likelihood of an output given its own CoT against its average likelihood across all sampled CoTs.

By maximizing this conditional MI through post-training, the authors aim to improve CoT monitorability. They demonstrate that this approach reduces the decoupling between CoT and final answers across various task settings.

Critical strengths
  1. Given our current limited understanding of internal mechanisms of llms, I think that directly improving CoT monitorability by treating it as an explicit optimization objective is an meaningful approach.
  2. The paper frames CoT monitoring as a representation transfer problem and demonstrates that CoT monitorability can be directly improve through a MI-based objective.
  3. The results in Table 3 are particularly interesting, as they show that the proposed training method also leads to improvements in activation probing.
Critical weaknesses

The authors' primary goal appears to be to predict a model's behavior, particularly undesirable behavior, by observing its CoT. However, the legibility and coverage metrics used in Figure 1 seem more closely related to CoT faithfulness, as they evaluate the completeness and interpretability of the reasoning process (faithfulness). In contrast, Table 1-3 primarily evaluate whether the model's behavior can be predicted from its CoT, which does not necessarily require the CoT to be faithful (predictive monitorability). In my view, the objective of this paper is just surrogate objective of predictive monitorability, authors should clarify the relationship between them.

For example, Authors can clarify how increased MI -> improved predictive monitorability -> imporved faithfulness are connected.

Questions for authors

Could you evaluate whether the proposed MI objective (equation 1) encourages answer leakage in CoT?

Since the objective rewards CoTs that make the final answer more predictable, the model might simply reveal the answer in its CoT without preforming valid reasoning.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5Gh6

Clarifying connections to broader CoT monitorability problem and faithfulness. Thank you for this suggestion. We have revised the Conclusion to focus on clarifying this.

Our goal is to improve predictive monitorability: whether a monitor can predict properties of the model’s behavior from its CoT. MI is indeed a nice surrogate objective for this, which we motivate from the transfer learning perspective. As discussed in Baker et al. [1, §5.1], CoT monitorability need not require fully faithful CoTs, but it does require them to contain the information needed for the monitoring task. Hence, we focus on direct gains in monitorability metrics.

Nevertheless, in the revised PDF we have included a new appendix with analyses showing that MI training indeed improves CoT faithfulness (Appendix I.3). Our high-level findings are:

  • On Haskins and MBPP, MI training increases both the causal influence of the CoT and its agreement with the final output.
  • On safety benchmarks, CIR does not increase appreciably, but CoT–output agreement improves on HarmBench and StrongREJECT. A finer analysis finds fewer misleading compliance plans at intermediate prefixes on all three benchmarks.

Our common response (Part 1, Part 2) presents these results in detail.

Answer leakage.

We perform an extensive analysis of final answer copying in Appendix I and find that this does not explain the observed monitorability gains. We also explore why MI training does not collapse to answer-only CoTs. We refer the reviewer to the common response (Part 1, Part 2) for a more comprehensive discussion, but in brief:

  • Answer-only CoTs are not guaranteed to be optimal given a constrained decoder (Appendix I.2.3).
  • Original, proper CoTs achieve competitive MI, thus limiting the incentive for drastic changes in the model’s behavior.
  • Removing detected answer copies preserves the monitorability gains. The gains also persist when evaluation is restricted to CoTs or safety prefixes with no detected copies (Figures 10 and 11).

References

[1] Baker et al. (2025). https://arxiv.org/abs/2503.11926

I read the authors' rebuttal and the new manuscript.

My initial review scored the paper as a weak accept, but after reading the rebuttal, I have some concerns about this paper.

  1. The model might be able to generate the same answer but with different reasoning. I mean, when we have lower answer entropy, how large are the nominal upper bound and the achievable bound?

  2. What is the basis for claiming that the divergence difference in Table 29 indicates an optimization barrier preventing answer collapse? The current values just show the difference between two decoders under the synthetic CoT condition. How can this explain the cost of distribution shift? According to the experiment in the general comment (answer copy), the Original + answer condition's MI is higher than that of the others. So, this literally shows that reasoning remains and answer copying will arise.

So, I think their explanations and claims are little bit ambiguous.

Thank you for your continued close engagement and your feedback. We apologize if any part of our exposition was unclear.

The basic point of App. I.2 and the answer copying general comment above is to show that:

  1. Putting the answer in the CoT does happen, but does not explain monitorability gains (Figs 9, 10). For instance, recall on hacks for MBPP is 35.6% after removing answers (37.4% before removal), which drastically improves over 1.4% for the base.
  2. CoT-only behaviour is competitive with answer-only and CoT+answer behaviour (Table 29). We also note that the χ2\chi^2 divergence penalises answer-only CoT more heavily. This supports the empirical finding that we observe a mix of CoT-only and some CoT+answer behaviour rather than collapse into CoTs that consist only of the answer or multiple copies of the answer. Both of these modes improve monitorability, and, per 1), for the CoT+answer mode the improved monitorability is not just because of the answer appearing in the CoT, but majorly due to genuine improvement in CoT monitorability. Appendix I.3 further supports this MI mechanism through our analysis of CoT-output consistency.
  3. The synthetic CoT experiment further suggests that MI continues to favor problem-specific CoTs even when (multiple) answer copies are available. CoTs unrelated to the specific problem take a heavy MI penalty, which is not offset by the MI gain of putting the answer in CoT.

We agree that Table 29 does not establish an optimization barrier, and have rephrased the last two sentences of App. I.2 to:

Empirically, the MI-trained models retain substantive and monitorable reasoning even when answer copies are removed from the CoT. We hypothesize that the models' pretrained inductive biases, potentially reinforced by the divergence regularization, favor such reasoning even when answer-only CoTs are competitive in MI.

Separately, we would note that the use of a divergence term to regularize potentially degenerate optima is common practice in RL with PPO/GRPO-style objectives, although our analysis here suggests answer-only CoT does not even attain the highest MI in our experiments.

Regarding the first point:

The model might be able to generate the same answer but with different reasoning. I mean, when we have lower answer entropy, how large are the nominal upper bound and the achievable bound?

The answer entropy $H(O|X)$ equals $I(O;Z|X) + H(O|Z,X)$, so it upper bounds the MI. If the answer is deterministic given the prompt X, then MI is zero regardless of variations in the reasoning. For non-deterministic models, generating a similar answer but with different – more monitorable – reasoning fits the ‘revelation’ mode (App. F) and the CoT-output consistency result of App. I.3. We note that we do not observe a collapse of the output distribution: e.g., for MBPP, MI training increases how many distinct code answers we see on the same prompt from 4.16 to 7.11 (over 8 rollouts), and decreases the probability that two answers exactly overlap from 32.1% to 5.1%.

If your question is about the nominal bound estimated in App. I.3: original CoTs reach 54% of the upper bound, answer-only reaches 56%, and CoT+answer reaches 57%.

If this does not answer your concern, could you clarify if your concern is about the amount of information available, about the gap between the MI of our models and the nominal upper bound, or about whether increased MI reflects consistent reasoning?

We hope our response answers your questions and would be happy to discuss further if any concerns remain.

Thank you for revising the paper and providing additional clarification.

Regarding my second question, I meant the nominal upper bound. My concern was that log⁡N\log N, where NN is the number of samples, could be larger than the actual entropy of the output distribution. Therefore, the gap between the measured MI and this nominal upper bound may not necessarily reflect the limitations of the decoder.

However, I agree that the relative ordering under the same evaluation conditions still shows that original + answer achieves higher MI than answer-only.

My original concern was whether MI training might incentivize answer copying rather than genuinely improving reasoning faithfulness, and how this possibility aligns with the paper's claims. I acknowledge that the additional analyses in Figures 10 and 11 partially address this concern by showing that the monitorability gains persist even after removing copied answers.

Overall, I appreciate the authors' efforts to address my questions.

I will maintain my original positive rating.

Thank you.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common response (1/2): Length controls and CoT faithfulness

Reviews noted that monitorability improvements might be due to length inflation or answer copying in CoT. In Appendix I, we test these explanations directly and show that neither provides a general explanation for the gains. Instead, we show that our method improves the causal influence of the CoT reasoning on Haskins and MBPP and generally improves the consistency of the CoT towards its output.

Length analysis

Increasing length alone does not explain the gains in most environments. We test a length-reward baseline, which simply provides a length bonus added to the task reward for compression recovery; the other settings use length reward alone (this is based on MacDermott et al. [1]) and compare their monitorability with MI training.

The length baseline is competitive with MI training in compression recovery (Figure 8).

  • On MBPP, matching CoT length does not reproduce the monitoring gain. The length-reward baseline achieves only 3.9% recall versus 37.4% for same-length CoTs. Restricting the original and MI-trained models to similar-length CoTs shows a similar result: 2.1% versus 32.0% recall. Similar analyses on Haskins, GPQA and all three safety benchmarks also show that the monitorability advantage of our method is preserved (Appendix I.1.1, Table 28).
  • On Haskins, the length reward produces substantially longer but less monitorable CoTs. The length control produces 370-token CoTs with 38.3% recall. For comparison, an MI checkpoint produces 120-token CoTs with 63.0% recall, compared with 32.3% for the starting model.
  • On all three safety benchmarks, MI improves monitoring with shorter CoTs, while rewarding length worsens monitoring. Both the CoT monitor and activation probe show this pattern. On HarmBench, for example, lower-tail MI reduces mean prefix length from 250.3 to 218.2 tokens, while increasing CoT-monitor balanced accuracy from 63.3% to 78.6%. The length reward increases mean prefix length to 256.2 tokens but reduces balanced accuracy to 58.6%.
  • The length baseline is also easily reward-hacked. On Haskins, the model learns to pad its reasoning with repetitive loops. After 50 training steps, 179 of 800 rollouts fail to finish, and another 37 exceed 5,000 reasoning tokens. We observe no comparable degeneration in the starting model or models trained with our method. On MBPP, continued training on the length reward leads to reward hacking by copying the output program into the CoT. This model reaches 38.4% recall, but its detections depend almost entirely on these copies.

Lack of improvements from length baselines is consistent with MacDermott et al. [1], who also found that rewarding reasoning length did not improve monitorability in their experiments. Their analysis also documents increased repetition and inconsistencies that can make longer reasoning harder to monitor.

Improved consistency and causal influence of CoT

MI training makes CoT more faithful to the model’s behavior. We test whether the final output follows the CoT’s stated conclusion or plan, and whether the reasoning causally influences the output (Appendix I.3). We use GPT-5.4 with medium reasoning effort for the agreement judgments. We measure causal influence using the causal importance of reasoning (CIR) metric from Yu et al. [2], which quantifies how truncating the CoT changes the probability of a specified binary outcome.

  • On MBPP and Haskins, CoT–output agreement and causal importance of reasoning both increase. Agreement rises from 6.8% to 78.6% on MBPP and from 70.5% to 100% on Haskins. Mean CIR increases 6.1× on MBPP and 2.8× on Haskins (Figure 12). The obfuscating MBPP model typically describes a general solution but produces code that hardcodes the revealed tests. After MI training, the reasoning more often describes what the code actually does.
  • On safety benchmarks, CIR does not improve, but other metrics show improved CoT–output consistency. In these environments, we compute CIR using safe/unsafe labels. We do not observe any improvement in CIR here, consistent with prior findings that on safety benchmarks, deliberation generally does not affect model’s decision ([3]). However, CoT–output agreement increases on HarmBench (+9.3 percentage points) and StrongREJECT (+6.5), and is unchanged on SORRY-Bench. A finer-grained analysis shows that MI training makes CoT less misleading. Among prefixes followed by safe answers, the share expressing a plan to comply falls from 68.8% to 23.6% on HarmBench, 51.2% to 44.0% on StrongREJECT, and 64.0% to 58.9% on SORRY-Bench (Figure 13). On HarmBench, these plans account for 93% of the starting policy’s false alarms, and the false-alarm rate falls from 54.1% to 24.0% after training.

References

[1] MacDermott et al. (2025). Reasoning Under Pressure. https://arxiv.org/abs/2512.00218

[2] Yu et al. (2026). https://arxiv.org/abs/2604.22074

[3] Do Thinking Tokens Help with Safety? https://arxiv.org/abs/2606.25013

AuthorsOpenReview ↗

Common response (2/2): Answer copying

Answer copying

Answer copying does not explain the monitorability gains. Copying behavior does occur in base models and increases after MI training, but removing copied answers has little effect in most environments and preserves the gains throughout (see the Figures 9 and 10). Copying is rare on open-ended safety benchmarks across both base and MI-trained models, but does occur in both model types in environments with short answers, such as GPQA. However, the gains also persist when evaluation is restricted to CoTs or safety prefixes with no detected copies (Figure 11).

Answer copying is not necessarily optimal for constrained decoders. A constrained decoder (like our LLM) may have biases that limit it achieving the Shannon capacity. Indeed, we empirically find that answer copying does not saturate MI under our constrained decoder.

Following the training objective, we divide each sequence-level MI score by the final answer’s token count. On 50 GPQA prompts with a group size of 32, answer-only CoTs achieve (102.7 ± 9.3) × 10⁻³ nats per output token for the MI-trained model, compared with (85.3 ± 8.6) × 10⁻³ for the base model. The MI-trained model’s score is only 55.8% of its upper bound of 184.1 × 10⁻³ nats per output token, obtained by averaging log(32)/|O| over the sampled answers.

Original CoTs from the MI-trained checkpoint already achieve (100.1 ± 9.5) × 10⁻³ nats per output token, or 97.4% of the answer-only score. At group size 8, they achieve 98.9%. Producing the full reasoning is therefore competitive with supplying the answer alone, and, empirically, is the behaviour that arises during training.

Appending the answer to the original CoT nevertheless provides a marginal gain over CoT only and achieves the highest mean and lower-tail MI scores and the lowest divergence: 0.072 ± 0.001, compared with 0.319 ± 0.002 for original CoTs and 0.446 ± 0.009 for answer-only CoTs. This is reflective of existing biases of the implicit decoder within the reasoning trained LLM, which prefers an elaborate CoT followed by answer drafting. Our training then builds on these existing biases: the MI reward favors CoTs that are more informative to the decoder, while the divergence constraint limits changes to the decoder’s output distribution. Hence, we do not observe any collapse towards answer-only CoTs. In fact, as shown in Figure 9, we generally do not observe any strong answer copying behaviors.

We also replace the original reasoning with a CoT from another GPQA question, keeping the final answer and frozen MI-trained decoder fixed. At group size 32, mean MI falls from 105.1 to −105.9 × 10⁻³ nats per output token. Repeating the answer five times still gives a negative score, −90.3. Divergence also increases, from 0.072 to 0.397. Thus, supplying the answer does not compensate for replacing the relevant reasoning.

The following tables report the results for group sizes 32 and 8.

Group size 32:

CoTMean MI (10⁻³ nats/output token)Lower-tail MI, α = 0.75 (10⁻³ nats/output token)χ² divergence, budget = 1.0
Original100.1 ± 9.577.8 ± 8.30.319 ± 0.002
Answer-only102.7 ± 9.380.9 ± 8.20.446 ± 0.009
Original + answer appended105.1 ± 9.583.3 ± 8.20.072 ± 0.001
Unrelated CoT + answer-105.9 ± 55.3-186.7 ± 69.90.397 ± 0.004
Unrelated CoT + answer (×5)-90.3 ± 48.1-171.1 ± 63.10.373 ± 0.003

Group size 8:

CoTMean MI (10⁻³ nats/output token)Lower-tail MI, α = 0.75 (10⁻³ nats/output token)χ² divergence, budget = 1.0
Original67.8 ± 6.455.0 ± 5.60.319 ± 0.002
Answer-only68.6 ± 6.256.1 ± 5.50.446 ± 0.009
Original + answer appended70.4 ± 6.457.9 ± 5.60.072 ± 0.001
Unrelated CoT + answer-75.3 ± 38.2-135.3 ± 51.20.397 ± 0.004
Unrelated CoT + answer (×5)-72.3 ± 35.9-132.4 ± 48.70.373 ± 0.003

So, on the whole:

  1. Answer-only CoTs are not guaranteed to be optimal given a constrained decoder.
  2. Original, proper CoTs achieve competitive MI gains, thus limiting the incentive for drastic changes in the model’s behavior.
  3. In general, replacing the reasoning with some unrelated reasoning is costly for the model under our objective, and appending repeated answers is insufficient to compensate for this.
AuthorsOpenReview ↗

Summary of the rebuttal process

We thank all reviewers for their reviews and engagement. For the convenience of the AC, we provide an overview of improvements to our paper made during the rebuttal period. We believe we have addressed most of the major concerns of the reviewers. The primary remaining concern seems to be the novelty point of reviewer SPZC, which we believe we comprehensively address in our comment here: https://openreview.net/forum?id=5xrNwUd8y4&noteId=6dzk7sxIXB.

  1. Improved ablations and baselines. We now include a full set of ablations of our χ2\chi^2 regularizer and lower-tail MI objective (Appendix J). These findings show that this constraint and the lower-tail objective make substantive contributions to the usefulness of MI-based monitorability post-training.
  2. Extensive investigation of the mechanism by which MI improves monitorability. Appendix I now analyses the mechanism by which MI improves monitorability in three parts:
    1. We exclude length as a confounder for monitorability. We compare our MI-trained checkpoints with models trained on length rewards that encourage longer CoT. Separately, we compare MI-trained and base model checkpoints on CoTs of similar length by rejection sampling. Both analyses show that CoT length does not explain our observed monitorability gains.
    2. We exclude answer copying as a confounder for monitorability. We show that our observed monitorability gains are not explained by the presence of answer copies in the CoT. Additionally, we present an analysis that suggests MI training favours using relevant CoTs that contain reasoning before stating the answer, and that degeneration into pure answer copying is not an optimum under our constrained decoders.
    3. We show that our objective improves causal influence and consistency of CoT across our various environments, and that these improvements correlate to improved monitorability. In particular, we find that MI training strongly improves CoT-output agreement, for instance by making reasoning in MBPP properly describe a resulting hack, and by reducing misleading compliance plans on the safety benchmarks. Overall, this suggests one of the effects of MI training is to improve faithfulness of CoT reasoning in the tested models.
  3. Timeliness of our work. We have clarified the timeliness of our work in the conclusion. CoT monitorability is a fragile opportunity for AI safety (Korbak et al., 2025) and CoT monitorability in frontier models has recently started to degrade. For instance, the Gemini 4 Argon report (p.124) states that “Gemini 4 Argon can successfully complete the harmful side task without detection 45.5% of the time, compared to 17.9% and 9.6% of the time for Gemini 3.8 Flash and Gemini 3.1 Pro”. The GPT-6 Astra System Card (sec. 9.2.2.3.2) shows that monitor recall on hacks in Impossible Coding Tasks drops from 94.2% in 5.6 Sol to 79.8% in 6 Astra, and to a substantially lower 61.2% when prompted to be less monitorable. This makes our paper timely in pushing for training-time interventions that may help preserve CoT monitorability. Importantly, we show that a single unsupervised objective can be effective at improving monitorability both for a) models whose monitorability has been degraded through realistic effects of training (compression, training against a monitor, out-of-context learning), and for b) undegraded models. As discussed above, we show these improvements majorly operate through improved CoT-output coupling: weak coupling is also a current problem in frontier models (Appendix G).

We thank the reviewers and AC again for their time and constructive engagement, which have substantially strengthened the paper. We believe that establishing a label-free approach to monitorability post-training, and developing the objective to make this procedure useful and stable, provides substantive scientific and practical value to the community at a time when preserving CoT monitorability is increasingly important.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.