- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper asks how much monitorability decreases when we do not have full access to the model's reasoning trace. It considers various experimental settings, starting with one where the user receives only the prompt, and then adding the model's response, reasoning summary, full reasoning trace, full next-token distribution, and hidden states. In each setting, this paper measures monitorability using "reader models". I gather that these readers are linear models that use surface text features as well as a text embedding, and predict the probability that the model's answer is correct. The actual results and contributions of the various setups are not clear; the authors report metrics for each setup, but come to no clear conclusion about how important it is to have full reasoning traces rather than some (though this does appear to be non-zero), and also no clear conclusion about how this connects to real-world monitorability.
Critical strengths
Solid motivation and high-level design (ultimately stymied by poor implementation). The idea of measuring how much monitorability we lose by having access to only prompts, responses, and summaries (this seems like the most realistic setting) is interesting and potentially relevant for safety. I do wonder if this is a realistic setup - the labs that tend to do this monitoring have full reasoning trace access - but for individuals interested in doing monitoring on their own, this is still a valuable question. And I think the incremental design is easy to understand and well-founded. Unfortunately, the implementation of these experiments seems quite poor. In particular, it's not clear what exactly the "reader" models in the setup are looking for in the prompts, responses, and summaries. What they probe for is really key here - we care about monitorability because we care about being able to monitor for and observe something. But what is that something?
Critical weaknesses
Confusing presentation. This paper is largely AI-written and it shows. The paper contains confusing compound nouns and strange (mis)uses of standard terms; see e.g. "the corresponding prompt-withheld ladder" or "The main held-out evaluation uses the first run of each question without selecting on its outcome". It's also littered with in-text statistics and confidence intervals that is not really helpful (sometimes, fewer numbers are better)! The text is overall strangely compressed and terse in a way that is very hard to understand for humans (although I'm sure that Claude would have no trouble). As a result, it took quite a lot of time to e.g., understand what exactly the reader models were doing, when this should be immediately obvious. This paper would need a substantial re-write to be acceptable to any human-focused venue.
Conceptual / experimental design issues. This paper's experimental design does not answer the question it sets out to tackle. Monitorability in reasoning traces tends to refer to things like monitoring for misalignment, as exemplified by scheming, reward hacking, etc. However, this paper uses "reader models" to probe for correctness of a model's answer. This quantity doesn't really seem related to the things that we care to monitor.
Even if we take the monitorability of correctness as an important target (say, because we want to verify the correctness of models' reasoning chains), I'm not sure that the reader models proposed do a good job of measuring that. Probes trained on surface-level feature + embeddings seem like a poor proxy for monitorability, and are unreflective of how CoT monitoring is done. The LLMs-as-judges seem more reasonable, but come with their own issues: the judges are stronger than most of the producer LMs you use to generate responses, so it's no wonder that they can do a good job at determining correctness of the producer given the prompt and producer's response.
Questions for authors
Clarification Questions:
- Why do you probe for final-answer correctness? How does this contribute to the high-level goal of ensuring monitorability (which generally seeks to monitor safety-relevant quantities)?
- Why do you use reader models trained on surface features + sentence embeddings? It seems weird to do this, as unless the sentence embeddings capture some correctness information, I'm not sure how a logistic regressor is going to provide a meaningful signal that isn't purely based on surface-level features. And such a weak reader model seems uninteresting and unrepresentative of how CoT monitoring is performed in practice.
- The LLM judges make more sense, if the goal is to answer the question "can we tell if the model's reasoning is correct based on a partial view of its reasoning?". But the results using LLMs seem entangled with the reader LLMs' own ability to solve the question. In particular, since the readers / judges are stronger than the producers, shouldn't it be unsurprising that they don't need the (weaker) producers' reasoning traces? It would make more sense to have weak judges as monitors here.
We thank the reviewers for their time and honest feedback.
We agree that the paper is harder to read than it should be. We disclosed our use of AI assistance and take full responsibility for the text. The next version will be shorter, better organized and easier to read.
Users of closed models see the answer and a short summary of the reasoning, not the reasoning itself. We wanted to know what they lose and therefore take a finished run and show a monitor more and more of it (question, answer, summary, full reasoning).
We use correctness because its labels do not depend on the monitor. We agree it is only one part of monitorability and do not claim the results carry over to scheming, or that they say anything about whether a summary is faithful. Still, "is this answer wrong?" is a question every user has.
It is a controlled baseline, not a realistic monitor. It is the same at every step and it cannot solve the question itself.
up9Q suspects it mostly picks up surface features. We agree, and showing this is one of our main results. The question alone gives an AUROC of 0.73 out of the final 0.81, and between two runs of the same question the classifier is barely better than chance.
This is not specific to GPT-5-mini. In new results, we find that the open Qwen3.5-27B also does best with only the question and the answer. And it flips when we hide the question: both monitors then improve with the summary and again with the full trace. Our reading is that a monitor that can check the answer itself does not need the reasoning, while one that cannot has to rely on it.
On the 90 words, we have now measured the summaries of six current OpenAI and Anthropic models. For five of them a typical summary has 70 to 120 words (ours average 77). We have not yet tested a separate summarizer, one that also sees the question, or longer summaries. This is the most important limitation of the paper.
We see no pattern with model size or accuracy. On MMLU-Pro the full trace adds about the same over the summary for all five producers, from 4B to 120B parameters.
We are happy to answer any further questions.