acceptodds
Under review as a conference paper at ICLR 2027

How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language Models

Abstract

A model's reasoning trace can expose errors and misbehavior absent from its final response, yet closed-source deployments may show users only that response and a short reasoning summary. Here, we ask how much monitoring performance this reduced access preserves and introduce an observability ladder that holds completed runs fixed and varies access to prompts, responses, reasoning summaries, full traces, and internal model features. We use final-answer correctness as an externally labeled test case across five open-weight models and three benchmarks, with each model writing a self-summary from its completed trace alone. Linear classifiers trained on text features and embeddings reach mean AUROC 0.73 from the prompt alone and 0.75 with the response. The summary then adds 0.02 (95% question-bootstrap CI [0.01, 0.03]), the full trace a further 0.04 [0.03, 0.05], and internal features 0.03 [0.02, 0.04], reaching 0.84. However, a summary-level reader given the trace's word and sentence counts without its text trails the same reader given the full text by 0.01 [0.01, 0.02], so part of the trace's gain is length. Without the prompt, the summary's gain jumps to 0.16 [0.13, 0.18], consistent with the summary standing in for the missing question, and for the linear classifiers, telling a correct from an incorrect run of the same MMLU-Pro question barely rises above chance (0.53 to 0.57). Much of the across-question performance, then, is attainable from the prompt alone, without observing the particular run. Two LLM readers recover far more within-question discrimination from summaries and traces than the linear classifiers when the prompt is withheld, yet reach their highest concordance from the prompt and response alone when it is visible, so what a summary reveals depends on the reader. We have measured one property and one kind of summary, but because the ladder holds the run fixed and varies only what the reader sees, it can be applied to misbehavior with independent labels, to provider summaries, and to stronger readers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning traces, reasoning summaries, monitorability, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_up9QOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks how much monitorability decreases when we do not have full access to the model's reasoning trace. It considers various experimental settings, starting with one where the user receives only the prompt, and then adding the model's response, reasoning summary, full reasoning trace, full next-token distribution, and hidden states. In each setting, this paper measures monitorability using "reader models". I gather that these readers are linear models that use surface text features as well as a text embedding, and predict the probability that the model's answer is correct. The actual results and contributions of the various setups are not clear; the authors report metrics for each setup, but come to no clear conclusion about how important it is to have full reasoning traces rather than some (though this does appear to be non-zero), and also no clear conclusion about how this connects to real-world monitorability.

Critical strengths

Solid motivation and high-level design (ultimately stymied by poor implementation). The idea of measuring how much monitorability we lose by having access to only prompts, responses, and summaries (this seems like the most realistic setting) is interesting and potentially relevant for safety. I do wonder if this is a realistic setup - the labs that tend to do this monitoring have full reasoning trace access - but for individuals interested in doing monitoring on their own, this is still a valuable question. And I think the incremental design is easy to understand and well-founded. Unfortunately, the implementation of these experiments seems quite poor. In particular, it's not clear what exactly the "reader" models in the setup are looking for in the prompts, responses, and summaries. What they probe for is really key here - we care about monitorability because we care about being able to monitor for and observe something. But what is that something?

Critical weaknesses

Confusing presentation. This paper is largely AI-written and it shows. The paper contains confusing compound nouns and strange (mis)uses of standard terms; see e.g. "the corresponding prompt-withheld ladder" or "The main held-out evaluation uses the first run of each question without selecting on its outcome". It's also littered with in-text statistics and confidence intervals that is not really helpful (sometimes, fewer numbers are better)! The text is overall strangely compressed and terse in a way that is very hard to understand for humans (although I'm sure that Claude would have no trouble). As a result, it took quite a lot of time to e.g., understand what exactly the reader models were doing, when this should be immediately obvious. This paper would need a substantial re-write to be acceptable to any human-focused venue.

Conceptual / experimental design issues. This paper's experimental design does not answer the question it sets out to tackle. Monitorability in reasoning traces tends to refer to things like monitoring for misalignment, as exemplified by scheming, reward hacking, etc. However, this paper uses "reader models" to probe for correctness of a model's answer. This quantity doesn't really seem related to the things that we care to monitor.

Even if we take the monitorability of correctness as an important target (say, because we want to verify the correctness of models' reasoning chains), I'm not sure that the reader models proposed do a good job of measuring that. Probes trained on surface-level feature + embeddings seem like a poor proxy for monitorability, and are unreflective of how CoT monitoring is done. The LLMs-as-judges seem more reasonable, but come with their own issues: the judges are stronger than most of the producer LMs you use to generate responses, so it's no wonder that they can do a good job at determining correctness of the producer given the prompt and producer's response.

Questions for authors

Clarification Questions:

  • Why do you probe for final-answer correctness? How does this contribute to the high-level goal of ensuring monitorability (which generally seeks to monitor safety-relevant quantities)?
  • Why do you use reader models trained on surface features + sentence embeddings? It seems weird to do this, as unless the sentence embeddings capture some correctness information, I'm not sure how a logistic regressor is going to provide a meaningful signal that isn't purely based on surface-level features. And such a weak reader model seems uninteresting and unrepresentative of how CoT monitoring is performed in practice.
  • The LLM judges make more sense, if the goal is to answer the question "can we tell if the model's reasoning is correct based on a partial view of its reasoning?". But the results using LLMs seem entangled with the reader LLMs' own ability to solve the question. In particular, since the readers / judges are stronger than the producers, shouldn't it be unsurprising that they don't need the (weaker) producers' reasoning traces? It would make more sense to have weak judges as monitors here.
Reviewer_sWNCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how much correctness-monitoring performance is preserved when a monitor receives a short reasoning summary instead of full CoT. The authors introduce an observability ladder that progressively expands access from the task prompt and final response to a reasoning summary, full CoT, and internal model features, while keeping the underlying model run fixed.

Critical strengths

This paper provides some interesting insights. The prompt-only baseline and repeated-run experiments reveal that substantial across-question prediction performance can arise without observing the actual reasoning process. The trace-length controls further show that simple metadata explains part of the apparent advantage of full CoT. Finally, the comparison between linear classifiers and LLM-based monitors demonstrates that the usefulness of additional reasoning text depends strongly on the monitoring procedure. These findings are informative for designing and interpreting future CoT-monitoring evaluations.

Critical weaknesses

All reasoning summaries in this paper are generated post-hoc by the same model that produced the original CoT. The summarizer receives only the completed trace, without the original question or final response. However, this is not necessarily representative of commercial reasoning-summary systems. For example, Anthropic explicitly states that summarized thinking is processed by a different model from the reasoning model. The choice of summary length also requires justification. The authors instruct the summarizer to produce at most 90 words, but it is unclear whether this budget corresponds to any commercial deployment setting. This paper does not examine how monitoring performance changes when the summary-generation budget is increased.

Questions for authors
  1. Have the authors examined summaries produced by an independent model? Is there any evidence about the effect of providing the original question or final response to the summarizer? This would help establish whether the measured information loss reflects summarization itself or the particular self-summarization procedure.
  2. Do the observed summary–trace gaps exhibit any consistent relationship with model size (e.g., larger models from Kimi & GLM), reasoning performance, or CoT length?
Reviewer_ybcjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tests how monitorable model output is to a reader if they have access only to the final response, a short reasoning summary, or the full trace. The authors have an “observability ladder” where they compare these three settings. They test this on 5 open-weight models and on 3 benchmarks, and generate CoT summaries from the models after they’ve responded do an answer.

The prompt < prompt + response < prompt + response + CoT summary < full trace in terms of monitorability, with the biggest jump being to the full trace. They get these scores by training linear classifiers by embedding the text and using these to predict final-answer correctness on each of the benchmarks.

Critical strengths

The paper correctly used a prompt-only baseline, and tested across multiple families of models and benchmarks. I like their point about monitorability depending on both the reader and the context available.

Critical weaknesses

First of all, this paper reads as largely AI-generated :(. Eg. even in the abstract: “We have measured one property and one kind of summary, but because the ladder holds the run fixed and varies only what the reader sees, it can be applied to misbehavior with independent labels, to provider summaries, and to stronger readers.” or see Table 1. It is 81 pages and almost every line of the Appendix seems AI-written. Titles for results sections are like “Summary construction matters for short displays” which aren’t very clear at all.

Second of all, this paper's methodology is extremely confusing. They consistently refer to a "reader" which, to the best of my knowledge trying to parse the text, is an embedding-based classifier! Their main method of comparing contexts for monitorability is by training a linear classifier using an embedding model to embed the text and add features like word and sentence counts, digit density, etc. I am very concerned by the usefulness of this method for monitorability -- eg. if the summary were to contain hallucinations, this would be unlikely to be caught since it doesn't reduce predictive power. Broadly, I think ability to predict final answer is only measures one part of monitorability, and I hoped they'd have considered more (https://arxiv.org/abs/2512.18311). Further, I think using a classifier on top of embeddings is a weak way to measure ability to predict final answer because embeddings throw away a lot of information. They don't seem to explain why they use this at all. They also do use two LLM readers, but when the GPT-5-min reader is given the trace as well as the summary, performance drops. I am not sure what their setup is, but they don't investigate this much which is extremely surprising if the goal of this paper is to measure monitorability to LLM CoT monitors.

Further, they don't provide any clear takeaways about what the findings mean, and what to do about them. They seem to propose the observability ladder, but it is not a particularly useful or novel framing in my opinion.

Beyond finding this a frustratingly hard read, I have come away not understanding the purpose of the paper, or the underlying motivation and contribution to AI safety or monitorability. The findings ultimately don’t seem particularly novel or interesting to the field of safety oversight.

Questions for authors
  1. What motivated writing this paper – why in particular are you interested in what a user can learn from a response vs. a summary? Is this meant to be a benchmark for CoT summaries? Why did you think this was more important than something else?
  2. Why did you choose a linear classifier on top of embeddings and surface features as the main way to measure monitorability? What should I conclude from a higher AUROC if the classifier can use signals like question difficulty or trace length?
  3. I was surprised that GPT-5-mini does worse when it gets the full trace on top of the summary. What do you think is driving this? Does the result hold if you explicitly ask the monitor to check the reasoning, or compare it to a baseline where the same model solves the question independently and checks whether the answers agree?
  4. What would you do differently when designing a summary or a monitor based on these results?

Comments on the paper

1 comment

We thank the reviewers for their time and honest feedback.

  1. The writing (ybcj, up9Q)

We agree that the paper is harder to read than it should be. We disclosed our use of AI assistance and take full responsibility for the text. The next version will be shorter, better organized and easier to read.

  1. Why this question, and why correctness (ybcj Q1, up9Q Q1)

Users of closed models see the answer and a short summary of the reasoning, not the reasoning itself. We wanted to know what they lose and therefore take a finished run and show a monitor more and more of it (question, answer, summary, full reasoning).

We use correctness because its labels do not depend on the monitor. We agree it is only one part of monitorability and do not claim the results carry over to scheming, or that they say anything about whether a summary is faithful. Still, "is this answer wrong?" is a question every user has.

  1. Why a linear classifier (ybcj Q2, up9Q Q2)

It is a controlled baseline, not a realistic monitor. It is the same at every step and it cannot solve the question itself.

up9Q suspects it mostly picks up surface features. We agree, and showing this is one of our main results. The question alone gives an AUROC of 0.73 out of the final 0.81, and between two runs of the same question the classifier is barely better than chance.

  1. Why the LLM monitors get worse with the reasoning (ybcj Q3, up9Q Q3)

This is not specific to GPT-5-mini. In new results, we find that the open Qwen3.5-27B also does best with only the question and the answer. And it flips when we hide the question: both monitors then improve with the summary and again with the full trace. Our reading is that a monitor that can check the answer itself does not need the reasoning, while one that cannot has to rely on it.

  1. The summaries (sWNC Q1)

On the 90 words, we have now measured the summaries of six current OpenAI and Anthropic models. For five of them a typical summary has 70 to 120 words (ours average 77). We have not yet tested a separate summarizer, one that also sees the question, or longer summaries. This is the most important limitation of the paper.

  1. Model size, accuracy and trace length (sWNC Q2)

We see no pattern with model size or accuracy. On MMLU-Pro the full trace adds about the same over the summary for all five producers, from 4B to 120B parameters.

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.