acceptodds
Under review as a conference paper at ICLR 2027

When Reading the Chain of Thought Falls Short: A Testbed for Reasoning Trace Analysis

Abstract

Reading the chain of thought (CoT) is a widely used safety technique for reasoning models, but it struggles when the CoT leaves out or misrepresents the factors driving a behavior. However, we lack benchmarks that focus on these cases where reading the CoT fails, so progress on alternative methods is hard to measure. To address this gap, we introduce and release nine novel CoT analysis tasks, each with in-distribution (ID) and out-of-distribution (OOD) test sets. All nine tasks are extremely challenging, with both prompt-optimized frontier LLM monitors and human reviewers frequently achieving no better than chance. We benchmark probes, term frequency methods, LLM monitors, and an LLM agent with interpretability affordances. We focus on OOD performance, since ID results often reflect dataset-specific shortcuts. We find that no method dominates: narrow classifiers, an LLM agent, and LLM monitors all win on different tasks. We provide a lower bound baseline for future work by ensembling all methods with a select-on-ID and score-on-OOD protocol; this ensemble beats the human baseline on 6 / 7 tasks. We believe that our testbed gives future CoT analysis methods a non-saturated hill to climb.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Chain of thought interpretability, Methods (probing, steering, causal interventions), Benchmarking Interpretability, Interpretability for AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LPi3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces nine tasks for analyzing reasoning traces in settings where surface reading of the chain of thought is insufficient. Seven binary tasks receive the main evaluation, each with in-distribution and out-of-distribution splits, while two exploratory tasks appear in the appendix. The benchmark compares text classifiers, activation probes, LLM monitors, a tool-using strategy agent, and human annotators. No method family dominates, and ID-selected methods often perform only modestly above chance OOD.

Critical strengths
  1. The paper targets a consequential gap in current safety evaluation: inferring model behavior when the explicit reasoning trace omits the relevant causal signal.

  2. Paired ID/OOD splits and selection on ID followed by scoring on OOD reduce the risk of celebrating dataset-specific shortcuts.

  3. The comparison spans lexical, activation-based, LLM, agentic, and human methods. The finding that different methods win different tasks is informative and creates a useful baseline for future work.

Critical weaknesses
  1. Most tasks are built from Qwen3-32B rollouts, with Gemma used for one specialized task, and several labels deliberately select extreme resampling regimes. It is unclear whether the resulting difficulty and method ranking transfer to other reasoning models or realistic monitoring deployments.

  2. Human annotators receive ten examples and no model internals, while some agent variants use larger labeled pools, repeated LLM calls, source-model interventions, logits, or activations. The claim that model methods beat humans therefore mixes information access, compute, and expertise.

  3. Several ground-truth labels are defined by extensive resampling, which prevents evaluation of resampling itself even though it is a natural and likely strong method. Some tasks may be unsolvable from the provided input, and the main comparisons often show point estimates without uncertainty intervals. This makes near-chance results hard to distinguish from insufficient information or split variance.

Questions for authors
  1. Do the task difficulty and method rankings replicate when the subject model, prompt family, and OOD construction are changed, ideally with repeated split generation and uncertainty estimates?

  2. Can the authors provide a resource-matched comparison that equalizes labeled examples and information access, together with an estimate of each task's attainable ceiling from the supplied input alone?

2 comments
AuthorsOpenReview ↗

Response to Reviewer LPi3

Thanks for your feedback!

W1, Q1 (one subject model, extreme regimes, and replication). On a second subject model, gpt-oss-120b, the classifier and the monitor come out in the same order on the three tasks where we ran both Section 4.4). We weren't able to replicate user preference. The labels do keep the two ends of a resampling statistic and discard the middle (Table 1 now says so). On reasoning termination we also scored the discarded prefixes: on them the attention probe's score has a rank correlation of 0.31 with the label's own statistic and of 0.56 with the share of continuations that end the reasoning within 100 tokens. On professor hint the probes carry little graded information (Appendix H.6). The one split variation we ran holds out each training domain in turn: on reasoning termination the attention probe's OOD AUROC stays between 0.95 and 0.98 (Appendix H). We selected the tasks for difficulty and do not claim that they resemble monitoring in deployment.

W2, Q2 (the human comparison mixes resources). We agree that the comparison between human annotators and our methods is unfair since annotators had fewer resources. We have withdrawn the corresponding claim. In the new matched comparison (four tasks, the annotators' own items), the annotators, a GPT-5.4 monitor given their instructions, and the classifiers each get ten labeled examples. The monitor and the TF-IDF classifier see text only, as the annotators did; the probes also read activations. Of eight paired comparisons (annotators against the monitor and against a ten-label probe) two are supported, both in the annotators' favor, and no method is supported above the annotators (Appendix E.4, Figure 13). The agent is not in this comparison, because it sees 40 examples and two of its tools query the subject model.

W3 (resampling excluded, solvability, and lack of intervals).

  • Resampling generates the labels, so it cannot be a method here: it is now the reference. One more draw of it predicts the label better than the selected classifier on five of the six sets where it can be computed (Section 4.3).
  • While we cannot measure the best score possible just from the inputs, we know that it is at least what queries to the subject model reach without any labels. On the three control tasks, those queries already beat the selected classifiers (0.91 against 0.66 on user preference, see Section 4.2).
  • Every score and difference now has a 95% interval.

Thanks for your response. I think part of my concerns are addressed. Overall, I think the paper is a borderline one for ICLR and I tend to maintain my original assessment.

Reviewer_6ACCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces nine CoT analysis tasks designed to evaluate settings where reasoning traces may not fully reveal the factors underlying model behavior. Labels are primarily constructed through repeated sampling and controlled interventions. The authors compare LLM monitors, activation probes, tool-augmented agents, and human annotators, emphasizing OOD evaluation. Results show that the tasks are challenging and no single method consistently dominates across them.

Critical strengths

S1. Novel and meaningful benchmark. The paper addresses an important limitation of CoT monitoring by introducing diverse tasks with behavior-based labels. The efforts to control shortcuts, such as CoT length and answer correctness, are particularly valuable.

S2. Comprehensive empirical evaluation. The comparison across LLM monitors, activation probes, and tool-augmented agents provides useful insights into their respective strengths and limitations. The emphasis on OOD performance and the release of evaluation artifacts further strengthen the contribution.

Critical weaknesses

W1. Limited source-model diversity. Almost all reasoning traces are generated by Qwen3-32B, with Gemma used only for self-deletion. Although the paper evaluates cross-domain generalization, it remains unclear whether the benchmark's difficulty and method rankings generalize across reasoning models.

W2. Unclear performance ceilings. For tasks whose labels depend on cross-rollout statistics, it remains unclear how much predictive information is available from individual reasoning traces. Additional analysis of the achievable performance ceilings would help distinguish limitations of current monitoring methods from inherent information constraints.

W3. Inconsistent OOD evaluation settings. The ID/OOD protocols differ across tasks. Some use different training domains with a shared test set, while others change the test domain. The professor-hint task also uses different labeling thresholds across domains. These differences complicate the interpretation of OOD performance.

W4. Evaluation reliability and method selection. Some test sets are relatively small, and multiple rollouts originate from the same prompts. More systematic uncertainty estimates would strengthen the comparisons. Additionally, the select-on-ID protocol requires clarification because Appendix B.1 states that agent methods lack corresponding ID scores.

Questions for authors

Q1. Have the authors evaluated whether similar task difficulties and method rankings hold for other reasoning models? Even a small-scale cross-model evaluation would strengthen the generality of the conclusions.

Q2. How can the authors distinguish monitoring failures from inherent information limitations in tasks whose labels depend on cross-rollout statistics? Are there analyses estimating the achievable performance from individual traces?

Q3. To what extent do differences in ID/OOD construction and labeling thresholds contribute to the observed performance degradation?

Q4. How were agent methods selected under the select-on-ID protocol if they lack ID scores? Could the authors also clarify the differences between the selected results reported in Figure 2 and Table 1?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6ACC

Thank you!

W1, Q1 (one subject model). We've rerun four tasks on a second subject model, gpt-oss-120b (Section 4.4, Figure 4). On the three tasks where both classifiers and the monitors were run, their order is the same on both models: the classifier is ahead on reasoning termination, and the monitor on professor hint and atypical answers. We couldn't replicate user preference on gpt-oss-120b since only 2 of 1,000 pairs met the rule.

W2, Q2 (ceilings). We now use no ceilings from the trace alone. For each task we now compare the methods' scores with a reference that may use what methods may not, such as one more draw of the resampling that defines the label (Figures 2 and 3, Section 4.3).

W3, Q3 (inconsistent OOD settings). The shared test set is gone: reasoning termination and self-deletion have separate ID test sets. The differing professor-hint thresholds were an error in our text, and the released data uses the same thresholds. Two shortcut controls you credited did not hold: CoT length predicted the label on user preference, professor hint and atypical answers (their validation and test sets are now balanced on it), and whether the professor's hint is correct still predicts the label (AUROC 0.71), which we disclose. Three label rules still differ slightly between ID and OOD sets (Table 1). Training inside the OOD domain, with the test question held out, raises the selected classifier's AUROC by at most 0.05 on the seven tasks (0.13 on the arm-matched set), so low OOD scores are not mainly a failure to transfer (Section 4.3).

W4, Q4 (reliability and selecting on ID). Every score now has a 95% interval over source questions (Section 3.4), and we have expanded or rebuilt five OOD test sets. The reviewed Figure 2 showed the ID-selected ensemble, while the bold entries of the reviewed Table 1 marked the best OOD score among the class bests; we have withdrawn both. We now run agents on OOD sets only and none is selected: we report all five variants, and no tool gives the agent a supported gain (Section 4.5).

Thank you for the detailed rebuttal and substantial revisions. The additional cross-model experiments, revised evaluation protocols, and statistical analyses address several of my concerns.

However, I still have reservations about the benchmark's generality and scope. In particular, the revised results show that three tasks do not meaningfully test CoT reading, while the main arm-matched finding remains limited to one source model and a specific intervention setting.

Overall, the revisions improve the paper's rigor, but I remain concerned about the breadth of its contributions. I will therefore maintain my original rating.

Reviewer_gs34OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that reading the chain of thought (CoT) is an incomplete safety tool and builds a testbed of nine CoT-analysis tasks. Seven tasks are binary, with in-distribution (ID) and out-of-distribution (OOD) splits. The authors compare probes, prompt-optimized LLM monitors, an LLM agent with interpretability tools, and human raters. They claim the tasks are hard, no method class dominates, the tools rarely help the agent, and an ensemble chosen on ID data beats humans on most tasks. Code and data are released.

Critical strengths
  1. The problem is important and the design is careful. Every task has paired ID and OOD splits, and Appendix C documents confounder controls. Such a released testbed would be useful, and this is its main value.
  2. The paper evaluates probes, monitors, an agent and humans under one protocol and reports useful negative results. Most agent tools do not beat the tool-free agent in Table 2, and the SAE as an agent tool does much worse than an SAE probe on the same features. These results would matter if the numbers were reliable (see Weakness 1).
Critical weaknesses
  1. The reported results contradict each other, so the headline claims cannot be checked. The caption of Figure 2 calls its bars the per-task best of all methods, but the text says Figure 2 is the select-on-ID ensemble. On user preference the figure shows 0.25, while Section 4.1 reports a per-task best of 0.59, and the caption still calls this win "decisive". So it is unclear what "6 of 7 tasks above humans" refers to. Agents are said to be run only on OOD, yet agent variants are selected by ID. The SAE probe is said not to be run on Gemma self-deletion, but it is reported and counted as the winner there. Monitor scores differ between Table 1 and Table 3, and the human scores differ between text and tables. Winners among category bests are picked by OOD score, so the winner counts still involve test-set selection.
  2. No uncertainty is reported, and the effective test sets are small. Monitors and agents are scored on 100 test examples, and OOD items are clustered within a small number of source questions. By my rough estimate the standard error is as large as many gaps the paper relies on. For two tasks ID and OOD share the same test set, so selecting on ID means selecting on test items. The "no method class dominates" and "above humans" findings are therefore not established.
  3. It is unclear whether the labels measure what the task names claim. For the two intervention tasks, a question-level switch rate is assigned to single rollouts, and the professor hint always points to the correct answer. The task may partly measure question-level uncertainty. The authors find this shortcut for user preference, where an entropy-only baseline reaches OOD AUROC 0.83, but do not control for it. The claim that reading fails also rests on non-expert raters with few reference examples, and on monitors whose near-zero scores on reasoning termination suggest a broken threshold rather than missing signal.
  4. It is unclear how well the testbed can rank future methods. The authors note that no method is clearly above chance on two tasks, which may reflect a real ceiling. If some tasks may be unsolvable and the gaps on the rest are within noise, the testbed may not yet separate good methods from weak ones.
Questions for authors
  1. Which numbers are correct for the ensemble in Figure 2, and how were agents selected by ID if they were only run on OOD?
  2. Can you report bootstrap confidence intervals for the main tables and the human baseline?
  3. How does a question-level uncertainty baseline do on the OOD sets for professor hint and user preference, and what is monitor AUROC on reasoning termination?
  4. Is there evidence, such as an oracle, that follow-up response and atypical answers are solvable from the given inputs?
1 comment
AuthorsOpenReview ↗

Response to Reviewer gs34

Thank you, your soundness concerns were apt and legitimate.

W1, Q1 (contradictory results). We no longer declare winners or count wins, and do not claim that "no method class dominates." We've withdrawn Figure 2. Agents had been run on ID test sets, and "selection by ID" chose between two variants only. Now agents run on OOD sets only and none is selected. The SAE probe was run on Gemma (the text said otherwise). Appendix K goes through each point.

W2, Q2 (uncertainty and small test sets). Every score now has a 95% cluster-bootstrap interval over source questions (Section 3.4). On the large sets, monitors are scored on 200 items and agents on 100. The two tasks that shared a test set now have separate ID test sets. We expanded four OOD sets: reasoning termination from 32 to 160 prompts (128 of them LLM-written), professor hint from 20 to 504 questions, atypical answers from 22 to 176, and CoT length from 56 to 148 prompts. We have withdrawn annotator scores on three tasks (their items covered 2 to 4 source questions), and the other four tasks have intervals.

W3, Q3 (what the labels measure). You were right about the two intervention tasks: the label belongs to the pair of question and suggestion, not to the rollout. The control rate of the suggested answer recovers the label (AUC 1.00 and 0.94). Forced-answer entropy (the baseline you asked for) scores 0.91 on user preference and 0.74 on professor hint out of distribution, against 0.66 and 0.58 for the selected classifiers. These tasks are now controls, and a new set asks the question on the rollout level. The hint now points to either option, but whether it's right or not still predicts the label (AUC 0.71), which is a shortcut that we disclose. On reasoning termination you were also right: three of the GPT-5.4 monitor's four templates call every item positive. The fourth, which validation selects, has AUC 0.60, and a newer monitor (GPT-6-astra) reaches 0.94. The claim that reading falls short now rests on two monitor models and an agent, not on the annotators.

W4, Q4 (ranking future methods and solvability). Table 3 now gives the smallest AUROC difference each set detects with 80% power: between two classifiers scored on every item, 0.02 to 0.08 on four sets and 0.11 to 0.28 on the others, and 0.12 to 0.28 between a classifier and a monitor. Section 6 assigns each set a use: three can rank methods, two detect only large effects, three are controls. Atypical answers is solvable, but not from the reasoning: the monitor scores 0.82 with the CoT and 0.81 without, so we turned it into a control. On follow-up response the selected classifier and monitor are not distinguishable from chance. Asking the model the follow-up question itself reaches 0.92, but that question defines the label, so it is a reference, not a method. The evidence that the inputs carry the signal is the best of four classifiers trained inside the OOD domain, at 0.75.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers. The reviews identified real errors, and the revised PDF corrects them.

Changes: every result on the seven binary tasks was recomputed from per-example predictions under one protocol (Section 3.4): selection on validation data, a 95% interval on every score, and paired comparisons, called supported if they hold after Holm adjustment. Five OOD test sets were enlarged or rebuilt, four tasks were repeated on a second subject model (gpt-oss-120b), and annotators were compared with methods given the same number of labeled examples. We have withdrawn the claims that methods beat human annotators on six of seven tasks.

After the revision, the paper now shows that:

  1. Reading the CoT falls short on one new set, built from user preference on one subject model and one phrasing of the user's view. Its two classes share question and final answer and differ only in whether a user's view was in the prompt; CoTs that mention the view are removed. A GPT-5.4 monitor is not distinguishable from chance there (AUROC 0.53) and a newer monitor (GPT-6-astra) is marginally above it (0.55 to 0.56), while the subject model's own log-probability of the CoT (surprisal) scores 0.63, a supported margin over the GPT-5.4 monitor (Section 4.1). With mentions kept, the monitor scores 0.70. On reasoning termination a GPT-5.4 monitor trails a probe (0.60 against 0.96), and a newer monitor does not (0.94).
  2. Three tasks (user preference, professor hint, atypical answers) do not test reading. Their label is determined by the question and the final answer, and a monitor scores about the same without the CoT. We keep them as controls (Section 4.2).
  3. One more draw of the resampling that defines a label ("one further sample") predicts the label better than the selected classifier on five of the six sets where it can be computed; on user preference its lead is within noise. The comparison favors the sample (Section 4.3), so we report it as a reference, not a method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.