acceptodds
Under review as a conference paper at ICLR 2027

Dissociating Internal Error Detection from Stated Verdicts in Small Language Models

Abstract

Small instruction-tuned models asked whether an arithmetic derivation is correct answer yes almost regardless of the derivation. We show that the evidence of the error is present in the model's own next-token probabilities and, at 1.5B and above, in the state its verdict is read from, and that the verdict uses little of it. On 1,280 seeded derivations with one planted wrong step, the model's own next-token surprisal separates corrupted from clean derivations with AUC 0.872 (Qwen2.5-0.5B-Instruct) and 0.893 (Qwen2.5-1.5B-Instruct) and finds the wrong step 75.2% and 79.2% of the time, while the Yes/No verdict on the same inputs reaches AUC 0.507 and 0.544. We split verification into three stages measured on the same problems: noticing the error when it is read, carrying the evidence forward, and reporting it in the answer. The 0.5B model loses the evidence within one step. In the 1.5B model, a linear probe on the final normalized state at the answer position, the vector the unembedding reads, detects corruption with AUC 0.803 regardless of the error's distance from the question, and the corruption direction is nearly orthogonal to the No-minus-Yes unembedding (cosine 0.069). On a subset of short derivations, removing that component lowers the verdict's AUC from 0.663 to 0.558 and amplifying it tenfold raises it to 0.787, while a random direction moves it by at most 0.034. The pattern persists at 3B and 7B, where surprisal reaches AUC 0.920 and 0.931 and the verdict 0.539 and 0.605; carry and the final-state probe vary with size and weight precision rather than growing with size. On ProcessBench, the digit surprisal of Qwen2.5-Math-1.5B-Instruct localizes the first error with an average F1 of 42.9, with no training and one threshold chosen on GSM8K, level with a 7B process reward model trained on step labels, while the same model asked for the index scores 0.7.

Primary area:
interpretability and explainable AI
Keywords:
verification, error detection, process reward models, interpretability, probing, activation steering, mathematical reasoning, small language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SKQeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs notice, carry, and report errors in multi-step arithmetic derivations. Using a controlled setup with one planted inconsistent step, it finds that next-token surprisal reliably detects the error, and in several models the hidden state at the answer position still encodes it. However, the one-word Yes/No verdict uses little of this evidence, revealing a detection–report dissociation. Consistent-twin and surface controls rule out simple number artifacts, and prompting step-by-step recomputation improves rejection. On ProcessBench, a training-free surprisal localizer from a 1.5B model is competitive with early process reward models, while verbal localization remains weak. The main bottleneck is often the readout/report stage, not internal error detection.

Critical strengths

1.The Notice-Carry-Report three-stage analysis framework proposed in this paper breaks down the language model error detection process into three separate stages: noticing, retaining, and reporting. This provides a clear layered research approach to explore why error detection fails and offers a useful methodological reference for studies in this area. 2. The paper uses paired samples and a consistent twin-control experimental design, strictly controlling confounding variables, allowing for accurate separation of the effects of 'arithmetic logic inconsistency' and 'rarity of numbers' on surprise in an artificial setting, with high internal validity in the controlled experiment.

Critical weaknesses

1.The research motivation and experimental scenario are mismatched, and the narrative loop is incomplete: The introduction starts from the research point of model self-verification failure in real scenarios and the field consensus that 'verification is easier than generation', but the core mechanism experiments are all based on highly controlled environments with artificially implanted single-step arithmetic errors, which are fundamentally different in type and distribution from naturally generated model errors, limiting external validity. The main scientific question of whether verification failure comes from perception failure or output reporting failure is not explicitly presented in the introduction, and the grand hypothesis raised at the start is neither verified nor addressed, causing a narrative disconnect and unclear research positioning. 2.The experimental design is biased, and the evidence for generalising the core conclusions is insufficient: Artificially constructed local numerical errors naturally align with the detection characteristics of next-token surprisal, so the positive results obtained in this scenario are not very convincing and cannot fully prove the model's general internal error perception ability. The most radical 'perception-report dissociation' mechanism conclusion only holds on artificial datasets; Chapter 8 only verifies the error localisation effect of surprisal and does not reproduce the core dissociation phenomenon in natural error scenarios, seriously lacking generalisation support. 3. The boundary between the model's inherent ability and the contribution of the reading method is unclear: The ProcessBench experiments in Chapter 8 fail to clearly separate the model's maths reasoning ability from the independent contribution of the surprisal reading method, making it unclear whether the performance comes from the surprisal method itself or is just another way of reading the internal error signal of the model, leaving the incremental contribution of the method undefined.

Questions for authors

Please add ablation control experiments to clarify the independent value of digit-token surprisal. On the same model and test set, provide a horizontal comparison of error localization performance across the following approaches: 1.Explicit verbal output of the error step number by the model 2.Average surprisal across all tokens 3.Average surprisal across digit tokens only 4.Other implicit uncertainty metrics (e.g., token entropy, local perplexity) This will demonstrate the gain and necessity of digit-token surprisal relative to other readout methods. Please explicitly define the positioning of the Section 8 experiments in the main text. Clarify that this section serves as an engineering application demonstration of the controlled mechanism findings, rather than a generalizability validation of the core perception-report dissociation phenomenon. Clearly delineate the scope of applicability of the conclusions to avoid readers misinterpreting the scope of contribution.

Reviewer_FCM5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies why small language models fail to verify arithmetic derivations and tests two hypotheses: one is that the error is never computed, the other is that the model computes it but never use it. To answer this, the authors build a controlled harness of 1,280 seeded derivations. In each corrupted copy, exactly one step has a wrong result, and every later step is recomputed from the wrong value. The authors define and measure three stages on the same forward pass. "Notice" measures the model's own surprisal on the written result at each step. "Carry" is a probe on later step positions for whether an error has occurred so far. "Report" compares a linear probe on the final normalized state at the answer position with the model's own Yes/No log-odds. The results show interestingly that, across seven models from three families (Qwen2.5 0.5B–7B, Llama-3.2 1B/3B, Gemma-2-2B), surprisal separates corrupted from clean derivations pretty well (AUC 0.86–0.93), while the one-word verdict (Yes/No) stays near chance (0.50-0.61). In four of these models, a probe on the answer state recovers much of the evidence (0.77–0.87), which the authors read as the evidence being present but barely used by the readout. A consistent-twin control shows that the surprisal gap comes from the line being inconsistent, not from the written numbers themselves, since the numbers from the twin samples are the same. Lastly, a training-free localizer built on digit-token surprisal reaches an average F1 of 42.9 on ProcessBench, comparable to several earlier process reward models.

Critical strengths
  1. The experimental harness is well controlled. Because the position and size of the error are fixed by construction, the internal measurements can be easily read as an effect of where the error is. The consistent twins are a good control for token-level confounds as well, and the evaluation is careful.
  2. The main dissociation between surprisal and the one-word verdict holds across model sizes and families, and across verdict biases in both directions: Qwen and Gemma accept almost everything, while Llama rejects almost everything. So the weak verdict is not simply a yes-bias, and the main conclusion of dissociation holds regardless of each family's default preference.
  3. The ProcessBench result is a useful practical finding. A label-free surprisal signal from a 1.5B model gives a cheap baseline for locating errors in real solutions, and the observation that restricting it to digit-bearing tokens roughly doubles F1 is worth knowing.
Critical weaknesses
  1. The paper is not well organized and written, which makes it hard to follow. The abstract and introduction are dense with numbers, and key terms that are used before defined. Also, the main text has no illustrative example or setup figure, and caveats are scattered throughout the results. I understand that researchers tend to use AI to draft or polish their work to save time and energy. However, the paper would benefit from a clearer narrative and a discussion of possible explanations for the findings, if it is reviewed more carefully by authors.
  2. I am concerned that calling surprisal as "Notice" overstates what it shows. Under teacher forcing, a high surprisal on a wrong result mostly reflects that the model can compute that step itself, so this is closer to generation ability than to error detection. The contrast between surprisal and the verdict therefore largely reinforces the already known generation–verification gap, which the paper itself cites. This leads to my concern about novelty of the work.
  3. The probe and the verdict are not directly comparable, and the verdict baseline is close to degenerate. The probe is a supervised classifier trained on in-distribution data, while the verdict is a zero-shot readout that is nearly constant (the models accept or reject almost everything). Metrics like "the verdict realizes 15% of the probe's discrimination" are hard to interpret. The fact that chain of thought raises the verdict AUC to 0.78 also suggests that much of the gap comes from how the verdict is elicited from small models, rather than from a bottleneck in the readout.
  4. The contribution is mainly descriptive and its novelty is limited. That internal states and probabilities carry more than the model's verbal output is already well established by prior work cited in the paper's "Related Work". The paper does not dig further into the reasons why the readout ignores the evidence. Also if authors are interested in performance of models of the same family but at various scales, other larger models like Gemma-2-9B or Llama-3.1-8B should be added into the experiment to make the claim more convincing.
  5. For the ProcessBench comparison, the statistic and threshold are selected on GSM8K, but the headline average includes GSM8K, so part of the headline number is already in-sample.
Questions for authors
  1. For steps where the model's own greedy generation of the result is wrong, is the surprisal gap still present? If not, "Notice" probably mostly reflects generation ability rather than error detection. This need further clearance.
  2. How does the final-state probe perform when trained on only a few labeled examples (e.g., 8 or 32)? This would help showing whether the evidence is "present but unused" or mainly found through supervision process.
  3. In the chain-of-thought condition, are rejections driven by recomputed values that disagree with the written ones? If the model mainly rejects by solving the problem again in text, this supports the view that the gap comes from elicitation rather than from a bottleneck in the readout
Reviewer_rzS9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies why small language models cannot identify errors in the arithmetic derivation task. To study this, the authors examine three stages of verification in their analysis: notice, carry, and report. Specifically, they use next-token surprisal to identify steps where error occured, linear probes to examine whether error-related information persists in hidden states at subsequent steps, and explicit Yes/No judgments to assess reporting. Across seven models, they find that surprisal reliably identifies errors and that some models retain error-related information in their hidden states, even when explicit verification performs poorly. They also demonstrate that digit-token surprisal achieves competitive error-localization performance against several trained process reward models on ProcessBench.

Critical strengths
  • The paper presents a well-controlled experimental setup with planted arithmetic errors, allowing systematic analysis of error detection across different positions and derivation lengths.
  • The experiments are conducted across seven models from three families, with useful controls such as consistent twins and cross-validated probing for generalization of the findings.
  • The surprisal-based error localization results on ProcessBench are promising.
Critical weaknesses
  • Overall, the paper presents a careful empirical study of error detection in small language models, supported by reasonably sound experiments. However, the novelty of the findings appears limited, as the findings largely characterize previously observed phenomena without providing substantial new insight into the causes of verification failures. For instance, as discussed in Introduction, Hu and Levy (2023) already demonstrated that probability-based measurements can outperform prompted judgments, while Sun et al. (2025) and Zhang et al. (2025) show that correctness-related information can be recovered from hidden states. In short, the paper provides a systematic characterization of these phenomena, however, it is unclear what substantially new understanding of verification failures emerges from the analysis.
  • The paper would benefit from more deeper mechanistic understanding of the reason behind why the error-related information doesn't influence the final verdict or demonstrate how these findings could be used to improve verfification.
  • The primary comparison relies on one-word Yes/No judgments, which may underestimate the models' verification capabilities. The substantial improvement from explicit step-by-step recomputation raises questions about how broadly the reported verification failures generalize to other prompting strategies.
Questions for authors

Could the authors extend the consistent-twin controls to the carry probes at subsequent steps? Since corrupted and clean derivations follow different numerical trajectories, it is unclear whether the probes detect the earlier arithmetic inconsistency or differences in subsequent values.

For example, consider a clean derivation (8 + 23 = 3, followed by 31 + 37 = 68), a corrupted derivation (8 + 23 = 68, followed by 68 + 37 = 105), and its consistent twin (8 + 60 = 68, followed by 68 + 37 = 105). If the twin scores similarly to the clean derivation despite sharing the corrupted derivation's subsequent numerical values, this would strengthen the evidence that the probes capture information about the original arithmetic inconsistency rather than merely differences in numerical trajectories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.