acceptodds
Under review as a conference paper at ICLR 2027

Predicting Reasoning Errors Before They Occur: A Prompt-Side Activation Direction Anticipates Correctness in a Large Language Model

Abstract

Responses produced by large language models are eloquent in a way that makes right answers indistinguishable from wrong ones. It is critical to predict the correctness of the model’s response, which significantly helps in either terminating the model from answering or routing it to a bigger model instead of producing incorrect answers eloquently. In this study, we use activation patching to identify a candidate layer for the arithmetic reasoning task. We utilize the hidden state activations at the candidate layer as a direction to develop a difference-in-means measure. We test the performance of the approach across three models: Mistral-7 B, Llama-2-13B, and Qwen 27B. The measure predicts subsequent correctness across all three models (AUC .627, .673, and .786 from smallest to largest model), and the signal emerges around the middle of the network and strengthens toward the task-relevant layers. The learned direction transfers to new formulations and an independent benchmark without retraining (AUC .810 and .864 on the largest model), although its decision threshold shifts across datasets and requires recalibration. These results place correctness information earlier in the computation, before the model commits to an answer, suggesting that reliability can be assessed before generation rather than only diagnosed after it. The principal methodological contribution of this paper is to utilize the activations of causally identified candidate layers and develop a lightweight difference-in-means measure to predict the correctness of models’ responses before the generation of answer tokens.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, large language models, error prediction, selective prediction, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tn6WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This article predicts whether LLM will correctly answer arithmetic problems before generation. It uses activation patching to select layers related to the task, and then constructs the correctness direction based on the difference between the mean activations related to correct and incorrect answers. The experiment covered three model families on GSM8K, Socratic GSM8K, and SVAMP.

Critical strengths

1.Predicting errors before generation can support early abstention or routing.

2.After calibration, the readout is simple and inexpensive.

3.Testing three model families and transferring the direction without refitting can provide useful evidence beyond a single setting.

Critical weaknesses

1.Activation patching and mean-difference directions are established techniques. The paper combines them together, but does not convincingly demonstrate that patching is a better layer choice than simple selection strategies.

2.In most cases, the performance of this method is not as good as that of linear probes. The confidence intervals largely overlap with those of post-generation baselines, and different data splits were used for the Qwen probe comparison.

3.The Qwen sample construction describes 160 error examples and an equal number of correct examples, totaling 320, instead of the reported 480. The AUC of its GSM8K probe is also reported as 0.751 in Table 2 and 0.738 in the discussion.

Questions for authors

Consistent with weaknesses

Reviewer_rdnYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores whether we can predict the correctness of a math problem's answer based on internal representations of the prompt, prior to generating that answer. The approach begins with selecting a layer through activation patching on a synthetic arithmetic task. It then uses the activation of the final prompt token from this layer to calculate the mean difference between correct and incorrect samples, using that as the scoring direction. The authors test this method on Qwen, Llama, and Mistral, comparing it with linear probes and post-generation confidence measures, providing in-domain results on GSM8K and transfer results on Socratic GSM8K and SVAMP (§3, Table 2).

Critical strengths

The pre-generation scoring is straightforward and has some empirical backing. Once the prompt activation is obtained, it only requires one vector dot product to generate a score, eliminating the need to wait for reasoning or answer generation. As shown in Table 2, the single-layer AUROC for GSM8K exceeds chance for all three models, with Qwen achieving 0.783. This suggests that this simple readout can indeed capture signals related to whether the later response will succeed or fail.

Critical weaknesses

W1: The paper should clarify its new contribution more clearly. Marks and Tegmark’s The Geometry of Truth (COLM 2024) used patching-based localization followed by probing, including difference-in-means methods. Afzal et al.’s Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion (Findings of ACL 2025) predicts the success or failure of math responses from internal representations before generation. Although the specific tasks and readout methods differ, the paper still needs a clearer methodological comparison to explain the added value of its combination of layer selection and readout.

In Table 2, the proposed method and the linear probes use different layers and scoring methods. This comparison does not isolate the contribution of layer selection. The paper lacks a control that keeps the data, calibration/evaluation split, and difference-in-means scoring method fixed while varying only the layer selection strategy. It therefore remains unclear how much additional benefit patching-based layer selection provides over simpler rules, such as choosing a fixed late layer.

W2: The evidence for cross-distribution generalization is still insufficient. The paper includes Socratic GSM8K in its transfer evaluation (§3.8). However, in the official data, this version has the same problem statements and final answers as the original GSM8K, with changes mainly in the reference solution format. Thus, the use of this dataset version does not by itself establish transfer to “new formulations.” The paper should clarify what actually changes in the input, including whether the few-shot examples and their reference solutions change. It should also clarify whether the evaluation problems overlap with the calibration set.

In addition, AUROC values for Socratic GSM8K and SVAMP are reported only as point estimates. The evaluation sample sizes, numbers of incorrect responses, and confidence intervals for each model are not reported (Table 2). This makes the statistical precision of the transfer results difficult to assess and limits the support for the claim of cross-distribution generalization.

Questions for authors

[Clarification] Could you address W1 by explaining which evidence in the current results independently supports the additional benefit of patching-based layer selection over simple layer selection rules?

[Clarification] Could you address W2 by clarifying the actual input differences between Socratic GSM8K and the original GSM8K, including whether the few-shot examples and their reference solutions change? Please also specify the source of the evaluation problems and whether they overlap with the calibration set. For each model on Socratic GSM8K and SVAMP, please report the evaluation sample size, number of incorrect responses, and AUROC confidence interval.

Reviewer_XuZVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about predicting if an LLM will get an arithmetic question right before it starts writing its answer. First, they use activation patching on simple addition questions to find a layer that is for arithmetic. Then they use the difference of means between correct and incorrect responses at that layer and consider it as a direction to predict correctness. Then, they test this on three different models with three math benchmarks and compare against linear probes and measure confidence after generation. The single-layer method gets AUCs of 0.621, 0.673, and 0.783 on GSM8K, and works best on Qwen.

Critical strengths

They motivate with an interesting and timely idea of catching errors before the model starts answering. This is useful for when incorrect answers are being generated and need to be stopped, or for detecting instances where the model needs to be changed to a stronger one for getting a better answer.

They compare their method with linear probes and confidence measures and test it on different models, which gives some evidence of generalizability. Their distinction between predicting correctness and causing correctness is important because even if a direction can predict whether an answer is going to be correct, this does not necessarily mean that the direction is responsible for the model generating that answer.

Critical weaknesses

In the introduction, lines 48-50, cite the claims you are raising; you are saying prior works … but there is no citation for them. Again, in the introduction before the third paragraph, you just jump into what you are doing and do not specify any gap, just talk about prior works and then move to your work.

The claim in lines 63-64 is not correct because if you change your benchmark category to something else, like commonsense, code, … you need to refit it. You are only working on the mathematics benchmark category.

In Section 2, they did not clearly separate methods that predict before generation from those that check after.

In lines 76-78, there are a lot more papers that you need to cite in your related work, like [2, 3].

In lines 88-90, you are drawing on your novelty, which is not the case; there are lots of papers that extract activation at a specific layer and then apply it on the same layer, like [1, 4]. You need to cite [4]; it is relevant to your work.

In line 92, there is also another closely related work that is [1].

In line 107, you should also cite [4].

In lines 117- 118, you did not cite [5], and this already investigates using representation directions for both detecting and controlling model properties, so the claim that previous approaches use these directions mainly for control might be too broad.

In Section 3.3, the layer is picked using just a simple problem like adding two numbers, but some of the GSM8K problems need reasoning steps.

In Section 3.3, patching shows that a layer affects answer probability, but this does not by itself show where arithmetic is computed. The selected layers are close to the output, where changes can affect answer scores.

Some of the numbers don’t add up. In Section 3.1, they say they collected 160 incorrect Qwen responses and 160 correct ones, but then report 480 examples. That should be 320.

In Table 2, the Qwen linear probe gets 0.751 AUC, but Section 5 uses 0.738 and says the probe was evaluated on a different split. That makes it hard to compare them.

The experiments just report AUC. Qwen gets around 96% of GSM8K right, so even with 0.783 AUC, it can’t tell how many wrong answers will get caught or how many right ones will get stopped for no reason.

The abstract reports 0.627 and 0.786 for Mistral and Qwen, which are the average projection results, while the single-layer numbers are 0.621 and 0.783. Since the paper focuses on the single layer, this should be clearer.

As mentioned in lines 401-404, correctness information in hidden states, linear probes, and difference-of-means directions is done in previous works. So show what this combination adds, something like better prediction or a new finding about reasoning, to have more contribution besides simplicity and negligible cost, that is mentioned in lines 406-407.

On a small note, please cite the models in line 137.

[1] Orgad, Hadas, et al. "Llms know more than they show: On the intrinsic representation of llm hallucinations." International Conference on Learning Representations. Vol. 2025. 2025. [2] Ji, Ziwei, et al. "LLM internal states reveal hallucination risk faced with a query." Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. 2024. [3] Sun, Yucheng, Alessandro Stolfo, and Mrinmaya Sachan. "Probing for arithmetic errors in language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. [4] Sun, Yucheng, Alessandro Stolfo, and Mrinmaya Sachan. "Probing for arithmetic errors in language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. [5] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).

Questions for authors

Some questions are mentioned in the weaknesses; other than those: In Sections 2 and 5, what is the difference between your approach and earlier methods that use activations to predict errors? Can you run the same difference of means method at different layers to show that patching-based selection is useful?

In Section 3.3, how are you sure that multi-step questions also pick the same layer that a simple math problem picks?

Can you explain the question difficulty comparison mentioned in Section 3.2?

In Section 3.1, how did you get 480 Qwen examples from 160 incorrect answers and 160 correct ones?

Why are the Qwen probe results different between Table 2 and Section 5?

In Sections 3.8 and 4.2, can you go beyond just showing AUC and show how the method does with a fixed threshold? For example, can you show how many wrong answers get caught and how many right ones get stopped at Qwen’s natural error rate? Does the same threshold hold up on Socratic GSM8K and SVAMP?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewers (Part 1/2): Layer-Selection Control

Part (1/2) We thank all the reviewers for their detailed feedback, which significantly helps in improving the paper. The comments can be broadly divided into three categories: (i) Whether the patching-based layer selection adds anything over simpler layer-selection strategies? (ii) How useful the readout is at a fixed operating threshold rather than just in terms of AUC? (iii) A few inconsistencies in reported numbers. We conducted two additional experiments to address the first two, and address the third in the final section. All tables referenced below (Tables A1, A2 and B1) are in the Appendix of the revised paper, and Table 2 refers to the main results table of the paper.

**Layer selection control ** The primary aim of the study is to investigate whether the causally implicated layer also encodes the signal of the models’ eventual correctness. The aim of the paper is not to test whether the causally implicated layer is the optimal choice. Instead of searching for the best layer for each model, the paper asks whether we can use the signal available at the candidate layer to predict the model’s eventual correctness. To check how the causally identified layer compares with other choices, we employ a controlled comparison using the same data, splits, and difference-of-means measure, varying only the layer used for the readout. We compared the candidate layer against the final layer; layers at 50%, 75%, and 90% network depth; the best layer selected through 5-fold CV on the calibration set; and 1000 random averages of 5 layers. We use paired bootstrap with 10,000 resamples for the comparisons.

We find that in four of the nine comparisons the difference is statistically significant and and alternative layer has an AUC greater than patched layer-based AUC; while other five cases are statistically insignificant (Table A1). However, no fixed layer-selection rule works consistently across models and datasets. The layer selected through cross-validation varies by model: the best CV layer for Mistral is at around 50% depth, whereas for Qwen it is at around 75% depth. Similarly, the 50% depth layer performs almost identically to the patching layer for Mistral, is slightly better on Qwen GSM8K but much worse on Qwen Socratic (−0.087) and SVAMP (−0.046) and is consistently better for Llama (Table A2). A fixed relative-depth rule therefore does not transfer reliably across models and datasets, whereas patching gives a model-specific way of selecting the layer. The aim of the paper is to test whether a layer that is causally important for the task at hand (in the paper’s context, the mathematical problems) has a signal that can predict eventual correctness. We sincerely thank the reviewers for suggesting that we test whether the candidate layer is the optimal layer and the identified dissociation between causal localisation and linear decodability (in terms of the optimal layer) through the suggested experiment is interesting in its own right. In future work, we aim to test whether the causally identified layer plays a causal role in models’ correctness, compared with the best-performing layer (in certain cases). This would show whether the causally identified layer matters more for getting the answer right, even if the best-performing layer predicts better (in cases where it does). One difference to note is that the new Qwen experiments use a natural-rate sample of 1000 randomly selected problems (700/300 split), rather than the error-enriched sample of 480 used in Table 2 of the paper. We made this change to evaluate operating-point performance at the model’s actual error rate, since metrics such as precision, recall, and false-positive rate depend on the proportion of incorrect answers in the sample. The enriched sample (160 incorrect and 320 correct) was suitable for the original AUC and top-k experiments but does not reflect the model’s natural error distribution (approximately 4–5% errors; Table A1).
For Mistral, the candidate layer (L29) is essentially tied with the other layer-selection rules (Table A1 and ). The only clear exception is the final layer, which performs worse on SVAMP (-0.027, p=0.025; Table B1). For Llama, the candidate layer outperforms the final layer across all three datasets, but the 50% depth layer outperforms the candidate layer (+0.031 to +0.054; Table A2). For Qwen, the 75% depth layer performs better on GSM8K (+0.048, p=0.003; Table A1), while the other comparisons are largely tied. Overall, patching identifies a layer that is sufficient for the readout, rather than necessarily the optimal layer.

AuthorsOpenReview ↗

Response to Reviewers (Part 2/2): Fixed-Threshold Evaluation and Other Clarifications

Part (2/2) As in Part 1, all tables referenced here are from the Appendix of the revised paper.

Fixed-threshold evaluation

We evaluated the readout at a fixed threshold selected on the GSM8K calibration set and applied it without modification to the GSM8K test set, Socratic, and SVAMP. The threshold was set to flag 80% of the incorrect answers in the calibration set. The figures below (Table B1) are means over 50 stratified 70/30 calibration/test splits. Test recall falls below the 80% calibration target in all cases, as the threshold is applied without re-tuning.

For the fixed-threshold evaluation, we get the following results across the three models:

  • Mistral (natural error rate: 55% on GSM8K, 54% on Socratic, 34% on SVAMP):
    • GSM8K: catches 73% of incorrect answers while stopping 59% of correct answers.
    • Socratic: catches 78% of incorrect answers while stopping 56% of correct answers.
    • SVAMP: catches 53% of incorrect answers while stopping 28% of correct answers.
  • Qwen (natural error rate: 5% on GSM8K, 2% on Socratic, 6% on SVAMP):
    • GSM8K: catches 56% of incorrect answers while stopping 17% of correct answers (15 incorrect answers).
    • Socratic: catches 98% of incorrect answers while stopping 14% of correct answers (6 incorrect answers).
    • SVAMP: catches 76% of incorrect answers while stopping 14% of correct answers (17 incorrect answers).
  • Llama (natural error rate: 65% on GSM8K, 66% on Socratic, 45% on SVAMP):
    • GSM8K: catches 76% of incorrect answers while stopping 51% of correct answers.
    • Socratic: catches 76% of incorrect answers while stopping 48% of correct answers.
    • SVAMP: catches 56% of incorrect answers while stopping 45% of correct answers.

The threshold seems to transfer reasonably well across datasets, although the false-stop rates vary across models and datasets. This is particularly noticeable for Mistral and Llama, where a relatively large fraction of correct answers are also stopped. Qwen has much lower false-stop rates, but this needs to be interpreted in the context of its much lower natural error rate and the small number of incorrect examples, especially on Socratic and SVAMP. Overall, these results give a better picture of how the readout would behave at a fixed operating point than AUC alone.

Other Clarifications

We have also addressed the other miscellaneous queries as follows:

  • Qwen sample count: The sample contains 160 incorrect and 320 correct answers (480 total). The original text has a typographically and incorrectly stated 160/160.
  • Qwen probe AUC: The correct value is 0.751 on the 480-example enriched sample (the 0.838 in Table A1 is on the natural-rate sample).
  • Abstract: We now consistently report the single-layer results.
  • Socratic dataset: We agree that this should not be presented as an OOD evaluation. The questions are identical to GSM8K, use the same prompt, and approximately 30 of the 300 examples overlap with our calibration set. We now treat Socratic as a second GSM8K sample rather than a transfer benchmark. Removing the overlapping examples barely changes the AUC. SVAMP is now our only OOD benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.