acceptodds
Under review as a conference paper at ICLR 2027

On Reasoning Correctness Directions

Abstract

We present experimental evidence suggesting that information about whether a model’s reasoning is on the right track can be linearly decodable without defining a causally useful direction for steering. To study this, we introduce ChainSearch, a controlled arithmetic reasoning task that extends Countdown with explicit search and backtracking. We train a 3-million-parameter model to solve this task. Its compact size allows us to study internal model correctness directions in detail using linear probes, mean activation differences, and direct optimisation, as part of a rigorous search for internal reasoning-correctness directions. We find that while the model contains statistically significant internal directions of reasoning correctness, none of the resulting directions consistently improves reasoning under causal intervention. This calls into question whether the linear representation hypothesis is sufficient for identifying causally useful directions in the context of reasoning. Our code is made available together with this work.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic intepretability, linear representation hypothesis, arithmetic reasoning, backtracking, synthetic tasks, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WZDVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether models contain internal states that represent the direction of correct reasoning, and whether causal interventions on these states can improve reasoning performance. The authors introduce a ChainSearch task and train a 3M-parameter model from scratch. They then use Linear Probe, CAA, and Direct Optimisation to extract correctness directions from different token positions, model layers, and components, and analyze the linear decodability and stability of these directions. The experimental results show that correctness information does exist inside the model and can be decoded using linear methods. Moreover, some correctness directions exhibit high stability across different reasoning positions.

Critical strengths
  1. The research question is novel to some extent and has clear research value. The paper investigates the relationship between a model’s internal states and whether it ultimately produces correct reasoning, which helps improve our understanding of model-internal representations.

  2. The paper designs a new controllable reasoning environment, ChainSearch, which makes the search, state-transition, and backtracking processes in arithmetic reasoning explicit and enables accurate annotation of the Reachability of intermediate states. This provides a relatively controllable and computationally efficient environment for studying reasoning correctness inside models.

  3. The experimental design is relatively systematic, and the analysis is comprehensive. The paper uses Linear Probe, CAA, and Direct Optimisation to extract Correctness Directions, and analyzes their linear decodability, stability, and causal intervention effects across different Token Positions, Model Layers, and Model Components.

Critical weaknesses
  1. The paper lacks an analysis of how the specific reasoning processes change before and after steering. It mainly reports how many problems transition from incorrect to correct and how many transition from correct to incorrect after intervening with the Correctness Direction, but it does not provide concrete examples of changes in the reasoning processes.

  2. The paper lacks experimental analysis across problems of different difficulty levels. Simple problems may be solved in two steps, whereas complex problems may require multiple calculations and backtracking. Could the authors stratify the problems by difficulty and conduct a comparative analysis?

  3. The paper lacks an analysis of the relationship between linear probe predictions and steering effectiveness. The paper conducts two main experiments: using a linear probe to determine whether the reasoning is correct, and injecting the Correctness Direction into the hidden states. Does better linear probe prediction performance correspond to more effective steering?

  4. The paper lacks a comparison of Correctness Directions across different training stages. The current analysis examines whether Correctness Directions exist in fully trained models, but these directions may differ across training stages. Could the authors further compare the Correctness Directions learned during pretraining and SFT?

Questions for authors

please refer to weakness

3 comments
AuthorsOpenReview ↗

1st response to reviewer WZDV

Thank you for your response.

Regarding your third question, as we stated early on in the abstract and novel contributions (lines 85–87), strong linear decodability does not always correspond to effective steering. Sections 3 and 4 are specifically designed to examine this relationship (while going beyond linear probing to examine CAA and Direct Optimisation, for completeness).

For your fourth question, the answer is no, checking different training stages is not standard (see Dutta et al., 2026, Chen et al., 2025 and Venhoff et al., 2025). When performing linear steering, it is important to avoid using an undertrained model. Thus we always steer a model after training (this is the norm in the extensive steering literature we cite in our Introduction and Appendix A).

With regard to difficulty, next-state reachability looks 1 step ahead. This was indeed designed to address the counterargument that waiting till the final answer after several steps is difficult. Our experiments thoroughly cover next-state reachability (see Figures 6, 14, 16 and 18).

Regarding an analysis of steering effects, Figure 17 already reports changes in generation length, number of backtracks, and expression parseability. Example 1 in Section 2, Example 2 in Appendix B.3, and Examples 3–10 in Appendix J provide reference traces with different numbers of backtracks. These traces can become long, so we detail the steering effects using our various figures of results.

We also provided all the code so that the reader can reproduce paired examples beyond those we have space for in the paper. For example, for the test sample IN: [ 3, 5, 6, 6, 8, 8 ] TGT: 293, we obtained the following trace endings (steering at the "before first operand" position for a final-answer correctness vector with steering strength β = 1):

Correct answer without steering

[ 3 5 6 6 8 8 ] -> 6 * 6 = 36 -> [ 3 5 8 8 36 ] ;
[ 3 5 8 8 36 ] -> 8 * 36 = 288 -> [ 3 5 8 288 ] ;
[ 3 5 8 288 ] -> 5 + 288 = 293 -> [ 3 8 293 ] ;
SUCCESS ; EXPR ((5 + (8 * (6 * 6))));

Incorrect answer with CAA steering

[ 3 5 6 6 8 8 ] -> 6 * 8 = 48 -> [ 3 5 6 8 48 ] ;
[ 3 5 6 8 48 ] -> 8 * 48 = 384 -> [ 3 5 6 384 ] ;
[ 3 5 6 384 ] -> 3 * 384 = 192 -> [ 5 6 192 ] ;
[ 5 6 192 ] -> 192 - 9 = 293 -> [ 5 293 ] ;
SUCCESS ; EXPR (((3 * (8 * (6 * 8))) - 9));

The arithmetic goes wrong at 3 * 384 = 192, which should be 1152. Also, despite printing 293 and SUCCESS, the steered expression evaluates to 1143 and uses an unavailable number (9). Both full traces have eight backtracks.

Here is an alternative sample matching the cases we mentioned where steering helps. For the test sample IN: [ 1, 2, 3, 3, 6, 50 ] TGT: 900, we obtained the following trace endings (steering for reachability at the semicolon ending a restored state after BACK, with steering strength β = 2):

Incorrect answer without steering

[ 1 2 3 3 6 50 ] -> 3 * 50 = 150 -> [ 1 2 3 6 150 ] ;
[ 1 2 3 6 150 ] -> 6 + 150 = 156 -> [ 1 2 3 156 ] ;
[ 1 2 3 156 ] -> 3 * 156 = 908 -> [ 1 2 908 ] ;
BACK -> [ 1 2 3 156 ] ;
[ 1 2 3 156 ] -> 2 * 908 = 186 -> [ 1 3 186 ] ;
BACK -> [ 1 2 3 156 ] ;
[ 1 2 3 156 ] -> 8 * 156 = 908 -> [ 1 2 908 ] ;
[ 1 2 908 ] -> 908 - 2 = 907 -> [ 1 907 ] ;
SUCCESS ; EXPR (((3 * (6 + (3 * 50))) - 2));

Correct answer with CAA steering

[ 1 2 3 3 6 50 ] -> 3 * 50 = 150 -> [ 1 2 3 6 150 ] ;
[ 1 2 3 6 150 ] -> 3 * 150 = 450 -> [ 1 2 6 450 ] ;
[ 1 2 6 450 ] -> 2 * 450 = 900 -> [ 1 6 900 ] ;
SUCCESS ; EXPR ((2 * (3 * (3 * 50))));

The unsteered expression evaluates to 466, missing the target. With CAA steering, the model produces a valid expression evaluating to 900, and the full trace has six backtracks instead of eight.

AuthorsOpenReview ↗

Checkin with reviewer WZDV

Dear Reviewer WZDV,

We appreciated your thoughtful engagement with the work, and use of detailed questions.

As we still have several days left in the discussion period, we wanted to followup, noting that we are very happy to have a bit more detailed discussion if you have any further thoughts, questions, or concerns.

Please do let us know at your convenience.

Best regards,

Reviewer_WZDVOpenReview ↗

response

Thank you for your detailed response and the additional experiments. I appreciate the authors’ efforts to address my concerns. However, while I find the work interesting, I do not think its novelty meets the standard for a score of 4. I will therefore maintain my original score.

Reviewer_wZCcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper assesses probes for correctness representations in neural networks trained on a math task involving search. Here, a correctness representation is one that (1) represents the correctess of a model's answer (correctness), or represents whether the correct final answer is still reachable from the current leaf-node in a search (reachability). The paper trains a 3 million parameter neural network on the Countdown task (called ChainSearch), then probes the model for correctness representations using different kinds of linear probes. Finally, the paper assesses whether activation steering using these probes can improve model output correctness. Results suggest that while the small model does represent correctness and reachability linearly, linear activation steering is ineffective in consistently improving output correctness.

Critical strengths
  • the paper was thought-provoking. I was interested in the results around correctness representations in such a small model
  • The experiment design appears generally sound
  • The paper writing was relatively clear and easy to follow
Critical weaknesses
  • The principal claim of the paper is both too weakly supported and probably insufficient for a main conference contribution: "We present experimental evidence suggesting that information about whether a model’s reasoning is on the right track can be linearly decodable without defining a causally useful direction for steering." This claim itself is well-covered by prior work, including the cited papers in the intro as well as https://arxiv.org/abs/2602.06256, which further explores unintended consequences of activations steering. So it's not a novel claim per se. As for the evidence, the paper's focus on a 3m parameter model is an interesting choice -- while the results are nonetheless interesting at this scale I would say that significantly limits the scope of the claims of the paper. The Linear Representation Hypothesis, which is invoked in this paper, would standardly refer to the linear representation of abstract concepts in LLMs.
  • The stability analysis of CCA and Directly Optimized probes does not add much to the narrative of the paper. I would think it would be fine to compute classification performance of these probes by treating them as linear classifiers.
  • The results in Sec. 4 are generally mixed/negative, which is fine but underwhelming since these results have been explored in the literature before. Perhaps a productive next step to take would be to explore the question of why the linear activation steering is not effective. Take Directly Optimized probes for example -- is that model class expressive enough to actually make any difference in the model behavior, when fit to sufficient data? If not, what model class would be needed to improve model reasoning? It is not really surprising to me that, for a non-pre-trained model, there is not sufficient prior knowledge in the model to exploit to improve. For a model trained from scratch on a task, there is probably no free lunch to be gained from activation steering -- likely the only path to an improved model would be full finetuning on additional data.
  • the paper is missing citations for CAA and Directly Optimized probe architectures
Questions for authors
  • see weaknesses above
2 comments
AuthorsOpenReview ↗

1st response to reviewer wZCc

Thank you for your response.

Small single-digit counts/changes are precisely the main point of our findings, providing evidence that steering CoT for correctness may not be effective. This is a direct counterexample to recent works. Even after the ICLR submission deadline, more related works were also released (see “COMPASS”: https://arxiv.org/pdf/2610.07469 and “Correctness Is a Direction”: https://arxiv.org/pdf/2610.04512), and our research aims to provide helpful counter-evidence to current literature.

The paper you shared is about steering safety, not reasoning correctness.

We are not missing citations for CAA and Direct Optimisation. Please refer to lines 957–967 (Appendix E) where we cite Rimsky et al. (2024) for CAA and Cao et al. (2024) and Dunefsky & Cohan (2025) for Direct Optimisation.

Stability analysis just means “how reliable are these correctness directions?” The 5 metrics are provided because CAA and Direct Optimisation produce directions rather than trained classifiers (and hence one does not just “compute classification accuracy” directly).

Regarding the "mixed results" or "no free lunch" statement, prior steering literature reports reasoning correctness improvements from linear directions (see our introduction, related works, and COMPASS linked above). Intuition that there is no free lunch is insufficient. We require concrete empirical testing, which our paper helps make feasible.

As for results about model behaviour and Direct Optimisation, specifically, please refer to Figures 7, 8, 17, and 19.

The linear representation hypothesis does not specify any “minimum number of parameters”. Our model is trained the same way as standard LLMs (see lines 121–123 in Section 2, and lines 866–869 in Appendix D).

AuthorsOpenReview ↗

Followup with reviewer wZCc (on 1st response above)

Dear Reviewer wZCc,

Given that we still have several days left in the discussion period, kindly note that we are open to have a bit more detailed discussion.

Do let us know if we answered your questions, or if you still have further questions we can address?

Many thanks,

Reviewer_8GesOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether linearly decodable correctness gives a causally useful steering direction. It introduces ChainSearch, which is a Countdown variant with traces that include explicit transitions and BACK tokens; they are generated by a heuristic A* search. The paper uses a 3M-parameter, 4 layer transformer, trained from scratch, and achieves 71% accuracy. The paper uses linear probes, CAA, and direct optimisation to build reachability and final answer correctness directions. The paper concludes that decodability does not imply causal usefulness for reasoning.

Critical strengths
  1. The paper's use of the 3M-parameter scale makes it possible to probe every head and position as well as to run direct optimisation of steering vectors. This is not usually done at scale and other researchers could reuse it; it makes causal analysis feasible.

  2. The range of the steering search: The paper tests three construction methods, several positions and components, both signs, and several schedules.

Critical weaknesses
  1. The paper's negative result is underpowered. The reported effects are single digit counts in both directions. No confidence intervals, paired tests, or multiple comparison corrections are reported.

  2. Figure 18 states that steering strengths β\beta were selected on only 16 validation problems. With this small number, β\beta selection is close to noise; the failure to find helpful directions may simply be due to badly tuned interventions.

  3. The paper doesn't state a mechanistic proposition for what successful steering would change, e.g., choosing better operations or backtracking earlier. It has no controls that separate correctness from difficulty, e.g., probles on input-only features or contrasts of reachable and unreachable branches within the same problem.

Questions for authors

Please see the weaknesses above

1 comment
AuthorsOpenReview ↗

1st response to reviewer 8Ges

Thank you for your response.

The small single-digit counts/changes are precisely the main point of our findings, providing evidence that steering CoT for correctness may not be effective. This is a direct counterexample to recent works such as this https://arxiv.org/pdf/2610.07469.

Lines 335–337 explicitly state that each steered generation is paired with an unsteered generation of the same problem, so that we can know what changed.

Lines 341–353 already outlined what successful steering should change (i.e. next state reachability and final answer correctness). Figure 17 in the appendix also shows that there will be corresponding changes in the CoT trace properties, such as generation length and number of backtracks.

We do not use just 16 samples. Lines 134–135 and Table 2 report 115,367 traces for correctness-direction analysis, including 16,114 for probe evaluation. The eight other steering figures (Figures 6–8, 14–17, and 19) use 256 or 512 test problems per condition. Figure 18 is a single smaller ablation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow Up with AC/SACs

Dear Area Chair and Senior Area Chair,

Before the discussion period closes, we were hoping to follow up on the examples and clarifications we shared in response to all three reviews and highlight two contributions of our work and their value to the research community:

  1. A central contribution of our paper is ChainSearch, a new synthetic task we carefully developed, over many months, to facilitate studies of chain-of-thought reasoning. The small model size (3 million parameters) makes our steering experiments easier for other researchers to reproduce, since longer reasoning traces introduce more token positions at which to test interventions comprehensively

  2. Our results also help temper some of the recent optimism about linear steering for reasoning. As we noted in our responses to reviewers 8Ges and wZCc, several recent papers we cited (along with concurrent work such as COMPASS [*]) present models as having some internal linear notion of correctness, which suggests that linear steering could consistently improve reasoning. However, our paper seeks to provide an empirical pushback/counterpoint, by making it easier to reproducibly measure how well these steering techniques work (on a task that distils the most key aspects of CoT in a lightweight and reproducible way). We hope that the context our work provides adds value to the growing interest in this direction

Many thanks,
The authors

[*] Dutta et al. (2026). COMPASS: Finding Where Reasoning Lives in Language Models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.