acceptodds
Under review as a conference paper at ICLR 2027

Holding the Answer Fixed: What Hidden-State Probes and Interventions Establish About Correctness

Abstract

Hidden-state probes are often interpreted as measuring whether a model's answer is correct. But correctness is relational: the same answer can be right for one task and wrong for another, while standard evaluations usually allow both the task and the answer to vary. We study what probe success establishes when answer identity is held fixed. We construct paired task variants in which the answer is identical but its checker outcome flips. Across six 7B–32B code and tool-use models, newly fitted probes almost perfectly distinguish these fixed-answer task conditions. Yet this strong decodability does not imply that an existing correctness score is reusable: probes learned from freely generated answers transfer inconsistently to the same candidate states and can even reverse their ordering under the tested shift. We uncover a second separation within a single frozen score. Two probes correctly order 93% and 95% of fixed-answer pairs, while incorrect answers still outrank correct answers from other problems in about one third of cross-problem comparisons; generation-matched mazes exhibit the same gap. Input controls show that the surviving signal can arise from different task-dependent sources. Finally, late whole-state patches reproduce donor outputs whether those outputs are correct or incorrect. Together, these results show that decodability, score reuse, fixed-answer responsiveness, cross-problem ranking, and output restoration are distinct properties. Correctness probes should therefore be evaluated against the comparison required by their intended use, rather than by pooled discrimination alone.

Primary area:
interpretability and explainable AI
Keywords:
hidden-state probing, correctness, language models, representation analysis, score transfer, activation patching, verification, fixed-answer evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w76DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether hidden-state "correctness" probes truly measure a model's generalized understanding of truth, or ior if they merely rely on superficial input cues and answer formatting. To isolate what these probes measure, the authors evaluate how models handle paired task variants where the model's answer remains exactly the same, but slight changes to the task constraints cause the answer's correctness outcome to flip from correct to incorrect.

Core Obersvations/Contributions:

  • Poor Score Reuse: Probes trained on freely generated answers fail to generalize. When applied to identical, supplied candidates, they perform inconsistently and can even reverse the correct/incorrect ordering.
  • Weak Global Ranking: A probe can reliably sort correct vs. incorrect answers within the same specific problem (e.g., 93–95% accuracy) but fails to reliably rank correctness across different problems.
  • Reliance on Shallow Cues: High probe accuracy is frequently driven by simple input compatibilities (like raw spatial occupancy in mazes or prompt instructions in code) rather than an abstract representation of truth.
  • Output Restoration ≠\neq Correctness: Whole-state activation patching successfully copies a donor model's exact output sequence, regardless of whether that copied answer is actually valid or correct for the recipient's specific task.
Critical strengths

1. Foundational Critique of Probing The paper exposes a critical flaw in interpretability research by proving that high probe scores often reflect shallow task cues rather than a genuine understanding of "correctness." This forces the field to re-evaluate what probe successes actually mean.

2. Elegant Experimental Design The "exact-output matching" method isolates the "correctness" variable by keeping the model's answer identical while slightly changing the task rules to flip its correctness. Testing this across diverse domains (mazes, graphs, and large code models) makes the findings convincing and provides a robust template for future evaluations.

Critical weaknesses

The paper demonstrates that late-layer, whole-state activation patches merely force the model to reproduce the donor's exact output, regardless of its correctness. However, this result is somewhat expected: late-layer hidden states are heavily correlated with the final token logits, so pasting them naturally dictates the output sequence. This coarse, late-stage intervention does not definitively rule out the existence of sparse, localized, or earlier-layer features that might still encode a task-general correctness signal, slightly weakening the conclusiveness of the intervention experiments.

Questions for authors
  1. Regarding the whole-state activation patching experiments: intervening at late layers (H+LH+L) heavily dictates the final token logits, making the restoration of the donor's exact output (regardless of correctness) somewhat expected. Did you investigate whether patching earlier layers or specific localized components (e.g., individual attention heads) might reveal a sparser, generalized correctness signal that does not strictly force the specific output sequence?

  2. You demonstrate a stark failure in score reuse when moving from generated answers (G) to supplied candidates (V). Have you tested whether training a probe on a mixed distribution of both G and V states allows it to isolate a generalized correctness direction?

Reviewer_B5TtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies what correctness probes measure when the answer itself is fixed. The authors construct paired tasks where the same answer is correct under a set of requirements and incorrect under another, and evaluate whether activation probes can distinguish these cases. They show that newly trained probes achieve almost perfect AUROC on fixed-answer code and tool-use tasks, but probes trained on freely generated answers transfer significantly worse. They also show that probes can correctly rank nearly all correct and incorrect pairs within the same problem while achieving only 0.67–0.68 pooled AUROC across problems. Their baselines reveal that some of the strong probing results can be explained by relatively simple task-dependent features. They also run activation patching experiments on maze and Sudoku reasoning models, showing that patching the entire state enables reproduction of both the correct and incorrect outputs. They use these experiments to distinguish the ability to decode correctness, generalization, within problem ranking, and output restoration, and argue that each one requires a different evaluation protocol.

Critical strengths

The fixed-answer evaluation protocol is a useful contribution, as it distinguishes whether a correctness probe responds to changes in task requirements from whether it simply exploits correlations with answer identity. The authors demonstrate that strong probe performance on ordinary evaluations does not necessarily generalize to this setting. They run several experiments across code, tool-use, and structured reasoning tasks, along with input and likelihood controls, to strengthen their claim that about the limitations of standard correctness-probing evaluations and provide a more precise framework for interpreting their results.

Critical weaknesses

The activation patching experiments are largely disconnected from the correctness-probing results. The authors replace the entire hidden state of the models and show that late patches reproduce the donor's output, whether correct or incorrect. This is unsurprising given the intervention, and does not tell us whether the correctness information identified by the probes plays any causal role in the model's behavior. These experiments also use Hierarchical Reasoning Models rather than the LLMs used in the main experiments, without clearly motivating this choice. The LLM patching experiments in Appendix I.2 also intervene at a different position from the main probes, so neither set of experiments directly tests the causal role of the identified correctness signal.

The authors show that probes fail to generalize under distribution shift which is not particularly novel. While the fixed-answer protocol is a useful evaluation technique, it is unclear what substantive conclusion the paper draws beyond demonstrating these known limitations of classifiers.

Questions for authors

What motivates the use of Hierarchical Reasoning Models for the activation patching experiments, rather than the LLMs used in the main probing experiments?

The main six-model transfer experiments compare probes trained on freely generated answers against hidden states from teacher-forced, supplied candidates. Does the same failure to generalize occur when evaluating on independently generated answers that are matched for answer identity but differ in correctness? This would help establish whether the observed transfer failure is due to answer identity rather than differences between generated and supplied answers.

Reviewer_evWjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies failure modes of probes that use model hidden state to judge whether the model's answer is correct or wrong. To this end, it builds datasets where for each answer, kept fixed (e.g. [1, 3, 5]), there are two tasks. In the first task, the answer is correct (e.g. order these numbers) and in the second it becomes wrong (e.g. order these numbers in reverse). The paper shows that using "classic" probes trained on a corpus of right/wrong responses may use particular answers or formats as shortcuts, hence not really detecting correctness, as thought. For example, most such trained probes perform poorly when tested on the newly created dataset (despite lower scores, however, some probes get almost every pair right, and yet perform poorly because their scores don't compare well across tasks). The paper then shows that training the probes on the newly created dataset can instead yield near perfect performance, suggesting what the authors call "decodability".

Critical strengths

The strengths are buried in a very poor presentation. Constructively, prior work seems to have shown that probes generalize poorly (should have been mentioned by the paper in its motivation). What this work shows is that building datasets that keep the answer fixed reveals whether a probe uses the task or not. Another strength is the systematic attempt to isolate probe failure modes leveraging the new datasets with with appropriate ablations and interventions.

Critical weaknesses

This paper is exceptionally hard to follow, with jargon or terms used before they're defined. This alone makes it below the standard for a venue such as ICLR. See more in the author comments below. The paper further does not explain its setting, it is written without a compass, as if the reader knows the work and the experiments in advance. Furthermore, i printed the paper to read it but the first figure is using so small font that it was impossible for me to parse it.

Content-wise:

The paper seems to focus on the learnability of correct/wrong probes from data alone, but why do we expect this to be the right way to build such probes? Forcing the right inductive bias (here correctness) with examples alone is difficult... as expected. and so shortcuts are expected, they are the norm not the exception, because the learning will simply find the shortest path to the solution, given the data, and has no reason to follow our "intention".

Any score that looks only at the answer has to give both rows the same number, so i do not understand the purpose of proposition 1 which is straightforward. And so is proposition 2 which is simple. The reader does not get the point behind them.

On Section 6, the fact that late patches reproduce the donor's (funny name btw) output, correct or not, is no news: as part of the hidden state is there... to decode the answer.

Questions for authors

Not questions, but examples of the lack of clarity, taking the abstract as an example: "the same candidate states" hasn't been defined "under the tested shift": you never said you tested a shift you say a "a second separation" but you didn't explicitly name a first leading the reader to go back and look. "maze": why are you talking about mazes here ? The abstract so far only mentions code and tool-use models, mazes appear only later in the main paper (and not even properly introduced)... "the surviving signal" same here... "late whole-state patches reproduce donor outputs"... And on the writing without compass: Results arrive before the set up is even explained (to an outside reader that reads the paper step by step and does not know the work as a whole) Why start 2.1 with the maze?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.