acceptodds
Under review as a conference paper at ICLR 2027

We Are Not Probing or Steering Concepts

27 views

Abstract

The Linear Representation Hypothesis associates high-level concepts with directions in language models, but it remains unclear how these concept-related linear structures are organized within the model. We propose the Answer-Basin Representation Hypothesis: the probability measure induced over answers by the model's continuation distribution organizes these linear structures, with its statistics represented along linear directions shared across questions. All continuations yielding the same answer form an answer basin, whose mass is their total probability. These basin masses define the pushforward probability measure over answers. We posit that concept-related linear structure emerges from differences in the answer measure rather than being determined by changes in concept labels. Experiments across models and tasks link concept-consistent effects and their reversals in probing and steering to the alignment between concept labels and the answer measure.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, Linear probing, Activation steering, Representation geometry, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9cyKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Answer-Basin Representation Hypothesis (ABRH) as an alternative perspective on the Linear Representation Hypothesis. It defines an answer measure over model outputs, where continuations leading to the same answer form an answer basin. ABRH suggests that shared linear directions in hidden states organize statistics of this measure, including answer concentration before generation and relative answer mass after generation. The paper further argues that concept-related probing and steering effects depend on how external concept labels align with the model's answer measure.

The authors evaluate this hypothesis across Qwen2.5 and Gemma-3 models on GSM8K, ARC-Challenge, BBQ, and SafetyBench. Their experiments show that linear probes can predict answer concentration and mass ordering, including when trained only on pairs of incorrect answers. They also demonstrate that changing the relationship between correctness labels and answer mass can reverse probing predictions and the relative effects of activation steering.

Critical strengths
  • The paper provides a relatively systematic formulation of answer basins, answer measures, collision concentration, and log-mass contrasts. Connecting pre-generation concentration and post-generation answer preferences through a common probabilistic perspective is an interesting conceptual contribution. This framework offers a potentially useful way to reconsider what linear probes and steering vectors actually capture, beyond their apparent association with external concept labels.

  • The alignment/conflict experiments are particularly interesting because correctness labels remain unchanged while their relationship with answer mass is reversed. The wrong-only experiments further show that a linear probe can recover answer-mass ordering without contrasting correct and incorrect labels during training. The steering experiments also highlight a practical concern: a vector constructed to favor correct answers can relatively increase the preference for incorrect answers when they have higher initial mass. These observations are valuable for understanding the reliability and limitations of representation-based interpretability methods.

Critical weaknesses
  1. The experiments do not sufficiently disentangle answer-mass information from concept-related representations, leaving the central interpretation underdetermined.

My main concern is that the experiments establish a relationship between answer mass and linear representations, but do not demonstrate that concept-related linear structure is primarily explained by answer mass rather than by external concepts. In Figure 3, Align and Conflict probes are trained using extreme positive or negative mass gaps while keeping correctness labels unchanged. However, a probe that predominantly learns answer-mass ordering would naturally produce the observed reversal when correctness and mass are positively associated in the test distribution. This result is compatible with the authors' interpretation, but it is also compatible with concept-related directions existing independently and being dominated by mass-related signals under the selected training conditions. The wrong-only experiments do not fully resolve this ambiguity. For example, in Table 2, Qwen2.5-7B achieves 78.03% Top-1 correctness with wrong-only supervision, compared with 79.36% under High-gap supervision. Yet the paper does not report the Top-1 accuracy obtained by directly selecting the highest-mass answer under the candidate measure. Since the mean-logp baseline uses a different scoring protocol (Appendix B.2), it cannot establish whether these probes provide information beyond recovering the model's own answer preferences.

A related concern applies to the pre-generation concentration results (Table 1). Answer concentration may correlate strongly with question difficulty or model confidence. Without controlling for these factors, it is difficult to determine what additional representational structure ABRH identifies beyond existing uncertainty or confidence-related signals, particularly given prior work on semantic entropy probes (Kossen et al., 2024) and question-only correctness probing (Cencerrado et al., 2025). More generally, Eqs. 5–6 formulate an idealized universal ordering, whereas the empirical results establish approximate statistical correlations (e.g., 63%–78% ordering accuracy in Table 1). Although the authors acknowledge the idealized nature of the hypothesis (L160–161), the interpretation and conclusions would benefit from more explicitly distinguishing the formal hypothesis from what is empirically supported. I consider this a central weakness because the paper's main contribution is not simply that answer mass can be linearly decoded, but that answer-measure statistics provide an alternative explanation of concept-related linear representations. The current experiments do not yet sufficiently distinguish these claims.

  1. The steering experiments and answer-measure constructions provide limited validation of the proposed mechanism and its generality.

The paper proposes an answer-measure transport hypothesis in Eq. 8, predicting that steering along a mass-oriented direction preferentially amplifies answers with higher initial mass. However, the main intervention in Figure 5 uses a correct-minus-wrong vector constructed from label-aligned source examples, rather than the explicitly mass-oriented direction defined in Appendix A.2. Because correctness and answer mass are already aligned in the source data, the observed reversal on conflict examples does not independently establish the proposed mass-transport mechanism. The theoretical analysis also does not fully bridge this gap. Proposition A.1 provides a change-of-measure identity, but does not establish the directional amplification predicted by Eq. 8. Thus, the central steering prediction remains largely empirical rather than theoretically derived. The experimental scope further limits the conclusions. Figure 5 evaluates only Gemma-3-4B-PT on BBQ and SafetyBench, using changes in length-normalized answer log-probability. Appendix C provides additional answer-letter logit results, but neither experiment directly measures changes in generated answers or safety-related behavior. Therefore, the results demonstrate shifts in model preference scores rather than verified changes in downstream generation behavior.

There is also an important distinction between the generation-based pushforward measure in Eq. 1 and the finite-choice candidate measures used in much of the evaluation. Both are explicitly defined by the paper, but it remains unclear how consistently the proposed representational principles transfer between these different constructions. The open-generation GSM8K experiments provide useful additional evidence, yet their alignment/conflict results are less consistent than those in the finite-choice settings (Table 3 and Appendix E, Figure 12). Finally, several representation analyses use different extraction layers (Appendices B and D), without a systematic layer-sensitivity analysis. This makes it difficult to assess how robust the proposed shared directions are to layer selection. Together, these limitations make the proposed mechanism plausible and interesting, but not yet sufficiently established to support the paper's broad interpretation of probing and steering.

Questions for authors
  • Can the authors evaluate correctness probing after controlling for answer mass, for example through mass-matched correct/incorrect comparisons or residualizing mass-related information from the representations? How much concept-related predictive signal remains? This would help distinguish whether mass-related directions replace or merely coexist with concept-related directions.

  • What is the held-out Top-1 correctness of directly selecting (arg⁡max⁡am~q(a))(\arg\max_a \tilde m_q(a)) under the candidate measure? How does this compare with the High-gap and Wrong-only probes in Table 2? This comparison seems necessary for interpreting the reported correctness gains independently of the model's existing answer preferences.

  • How robust is the pre-generation concentration readout after controlling for question difficulty, correctness, or other uncertainty measures, such as semantic entropy? In particular, do confidently incorrect questions exhibit the same concentration-related linear structure?

  • Can the authors directly evaluate the relative answer-mass amplification predicted by Eq. 8, preferably using a mass-oriented steering direction with appropriate control vectors? Do the observed preference-score changes also translate into changes in actual answer selection or generated outputs?

  • How sensitive are the probing and steering findings to the selected layers and answer-measure protocols? In particular, which conclusions remain consistent when moving from finite-choice candidate scoring to generation-based answer measures?

5 comments

We organize our response around Q1–Q5, addressing overlapping concerns from W1 and W2 together.

1. The organizing role of the answer measure and controlled correctness probing (W1, Q1)

The organizing explanation and independent correctness information (W1, Q1)

The reviewer suggests that "correctness directions may exist independently but be dominated by mass signals under the selected training conditions." This requires distinguishing two different questions. Our central contribution is to identify what primarily organizes the shared linear structures used by probing and steering, thereby explaining why these directions exhibit concept-level effects and why those effects succeed, fail, or reverse. Our main experiments test this organizing role of the answer measure, rather than proving that the model contains no independent correctness information. We additionally investigate the latter issue through correctness probing with controls on mass.

The reviewer argues that the experiments do not sufficiently establish an organizing explanation beyond linear decodability, but our evidence extends well beyond linear decodability. Figure 1 uses unsupervised PCA: without training a correctness probe, correct–incorrect geometric separation already decreases as the mass gap narrows, showing that this relationship precedes any probe fitting. Figure 3 keeps correctness-labeling rules unchanged and reverses only the training label–mass relationship, after which correctness performance reverses. The observation that "a probe learning mass ordering would naturally reverse" already adopts the answer-measure explanation emphasized by ABRH; the interpretation that "correctness supervision yields a correctness direction" alone does not predict this reversal. Wrong-only experiments further show that correct-answer selection can emerge without correct–incorrect supervision contrasts. Together, these experiments test whether concept labels or the answer measure explain the geometry, orientation, and generalization of these directions, beyond whether a statistic is linearly decodable.

Correctness readout after controlling for mass (W1, Q1)

To address whether correctness remains readable after controlling for mass, we conduct an additional experiment on ARC-C. We select correct–incorrect answer pairs with very small mass gaps, balance the numbers of pairs in which the correct or incorrect answer has higher mass, and match the gaps between these groups. This jointly limits mass differences and their association with correctness labels. We train probes using correctness supervision, maintaining question-disjoint training, validation, and test splits:

ModelTraining pairsTest pairsCorrectness discrimination accuracy95% interval
Qwen2.5-7B425646.43%32.20%–60.38%
Gemma-3-12B11413444.78%35.77%–53.29%

For reference, probes trained with mass-ordering supervision using the same number of training pairs under the normal gap distribution, and evaluated on the full test set, achieve ordering accuracy significantly above chance. When mass differences are restricted and their label association is balanced, the correctness probes do not show clear above-chance correctness discrimination.

The formal hypothesis and the scope of empirical support (W1)

Equations 5–6 state an idealized ordering relationship. The experiments are additionally affected by two sources of error: answer-measure estimation and approximation errors, including finite sampling and the approximation of the target answer measure through candidate scoring; and probe fitting and generalization errors. Observed ordering accuracy therefore measures how closely the practical readout follows the proposed ordering, rather than proving an exception-free relationship. We will explicitly distinguish the formal hypothesis from the scope of empirical support in the main text and conclusion.

2. The highest-mass baseline and the interpretation of probe correctness (W1, Q2)

The reviewer requests a comparison between directly selecting the highest-mass answer and probe correctness; this comparison supports our interpretation. On the same test questions, directly selecting the highest-mass answer gives the correctness level associated with fully recovering the answer ordering:

ModelHighest-mass selectionHigh (Table 2)Wrong-only (Table 2)
Qwen2.5-7B90.39%79.36%78.03%
Qwen2.5-14B93.73%82.06%80.95%
Gemma-3-4B-PT76.91%73.03%69.70%
Gemma-3-12B-PT91.33%86.56%83.28%

All probes fall below this reference, with no indication of performance beyond the model's own answer preferences. The probes capture linear structure organized by mass differences and use it to recover answer ordering. Wrong-only results further show that this structure can be learned without correct–incorrect supervision contrasts. Their correctness performance arises from the alignment between the recovered answer ordering and correctness labels, rather than the probes independently learning correctness information.


3. Pre-generation concentration, difficulty, and uncertainty (W1, Q3)

Operational definitions and structural explanations of difficulty and uncertainty (W1, Q3)

The reviewer suggests that concentration may correlate with difficulty or confidence, but a prior question must be answered: what exactly does confidence mean here? The probability of a single continuation, the total mass of an answer, concentration across answers, and the probability mass of the correct-answer set are different quantities. For example, a particular wording can have low probability while all wordings yielding the same answer collectively have high probability. An answer distribution can also be highly concentrated on an incorrect answer. Calling all these quantities confidence does not distinguish their properties, much less identify which structure a hidden-state direction actually corresponds to.

The answer measure provides a unified and explicit foundation for these distinctions: the mass of an individual answer, concentration of the answer distribution, and correct-answer mass obtained by incorporating external labels. ABRH further proposes how these properties organize shared linear structure and thereby predicts the concept-level effects of the resulting directions. Keeping correctness supervision unchanged while changing the source label–mass relationship can reverse probe behavior. Holding a steering vector fixed while changing the target label–mass relationship can shift relative amplification from correct to incorrect answers. These are testable structural predictions.

Controlling for difficulty or confidence requires an explicit operational definition. If confidence specifically denotes mass or concentration, it already corresponds to an object studied in this paper. Renaming these objects confidence does not provide another explanation of shared linear directions, cross-question generalization, or intervention effects. An alternative explanation must specify its quantities and structural relationships and explain how they predict these controlled changes.

Distinction from semantic-entropy and correctness-readout studies (W1, Q3)

The reviewer cites Kossen et al. (2024) and Cencerrado et al. (2025), but showing that activations predict semantic entropy or correctness is a different question from explaining what primarily organizes the linear structures supporting those predictions. Predicting correctness still leaves open why the direction exhibits correctness-related behavior, what conditions that behavior depends on, and when it fails.

ABRH connects pre-generation concentration readout, post-generation mass ordering, and relative gains under steering, and uses the label–mass relationship to make specific predictions. Its contribution advances from "a signal can be decoded" to proposing and testing an organizing principle that connects multiple forms of linear structure and explains concept-level effects, failures, and reversals.

4. Steering directions, relative gains, and random controls (W2, Q4)

The relationship between correct-minus-wrong and mass-oriented directions (W2, Q4)

The reviewer argues that the correct-minus-wrong vector in Figure 5 differs from the mass-oriented direction in Appendix A.2, but they coincide term by term on the aligned source data used in our experiments. Appendix B.4 explicitly states that the correct answer has higher mass in every source pair. Consequently, sgn⁡(Δlog⁡m)\operatorname{sgn}(\Delta\log m) in Equation 16 is always +1+1, and its empirical average is exactly the correct-minus-wrong vector used in Figure 5.

Satisfying both the label-defined and mass-oriented definitions is central to this controlled design. We preserve the conventional correct-minus-wrong construction and the meaning of correctness labels, while using aligned sources so that the resulting vector also satisfies Equation 16. We then hold this vector fixed and compare aligned and conflict targets, testing whether its concept-level effect reverses with the target label–mass relationship. The central test is whether the same correctness-defined vector shifts its relative amplification from correct to incorrect answers as target mass ordering changes.

The roles of Proposition A.1 and Equation 8 (W2)

We explicitly introduces Equation 8 as an additional transport hypothesis. Proposition A.1 establishes the transformation relationship between answer measures before and after intervention; Equation 8 proposes a directional prediction for relative gains; and the steering experiments test that prediction. They serve the distinct roles of mathematical description, hypothesis formulation, and empirical evaluation. We do not present Equation 8 as a corollary of Proposition A.1.

Relative amplification versus answer selection and generation (W2, Q4)

We deliberately align the evaluation with the mechanistic quantity stated in Equation 8: the relative mass gains of answers under intervention. Answer flips are coarse-grained outcomes influenced by initial answer margins and the decision rule. They cannot distinguish an absence of effect from systematic amplification that does not cross the decision boundary. In particular, for deterministic binary selection, Equation 8 predicts further amplification of the initially higher-mass answer. The absence of a flip therefore need not indicate an absence of intervention effects; it may correspond precisely to relative amplification consistent with the prediction.

Continuous relative gains capture both direction and magnitude and directly test Equation 8. Even when the final choice remains unchanged, they identify which answer receives greater relative amplification. Free-generation outputs are additionally affected by subsequent token-conditional distributions, autoregressive feedback, and sampling, and thus reflect the combined outcome of the entire generation process.

Original, reversed, and norm-matched random directions (W2, Q4)

We add control-vector experiments covering two models. Consistent with the protocol in Table 1, each question retains two substantive options. We compare the original direction, its reverse, and multiple norm-matched random directions:

ModelIntervention directionMean relative gainProportion with greater gain for the higher-mass answer (Aligned / Conflict)
Gemma-3-4B-PTOriginal0.010895% / 84%
Reversed−0.01054% / 16%
Norm-matched random (mean)0.0006—
Qwen2.5-7BOriginal0.033295% / 82%
Reversed−0.03445% / 17%
Norm-matched random (mean)0.0037—

The original directions produce significantly greater relative gains than the random controls, with paired confidence intervals excluding zero. Reversing the intervention reverses the gain orientation. In particular, on conflict targets, the same correctness-defined vector predominantly gives greater relative amplification to the higher-mass incorrect answer. A correct-minus-wrong construction does not guarantee a fixed correctness-enhancing effect; target mass ordering predicts which answer receives the larger relative gain.

Evidence across models and tasks (W2)

Figure 5 provides fixed-vector intervention evidence across two tasks, while the additional controls test directional effects in two models. Figure 6 further shows across multiple models and tasks that reversing the source label–mass relationship produces opposing projected directions while preserving the meaning of correctness labels. Source geometry, reversals of target concept-level effects under a fixed vector, and comparisons among original, reversed, and random directions jointly support the organizing role of the answer measure in steering directions and their concept-level effects.

Reviewer_q5kxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyses the directions of probes that seem to represent correctness, bias or safety, and claims that they instead track the model's answers preferences. The authors group continuations by their final answer, adding their probabilities, and use this to build "answer-basin". Their main result is that changing the relationship between correctness and model preference in the training examples for the probes changes their behavior.

Critical strengths
  • The alignment-versus-conflict experiment suggests that a probe trained to detect correctness may instead learn whether the model assigns an answer higher or lower probability.
  • Probes trained only on pairs of wrong answers can still rank answers by model preference. This shows that learning this ranking does not require correct-versus-incorrect training examples.
  • The probing experiments cover multiple models and tasks.
Critical weaknesses
  • All in all, the results from the paper are compatible with the probes basically learning to read the confidence of the model in its top guess, which is not really novel in the interpretability community.
  • Thus, the title doesn't really reflect the work and it's a bit of an over-claim. There are many different papers in the field out there showing that concepts can be retrieved from linear directions in activation space.
  • Also, the fact that the probe trained by the authors represents confidence instead of correctness does not necessarily mean that correctness is not represented in the model.
  • It is unclear to me what the answer-basin framework adds to the work. Everything in the paper can be better explained in layman's terms without this extra layer of formalization.
  • The paper reports ordering accuracy and rank correlation, based on the probes trained, but doesn't present how accurate are the numerical predictions of the probe. How close or far is the probe from the target? With two possible answers, top-answer confidence gives exactly the same ranking as concentration.
  • The PCA plot is not really informative as to what these directions represent. Again, this PCA analysis could be picking up on the direction that the model uses to store its confidence on the top guess.
  • And the results on probes trained with two wrong answers could also be explained with the same confidence direction.
Questions for authors
  • What is the actual training and eval accuracy of the before-answer and after-answer probes?
  • Can the probes distinguish correct from wrong answers when their probabilities are close?
  • What evidence is there that the probes are not just learning to read the model's confidence on the answer?
6 comments

1. Motivation, contribution, and title (W2, W3, and the novelty concern in W1)

The reviewer's characterization of our contribution—that "concept directions actually encode confidence"—does not match our research question or claim. Our question arises from the use of LRH to motivate probing and steering (Park et al., 2024). Connections between concept directions, readout, and intervention have been established under explicit counterfactual conditions. These results do not automatically guarantee that a direction trained with external labels such as correctness or safety is organized by the corresponding concept. We ask what structure these methods actually identify and why that structure exhibits concept-level effects.

The reviewer notes that the paper cannot prove the absence of correctness representations. But the absence of correctness information has never been our claim. ABRH proposes that the linear structures under study are primarily organized by the model's own answer measure, with their concept-level behavior depending on the relationship between this measure and external labels. This is a different question from whether additional correctness information exists.

The reviewer invokes prior findings that concepts can be extracted along linear directions. However, that finding does not explain why probe behavior fails or reverses despite unchanged concept labels (Figure 3), why the same fixed steering vector produces opposite concept-level effects (Figure 5), or why correct-answer selection remains possible without correct–incorrect training contrasts (Tables 2 and 4). These phenomena are what our paper seeks to explain and predict, beyond demonstrating that a direction has concept-level effects.

The title emphasizes this interpretive shift: concept-level effects are phenomena to be explained through the model's internal structure and its relationship to labels; they cannot directly establish the direction's organizing principle.


2. The confidence explanation and the answer-basin framework (W1, W4, W6, W7, Q3)

The reviewer summarizes our results as confidence, but confidence has no single operational definition across the relevant literature. It may refer to the probability of a continuation, token-averaged log-probability, the total probability mass of an answer aggregated across continuations, or an estimate of answer correctness. These quantities are not equivalent. If prior papers, under an explicit operational definition of confidence, establish the probe reversals, opposite concept-level effects of a fixed steering vector, and Wrong-only generalization examined here, we ask the reviewer to cite those papers and explain the correspondence. Describing the results collectively as confidence does not establish that our organizing principle and empirical evidence have already been covered.

This also addresses the concern that the answer-basin framework is unnecessary. Explaining representational structure requires an explicit, measurable object; an ambiguous description is insufficient. We define basin mass as the total probability of all continuations yielding the same answer, thereby constructing an answer measure. ABRH further proposes that the linear structures under study are primarily organized by statistics of this measure, with their concept-level behavior depending on its relationship to external labels. This definition connects pre-generation concentration, post-generation mass contrasts, and relative amplification under intervention to a common object, supporting explanations and predictions of concept-consistent behavior, failure, and reversal. Informal language can summarize the intuition, but cannot replace explicit objects, relationships, and predictions.

Regarding the suggestion that PCA and Wrong-only results can also be interpreted as confidence directions, the key question is whether that interpretation explains the controlled relationships in the experiments. In PCA, concept-label meanings remain unchanged while geometric separation varies with the mass gap. In Wrong-only training, there are no correct–incorrect contrasts: supervision uses only mass ordering among incorrect answers, yet the resulting directions generalize across questions and support correct-answer selection. Together, these results support the answer measure as the primary organizing factor of the linear structures studied, beyond showing that a confidence score can be decoded.

Thus, our evidence for Q3 comprises these geometric and generalization controls, the Align/Conflict readout reversals, and the opposite concept-level effects of a fixed steering vector. These experiments test what primarily organizes the linear structure, rather than excluding every quantity that could be called confidence. A confidence-based alternative explanation must specify its operational definition and explain these controlled changes.

3. Numerical precision and the two-answer case (W5)

Equation 5 is an ordering statement, not a numerical fit. It states that questions with higher concentration have larger projections along a shared direction; it does not state that projection values equal concentration. Order Acc. and Spearman's ρ\rho therefore directly match the hypothesis. Numerical reconstruction accuracy is not a requirement of Equation 5.

For a two-answer measure, concentration and the highest answer mass indeed induce the same ordering. This is a property of binary distributions. If the reviewer's top-answer confidence refers to max⁡amq(a)\max_a m_q(a), then both it and C2C_2 are statistics of the answer measure, rather than an alternative organizing explanation independent of that measure. ABRH concerns the relationship between shared linear directions before generation and statistics of the future answer measure; it does not require C2C_2 to be the only readable statistic.

We choose C2C_2 because it equals the expected mass of the answer realized when sampling from the answer measure (Equation 2). This connects pre-generation concentration and post-generation basin mass to the same object; the maximum answer mass does not generally have this property. Furthermore, GSM8K has an open answer space, and SafetyBench includes questions with three or four options. In these multi-answer settings, the two statistics need not induce the same ordering, while the results in Table 1 continue to hold.


4. Probe training and test accuracy (Q1)

The reported results are held-out test results with no overlap with the training set. Pre-generation test results across models and datasets appear in Table 1. After generation, both Align and Conflict achieve nearly perfect accuracy on their training answer pairs; test performance across models and conditions is reported in Tables 2 and 4 and Figure 3. Training metrics for the remaining experiments will be presented alongside their corresponding test results in the revised appendix.


5. Can probes distinguish correct from incorrect answers when their probabilities are close? (Q2)

When answer probabilities are close, probes retain the ability to distinguish mass ordering, but their correctness performance reverses with label–mass orientation. This directly illustrates the paper's central argument.

We train probes using mass-ordering supervision and select correct–incorrect answer pairs with very small mass gaps from an independent test set. We evaluate the two orientations separately:

ModelTest-pair typeNumber of pairsMass-ordering accuracyCorrectness discrimination accuracy
Qwen2.5-7BCorrect answer has higher mass7659.47%59.47%
Incorrect answer has higher mass2656.92%43.08%
Qwen2.5-14BCorrect answer has higher mass4371.16%71.16%
Incorrect answer has higher mass1772.94%27.06%

Results are averaged over multiple training runs, and each trained probe remains fixed across the two test conditions. Mass-ordering accuracy is similar between the two groups, whereas correctness performance differs substantially: when the correct answer has higher mass, the probe tends to select it; when the incorrect answer has higher mass, the same probe instead tends to favor the incorrect answer. A direction's concept-level behavior depends on the alignment between the answer measure it reads out and external concept labels.

Additional experiment: correctness probing under mass controls.

We additionally test whether correctness supervision recovers a reliable correctness signal after controlling the mass-order–label association. Unlike the preceding experiment, which uses mass-ordering supervision, this experiment trains probes directly with correctness labels. On ARC-C, we select correct–incorrect pairs with very small mass gaps, balance the numbers of pairs in which the correct or incorrect answer has higher mass, and match the gap distributions between the two groups. Training, validation, and test splits are question-disjoint.

ModelTraining pairsTest pairsCorrectness discrimination accuracy95% interval
Qwen2.5-7B425646.43%32.20%–60.38%
Gemma-3-12B11413444.78%35.77%–53.29%

Under these controls, neither model's correctness probe shows clear discrimination above the 50% chance level. This also directly addresses W3's suggestion that correctness may remain independently decodable.

Together, these experiments examine both how mass readout acquires correctness-related behavior and whether correctness supervision recovers a reliable signal after the mass-order–label association is balanced. The first shows that a fixed mass readout exhibits opposite correctness behavior across the two orientations; the second does not recover clear above-chance correctness discrimination under the reported mass controls. These findings support evaluating concept-level behavior through the relationship between the answer measure and external labels, rather than identifying a direction's organizing principle from its supervision labels alone.

Reviewer_AmQjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Answer-Basin Representation Hypothesis: the linear structure that probing and steering methods pick up in LLM hidden states is organized by statistics of the model's answer probability measure, rather than by external concept labels.

Critical strengths
  1. The claim that "concept directions" are largely reading out answer-distribution statistics rather than concept semantics is a reasonable and interesting idea.
  2. The overall writing and organization are good and easy to follow.
Critical weaknesses
  1. The paper argues that the linear structure in hidden states is organized by statistics of the model's answer distribution rather than by concept labels, thereby challenging the connection between concept directions and correctness. However, it never directly reports the correlation between correctness and basin mass itself.
  2. In Figure 3, the label-mass composition of the held-out set is never reported, and the natural assumption is that it is alignment-dominated. Meanwhile, the conflict-trained probe is trained on pairs where the correct answer always carries lower mass, so its training distribution mismatches the held-out distribution by construction, and its poor performance is then partly expected rather than surprising. More importantly, the reversal does not distinguish between two explanations. The first is the paper's claim that the representation itself is organized by basin mass and carries little correctness information. The second is shortcut learning: the probe may simply learn to avoid high-mass answers. Both explanations predict the same reversal, so the experiment cannot tell whether the dominance of mass reflects the representation itself or the constructed training distribution, and it cannot exclude that the conflict probe still carries correctness information. Similar concern may apply to other align/conflict experiments.
  3. The contribution is limited. Many prior works have already shown that distributional statistics of model outputs are linearly predictable from hidden states.
  4. The training protocols for the post-generation probes are under-specified. It would be helpful to state them more clearly.
Questions for authors

See weaknesses

5 comments

We thank the reviewer for the feedback and for recognizing the interest of our central hypothesis. We address the four weaknesses below.


W1: Correlation between correctness and basin mass

We did not overlook the correlation between correctness and mass; we systematically manipulated it as a central experimental variable. A correlation coefficient alone cannot answer our question: does a probe's ability to predict correctness arise from correctness itself, or from mass-related structure that happens to align with correctness? Distinguishing these possibilities requires changing the label-mass relationship and observing the consequences.

Figure 3 does precisely this: it holds the labeling rule, training objective, and test set fixed, while reversing only the correspondence between correctness and mass in the training data. If the probe reads out correctness itself, this reversal should not affect its performance on the same test set. Yet the results show the opposite: the Conflict probe systematically favors incorrect answers, with accuracy far below chance. This reversal is a testable prediction of ABRH, not merely a statistic to be reported. Reporting a correlation coefficient cannot replace this causal test based on controlled manipulation.


W2: Align/Conflict experiments, distribution mismatch, and shortcut learning

The reviewer describes the Conflict probe's failure as "expected," but anticipating this failure requires considering the label-mass relationship, precisely the analytical perspective introduced in this paper. Identical correctness labels and an identical training objective alone do not predict that a probe will systematically favor incorrect answers on the same test set. In other words, the framework the reviewer invokes to explain this result is precisely the one proposed in this paper.

Regarding the composition of the test set, both probes are evaluated on exactly the same, unfiltered standard held-out set, where correctness and mass are positively correlated overall. Align/Conflict filtering is applied only to the training set.

Both "alternative explanations" proposed by the reviewer involve a misreading of our claim:

  • The first explanation attributes to us the claim that "representations contain almost no correctness information"; this is not our claim. ABRH asks what primarily organizes the linear structures under study and which conditions their concept-level behavior depends on. This is a different question from whether representations also contain other correctness information.

  • The second explanation says that the probe "learns to avoid high-mass answers"; this simply redescribes the reverse mass readout under the Conflict condition, rather than providing an independent alternative mechanism. "What structure the representations provide" and "which orientation the training labels induce in the readout" are two parts of the same explanatory chain, not two mutually exclusive theories. Calling the latter a shortcut does not challenge our structural explanation.

The Wrong-only experiments provide further support: even without any contrast between correct and incorrect labels, training exclusively on incorrect answers still yields a mass ordering that generalizes across questions, and the resulting direction can still select correct answers in standard evaluation. This shows that mass-related structure exists in the representations independently of correctness labels.

The intervention experiment in Figure 5 provides another perspective: the same fixed vector favors correct answers on aligned targets and incorrect answers on conflict targets. If the linear structure were organized by correctness rather than mass, the same vector should not produce concept-level effects in opposite directions.

W3: Contribution relative to prior work on linear readouts of distributional statistics

The reviewer characterizes our contribution as "distributional statistics can be linearly predicted from hidden states," but this is neither the question this paper poses nor the contribution it makes.

The paper's central question is what primarily organizes linear structures interpreted as concept directions, and when their concept-consistent behavior holds or fails. Training an effective direction with correctness labels does not establish correctness as its organizing principle: as Figure 3 shows, the same labeling rule can yield directions with opposite orientations.

The prior finding that "distributional statistics are linearly predictable" cannot answer any of the following questions:

  • Why does the same correctness supervision yield directions with opposite orientations?
  • Why does the same fixed steering vector produce opposite concept-level effects?
  • Why can a direction learned without correct-incorrect contrasts still select correct answers?

By taking the answer measure as the unifying organizing object, ABRH provides a common explanation and testable predictions for these phenomena. These predictions are consistently validated across multiple probing and steering experiments, both before and after answer generation. Novelty should be assessed with respect to this organizing principle and its systematic supporting evidence, rather than the different question of whether distributional statistics admit linear readout.


W4: Training protocols for post-generation probes

Thank you for pointing this out. We agree that the main text does not describe the training protocols in sufficient detail and will provide a complete specification in the revision.

The core method is as follows. A post-generation probe takes the hidden-state difference between two answers to the same question as input and is fitted using L2-regularized logistic regression. Hidden states are first standardized using statistics estimated solely from the training set. Pair orientation depends on the experimental condition: correct-minus-wrong in Figure 3, and high-minus-low mass in Tables 2 and 4.

All conditions share the same probe formulation, preprocessing pipeline, and hyperparameter search and selection procedures for the extraction layer and regularization strength. The core implementation is already provided in the supplementary material.

In the revision, we will explicitly describe the training method and shared selection procedures in the main text, and add the hyperparameter search ranges, selection criteria, and training-set sizes for each condition to the appendix.

The rebuttal states: “We did not overlook the correlation between correctness and mass ... Figure 3 does precisely this: ... If the probe reads out correctness itself, this reversal should not affect its performance on the same test set.” However, the Align and Conflict training sets are explicitly constructed so that mass may become exceptionally predictive of the training labels, with the correlation reversed between the two sets. A finite-sample probe may therefore learn mass even if correctness remains independently decodable. On an alignment-dominated test set, a Conflict probe that learns “correct answers always have low mass” is expected to perform below chance. Overall, I do not think Figure 3 establishes that the correlation between correctness and mass reflects the organization of the representation itself. It may only show that the fitted probe exploits the constructed training correlation. I maintain my score.

Comments on the paper

1 comment

Global Response (To All Reviewers and the Area Chair)

We thank all reviewers and the Area Chair (AC) for their thorough reviews. In this global response, we clarify our motivation, the core claim of ABRH, and the scope of our study.

Much of the current work on probing and steering is inspired by the Linear Representation Hypothesis (LRH). Under well-specified counterfactual conditions, Park et al. (2024) established a theoretical link between concept directions, readout, and intervention. In practice, however, researchers typically train linear probes on labeled question-answer pairs or construct steering vectors from representation differences between positive and negative examples. When such a direction succeeds on a test set, it is taken as a "correctness direction" or "safety direction" inside the model. Yet empirical effectiveness does not equate to the underlying linear structure being organized around that semantic concept. Our paper begins with this untested equivalence.

To investigate what structure these methods actually capture, we start from the model's own generative distribution. Given a question, a language model can produce multiple continuations that arrive at the same final answer. Aggregating the probabilities of continuations yielding the same answer gives that answer's probability mass; across all answers, these masses form the model's answer measure. ABRH posits that the statistical properties of this answer measure are the primary organizing factor of the linear structure under study.

Under this hypothesis, both methods fit into a single framework: pre-generation hidden states reflect the concentration of the upcoming answer distribution, post-generation states reflect the relative probability mass across candidate answers, and steering shifts the model's relative preference among them. Whether a direction externally exhibits a "correctness" effect depends on how internal answer preferences align with external labels—when aligned, the direction yields concept-consistent readout or amplification; when in conflict, its conceptual effect fails or even reverses. This relationship provides a common basis for understanding both the successes and failures of probing and steering.

We also want to be explicit about what this paper does not investigate or claim. Our goal is not to prove whether probes can read out correctness, nor to explain how models acquire knowledge or judge truth and falsity—nor do we claim more accurate probes or stronger error-correction algorithms as our contribution. Correctness serves as an easily annotated, comparable external attribute that offers an entry point for examining the link between internal answer structure and external labels. Nor do we deny that other concept directions may exist in representation space—but their existence does not answer what primarily organizes the structure extracted by current methods. Our focus is on this dominant factor and how it governs directional behavior.

Similarly, summarizing this structure as "confidence" requires specifying what that term actually means. Across prior work, confidence may refer to single-continuation probability, total answer mass, or the probability of answering correctly—quantities that are not equivalent. A loose appeal to confidence neither specifies which probabilistic property corresponds to the shared linear directions nor predicts when their conceptual effects hold or fail. ABRH provides a well-defined probabilistic object and testable structural relationships.

Accordingly, what we challenge is the direct interpretation—drawn by appealing to LRH—that the empirical success of probing or steering means reading out or controlling the corresponding semantic concept. Conceptual effects themselves require explanation, especially when a model's preferences conflict with external objectives and correction is needed. ABRH's core contribution is to use the model's answer measure as a unifying object to link the structure exploited by probing and steering with their successes, failures, and reversals. We hope this clarifies our scope so the AC and reviewers can evaluate whether this work advances our understanding of what these directions represent and when their conceptual effects are reliable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.