acceptodds
Under review as a conference paper at ICLR 2027

Same Data, Different Target: How the Prediction Target Can Separate Probing from Steering

Abstract

A linear probe can decode a fact while steering along it barely changes the model's output, and such a failure is easily read as the model not using the fact. We ask whether it can instead reflect what the model's readout, its output layer, was trained to predict. We train small blocks-world transformers that share data, initialisation and batch order and differ only in whether the readout predicts the current, next or previous state. Under the next- or previous-state target, clear(x) stays decodable, but its difference-in-means direction aligns more with the readout row of holding(x) than with its own. In our random play, holding(x) is possible one step away only if clear(x) holds now, and a model fitted to the labels alone predicts where the direction goes. How much steering along it changes the clear(x) output depends on its component on that output's row relative to the outputs' logit margins: it is partial where next-state outputs are uncertain and, in distribution at our primary strength, near zero where previous-state outputs are confident, under 3% of what the row itself achieves. The alignment replicates on new seeds and on one encoding whose action counts do not fix the state, and in Othello the target sets the sign of a probe direction's alignment with its own row. A steering failure can therefore reflect the readout's target rather than the representation; where the readout is hidden, a probe fitted to the model's own outputs diagnoses the alignment.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear probes, activation steering, decodability-steerability gap, prediction target, readout geometry, world models, Blocksworld

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_n42bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how the prediction target of a transformer can explain the gap between concept decodability and steerability when training probes. The authors train transformers on a Blocksworld task, varying whether the model predicts the current, next, or previous state. They show that a concept can remain accurately decodable while its probe direction becomes more aligned with the readout of a related concept than with its own. They further investigate this effect using an unconstrained features model, show how readout alignment and logit margins determine steering effectiveness, and conduct additional experiments on Othello and LoRA for language models.

Critical strengths
  1. The decodability-steerability gap is an important problem in interpretability. Investigating the role of the prediction target provides an interesting and novel perspective that complements existing work.
  2. The setup allows the authors to isolate the effect of the training target. The evaluation is rigorous, with several controls and complementary analyses. I also appreciate the efforts to extend the findings beyond the primary setting.
  3. The UFM analysis, the distinction between readout alignment and margins, and the comparison between state and output probes provide different perspectives on the phenomenon.
  4. The figures, particularly Figure 1, communicate the results effectively. The paper also acknowledges results that do not meet the authors' criteria.
Critical weaknesses
  1. My main concern is the extent to which the proposed mechanism explains the decodability steerability gaps observed in practice. The setup provides a controlled example of how a gap can arise, but it remains unclear how representative this explanation is of failures observed in realistic models, and whether such target mismatches are common in practical interpretability settings or whether they explain existing empirical observations.

  2. Limited understanding of the mechanism and its generality. The similarity between the next- and previous-state results appears closely related to the particular dependencies between clear(x) and holding(x) in Blocksworld. It would be useful to understand whether the authors can characterize more general conditions under which changing the prediction target produces such results.

  3. The paper is generally well written, but several important concepts and experimental choices are discussed before they are properly introduced or motivated, which makes reading a bit confusing: introduction uses clear(x) and holding(x) before explaining the task, the motivation for previous-state objective only becomes clear later, and aspects of the Othello setup are explained after the relevant results are introduced.

Questions for authors
  1. How common is the proposed mechanism in realistic models? Can the authors connect it to existing observations of steering failures?
  2. Generality: The results seem tied to the dependencies between clear(x) and holding(x). Under what general conditions does changing the prediction target produce similar alignment shifts?
  3. Experimental motivation: Could the authors better motivate the current/next/previous-state comparison and explain how the additional experiments, particularly Othello, support the main claim?

I am currently leaning toward weak reject, but my score could increase substantially if uncertainty about the broader significance and generality of the findings (Q1, Q2) are mitigated.

1 comment
AuthorsOpenReview ↗

Response to Reviewer n42b

Thank you for the review. We address your questions in turn.

Q1, how common is it? We cannot give a prevalence and the revision does not claim one. The condition is that the probed fact differs from what the readout was trained to predict. The revision adds one test at language-model scale (Appendix L), in one public pair of 210M models trained identically except for the objective and its attention mask. For three facts about the previous token, each model's direction at its readout lines up with the label contrast of its own target, next token or masked current token, and its direction two blocks earlier raises that target beyond random directions of its norm, and the other model's much less, if at all. This is one pair and three local facts. The profile part is close to definitional and the steering is the empirical content, and objective and attention mask change together. We have not shown that this explains any specific published steering failure. Others report related findings. The best probing direction is not the best steering direction in three instruction-tuned models (Sadeghi & Becker, 2026, concurrent), and in AxBench (Wu et al., 2025) a linear probe ranks among the best methods for concept detection and among the weakest for steering. Neither ties these findings to the prediction target.

Q2, general conditions. The revision adds a general rule (Section 6). At an affine readout, the inner product of a fact's difference-in-means direction with an output's row equals, exactly, the gap in that output's logit between the fact's two classes. In a model that has learned its target, the profile of the direction over all rows should follow the contrast of the target's labels across the fact, computed from labels; the next-state log-odds form uses one model-derived clipping magnitude. Facts that most actions leave unchanged, such as on_table(x) and on(x,y), barely move, while clear(x) rotates toward holding(x). Against nine criteria fixed in advance, on fresh data, eight held, including every scored blocks-world own-row sign, the hand_empty reversal under a no-op policy and each Othello model's median sign. One failed: under the next-state target the fit of clear(x) stayed at 0.75–0.79 in 4 of 5 cells, below 0.80. The identity is standard and cited, so what is new is the label-contrast forecast and its test.

Q3, motivation. Moving to the next-state target changes the label statistics and adds uncertainty at once. The previous-state target is deterministic, so it separates the two. Othello tests the claim in a second domain where the labels predict the opposite sign, and Section 6 shows that the same rule gives that sign. The revision states these reasons where each target is introduced, and introduces the task and the Othello setup before their results.

Reviewer_4drHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors ask how prediction targets of a model impact the alignment between probed concept directions in the residual stream and their corresponding readout directions. They test this in transformers trained on a toy task (block-world) which admits a natural notion of concept direction at the affine readout. For the clear(x) concept, they show that its probe direction aligns strongly with its corresponding readout row when the model is trained to predict the current target. When trained to predict next and previous targets, the direction rotates to align more with the holding(x) readout direction. As holding(x) is always followed by and preceded by clear(x) in the data, the authors wonder whether the observed rotation can be explained by the particular statistical coupling of the task itself rather than particular biases of the transformer. Using an unconstrained features model (UFM) they provide evidence that it does.

The authors then ask how this target-induced rotation of clear(x) geometry away from the clear(x) readout direction impacts steering of the clear(x) output. To do this, they measure "flip rate" as a proxy of steering success and determine that output margins (which are computed from logits) largely determine steering rate. Measuring flip rate across targets, they show that the previous clear(x) output is the least steerable when steered along the difference-in-means probe direction. Across all steering experiments, the authors compare four steering directions (including DiM) and show that the output probe direction largely tracks the readout row direction. To check that the probe directions are more meaningful to the model than just alignment to readout, they steer from one block upstream and show that downstream computation is still sensitive to the direction.

Finally, as part of demonstrating scope beyond blocks-world, the authors run similar experiments in Othello and show that in a task which bears some structural similarities to the blocks-world task, the identity of the target sets the sign of the alignment between the probe and readout direction.

Critical strengths

The paper is honest about its contributions and the authors take care when running experiments and analyses to not trick themselves. They design and run reasonable controls, report when things don't pan out as expected, and even define pre-specified criteria for how to interpret a result. The matched transformer experiments provide a clean method to isolate the readout target as the source of variation in the concept geometry, and the idea to use the UFM as a way to distinguish between the geometric bias coming more from the transformer architecture and optimizer or more from the data statistics was both clever and likely useful for guiding future intuitions about the relationship between task structure and geometry. As far as I can tell, the contributions as stated are all supported by the evidence.

More broadly, I like the approach of starting in a toy setting to study a phenomenon of interest. This makes the analysis more tractable and can be fertile ground for developing intuition that can then be formalized and tested against more complex settings.

The above strengths are what keep my current score above a clear reject considering the weaknesses I discuss below.

Critical weaknesses

While this paper is honest about its contributions, I don't feel these contributions are sufficient to warrant acceptance. This paper mostly presents empirical facts about the relationship between a probed concept direction, d, and task structure in blocks-world and Othello. While I can't say I would have predicted the specific results of the experiments, I don't feel like these facts taken together, and the provided explanations for them, help me generalize to other settings even within the restricted domain analyzed. For instance, what would happen if we changed the probed concept to be something other than clear(x)? How much do the results depend on this particular concept choice and its particular relationship to the shifted target concepts? There is some evidence that it may depend a lot, such as the fact that the on_table(x) probe did not seem to rotate when the target shifted but this is not addressed in much detail. Combined with the Othello results, it feels like there is some shared structure underpinning the observations in both settings, namely, there are similar consequences for the geometry of a probed fact B at an affine readout when the target A requires a particular value of B. However, this structure is never satisfyingly surfaced nor made central.

For me to see this work as a meaningful contribution to the field of interpretability, I would want to see either novel and surprising structure discovered and characterized in a model and task setting that the field cares about, new methods for discovering and characterizing known or novel structure, or the development/validation of theory that explains the nature of structure in the toy setting that has clear potential to be built upon. While the experiments are sound, the work doesn't convincingly do any of these things. In fact, the intuitions that were built in the toy setting failed the paper's own criterion once translated to a real language model. This is not altogether surprising as the paper's story seems to rest on there being some readout direction that "naturally" corresponds to the concept being probed and steered (i.e. the target vector is indexed by the same concepts that are probed). While this may hold for blocks-world and to a lesser extent for Othello, we should not expect this to hold in general. In the Qwen experiment, the authors seem to implicitly acknowledge the loss of this concept-output value correspondence in that they modify the "flip-rate" measure to not need a readout row. However, the consequences of removing this particular feature of the blocks-world setting is not explicitly discussed. There is a broader question here of how a given direction in activation space at a particular layer impacts downstream behavior, or, in other words, how the downstream layers of a network interpret information at the current layer. This is an important question, but the paper is asking, if at all, a very scoped version of it, for a particular model class, task setting, and intervention point. Without the model class and task setting being more relevant to the broader research community, it is hard for me to see this as a meaningful contribution to the field.

Beyond scope and significance, there were some framing and presentational issues that I think could be improved. The framing issue for me rests on this line: "We would naturally conclude that the model does not use the fact." When they say "we" here, who are the authors referring to? It is a bit unclear to me how well motivated this statement is and it feels fairly important for motivating the main line of inquiry a few sentences later. Furthermore, is this the standard way of defining a steering failure in the literature? The authors don't explicitly say nor do they connect the field's notion of steering failure to the flip rate measure they define. There are also some presentational issues, a main one being what feels like an unjustified assumption of familiarity with the blocks-world task especially given how important it seems to be for understanding the main results. Having a clearer picture up front for why they chose the blocks-world task, what they are referring to when they introduce clear(x) and holding(x), and why they only probe clear(x) would be helpful for comprehension. Other presentational issues include not stating what an "example" is for the UFM fitting, why an alpha of 2 is chosen for steering beyond Figure 1, and what it means in Section 4.3 to not have access to the readout and why this matters.

Questions for authors
  1. Do you have any understanding of a more general relationship that would explain the degree to which we do or don't see geometric alignment between a probed fact direction and a readout direction? Ideally this understanding would help connect why we see similar behavior between the blocks-world and Othello setting, would explain why we don't see rotation for the on_table(x) predicate, and would hopefully allow us to make predictions about, for a given task, what the geometric alignment and steerability would be for any probed fact and target.

  2. It would increase the relevance of this work for me if I could better understand how the insights here apply to models that do not have a clear concept index in the target space, such as with Qwen. In that setting, where there are no concept readout rows, how do you see these analyses carrying over?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4drH

Thank you for the review. A new Section 6 answers your question about a general relationship.

A general relationship. At an affine readout, by a standard identity (cited in the revision), the inner product of a fact's difference-in-means direction D_B with row k equals the mean difference of logit k between the two classes of B. If the model has learned its target, these logits follow its labels, so the profile of D_B over all rows should follow the label contrast P(A_k | B) − P(A_k | not B), where A_k is the label of output k, computed from labels, with a model-derived clipping magnitude for the next-state log-odds form.

Which facts rotate. A fact keeps its row when its own shifted label still differs sharply across it. Most actions leave on_table(x) and on(x,y) unchanged, so they barely move, while clear(x) loses own contrast and gains contrast with holding(x). It also predicted an unreported hand_empty reversal and the Othello sign.

The test. Against nine criteria fixed in advance, on fresh data, eight held. The rule fits every current- and previous-state profile (family medians 0.89–0.99), beats forecasts that ignore the shift or keep each fact on its own row, and predicts every scored blocks-world own-row sign and each Othello model's median sign. One failed: under the next-state target the fit of clear(x) was 0.75–0.79 in 4 of 5 cells, below 0.80, and holding(x) reached it in 4 of 6 models in one cell. The log-odds form was derived on development data, and the rule overestimates the own row's share by 0.03–0.12 for the deterministic targets. Our earlier criterion asking on_table(x) and on(x,y) to show the clear(x) effect failed, as predicted.

What is new. The identity is standard, so agreement says only that competent logits follow the labels. The contribution is this label-contrast forecast and its test.

Models without concept rows. The identity needs one row per output, so a language model's token unembeddings play that role. The revision adds one test (Appendix L) in one public pair of 210M models trained identically except for the objective and its attention mask, next-token with a causal mask against masked-token with bidirectional attention, for three facts about the previous token. Each model's direction at its readout lines up with the label contrast of its own target on held-out text, and its direction two blocks earlier raises that target beyond 20 random directions of its norm, and the other model's much less, if at all (18 of 18 pre-specified criteria). The profile part is close to definitional, and the steering is the empirical content. It does not show that the objective changes how the fact is represented, each model is pushed by its own direction, not a matched budget, and target and visibility of the predicted token stay coupled. The earlier failed language-model criterion stands.

Presentation. The sentence you questioned now cites the reading of causal probing tests, linked to our flip rate. A figure introduces the task first, and we explain the focus on clear(x), the UFM example, α = 2 and the hidden readout.

Reviewer_9f6dOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is on using different prediction targets with the same data in the context of disentangling probing from steering.

Critical strengths

I found this paper hard to parse. See below.

Critical weaknesses

The paper is dense and essentially unreadable, so I cannot follow the argumentation. For example, even the worked example is difficult to understand. This should be simple & straightforward. As such, I cannot provide a detailed review and recommend rejection because of poor intelligibility.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9f6d

Thank you for the feedback. Readability was also raised by the other reviews, and we rewrote the opening of the paper. The task is now introduced in plain words with a figure before its facts are used, the worked example follows one action, and each target's purpose is stated when it first appears. If you have time, Sections 1–2 alone should show whether the problem you describe has been resolved.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revision

The revision makes the following changes to the reviewed version. New: Section 6 and Appendix K, a rule based on label contrasts, with one model-derived clipping magnitude for the next-state target, that forecasts which readout rows each fact's direction aligns with, tested once on fresh data against nine pre-specified criteria (eight met; the next-state profile fit for clear(x) failed, and this is reported), with new rows in Table 18, including an earlier failed criterion. New Appendix L: in one public pair of 210M language models differing only in objective and attention mask, for three facts about the previous token, each model's direction at its readout lines up with its own target's label contrast, and its direction two blocks earlier raises that target and the other model's much less, if at all (18 of 18 pre-specified criteria), with its limits stated. Related work now cites language-model evidence that the best probing direction is not the best steering direction (a concurrent preprint) and that a linear probe among the best concept detectors is among the weakest at steering (AxBench). Rewritten for clarity: the opening of Sections 1–2 (a new Figure 2 introducing the task, a simpler worked example, why there are three targets, why we focus on clear(x)); the sentence on how a steering failure is read; Othello introduced in Section 2; what an example is in the UFM; how the steering strength was set; what fitting without access to the readout means. The Conclusion is shorter, and Table 18 continues across pages. To fit, the invariant-cosine ranges moved from Section 3.1 to Appendix D. Unchanged: the title, the abstract, the existing claims, numbers and pre-specified criteria.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.