- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how the prediction target of a transformer can explain the gap between concept decodability and steerability when training probes. The authors train transformers on a Blocksworld task, varying whether the model predicts the current, next, or previous state. They show that a concept can remain accurately decodable while its probe direction becomes more aligned with the readout of a related concept than with its own. They further investigate this effect using an unconstrained features model, show how readout alignment and logit margins determine steering effectiveness, and conduct additional experiments on Othello and LoRA for language models.
Critical strengths
- The decodability-steerability gap is an important problem in interpretability. Investigating the role of the prediction target provides an interesting and novel perspective that complements existing work.
- The setup allows the authors to isolate the effect of the training target. The evaluation is rigorous, with several controls and complementary analyses. I also appreciate the efforts to extend the findings beyond the primary setting.
- The UFM analysis, the distinction between readout alignment and margins, and the comparison between state and output probes provide different perspectives on the phenomenon.
- The figures, particularly Figure 1, communicate the results effectively. The paper also acknowledges results that do not meet the authors' criteria.
Critical weaknesses
-
My main concern is the extent to which the proposed mechanism explains the decodability steerability gaps observed in practice. The setup provides a controlled example of how a gap can arise, but it remains unclear how representative this explanation is of failures observed in realistic models, and whether such target mismatches are common in practical interpretability settings or whether they explain existing empirical observations.
-
Limited understanding of the mechanism and its generality. The similarity between the next- and previous-state results appears closely related to the particular dependencies between clear(x) and holding(x) in Blocksworld. It would be useful to understand whether the authors can characterize more general conditions under which changing the prediction target produces such results.
-
The paper is generally well written, but several important concepts and experimental choices are discussed before they are properly introduced or motivated, which makes reading a bit confusing: introduction uses clear(x) and holding(x) before explaining the task, the motivation for previous-state objective only becomes clear later, and aspects of the Othello setup are explained after the relevant results are introduced.
Questions for authors
- How common is the proposed mechanism in realistic models? Can the authors connect it to existing observations of steering failures?
- Generality: The results seem tied to the dependencies between clear(x) and holding(x). Under what general conditions does changing the prediction target produce similar alignment shifts?
- Experimental motivation: Could the authors better motivate the current/next/previous-state comparison and explain how the additional experiments, particularly Othello, support the main claim?
I am currently leaning toward weak reject, but my score could increase substantially if uncertainty about the broader significance and generality of the findings (Q1, Q2) are mitigated.
Response to Reviewer n42b
Thank you for the review. We address your questions in turn.
Q1, how common is it? We cannot give a prevalence and the revision does not claim one. The condition is that the probed fact differs from what the readout was trained to predict. The revision adds one test at language-model scale (Appendix L), in one public pair of 210M models trained identically except for the objective and its attention mask. For three facts about the previous token, each model's direction at its readout lines up with the label contrast of its own target, next token or masked current token, and its direction two blocks earlier raises that target beyond random directions of its norm, and the other model's much less, if at all. This is one pair and three local facts. The profile part is close to definitional and the steering is the empirical content, and objective and attention mask change together. We have not shown that this explains any specific published steering failure. Others report related findings. The best probing direction is not the best steering direction in three instruction-tuned models (Sadeghi & Becker, 2026, concurrent), and in AxBench (Wu et al., 2025) a linear probe ranks among the best methods for concept detection and among the weakest for steering. Neither ties these findings to the prediction target.
Q2, general conditions. The revision adds a general rule (Section 6). At an affine readout, the inner product of a fact's difference-in-means direction with an output's row equals, exactly, the gap in that output's logit between the fact's two classes. In a model that has learned its target, the profile of the direction over all rows should follow the contrast of the target's labels across the fact, computed from labels; the next-state log-odds form uses one model-derived clipping magnitude. Facts that most actions leave unchanged, such as on_table(x) and on(x,y), barely move, while clear(x) rotates toward holding(x). Against nine criteria fixed in advance, on fresh data, eight held, including every scored blocks-world own-row sign, the hand_empty reversal under a no-op policy and each Othello model's median sign. One failed: under the next-state target the fit of clear(x) stayed at 0.75–0.79 in 4 of 5 cells, below 0.80. The identity is standard and cited, so what is new is the label-contrast forecast and its test.
Q3, motivation. Moving to the next-state target changes the label statistics and adds uncertainty at once. The previous-state target is deterministic, so it separates the two. Othello tests the claim in a second domain where the labels predict the opposite sign, and Section 6 shows that the same rule gives that sign. The revision states these reasons where each target is introduced, and introduces the task and the Othello setup before their results.