acceptodds
Under review as a conference paper at ICLR 2027

Lagged Coupling: Decodability and Probe-Direction Interventions Across Training

Abstract

Information that is linearly decodable from a language model need not be accessible through an intervention along the decoding direction. We track this distinction through training across six Pythia models (160M–12B parameters), eight checkpoints, and four synthetic task families. At each model–checkpoint–task combination, we measure internal probe AUROC, output-level answer-score AUROC, and the effect of a single-site additive edit along the probe direction relative to norm- and site-matched random directions. The three measurements dissociate. Internal decodability is near ceiling (probe AUROC ) from the first measured checkpoint; output-level discrimination, tabulated for the four largest models, rises by up to 0.43 AUROC over training; and 43 of 48 intervention summaries fall within the random-direction band , with no sustained positive effect and a consistently negative alignment at the two earliest checkpoints. Over the same period, the RMS activation along the probe direction grows by up to 56.8-fold without a corresponding intervention benefit. We call this dissociation lagged coupling. A first-order analysis explains why it is expected: the standardized intervention effect equals times the cosine between the probe direction and the model's mean readout gradient, independently of the edit norm, and near-perfect decodability places no lower bound on that cosine when the representation is anisotropic. Probe accuracy is therefore not a proxy for steerability, and decodability and intervention efficacy should be evaluated separately throughout training.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear probes, activation steering, developmental scaling, Pythia, causal efficacy, preregistration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AAzDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents data on linear-probe decodability, output discrimination, and probe-direction interventions across six sizes of Pythia, eight training checkpoints, and four binary tasks. It reports near-ceiling decodability very early on, intervention scores below reference early on and then moving towards a score for random-direction reference later on. Two propositions explore how decodability and intervention efficacy change under various assumptions.

Critical strengths
  1. The experimental design addresses an important question that is typically not addressed by final-checkpoint analyses, particularly when probe accuracy is useful for steering.

  2. The propositions establish a conditional distinction between decodability and intervention efficacy, providing a possible geometric explanation for their separation.

Critical weaknesses
  1. The reported aggregates do not establish consistency across tasks. Each cell is a median of four signed effects (one for each task), which could conceal opposing task-level effects. In addition, random-direction spread needs to be distinguished from sampling uncertainty. The null-distribution and dependence assumptions underlying Section 5.2's sign calculation and independent-normal median argument need to be justified.

  2. The intervention target lacks an explicit token and sign convention. Although Equation 3 suggests that a positive-class token is fixed, the prompt and label mapping are not specified. If each example's correct token is used instead, opposing class effects could cancel out in the signed average. This is a conditional concern that could be resolved by providing an explicit fixed-class definition.

Questions for authors
  1. [Clarification] Do the task-level effects and sampling uncertainty support the same trajectory, and what justifies the null assumptions in Section 5.2?
  2. [Clarification] Is the answer token fixed across classes, and how is the probe direction oriented? An example from each class would resolve the ambiguity.
Reviewer_G6dtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether information that is easily decodable from a language model's internal representations is also useful for steering its behavior. The authors train linear probes on activations from Pythia models at multiple stages of training and compare probe accuracy with the effect of interventions applied along the probe direction. They find that probe performance becomes nearly perfect very early in training, while task performance continues to improve substantially. However, interventions based on probe directions have little effect on model outputs and often perform no better than random directions. The authors call this phenomenon lagged coupling. They argue that decodability and intervention efficacy depend on different geometric properties of the representation space: a concept can be easy to decode without being aligned with the directions that drive the model's final decisions. The results suggest that high probe accuracy alone is not sufficient evidence that a representation provides an effective target for behavioral steering.

Critical strengths
  • The empirical study is systematic and spans multiple model sizes and checkpoints.
  • The phenomenon of lagged coupling is clearly interesting.
  • Most of the results look reproducible within the evaluated setting.
  • The discussion touches on an important misconception in interpretability research.
  • The work raises worthwhile questions about what probes actually measure.
Critical weaknesses

Clarity

As mentioned earlier, the clarity is sometime an obstacle. I won't detail all the paragraphs I found difficult to understand, but let me provide one example.

The internal decodability is an important concept in the paper. Its definition appears page 4 without any comment here or before. Since one important point (If I understood correctly) is not the definition of decodability, but its interpretation in the litterature. An unformal definition (with words) could be introduced before and could clarify the purpose of the work. It is also true with the two other measures defined in section 3.1. It is important for the reader to understand how they interact to really receive the "message". These remarks may look a bit shallow but at the end I think the paper could interesting if I could really understand the its construction.

The claim appears stronger than the evidence supports

This is an illustration of the clarity issue. For me, the claim of the paper is "decodable does not imply steerable." However, the experiments only evaluate a very specific intervention paradigm: additive perturbations along logistic-probe directions. Failure of this intervention method is not sufficient to conclude that the underlying representation lacks causal relevance. Indeed, a large body of recent work claimed that that the success can depend on the choice of representation direction, intervention location, scaling strategy, training procedure, ... In this paper, this line of work is not really discuss. Does this work encompass all the recent appoaches ?

Experimental setup

Once again ! It is not clear. When the only sentence in the main part of the paper that describes the tasks is : "four synthetic task families probe different ways in which a decodable variable can fail to reach the answer." Even with table 2, it is difficult for the reader to really assess the results.

Baselines

What about baselines with stronger intervention ? Is it possible ? Meaningful ?

Questions for authors

See Weakness

Reviewer_EfkwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an investigation into the misalignment between linear probe performance and steering effectiveness and provides a theoretical analysis aimed at explaining this misalignment. The analysis includes: (1) the finding that the effect of steering (zez_e) does not depend on the strength of editing (λ\lambda); and (2) a constructed theoretical example where a probe is highly accurate but ineffective as a steering direction. Then, the authors provide an empirical study on Pythia pre-training checkpoints and identify three findings: (1) probes become very accurate early on (~1,000 steps), yet overall output quality improves much later; (2) steering effectiveness does not scale with pre-training; and (3) larger activation magnitudes do not mean more effective steering.

Although their theoretical analysis and empirical observations appear technically sound, the paper offers little in terms of contributions or useful insights for the community. While they provide one possible explanation for the misalignment, their empirical observations do not provide evidence suggesting that this is what is happening under the hood. Again, the three findings, while all correct, do not appear to support the theory. Furthermore, after reading the paper, I was left with the message that there could be a misalignment between probe performance and steering effectiveness, which is well known in the community, but without any crucial insights on how to address it.

Critical strengths
  • The authors investigate an important problem for the steering community.
  • The main claims and observations of the paper appear technically sound.
Critical weaknesses

Major Weaknesses

  • The authors provide a possible explanation for the misalignment between steering effectiveness and probe quality, but do not present evidence supporting this explanation in their empirical study.
  • While the main claims appear sound, they do not provide insights for the community to address the misalignment and improve current steering methods.
  • It is not clear what the synthetic tasks are. No prompts, examples, or task construction details are provided anywhere in the paper. Therefore, it is not clear what those performance results actually entail. I believe this is a major limitation.

Minor Points

  • The checkpoint selection is rather limited. Have you considered OLMo or SmolLM checkpoints?
  • The paper is rather hard to follow, and much of the terminology could benefit from substantial refinement. "Dose," for example, is not a commonly used term for referring to steering strength. The usual terms are "strength," "steering multiplier," or "coefficient."
Questions for authors

Please see the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.