acceptodds
Under review as a conference paper at ICLR 2027

Linear Readability Does Not Guarantee Writability: Evidence from Controlled Counterfactual Recomposition

Abstract

Linear probing reveals information encoded in large language models, but whether probe-derived directions can reliably steer model decisions remains unclear. To investigate this question, we carefully design a controlled experiment on tasks constructed from ALFWorld and WebShop trajectories, whose action lists allow us to define explicit state changes and determine the corresponding answers. In the experiments, we combine the source input's queried state with the target input's decision rule and answer mapping, which we term counterfactual recomposition, to test whether steering supports the intended recomputation rather than merely copying an answer or source plan. Simultaneously, we learn and validate a reference steering subspace to ensure that steering can produce the required counterfactual answer at the selected location. This reference enables a controlled comparison with probe-derived steering in the complement space under fixed intervention conditions. Across four models and both environments, probing accuracy remains near-perfect in the complement space, while steering with direct probe directions, covariance-transformed directions, or combinations of multiple probe directions achieves near-zero counterfactual recomposition accuracy. These results reveal a critical finding: high readability alone does not guarantee writability through probe-derived interventions even with effective steering available, which provides valuable insight for developing reliable steering methods.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Linear Probing, Activation Steering, Counterfactual Recomposition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1jywOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether information that can be accurately extracted from LLM activations using linear probes can also be used to steer model behavior. The authors introduce a controlled counterfactual recomposition task using ALFWorld and WebShop trajectories, along with a learned reference steering subspace that demonstrates successful interventions are possible. Across four models, they find that probing accuracy remains nearly perfect even when probe-derived steering directions fail to produce the desired counterfactual answers.

Critical strengths

Careful experimental design: The counterfactual recomposition setup distinguishes genuine recomputation from simply copying an answer or source plan. Using a learned reference steering subspace as a positive control makes the comparison more convincing by showing that successful steering is achievable at the same intervention location.

Consistent experimental evidence: The results are consistent across four models and two environments, with additional tests involving covariance transformations and multiple probe directions. The finding that highly readable information can still fail to support effective probe-derived steering is relevant to activation-based interpretability methods.

Critical weaknesses

Limited interpretation of the read-write gap: The reference subspace is explicitly optimized for successful steering, while the probe-derived directions are trained only to predict states. Since the complement space also excludes the learned reference directions, the results do not establish whether other steering methods could successfully use the remaining readable information.

Limited task generalization: The experiments use simplified decision rules, four action categories, and single-digit answers rather than naturalistic agent behavior. It remains unclear whether the same findings hold for more complex reasoning tasks, longer interactions, or free-form generation.

Questions for authors

Could the authors test whether a separately optimized steering subspace within the complement space can achieve successful counterfactual recomposition? This would help clarify whether the failure comes from the information being unsuitable for steering or from the specific probe-derived methods.

How well do the findings generalize beyond the simplified decision tasks, particularly to settings involving multi-step reasoning or free-form responses?

Reviewer_tPLcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether information that can be read from LLM representations through linear probing can also be used to steer model behavior.

Critical strengths
  1. This paper investigates whether readable information can actually be used to control model behavior, which is an important problem.
  2. The counterfactual recomposition setting is reasonable, which helps rule out simply copying the original answer or source plan.
Critical weaknesses
  1. Similar investigations have been explored in previous papers [1, 2], for example, CausalGym[1] also discovers that features found by linear probing are not the most effective steering ones compared to DAS. Another recent work [2] also indicates that “read best is not steer best”. Thus, the core contribution of this work is more to confirm this phenomenon, which is somewhat incremental novelty. Please add some discussion of these similar studies, which could strengthen this paper.
  2. While the experiment results are promising on two datasets, could you add one more dataset, such as ScienceWorld or CausalGym? If the results are still consistent, I would like to raise my score.

References: [1] Arora et al., CausalGym: Benchmarking causal interpretability methods on linguistic tasks, ACL 2024 [2] Wang et al., Read-Best Is Not Steer-Best: A Probing--Steering Layer Dissociation in Omni-Modal Large Language Models, Arxiv 2026

Questions for authors

N/A

Reviewer_gb2kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This is an experimental paper that shows that, even if steering vectors exist, linear probes do not find them.

Critical strengths

The authors put in a lot of effort to ensure that the experimental set-up is proper, and the maths that describes the process also appears sound.

Critical weaknesses

The main critical weakness for me about this submission is that the finding is not new. It has been known since linear probes were first devised that they did not reflect anything about what happens under interventions, by their very nature. Steerability is an entirely different property, and it should be clear with a rudimentary understanding on causality. The authors themselves cite a few papers that demonstrate this. Unless I'm missing something, this submission does not report anything that the community did not know about.

The mathematics is thorough but entirely consists of listing the definitions, rather than proving any results.

Questions for authors

No questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.