- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether information that can be accurately extracted from LLM activations using linear probes can also be used to steer model behavior. The authors introduce a controlled counterfactual recomposition task using ALFWorld and WebShop trajectories, along with a learned reference steering subspace that demonstrates successful interventions are possible. Across four models, they find that probing accuracy remains nearly perfect even when probe-derived steering directions fail to produce the desired counterfactual answers.
Critical strengths
Careful experimental design: The counterfactual recomposition setup distinguishes genuine recomputation from simply copying an answer or source plan. Using a learned reference steering subspace as a positive control makes the comparison more convincing by showing that successful steering is achievable at the same intervention location.
Consistent experimental evidence: The results are consistent across four models and two environments, with additional tests involving covariance transformations and multiple probe directions. The finding that highly readable information can still fail to support effective probe-derived steering is relevant to activation-based interpretability methods.
Critical weaknesses
Limited interpretation of the read-write gap: The reference subspace is explicitly optimized for successful steering, while the probe-derived directions are trained only to predict states. Since the complement space also excludes the learned reference directions, the results do not establish whether other steering methods could successfully use the remaining readable information.
Limited task generalization: The experiments use simplified decision rules, four action categories, and single-digit answers rather than naturalistic agent behavior. It remains unclear whether the same findings hold for more complex reasoning tasks, longer interactions, or free-form generation.
Questions for authors
Could the authors test whether a separately optimized steering subspace within the complement space can achieve successful counterfactual recomposition? This would help clarify whether the failure comes from the information being unsuitable for steering or from the specific probe-derived methods.
How well do the findings generalize beyond the simplified decision tasks, particularly to settings involving multi-step reasoning or free-form responses?