acceptodds
Under review as a conference paper at ICLR 2027

Expressed Interpretation Vectors in LLMs

Abstract

Large language models can often glean the intent of a user's question despite typographical errors. However, past work has shown that sufficiently corrupted prompts can induce a lack of attempted interpretation. In this work, we explore whether this behavior is linearly represented in the activation space of models. To this end, we construct a contrastive dataset of prompts by corrupting questions until models transition from interpretation to non-interpretation, extract candidate directions via difference-in-means, and perform selection by evaluating the behavioral effects of each one during steering. We find that across nine instruction-tuned models, projecting the direction out of the residual stream increases attempted interpretation by 39 to 70 percentage points on held-out corrupted TriviaQA questions, while adding it induces non-interpretation. The effects of these interventions transfer to SimpleQA, and in the largest model of each family, to word-level omission and OCR corruptions without re-extracting the direction. Notably, across six utility benchmarks, direction removal only changes scores by a mean absolute change of 0.74 percentage points, showing that general capabilities are left largely intact. We also demonstrate that the direction is separable from the refusal direction. Ultimately, our results suggest that whether a model attempts to interpret a degraded prompt is mediated by a low-rank subspace: several directions within it restore interpretation, but only directions built from our interpretation contrast reliably suppress it.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation steering, difference-in-means

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XQpPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose to extract directions corresponding to the transition from interpretation to non-interpretation of increasingly corrupted text inputs in LLMs. They construct a contrastive dataset by progressively corrupting inputs and identifying transition pairs between interpretation and non-interpretation. They then extract directions using the difference-in-means method and validate the effects of adding or removing these directions. They thoroughly investigate generalization across datasets and corruption mechanisms, and compare the effects with random and refusal directions. Experiments are conducted on several models from different families and sizes.

Critical strengths
  • Interesting and thorough experiments, including generalization across datasets and corruption mechanisms.

  • Informative comparisons with random and refusal directions, as well as evaluations showing that general benchmark performance is largely preserved.

  • Clever construction of the contrastive dataset.

Critical weaknesses

The main weakness is the limited methodological novelty. Similar difference-in-means approaches have already been used to extract steering directions for behaviors such as refusal and sycophancy, as the authors acknowledge. The contribution is therefore primarily empirical.

Questions for authors
  • Since the interventions are applied throughout generation, could there be a compounding effect that causes outputs to drift over longer generations? Have the authors investigated the effect of generation length on output quality?

  • I found the effects of random directions in Table 2 somewhat surprising. Adding and removing random directions generally shift behavior in the intended direction (toward non-interpretation and interpretation, respectively). Do the authors have an intuition of why ?

  • In Section 3.2, the authors compute activation differences at matched token positions. Could they clarify whether the last five positions always correspond to fixed chat-template tokens? Otherwise, corruption-induced changes in tokenization could misalign the positions, potentially resulting in differences between representations of semantically different tokens.

Reviewer_UAYQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs' tendency to attempt interpreting corrupted prompts is represented by a linear direction in their residual-stream activations. The authors construct model-specific contrastive datasets by progressively corrupting TriviaQA questions until models transition from interpretation to non-interpretation. They then extract candidate directions using difference-in-means and select them through bidirectional activation steering.

Experiments across nine instruction-tuned models from three families show that projecting out the selected directions increases attempted interpretation by 39.0–69.8 percentage points on held-out transition endpoints, while adding these directions suppresses interpretation. The effects transfer to SimpleQA and, for the largest models of each family, to word-omission and OCR corruptions. Additional analyses compare the extracted directions with random and refusal directions, investigate their low-rank structure, and assess effects on downstream utility and factual correctness.

Critical strengths
  1. Consistent empirical results across models and corruption settings.

The paper provides substantial empirical evidence across nine models from three different families. The bidirectional steering effects are consistent, and the transfer experiments on SimpleQA and additional corruption types strengthen the claim that the directions capture a behavior beyond the specific typographical perturbations used for extraction. Importantly, these transfer experiments reuse the original directions and intervention settings without refitting.

  1. Careful analysis of intervention specificity and downstream effects.

The comparisons against norm-matched random directions, refusal directions, and alternative activation subspaces are valuable. The paper also examines whether the intervention affects factual correctness, safety refusal, and standard benchmark performance. These analyses make the empirical findings more informative than a straightforward application of difference-in-means steering.

Critical weaknesses
  1. The distinction between expressed interpretation and genuine semantic interpretation remains insufficiently established.

The primary evaluation measures whether a model attempts an interpretation, rather than whether it correctly recovers the intended meaning. As reported in Section 5, although direction removal increases attempted interpretation by 59.9 percentage points in the aggregated analysis, correct question reconstruction increases by only 10.9 points (from 5.0% to 15.9%), and correct answers increase by only 7.2 points (from 1.3% to 8.4%).

The authors appropriately acknowledge this distinction. Nevertheless, the results leave open whether the extracted direction specifically captures interpretation or more general tendencies toward answer commitment, guessing, or reduced abstention. The partial overlap with refusal-related behavior reinforces this concern. This limits the mechanistic interpretation of the extracted feature, even though the behavioral steering effects are convincing.

  1. The evaluation is heavily concentrated on model-specific interpretation boundaries.

The main evaluation uses transition endpoints identified by progressively corrupting questions until each model stops attempting interpretation. These examples are deliberately selected near a behavioral decision boundary, where relatively small activation interventions may cause large changes in the measured outcome.

While this is a reasonable experimental design for studying causal steering, the reported effect sizes may not generalize to corrupted inputs outside these selected boundaries. Indeed, Section 5 reports substantially smaller improvements of 2.8–21.8 percentage points on corruption depths six through eight in seven of nine models, compared with the headline improvements of 39.0–69.8 points on transition endpoints.

This distinction limits how broadly the results support the claim that the identified direction governs expressed interpretation across degraded inputs.

Questions for authors

Q1. Generalization beyond selected transition boundaries.

How much of the reported intervention effect persists on corrupted questions sampled independently of the model's interpretation/non-interpretation transition? In particular, could the authors report attempted interpretation, correct question reconstruction, and answer correctness across different corruption severities without conditioning on transition endpoints? This would help determine whether the large reported effects primarily reflect sensitivity near a decision boundary.

Q2. Specificity relative to abstention and guessing.

Can the authors provide additional evidence distinguishing the extracted interpretation direction from a more general willingness-to-answer or abstention-related direction? In particular, how much of the addition and removal effects remains after removing the refusal-aligned component, and does the intervention improve correct interpretation rather than merely increasing speculative responses? Clarifying this distinction would substantially strengthen the claimed mechanistic interpretation.

Reviewer_pCVeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLMs can often understand user's intent despite errors and typos in the prompts. The paper studies whether we can extract a representation for interpretation vs non-interpretation of user's queries. They gradually corrupt questions in datasets such as TriviaQA and extract the interpretation vector. They use difference-in-mean steering to improve interpretation of prompts or suppress it. They show that across nine instruction-tuned models, projecting out the extracted vector increases attempted interpretation by 39 to 70 percentage points on held-out corrupted TriviaQA questions. Applying this steering does not degrade capabilities performance on EleutherAI’s lm-evaluation-harness that contains datasets such as MMLU, HellaSwag, ARC Challenge, TruthfulQA, WinoGrande and GSK8K. They also show transferability of this steering vector to datasets such as SimpleQA. They show that this direction is independent of the refusal direction.

Critical strengths
  1. Experiment setup: The paper shows that interpretation direction is linearly represented and models can be steered to increase or reduce interpretation of prompts. They have strong baselines: norm-matched random directions, refusal direction, unrelated contrast direction, principal components) to show that the interpretation is specific to this direction rather than the other tested perturbations.

  2. Generalization: They are able to show transferability by testing on SimpleQA without re-extraction. They are also able to show improvements on different corruption types not included during steering. They additionally show that this steering affects general capabilities benchmark performance by only 0.74 points.

Critical weaknesses
  1. Separability from refusal: The paper claims “We also demonstrate that the direction is separable from the refusal direction”. However the analysis suggests that there is some overlap between refusal and “interpretation” directions. Section 5 shows that adding interpretation direction to harmless instructions makes the model refuse ~25% of the time in Llama models, compared to 0% in baseline. The result also doesn’t hold on Qwen3 1.7B, where projecting out the interpretation direction lowers refusal from 51% to 4% on Jailbreakbench. The separability claim in Abstract and Contributions section should be revised based on the Results. Additionally, Wollschläger et al. (2025) show that refusal is mediated by multiple directions. For the separability from refusal claim, it would be better to test against the refusal subspace instead of a single direction.

  2. Differentiation between non-interpretation and “refusal” in the model evals: The paper has two different evaluation settings for interpretation vs refusal: LLM judge to check if model attempted to interpret the answer vs substring match for refusal. There could be overlap between these two and neither of the eval metrics accounts for that. There is no clear distinction between whether the model understands and refuses or just doesn’t understand the question. This could also explain why adding the interpretation direction raises refusal on harmless instructions.

Questions for authors
  1. The Appendix seems to be missing from the submission. Additionally, there are multiple tables references in the paper that are missing: Table 4 referenced in Section 4, Table 6, 16, 24 referenced in Section 5.
  2. What were the alpha values for refusal direction and interpretation direction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.