acceptodds
Withdrawn from ICLR 2027

One Concept, Two Facets: Rethinking the Readout-vs-Control Verdict in Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Linear probes, classifiers fit to a model's intermediate representations, decode high-level concepts from residual-stream activations with near-perfect accuracy, yet a decodable direction does not always steer generation. Prior work attributes this gap mainly to properties of the concept, but cannot explain why the same concept yields opposite verdicts under different extraction methods. We show that the readout/control verdict is jointly set by the concept's structure and the extraction pipeline, and distinguish two poles: a readout direction whose activation traces the surface tokens realizing a concept, and a control variable direction that shapes downstream generation independently of those tokens. To test this claim, we build a concept bank balanced by concept structure and develop a structure-adaptive protocol that derives the steering layer, budget and coherence threshold from the model under test, scoring effects against a norm-matched random-direction null across five independent model families. Concept structure is positively correlated with steerability in all five families on both splits, and the held-out correlation is significant in every family. Holding the concept and data fixed, selecting attention heads by separability recovers the readout facet while an unselected residual difference-of-means recovers the control facet. Alignment audits that read concepts off such directions may therefore miss what the model acts on.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, large language models, linear probing, representation engineering

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_qkCrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why a linearly decodable direction in the residual stream sometimes steers generation and sometimes does not. It argues the readout/control verdict is set jointly by (i) concept structure (generation-mode concepts such as tone/register vs. topic concepts) and (ii) the extraction pipeline (separability-gated head-level selection vs. an unselected residual difference-of-means).

Critical strengths
  • I think experiments part are good. Random baseline directions go through the exact same pipeline, two independent judges are used, and the constraint experiment needs no LLM judge at all. The same-subject comparison with token ablation (Table 1) is the cleanest evidence in the paper.
Critical weaknesses
  • The title claim, that the same concept flips from readout to control depending on the pipeline, gets one paragraph with no numbers, and that comparison changes both where the vector is injected and whether layers are selected. The main result, that style steers more easily than topic, is fairly expected, and it fades in the largest models (Qwen3-14B, Gemma-2-27B), which undercuts calling concept type a fundamental determinant.
  • The presentation are somewhat not that careful (figure 1/2, alg section). Tightening the presentation would improve paper's quality.
Questions for authors

N/A

Reviewer_Y8EZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to study why the linear probes to decoding the concept from the residual stream / representations of the model cannot often be used for steering the model for or against that concept/behavior. The paper claims that there are two variables, the concept's structure and extraction pipeline, that control whether a direction can steer a model's behavior.

Critical strengths

None.

Critical weaknesses
  • The method and experiments sections are poorly written and hard to understand the motivation of the method and interpretation of what the results mean
  • The main figures are all AI-generated (which is not a problem of its own), but illustrative of what the paper is doing and not helpful in understanding this paper in any way
  • Section 3.2 needs more context about where and how these diagnostic batteries are applied. Overall, the method section does not motivate why we would care about this and how the method answers the question that was raised by the authors in the introduction to understand why decodability doesn't mean steerability.
  • Section 3.3 and 3.5, they just cite the relevant dataset or models without explicitly listing out the dataset and models used in the paper. This lack of details and care in the paper makes it further hard to read.
  • In lines 317-319 for the experimental results, these use a different method from Li et al. 2023 instead of the method described in Section 3.1, so confused again what this paper is doing
Questions for authors

I hope the authors can answer the weaknesses raised above on top of this.

  • In line 144, "Let F and N be concept-framed and neutral continuations of the same prompts." What does it mean by neutral continuations?
Reviewer_Pg7BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks why some concept directions that are linearly decodable from an LLM’s residual stream can steer generation while others cannot. The authors distinguish a “readout” direction, which tracks the surface tokens realizing a concept, from a “control” direction, which shapes generation independently of those tokens, and argue that which one emerges depends jointly on concept structure (mode vs. topic) and the extraction pipeline. To test this, they build a type-balanced concept bank and a structure-adaptive steering protocol that sets the layer, budget, and coherence threshold per model and concept, scoring effects against norm-matched random-direction nulls. Experiments include case studies, a content-matched topic-vs-stance comparison, source-token ablation, and a five-family evaluation reporting a pooled correlation of r = +0.42 between concept type and steerability, with separability and probe accuracy showing no predictive signal once concept type is controlled for.

Critical strengths
  1. The main effect holds on held-out concepts with matched nulls (Section 3.4, Table 2). Random directions pass through the same selection and gating as concept directions, and the concept-type correlation replicates on a disjoint held-out split in all five families. This makes the core finding unlikely to be a tuning artifact.

  2. Geometric steerability predictors are shown to be confounded by concept type (Section 4.5, Table B.3). Once concept type is controlled for, separability and probe accuracy carry no signal (partial r = −0.03). Since prior predictors were evaluated on unbalanced banks, this is a useful corrective for the steering literature.

Critical weaknesses
  1. The headline claim lacks quantitative support (Section 4.2, "Two facets from one concept"). The title and first contribution state that the extraction pipeline alone flips the readout/control verdict for the same concept, but this is supported by a single paragraph with no numbers, tables, or specification of which concepts were tested. It also uses head-level ITI-style selection rather than the layer-level gate of Eq. 5. Most quantitative evidence instead fixes the pipeline and varies concept type, a dependence already noted by prior work (Bas & Novak, 2025; Arad et al., 2025). This matters because the paper's main novelty claim is not the one its experiments establish.

  2. The experimental evidence is fragmented and the claims outrun it (Sections 4.2–4.4, Table 2, Appendix B). The case studies predate the adaptive protocol and use fixed per-experiment settings that are not reported, and the random null differs across experiments (30 directions, 5 directions with sd floored at 0.5, 20 direction×seed samples). The abstract states that the correlation is positive in all families on both splits, yet the main-split CIs for OLMo and Mistral include zero, and the effect weakens at the largest models. This makes it hard to read the experiments as a single coherent test of the stated hypothesis.

Questions for authors
  1. For the "two facets from one concept" result (Section 4.2), which concepts were tested, and what are the swing, z-score against the random null, and probe accuracy for the head-selected vs. unselected directions? Does the same reversal occur with the layer-level gate of Eq. 5, or only with head-level selection?
  2. What fixed settings (layer, α, number of prompts) were used in the case studies of Sections 4.2–4.4? Do the frame-control and content-matched results (Table 1, Figure 3) still hold when rerun under the structure-adaptive protocol with a common random null?
  3. Beyond the observation that mode concepts steer better than topic concepts, what does this work establish that is not already implied by Bas & Novak (2025) and Arad et al. (2025)?
  4. Since the judge may score tone or stance shifts more easily than topic shifts, a judge-free or human-rated check on a subset of mode and topic concepts would help rule out judge sensitivity as a confound.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.