acceptodds
Under review as a conference paper at ICLR 2027

Capacity, Responsiveness and Alignment: What Makes a Latent Structure Actionable

Abstract

Localizing latent structures in the activation space of language models (LMs) is central to understanding and controlling their behavior. Yet, localized structures can differ substantially in their causal influence, raising the question of what makes a structure actionable. We tackle this question by casting causal influence as a product of three factors and showing empirically that they act as interpretable, distinct constraints: capacity, measuring the sensitivity of the model's output to movement along the structure, responsiveness, capturing how promotable the concept is given the current context, and alignment, reflecting how well the structure aligns with the context-specific representation of the concept. Across 4 LM families and 50 concepts, we observe that causal effectiveness requires all factors to be high; low capacity and responsiveness reduce it by 84% and 95%, respectively, while low alignment can reverse it, suppressing concept expression. Moreover, we find that causality is context-dependent rather than an intrinsic property of the structure, with causally effective directions forming a low-dimensional subspace that varies across contexts. By restricting the training of linear probes to this subspace, we introduce causal probes that achieve 17%-118% improvement in steering across models, with only 3% reduction in concept detection.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation steering, causal interventions, representation geometry, latent representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_D9fpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what determines the causal effectiveness of latent structures in language model representations. The authors decompose the first-order causal influence of an intervention into three factors: capacity, responsiveness, and alignment. Experiments across four language models and 50 concepts suggest that these factors jointly constrain intervention effectiveness and that effective steering directions are context-dependent. Building on these findings, the authors introduce Causal Probes, which restrict linear probes to high-capacity subspaces to improve steering performance while largely preserving concept detection accuracy.

Critical strengths

1.The paper provides an interpretable framework that decomposes causal influence into capacity, responsiveness, and alignment. The empirical analysis across models, layers, and contexts offers a unified perspective on the gap between concept detection and behavioral control.

2.The paper translates its theoretical findings into a practical method through Causal Probes. The proposed approach improves steering performance over standard linear probes while introducing only a modest reduction in concept detection accuracy, demonstrating the practical relevance of the analysis.

Critical weaknesses

1.The concept-expression metric is constructed from tokens appearing in positive examples but not in neutral examples. Such an occurrence-based definition may not faithfully capture complex semantic concepts and could be sensitive to sampling, token frequency, and tokenization. Since this metric directly underlies the three-factor analysis, its validity is critical to the paper's central conclusions.

2.Although Causal Probes substantially outperform standard linear probes, Figure 6 shows that their performance is often comparable to DiffMeans, leaving their practical advantage over strong existing baselines less conclusive. Moreover, constructing high-capacity subspaces introduces additional Jacobian-related computation without a systematic cost comparison. The reliance on synthetic concept examples and a limited number of generation prefixes also restricts conclusions about broader applicability.

Questions for authors

1.The concept covector is constructed from tokens appearing in positive but not neutral examples. How do the authors ensure that this representation captures the intended semantic concept rather than dataset-specific lexical or frequency differences? Do the three factors and the main conclusions remain stable under alternative concept representations?

2.Since the three factors arise from the covariance–correlation identity, a small factor necessarily constrains their product. To what extent do the empirical results establish distinct causal mechanisms beyond this algebraic relationship? Could the authors clarify the main conceptual novelty relative to existing Jacobian- and Fisher-geometry-based analyses?

3.The three-factor analysis relies on a first-order approximation, whereas the steering experiments consider a wide range of intervention strengths. Although the predicted and observed KL divergences are compared, this does not directly establish the accuracy of predicted concept-expression changes. Could the authors evaluate how well the approximation predicts the sign and magnitude of concept-expression changes under finite interventions?

4.When computing Jacobians for future positions, the authors fix intermediate tokens through teacher forcing. However, interventions during free-running autoregressive generation may alter earlier tokens and subsequently change the context and hidden states. How do the authors verify that the causal influence estimated under teacher forcing accurately reflects the cumulative effects observed during actual generation?

2 comments

We thank the reviewer for their thoughtful feedback. We appreciate their recognition of our framework in clarifying the gap between concept detection and control, as well as the practical contribution of causal probes. Below we address the concerns, additional results are at https://anonymous.4open.science/r/iclr2027-8D58/ and will be incorporated into the paper before discussion ends.

W1 + Q1. Concept Expression Metric

While the token-based covector may not capture the full semantics of complex concepts, our goal is not to introduce a new method for representing concepts, but to understand the conditions under which their expression can be causally influenced. Importantly, the decomposition holds for any construction of an output covector.

We have now run a new analysis to show robustness to sampling and dataset choice: We conducted a five-fold heldout experiment, training DiffMeans on four folds and constructing the covector from the fifth and measuring the directions effect on the covector. The localized direction produced 4.2× greater promotion of held-out concept tokens than unrelated directions across all settings, demonstrating robustness even for tokens missing from the training covector.

W2. Causal Probe Performance

Our goal with causal probes was to show the theoretical insights yield practical gains, rather than achieving state-of-the-art steering. Linear probes are used widely in both research and industry. Still, they have a well-known gap between concept detection and causal influence. Our experiments show significant progress towards minimizing this gap, which demonstrates the utility of our findings. Regarding computational cost, the high-capacity subspace is constructed offline. At each training position, we apply JFJ to K random sketch vectors using jacobian-vector products, then keep the top d directions. This avoids producing the full Jacobian, and the cost scales linearly with training position count. Once the probe is trained, intervention-time cost is unchanged.

Q2. Factors as Distinct Causal Mechanisms

Despite the factors arising from an identity, our contribution is identifying and empirically showing distinct constraints on causal effectiveness: capacity is concentrated in low-dimensional subspaces, responsiveness constrains promoteable outputs, and alignment varies substantially across contexts. We further show causal applications in Section 5, greatly improving causality of probes.

Prior Jacobian and Fisher based papers design intervention/steering methods or more recently identify intermediate variables. In contrast, this paper studies the gap between concept detection and causality: We explain why the causal effectiveness of a given latent structure varies across contexts and identify the constraints governing this variation. This work highlights problems with a lot of common assumptions made today regarding both localization and steering, especially that identifying a global concept direction is sufficient for reliable causal control across contexts.

Q3. Approximation of Concept Expression

We extended our existing quality of estimation analysis of the first-order approximation. We conducted a direct comparison between the product of the three factors and the observed finite change in concept expression across ten intervention strengths.

ModelMaximum rMedian r (IQR)
Gemma0.880.48 (0.41–0.52)
Llama0.750.72 (0.66–0.74)
Qwen0.810.39 (0.35–0.42)
Mistral0.800.78 (0.77–0.80)

The results support that the approximation remains accurate at finite interventions with its accuracy eventually deteriorating for sufficiently large interventions. Full results are at the anonymous link.

Q4. Teacher Forcing

In Sections 3 & 4, we use teacher forcing to study the intervention’s effect on the model's downstream computation following input processing rather than approximate effect on generation. Additionally, the paper contains two experiments that support teacher-forcing formulation:

Section 5 relies on teacher-forced Jacobians to construct the high-capacity subspaces and evaluated on generation. Its significant improvement shows that the approximated subspaces improve steering beyond the teacher-forced setting. Thus, while teacher forcing is an approximation, our practical results demonstrate the framework remains useful during actual generation.

In Appendix E.3 we use the teacher-forced formulation to identify positions with the highest predicted causal effect for steering. In all settings, intervening at these positions outperforms all the controls. The results demonstrate the teacher-forced approximation provides meaningful predictions of successful steering.

We thank the reviewer again for the engaging discussion and feedback. We would be happy to discuss any remaining points or answer any additional questions.

Thank you again for your helpful feedback on our work. As the discussion period is ending, we would greatly appreciate it if you could take a moment to review our response. Many thanks,

Authors

Reviewer_DPD7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the causality of activation-level interventions through the first-order change in concept relevant portion of output distribution along the existing line of work using Jacobian-based analysis. The authors study the mechanisms behind causal effects by decomposing the first-order change into 3 interpretable factors. Across 4 LLM families, the interpretations are observed to hold consistently.

Critical strengths
  1. The claims on the causal effects of each of three factors are backed well by empirical evidence, going beyond speculative hand-wavy interpretation of, say, some terms in their decomposition.
  2. The problem of the study is of increasing relevance in the field: understanding what and when we gain something usable from the observed concept-relevant structures.
Critical weaknesses
  1. I think the summarizing claim "interventions have limited ability to promote outputs with little support in the current context" (lines 290-292) needs a more careful statement. To be more precise, some concepts can be unlikely under any (non-adversarial) context such as safety/harmfulness as models can be specifically safety-tuned. But activation steering-based jailbreaking has been shown to be largely feasible. Do authors mean such scenarios has to hurt fluency significantly to achieve a notable causal effect? If yes, it would be useful to isolate such concept and separately report the analysis as in Figure 3.
  2. The Introduction section starts with listing some examples of latent concept-relevant structures as directions, subspaces, and manifolds and frame their study in terms of a generic sturture S\mathcal{S}. However, the study apparently only considers directions (a.k.a. linear structures) as vv is said to be intervention along S\mathcal{S}, creating some friction in understanding the scope of the study. Can authors clarify the object S\mathcal{S} and intervention along this object in the framework?
Questions for authors

See weaknesses.

3 comments

We thank the reviewer for the careful reading and thoughtful feedback. We appreciate that they found the claims to be backed well by the empirical study and that the research question is increasingly important. We address the weaknesses below with full results for the additional analysis at: https://anonymous.4open.science/r/iclr2027-8D58/ , which we will add to the submission PDF before the discussion period ends.

W1. Responsiveness Experiment

Our intention was not to claim that every intervention “has to” sacrifice fluency to promote low-probability outputs. Rather, we focused on the “natural” behavior of the model in non-adversarial settings, where a localized direction is applied across contexts without optimizing for specific individual intervention sites. Targeted interventions, such as jailbreaking, may exploit specific mechanisms to overcome this limitation. Our claim is instead that fluency-preserving interventions mainly redistribute probability mass among already supported tokens.

We have now further tested this by asking how often concept tokens that started outside the top 1%, 2.5%, or 5% moved into these ranges after intervention. While responses remained fluent, only 2.44%, 4.17%, and 5.89% crossed these thresholds, and they contributed just 0.91%, 0.46%, and 0.24% of the resulting concept mass. Once fluency fell below 0.8, their mass contributions increased to 12.52%, 8.63%, and 5.98%. Thus, concept tokens rarely move into the top predictions while preserving fluency, and they gain substantial mass mainly after fluency deteriorates. The full plots showing these trends are available in the anonymous link.

W2. Intervention Along a Structure

While localization methods may identify directions, subspaces, or more complex structures [1–5], our analysis concerns the resulting change to the residual-stream activation. Any intervention modifying the activation at a single site, which is standard practice in activation intervention or steering methods, can be represented as an additive displacement, regardless of how it is constructed. Namely, at a given activation $h$, any intervention producing a modified activation h' can be represented by the displacement h'-h, regardless of the underlying latent structure that produced the displacement. Our framework therefore does not assume that the underlying structure or intervention mechanism is linear, but instead characterizes the local causal effect of the resulting displacement.

We have now clarified this distinction in the paper, revising line 99 and a footnote to make it explicit:

Line 99:

To quantify the causal effect of a latent structure S on c, we consider how intervening along S changes concept expression. We use additive interventions since modifying the activation at a single layer can be represented as an additive displacement, regardless of how it is constructed. For an intervention vector v along S at h, we obtain a distribution change p=f(h+v)-f(h), where f() denotes the remaining computation from $\mathbf h$ to the output.

And with a footnote:

At a given activation $h$, any intervention producing a modified activation h' can be represented by the displacement h'-h, regardless of the underlying latent structure or intervention method that produced the displacement.

[1] ReFT: Representation Finetuning for Language Models
[2] The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence
[3] From Directions to Regions: Decomposing Activations in Language Models via Local Geometry
[4] Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution
[5] Contrastive Covariance Subspaces for Activation Steering

We thank the reviewer again for the engaging discussion. We would be happy to discuss any remaining points or answer any additional questions.

Thanks for your rebuttal and responses to my questions. The concerns were already minor and authors address them adequately. Thus, I maintain my positive score of 3.

Thank you for your positive assessment and for taking the time to consider our responses. We're glad that our clarifications adequately addressed your concerns.

Reviewer_NuVMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why concept directions found by probing or DiffMeans often steer weakly or inconsistently. Pulling the effect of an intervention vv back through the model's Jacobian gives its first-order effect on concept-token mass. This is composed as a product of capacity, responsiveness, and alignment. Capacity is how much the output can move along vv, responsiveness is how much concept mass there is to move, and alignment is whether the move points toward the concept. On four 7-9B models and 50 concepts, the authors reports three findings. Capacity is concentrated in about 23-47 effective directions. Fluent interventions mostly reshuffle tokens that were already likely. The best direction for a concept differs a lot between contexts. Bucketing samples by factor quartiles shows that a low value of any factor shrinks steering, and low alignment reverses it. As an application, "causal probes" restricted to the top capacity subspace steer better than plain linear probes, at a cost of about 3 points of detection accuracy.

Critical strengths
  1. Exact first-order decomposition of how an activation space intervention changes concept expression: this gives a precise vocabulary for why a well-detected direction can fail to steer in a given context (e.g., little downstream sensitivity, little baseline concept mass, misorientation). I found this very informative and insightful.
  2. Evidence that the direction that best promotes a concept is context-specific: a locally optimal direction moved to another context keeps only about 10%-40% of that context's own alignment, and less in early layers. This explains why a fixed steering vector varies across inputs.
Critical weaknesses
  1. The main claim that factors "act as interpretable, distinct constraints" (abstract) is only a first-order approximation, the data shows that the factors co-vary. α\alpha and the fig5 normalization are not reported so one cannot tell how far the finite α\alpha effect departs from the first-order product.
  2. The responsiveness experiment does not test factor II as defined (section 3.3): σp(rc)\sigma_p(r_c) depends on the baseline mass of the concept tokens, but Eq. 9 measures where all positive probability change goes across the whole vocabulary. rcr_c does not appear in it.
Questions for authors
  1. Could you report pairwise and partial correlations between the three factors, since fig10 suggests strong co-variation, and a joint regression of the effect on all three?
  2. In Sec. 4, which α\alpha was used for the DiffMeans intervention, and how is "normalized concept expression" normalized and aggregated across models and layers? How well does the first-order approximation predict the observed finite α\alpha change?
  3. Do causal probes still beat probes constrained to a random rank-128 subspace, or to the top-128 PCA subspace of the training activations? Appendix F3 reports all ranks: is r=128 in Fig. 6 simply the largest rank tested, or was it selected on validation prefixes?
  4. How large is VcV_c per concept and tokenizer, and how much do the concept sets overlap, given that they are built from only 304 neutral completions (Appendix A1 and A2)?
  5. The capacity subspace is described as context-dependent (Sec. 3.2), yet causal probes average GG over contexts (Appendix F1). How much do the top eigenvectors of GhG_h overlap across contexts?
  6. How do the cross-model results change if α\alpha is scaled by each layer's residual-stream norm? This covers Gemma's confinement to the baseline support in Sec. 3.3 and its near-zero late-layer steering in fig6. Fig8 shows the same α\alpha grid gives KL values about 1000x apart between Gemma and Mistral.
5 comments

We apologize for the long response, we wished to fully answer all questions posed and greatly appreciate the effort in reviewing our work and the thoughtful and constructive feedback. We especially appreciate the reviewer’s recognition that the findings are informative and insightful, highlighting the limitation of current methods to account for casualty’s context dependence. We address and provide additional results for the questions and weaknesses below with full results at: https://anonymous.4open.science/r/iclr2027-8D58/ . We will include all the new results in the camera-ready version and will update the PDF by the end of the discussion phase.

W1. Distinct Constraints

In our main claim, we write "act as interpretable, distinct constraints", however our intention is not to claim that they are statistically independent but to show they capture different constraints and can vary independently (whether they are high or low). Still, we have conducted the analyses suggested in Q1 and found that the pairwise and partial correlations between the factors are near 0 (see our response to Q1), providing further support for the intended claim.

As for the difference between the finite effect and the first-order product we show that the two have moderate positive spearman and pearson correlations – see full results in our response to Q2.

W2. Responsiveness Experiment

We thank the reviewer for the suggestion! We repeated the experiment while measuring whether concept tokens selected by the concept covector move from outside to support in the top 1%, 2.5%, or 5% of the vocabulary. Under a fluency threshold of 0.8, only 2.44%, 4.17%, and 5.89% of concept tokens moved into the top 1%, 2.5%, and 5% vocabulary supports, accounting for just 0.91%, 0.46%, and 0.24% of the concept probability mass inside each respective range; nearly all remaining mass belonged to tokens not related to the concept that were already present in the support before intervention.

For non-fluent outputs (resulting from stronger interventions), the crossing rates increased to 8.49%, 12.93%, and 17.15% for the same three ranges. More importantly, the newly entering tokens now accounted for 12.52%, 8.63%, and 5.98% of the concept probability mass. Thus, interventions introduce substantial probability mass from previously unlikely concept tokens mainly when fluency deteriorates. Full plots showing these trends are available in the anonymous link.

Q1. Factor correlations and joint regression

We measured pairwise and partial correlations between the three factors. Pearson correlations are 0.083 (Responsiveness–Capacity, 95% CI [0.080, 0.086]), 0.082 (Responsiveness–Alignment, [0.074, 0.091]), and approximately 0 (Capacity–Alignment, [−0.002, 0.002]). Partial correlations are similar: 0.083, 0.083, and −0.007, respectively. This suggests that the factors are only very weakly correlated.

We also performed fivefold held-out regression to predict actual finite intervention effects. For interventions that reduce fluency to 0.8, the joint additive model achieves (R^2=0.112), compared with 0.004, 0.05, and 0.07 for Responsiveness, Capacity, and Alignment on their own. Using their product improves this to (R^2=0.322). The results suggest the factors capture different information and their product is more informative than individually.

Q2. Intervention strength, normalization, and finite effects

The reviewer requested a clarification on the normalization procedure in Section 4. We median-aggregate concept-expression changes over contexts and concepts, normalize within each model and source offset by the maximum absolute value across layers and factor buckets, and then average equally across models. Per-model results are in the appendix.

Following the reviewer’s suggestion, we extended our existing quality of estimation analysis of the first-order approximation beyond KL. We conducted a direct comparison between the product of the three factors and the observed finite change in concept expression across ten intervention strengths.

ModelAvg. Max r across layersMedian r across layers (IQR)
Gemma0.880.48 (0.41–0.52)
Llama0.750.72 (0.66–0.74)
Qwen0.810.39 (0.35–0.42)
Mistral0.800.78 (0.77–0.80)

The results support that the approximation remains accurate at finite interventions with its accuracy eventually deteriorating for sufficiently large interventions. Full results are at the anonymous link.

Q3. Random Subspace Control For Causal Probe

Following the reviewer’s suggestion, we now include a control consisting of a random subspace of equal rank. Since this analysis is computationally heavy, we include here results for Gemma-2-9B and Mistral-7B at a layer ratio of 0.2 and 0.6 and will include the rest of the settings in the final draft before discussion ends. The trends observed are consistent: random subspaces are the weakest method across all settings, significantly underperforming regular probes. Full results are in the anonymous link.

Regarding the selected rank, we chose to include rank 128 as it is the largest rank while keeping the rest in the appendix for full transparency.

Q4. Concept covector construction

We examined the size and overlap of the concept covector across all 50 concepts and the four tokenizers of the different models. Median size ranges from 351 to 377 tokens or 0.137%–1.178% of the vocabularies. The median pairwise Jaccard similarity ranges from 0.064 to 0.072. We would be happy to measure any additional statistics the reviewer wishes to see.

Q5. Capacity-subspace overlap

We measured how much the high-capacity subspaces overlap across contexts (mean squared cosine of the principal angles). At ranks 8, 16, and 32, the median overlaps are 0.019, 0.032, and 0.050. These are about 6 to 9 times higher than random subspace overlap. We also compared local subspaces with the concept-averaged subspaces used to restrict the causal probes, finding overlaps of 0.067, 0.085, and 0.099.

These results further support our finding that causality is context-dependent, however, high-capacity subspaces share some structure, which improved the causality of probes.

Q6. Matched Residual Stream Norms

Our experiments cover a broad range of 11-14 intervention strengths between 0.3 and 128. In all models and layers, at the lower range of intervention strength the causal effect is minimal, and at the upper range the fluency falls to below 0.5. Thus, the sweep covers the behaviorally important range across all settings. None of the experiments compare across models, rather, each is compared at all coefficients to account for the varying norm. We would appreciate clarification regarding the question if our answer was insufficient.

Thank you for the careful and extensive response. My main concerns are addressed well. The near-zero pairwise and partial factor correlations, together with the product model’s higher R2R^2 (0.32 vs. 0.11 additive), support the distinct-constraints claim. The responsiveness experiment re-done on concept tokens directly tests the factor as defined. The random-subspace control, the normalization details, and the co-vector and overlap statistics are all helpful.

A few points remain: the first-order approximation is only moderately accurate for Gemma and Qwen (median r≈0.4r \approx 0.4), so “remains accurate” should be qualified, the top-128 PCA control was not run, and on Q6, I was asking whether effects reported against α/∥h∥\alpha / \|h\| would show that Gemma’s confinement to the baseline support is a scale artifact.

Please also include the new results in the revised PDF. Given these improvements, I am raising my score from 2 to 3.

Comments on the paper

1 comment

We would like to thank all three reviewers again for their careful reading and constructive feedback. We have now updated the submission PDF to include the additional experiments and analyses conducted during the rebuttal, directly addressing the main concerns raised.

The main additions are:

  1. The three factors capture distinct and complementary constraints. We measured pairwise and partial correlations, finding very weak dependence between the factors. Fivefold held-out regressions further show that their product predicts finite intervention effects substantially better than individual factors or an additive model (R^2=0.322 vs. 0.112), supporting their complementary roles beyond the algebraic decomposition. (Section 4, Appendix E.4)
  2. First-order predictions are informative over finite intervention regimes. We compared predicted and observed concept-expression changes across ten intervention strengths and four models. We identify finite ranges of intervention strengths where predictions correlate strongly with observed effects (mean peak Pearson r=0.75-0.87 across models), while predictive accuracy varies with intervention strength and deteriorates outside these regimes. These results demonstrate the predictive utility of the first-order approximation while clarifying its limitations. (Appendix E.5)
  3. Responsiveness limits the promotion of previously unsupported concept tokens. We measured whether concept tokens outside the baseline top vocabulary predictions enter this support following intervention. These tokens contribute very little probability mass under fluency-preserving interventions, with larger contributions as fluency deteriorates. This further supports our finding that interventions predominantly promote outputs already supported by the context. (Section 3.3, Appendix C.2)
  4. Held-out validation supports the robustness of concept readouts. To test whether our token-based readouts capture concept expression beyond the data used to construct them, we conducted a fivefold experiment, learning intervention directions on training data and constructing concept readouts independently from held-out examples. At matched KL, the learned directions promote held-out concept tokens 4.2× more than unrelated directions (2.8× even for tokens absent from training readouts), supporting the validity of our concept measure. We also report readout sizes and overlap statistics. (Appendix A.2)
  5. Capacity subspaces and causal probes receive additional validation. We measured cross-context overlap of high-capacity subspaces, finding limited but meaningful shared structure above random baselines, consistent with context-dependent capacity. We also introduced equal-rank random-subspace controls, which perform substantially worse than causal probes, showing that the steering gains arise from identifying high-capacity directions rather than dimensionality reduction alone. (Section 5, Appendices B.4 and F.5)

Together, these additions provide stronger empirical support for the paper's central contributions:

  • explaining the constraints governing causal effectiveness
  • demonstrating their context dependence
  • translating these insights into practical improvements in steering.

We thank the reviewers again for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.