- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates what determines the causal effectiveness of latent structures in language model representations. The authors decompose the first-order causal influence of an intervention into three factors: capacity, responsiveness, and alignment. Experiments across four language models and 50 concepts suggest that these factors jointly constrain intervention effectiveness and that effective steering directions are context-dependent. Building on these findings, the authors introduce Causal Probes, which restrict linear probes to high-capacity subspaces to improve steering performance while largely preserving concept detection accuracy.
Critical strengths
1.The paper provides an interpretable framework that decomposes causal influence into capacity, responsiveness, and alignment. The empirical analysis across models, layers, and contexts offers a unified perspective on the gap between concept detection and behavioral control.
2.The paper translates its theoretical findings into a practical method through Causal Probes. The proposed approach improves steering performance over standard linear probes while introducing only a modest reduction in concept detection accuracy, demonstrating the practical relevance of the analysis.
Critical weaknesses
1.The concept-expression metric is constructed from tokens appearing in positive examples but not in neutral examples. Such an occurrence-based definition may not faithfully capture complex semantic concepts and could be sensitive to sampling, token frequency, and tokenization. Since this metric directly underlies the three-factor analysis, its validity is critical to the paper's central conclusions.
2.Although Causal Probes substantially outperform standard linear probes, Figure 6 shows that their performance is often comparable to DiffMeans, leaving their practical advantage over strong existing baselines less conclusive. Moreover, constructing high-capacity subspaces introduces additional Jacobian-related computation without a systematic cost comparison. The reliance on synthetic concept examples and a limited number of generation prefixes also restricts conclusions about broader applicability.
Questions for authors
1.The concept covector is constructed from tokens appearing in positive but not neutral examples. How do the authors ensure that this representation captures the intended semantic concept rather than dataset-specific lexical or frequency differences? Do the three factors and the main conclusions remain stable under alternative concept representations?
2.Since the three factors arise from the covariance–correlation identity, a small factor necessarily constrains their product. To what extent do the empirical results establish distinct causal mechanisms beyond this algebraic relationship? Could the authors clarify the main conceptual novelty relative to existing Jacobian- and Fisher-geometry-based analyses?
3.The three-factor analysis relies on a first-order approximation, whereas the steering experiments consider a wide range of intervention strengths. Although the predicted and observed KL divergences are compared, this does not directly establish the accuracy of predicted concept-expression changes. Could the authors evaluate how well the approximation predicts the sign and magnitude of concept-expression changes under finite interventions?
4.When computing Jacobians for future positions, the authors fix intermediate tokens through teacher forcing. However, interventions during free-running autoregressive generation may alter earlier tokens and subsequently change the context and hidden states. How do the authors verify that the causal influence estimated under teacher forcing accurately reflects the cumulative effects observed during actual generation?
2 comments
Thank you again for your helpful feedback on our work. As the discussion period is ending, we would greatly appreciate it if you could take a moment to review our response. Many thanks,
Authors
We thank the reviewer for their thoughtful feedback. We appreciate their recognition of our framework in clarifying the gap between concept detection and control, as well as the practical contribution of causal probes. Below we address the concerns, additional results are at https://anonymous.4open.science/r/iclr2027-8D58/ and will be incorporated into the paper before discussion ends.
W1 + Q1. Concept Expression Metric
While the token-based covector may not capture the full semantics of complex concepts, our goal is not to introduce a new method for representing concepts, but to understand the conditions under which their expression can be causally influenced. Importantly, the decomposition holds for any construction of an output covector.
We have now run a new analysis to show robustness to sampling and dataset choice: We conducted a five-fold heldout experiment, training DiffMeans on four folds and constructing the covector from the fifth and measuring the directions effect on the covector. The localized direction produced 4.2× greater promotion of held-out concept tokens than unrelated directions across all settings, demonstrating robustness even for tokens missing from the training covector.
W2. Causal Probe Performance
Our goal with causal probes was to show the theoretical insights yield practical gains, rather than achieving state-of-the-art steering. Linear probes are used widely in both research and industry. Still, they have a well-known gap between concept detection and causal influence. Our experiments show significant progress towards minimizing this gap, which demonstrates the utility of our findings. Regarding computational cost, the high-capacity subspace is constructed offline. At each training position, we apply JFJ to K random sketch vectors using jacobian-vector products, then keep the top d directions. This avoids producing the full Jacobian, and the cost scales linearly with training position count. Once the probe is trained, intervention-time cost is unchanged.
Q2. Factors as Distinct Causal Mechanisms
Despite the factors arising from an identity, our contribution is identifying and empirically showing distinct constraints on causal effectiveness: capacity is concentrated in low-dimensional subspaces, responsiveness constrains promoteable outputs, and alignment varies substantially across contexts. We further show causal applications in Section 5, greatly improving causality of probes.
Prior Jacobian and Fisher based papers design intervention/steering methods or more recently identify intermediate variables. In contrast, this paper studies the gap between concept detection and causality: We explain why the causal effectiveness of a given latent structure varies across contexts and identify the constraints governing this variation. This work highlights problems with a lot of common assumptions made today regarding both localization and steering, especially that identifying a global concept direction is sufficient for reliable causal control across contexts.
Q3. Approximation of Concept Expression
We extended our existing quality of estimation analysis of the first-order approximation. We conducted a direct comparison between the product of the three factors and the observed finite change in concept expression across ten intervention strengths.
The results support that the approximation remains accurate at finite interventions with its accuracy eventually deteriorating for sufficiently large interventions. Full results are at the anonymous link.
Q4. Teacher Forcing
In Sections 3 & 4, we use teacher forcing to study the intervention’s effect on the model's downstream computation following input processing rather than approximate effect on generation. Additionally, the paper contains two experiments that support teacher-forcing formulation:
Section 5 relies on teacher-forced Jacobians to construct the high-capacity subspaces and evaluated on generation. Its significant improvement shows that the approximated subspaces improve steering beyond the teacher-forced setting. Thus, while teacher forcing is an approximation, our practical results demonstrate the framework remains useful during actual generation.
In Appendix E.3 we use the teacher-forced formulation to identify positions with the highest predicted causal effect for steering. In all settings, intervening at these positions outperforms all the controls. The results demonstrate the teacher-forced approximation provides meaningful predictions of successful steering.
We thank the reviewer again for the engaging discussion and feedback. We would be happy to discuss any remaining points or answer any additional questions.