acceptodds
Under review as a conference paper at ICLR 2027

The Limits of Directional Alignment as Evidence for a Shared Mechanism

Abstract

The directional alignment of residual stream directions in language models has been put forth as evidence of a shared underlying mechanism. Since directional alignment and causal transfer may both occur at the same time, it is unclear whether directional alignment alone is enough to support a shared underlying mechanism. This question is tackled in the context of self-preservation, using a role × evidence conflict contrast in lieu of self-preservation contrast, and hence the directions obtained cannot be attributed solely to self-relevance/self-preservation. The rank-1 directions of Qwen3-8B trained separately across mutually exclusive scenarios in English, Chinese, Spanish, and Hindi languages display high alignment of 0.76-0.90 signed cosine similarities, compared with 0.26-0.42 under label permutation null. The selective causal control test is carried out via ablation of directions at each depth level. Role-conditioned decision gap falls within the label permutation null distribution in all but one language × depth condition. The latter also violates specificity the most. The intervention procedure causes a reduction of 0.844 in refusal on held out prompts as a positive control, indicating the capability of the intervention to influence model behavior. Further limitations of role-conditioned design are explored, including near-determinism not attributable to the stimulus strength, sensitivity to order of options in small gaps of evidence, and dependency of roles on presentation. The findings point to the conclusion that cross-lingual directional alignment alone does not constitute evidence for shared causation.

Primary area:
causal reasoning
Keywords:
Mechanistic Interpretability, Representation Alignment, Causal Interventions, Residual Stream, Multilingual Language Models, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wNTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper examines whether cross-lingual alignment of activation directions indicates shared causal mechanisms in LLMs. Using Qwen3-8B, the authors extract role–evidence conflict directions across four languages and evaluate their geometric similarity and causal effects. Despite strong directional alignment (cosine similarities of 0.76–0.90), ablation experiments show no distinguishable selective effect beyond controls in 17 of 18 conditions. Further analyses reveal substantial sensitivity to option presentation order and instability in behavioral measurements. The findings suggest that directional alignment alone is insufficient to establish shared causal mechanisms.

Critical strengths
  1. Careful evaluation of cross-lingual directional alignment. The paper provides a relatively well-controlled analysis of cross-lingual activation directions. In particular, fitting directions on mutually disjoint base scenarios and comparing their similarities against label-permutation references in the same representation space help address potential confounding factors such as shared content and representational anisotropy. These experiments make the observed cross-lingual alignment more convincing than comparisons based solely on raw cosine similarity.

  2. Informative analysis of behavioral measurement sensitivity. The paper provides useful evidence that model decisions are highly sensitive to option presentation order, even when the underlying evidence remains unchanged. The observed decision reversals and sign changes in the role–evidence interaction highlight an important methodological challenge in evaluating causal mechanisms through behavioral measurements. These findings are valuable for interpreting activation-based interventions, and the authors appropriately acknowledge several limitations of their experimental design.

Critical weaknesses
  1. Alignment between activation extraction and causal evaluation. The extracted direction targets the role–evidence interaction II, while the causal intervention evaluates the incumbent–neutral gap GG. Although the authors acknowledge this distinction, it would be helpful to clarify how changes in GG reflect the causal role of the extracted direction. Could the authors further evaluate intervention effects on II, particularly given its sensitivity to option presentation order?

  2. Further validation of the causal findings. The strong cross-lingual directional alignment is interesting, but the causal evidence could be strengthened. Would cross-lingual transfer experiments help establish a more direct connection between directional alignment and shared mechanisms? Additionally, more extensive permutation controls and base-disjoint evaluation could help clarify whether the largely null intervention results reflect limited causal effects or the sensitivity of the experimental setup.

Questions for authors

Q1. Matched causal evaluation. Can the authors evaluate the effect of directional ablation directly on the role–evidence interaction (I), rather than only on the incumbent–neutral gap (G)? In particular, reporting intervention effects separately for the two option-presentation orders and individual role–evidence conditions would help determine whether the extracted directions causally influence the intended behavioral construct. This is the most important question for my assessment.

Q2. Reliability of the causal null results. Can the authors provide stronger calibration of the intervention test, for example by fitting true and permutation-control directions on the same fixed data partition, preserving base-scenario independence, and increasing the number of permutation trials? A sensitivity analysis demonstrating the ability to recover a known selective effect would also help establish whether the reported null results are informative.

Q3. Cross-lingual causal transfer. Have the authors examined whether activation directions extracted from one language produce corresponding behavioral effects when applied to another language? A source-to-target transfer matrix, ideally using the matched interaction endpoint, would substantially strengthen the connection between the geometric alignment results and the paper's central research question.

1 comment

We thank the reviewer for the careful summary and for recognizing the disjoint-scenario calibration and the order-sensitivity analysis. We agree that the mismatch between the extracted contrast (I) and the intervention endpoint (G) is the central limitation of the causal test.

How G relates to the extracted direction. G does not directly test the causal role of the direction. The direction is extracted from the role × evidence interaction I, whereas G is the incumbent–neutral gap and involves the neutral role, which the extraction never sees. A change in G therefore cannot show that the interaction is responsible, and the absence of a change in G cannot show that mediation is absent. The original manuscript noted this distinction; the revision states it more plainly in Sections 3.2 and 3.4 and limits the causal claim accordingly: no selective effect was detected on G under the tested intervention (Sections 4.3 and 5).

Q1. Matched causal evaluation. We agree this is the most informative test. Measuring the effect of ablation on I, separately for each option order and each role × evidence cell, requires held-out ablation results for all of these conditions, which the current revision does not include. Given the sign reversal of I across orders (Section 4.4), we agree that such an analysis must be reported per order, and Section 5 now states this as the first requirement of a decisive test.

Q2. Calibration of the null. We agree. The current controls permute labels across all trials rather than within base scenarios, are not fitted on the same rows as the true directions, use 20 draws (resolution 1/21), and are evaluated on a split that is not base-disjoint; no planted-effect power analysis was run. Section 4.3 lists these limitations and treats the intervention results as exploratory, and Section 5 specifies the corrected design: within-base permutation on a fixed, base-disjoint row set, at least 200 draws, and a power check that recovers a planted effect. The positive control (refusal reduced by 0.844 on held-out prompts) shows only that the ablation code can change behavior at global scope; it does not establish power for this task.

Q3. Cross-lingual causal transfer. We agree that a source-to-target transfer matrix on the matched endpoint is the most direct test of the paper's central question. It is not included in the current manuscript, and Section 5 and Appendix D state this explicitly. For this reason the conclusion is deliberately limited: cross-lingual alignment alone is not sufficient evidence for a shared causal mechanism. The paper does not claim that the languages lack one.

Additional checks in the revision. We reproduced the main results from the saved data and code (new Appendix E). The alignment (0.76–0.90) and permutation reference (0.26–0.42) replicate; the ablation sweep covers 18 of 20 cells, with index-29 values matching the archived outputs; the option-order effects and I match exactly; and a GPU rerun of the unmodified model matches the stored decisions in 99.2–99.7% of trials.

In response to Reviewer gQWw, Sections 1–3 have also been completely rewritten for clarity, with all changes marked in blue; the definitions of I and G now appear in Sections 3.2 and 3.4 before they are used.

We hope these clarifications address the reviewer's concerns about how the causal results should be read, and we would welcome any further questions.

Reviewer_gQWwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The work challenges a widespread assumption in mechanistic interpretability that finding a high cosine similarity between activation patterns (for example, across different languages) implies that they share an underlying causal mechanism.

Critical strengths
  • Demonstrating that high geometric alignment may not imply causal control is a valuable contribution toward preventing hasty conclusions.
Critical weaknesses

Clarity and Presentation

The manuscript has serious deficiencies that make it difficult to understand from the very first sections. Many technical concepts and key ideas are introduced abruptly without prior definition, and the writing is often ambiguous or grammatically confusing.
  1. Readability

    • Erroneous or confusing logical connections and imprecise wording:
      • In line 39, it is not clear what the word "Here" refers to.
      • At the beginning of page 2 (line 56), the connector "Rather" ("Rather, we make the comparison between those instances...") is used without any prior negation or clear alternative to justify that contrast.
      • In lines 104–105, the statement that "the cosine must be evaluated relative to a baseline... not simply because it is high" is confusing and lacks technical precision.
    • Syntactic and Conceptual Ambiguity:
      • The sentence "It depends on the structure of the feature space and features which contribute a lot to the similarity calculation may not be the important ones in terms of behavior" (lines 41–43) is worded ambiguously.
      • In lines 52–53, the sentence "In terms of score-only comprehension and the use of bootstrapping while keeping mirrored scenarios in the base cluster, the incumbent-arbiter probabilities are positive for all four languages" is poorly constructed and introduces concepts ("score-only comprehension", "mirrored scenarios in the base cluster") without explaining what they are.
    • References taken out of context (use of definite articles without antecedents): The text assumes that the reader is already familiar with specific references that have not been introduced:
      • Line 90: "The key idea in probing lies in..." does not specify which key idea is being referred to.
      • Line 96: The terms "treatment" and "control" arm are mentioned without explaining which experimental configurations are being referred to.
      • Line 99: There is mention of "The refusal experiments" without clarifying which experiments in the literature or in the article itself are being referred to.
      • Line 103: "Our findings related to anisotropy and dominant dimensions" are mentioned in the middle of the literature review without having yet presented any of their own results.
  2. Task definition

    • Inadequate explanation of the experimental design. The first paragraph of Section 3.1 is very poorly explained:
      • It refers to "benchmark-score lists" without clarifying what 'scores' are being referred to.
      • In the sentence "The non-tie scenarios classify the evidence as better for the incumbent (+) or better for the challenger (-)", what are "non-tie scenarios"? It is not defined what exactly it means to classify the evidence as better for the incumbent ("+") or for the challenger ("−").
      • The terms "retain-first order (rsrs)", "replace-first order (srsr)", "each order oo", and "the two option tokens" (lines 128–129) are introduced without ever having shown an example of the prompt or defined what the options are or what the first and second orders refer to. This omission carries the confusion through to Section 3.3, where "averaged across the two orders of options" is mentioned again without the concept having been clarified.
    • Lack of precision in the mathematical formulation. In Section 3.2, mathematical notation and concepts become so obscure that you lose your train of thought while reading.
      • In "Conflict comprises (inc, -) and (cha, +); alignment comprises (inc, +) and (cha, -)" (line 146), the plus and minus signs appear without a clear formal definition nor what 'conflict' and 'alignment' are.
      • On line 155, the phrase “For any scalar response gg ...” is introduced without explaining what that scalar response is.
    • Strange data partitioning. In Section 3.3, the phrase "The intervention partition used for archiving is distributed 300 ways in favor of training, and 200 ways in favor of held-out evaluation" is completely confusing and makes it impossible to understand how the data has been divided.

Typos

  • Line 37: There is an extra closing parenthesis in the citation "Wang et al. (2025))".
  • Line 194: Typographical error in "200 drwas".
Questions for authors

As it stands, the lack of clarity in Sections 1 through 3 makes it impossible to verify the validity of the experimental design and its connection to the results in Section 4. For a positive reevaluation of the work during the rebuttal period, you may completely rewrite Sections 1, 2, and 3 in the revised PDF, including a diagram or example of the task and defining all terms before using them.

Reviewer_Xp8GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Unclear what this paper is trying to do. I used Pangram on 3 random paragraphs from this paper and they all came back 100% AI generated. I refuse to spend time and effort reviewing AI slop. If the ACs/authors/other reviewers disagree, please let me know and I will review this paper more carefully.

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

1 comment

We thank the reviewer for raising this concern and take it seriously. Since the review invites authors to say so if they disagree, we respectfully do, and we would welcome a full review of the revised manuscript.

What the paper does. In interpretability research, directions in a model's residual stream that look alike across settings are sometimes taken as evidence of a shared mechanism. This paper tests that inference in Qwen3-8B across four languages. Directions fitted on disjoint scenarios are strongly aligned (signed cosine 0.76–0.90, against 0.26–0.42 under label permutation), yet ablating them produces no selective effect beyond control directions in 17 of 18 completed conditions, although the same ablation code reduces refusal by 0.844. The paper argues that alignment alone is not sufficient evidence of a shared causal mechanism, and it identifies properties of the behavioral readout (near-deterministic decisions and option-order sensitivity) that limit such tests.

AI use. The research question, experimental design, and experiments were conceived and carried out by the authors, who take full responsibility for all content. As ICLR 2027 policy requires, the paper's AI use statement discloses that AI tools assisted with generating some synthetic scenarios, with parts of the experimental and analysis code, with methodological critique, and with drafting and editing text. Every reported number was checked by rerunning the computations on the saved outputs; the new Appendix E reports this replication, including a GPU rerun of the unmodified model that matches the stored decisions in 99.2–99.7% of trials.

Writing. As non-native English speakers, we used AI tools in part to improve the language of the manuscript; we acknowledge that the first submission was nevertheless difficult to read. We respectfully note that a detector score reflects the style of the text rather than the validity of the experiments or results. The manuscript has since been substantially revised for clarity: Sections 1–3 were completely rewritten in response to Reviewer gQWw's detailed comments, and all changes are marked in blue.

We would be grateful if the reviewer would assess the revised manuscript on its technical content, and we would welcome the AC's guidance on the ethics flag. We are happy to answer any questions.

Comments on the paper

1 comment

We thank all reviewers for their time and comments. We have uploaded a revised PDF in which all changed text is shown in blue. Line, table, and section numbers in the reviews refer to the original submission; because two tables were added in Section 3, the ablation results are now in Table 5.

Main changes

  • Sections 1–3 completely rewritten for clarity (Reviewer gQWw). Every term is now defined before use. Table 1 shows the prompt structure, Appendix F gives an English example of the decision prompt, and Table 2 defines the aligned and conflict classes used to extract the direction.
  • Matched interaction I versus ablation endpoint G (Reviewer wNTt). The manuscript already distinguished these; the revision states the distinction more plainly in Sections 3.2 and 3.4, and Section 5 lists the experiments a decisive test requires.
  • Corrections. A LaTeX error that truncated a sentence in Section 4.1 has been fixed; the statement about Hindi depths in Section 4.3 has been corrected; an overstated sentence in Section 5 has been revised; and typos have been fixed.
  • AI use statement clarified. It now states the authors' own contribution and lists the AI assistance specifically, including for the present revision (Reviewer Xp8G).
  • Replication (new Appendix E). We reproduced the main results from the saved data and code. The alignment (0.76–0.90) and permutation reference (0.26–0.42) replicate; the ablation sweep covers 18 of 20 cells, with index-29 values matching the archived outputs; the option-order effects and I match exactly; and a GPU rerun of the unmodified model matches the stored decisions in 99.2–99.7% of trials.

Detailed responses are posted under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.