- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper examines whether cross-lingual alignment of activation directions indicates shared causal mechanisms in LLMs. Using Qwen3-8B, the authors extract role–evidence conflict directions across four languages and evaluate their geometric similarity and causal effects. Despite strong directional alignment (cosine similarities of 0.76–0.90), ablation experiments show no distinguishable selective effect beyond controls in 17 of 18 conditions. Further analyses reveal substantial sensitivity to option presentation order and instability in behavioral measurements. The findings suggest that directional alignment alone is insufficient to establish shared causal mechanisms.
Critical strengths
-
Careful evaluation of cross-lingual directional alignment. The paper provides a relatively well-controlled analysis of cross-lingual activation directions. In particular, fitting directions on mutually disjoint base scenarios and comparing their similarities against label-permutation references in the same representation space help address potential confounding factors such as shared content and representational anisotropy. These experiments make the observed cross-lingual alignment more convincing than comparisons based solely on raw cosine similarity.
-
Informative analysis of behavioral measurement sensitivity. The paper provides useful evidence that model decisions are highly sensitive to option presentation order, even when the underlying evidence remains unchanged. The observed decision reversals and sign changes in the role–evidence interaction highlight an important methodological challenge in evaluating causal mechanisms through behavioral measurements. These findings are valuable for interpreting activation-based interventions, and the authors appropriately acknowledge several limitations of their experimental design.
Critical weaknesses
-
Alignment between activation extraction and causal evaluation. The extracted direction targets the role–evidence interaction , while the causal intervention evaluates the incumbent–neutral gap . Although the authors acknowledge this distinction, it would be helpful to clarify how changes in reflect the causal role of the extracted direction. Could the authors further evaluate intervention effects on , particularly given its sensitivity to option presentation order?
-
Further validation of the causal findings. The strong cross-lingual directional alignment is interesting, but the causal evidence could be strengthened. Would cross-lingual transfer experiments help establish a more direct connection between directional alignment and shared mechanisms? Additionally, more extensive permutation controls and base-disjoint evaluation could help clarify whether the largely null intervention results reflect limited causal effects or the sensitivity of the experimental setup.
Questions for authors
Q1. Matched causal evaluation. Can the authors evaluate the effect of directional ablation directly on the role–evidence interaction (I), rather than only on the incumbent–neutral gap (G)? In particular, reporting intervention effects separately for the two option-presentation orders and individual role–evidence conditions would help determine whether the extracted directions causally influence the intended behavioral construct. This is the most important question for my assessment.
Q2. Reliability of the causal null results. Can the authors provide stronger calibration of the intervention test, for example by fitting true and permutation-control directions on the same fixed data partition, preserving base-scenario independence, and increasing the number of permutation trials? A sensitivity analysis demonstrating the ability to recover a known selective effect would also help establish whether the reported null results are informative.
Q3. Cross-lingual causal transfer. Have the authors examined whether activation directions extracted from one language produce corresponding behavioral effects when applied to another language? A source-to-target transfer matrix, ideally using the matched interaction endpoint, would substantially strengthen the connection between the geometric alignment results and the paper's central research question.
We thank the reviewer for the careful summary and for recognizing the disjoint-scenario calibration and the order-sensitivity analysis. We agree that the mismatch between the extracted contrast (I) and the intervention endpoint (G) is the central limitation of the causal test.
How G relates to the extracted direction. G does not directly test the causal role of the direction. The direction is extracted from the role × evidence interaction I, whereas G is the incumbent–neutral gap and involves the neutral role, which the extraction never sees. A change in G therefore cannot show that the interaction is responsible, and the absence of a change in G cannot show that mediation is absent. The original manuscript noted this distinction; the revision states it more plainly in Sections 3.2 and 3.4 and limits the causal claim accordingly: no selective effect was detected on G under the tested intervention (Sections 4.3 and 5).
Q1. Matched causal evaluation. We agree this is the most informative test. Measuring the effect of ablation on I, separately for each option order and each role × evidence cell, requires held-out ablation results for all of these conditions, which the current revision does not include. Given the sign reversal of I across orders (Section 4.4), we agree that such an analysis must be reported per order, and Section 5 now states this as the first requirement of a decisive test.
Q2. Calibration of the null. We agree. The current controls permute labels across all trials rather than within base scenarios, are not fitted on the same rows as the true directions, use 20 draws (resolution 1/21), and are evaluated on a split that is not base-disjoint; no planted-effect power analysis was run. Section 4.3 lists these limitations and treats the intervention results as exploratory, and Section 5 specifies the corrected design: within-base permutation on a fixed, base-disjoint row set, at least 200 draws, and a power check that recovers a planted effect. The positive control (refusal reduced by 0.844 on held-out prompts) shows only that the ablation code can change behavior at global scope; it does not establish power for this task.
Q3. Cross-lingual causal transfer. We agree that a source-to-target transfer matrix on the matched endpoint is the most direct test of the paper's central question. It is not included in the current manuscript, and Section 5 and Appendix D state this explicitly. For this reason the conclusion is deliberately limited: cross-lingual alignment alone is not sufficient evidence for a shared causal mechanism. The paper does not claim that the languages lack one.
Additional checks in the revision. We reproduced the main results from the saved data and code (new Appendix E). The alignment (0.76–0.90) and permutation reference (0.26–0.42) replicate; the ablation sweep covers 18 of 20 cells, with index-29 values matching the archived outputs; the option-order effects and I match exactly; and a GPU rerun of the unmodified model matches the stored decisions in 99.2–99.7% of trials.
In response to Reviewer gQWw, Sections 1–3 have also been completely rewritten for clarity, with all changes marked in blue; the definitions of I and G now appear in Sections 3.2 and 3.4 before they are used.
We hope these clarifications address the reviewer's concerns about how the causal results should be read, and we would welcome any further questions.