acceptodds
Under review as a conference paper at ICLR 2027

CRANE: Functional Selectivity of Language-Conditioned Neurons in Multilingual Large Language Models

Abstract

Multilingual large language models operate across languages within a shared parameter space, yet it remains unclear how functionally selective language-related internal components actually are. Prior work has mainly identified language-related neurons through activation statistics, but activation selectivity does not directly reveal how these components functionally affect different languages. We propose CRANE, a functional interpretability framework that combines relevance attribution with controlled neuron interventions. CRANE uses AttnLRP to estimate the contribution of MLP components to language-conditioned predictions, separates candidate identification from functional validation, and evaluates target and non-target language behavior under matched neuron-masking budgets to characterize their functional selectivity. We evaluate CRANE on LLaMA2-7B Base across English, Chinese, and Vietnamese using natural language understanding and open-ended generation tasks. Across all evaluated settings, CRANE-identified components exhibit stronger target-language functional effects than those selected by the activation-based LAPE baseline. However, these effects are not strictly confined to the target language, and some non-target languages are also substantially affected. The results indicate that language-related internal components exhibit measurable target-language functional biases, but these biases overlap with cross-lingual functional effects rather than forming fully isolated language-specific modules. We further transfer Base-identified components directly to LLaMA2-7B Chat and observe that part of their functional influence persists after instruction tuning. Overall, CRANE provides an interpretability perspective that moves from correlational identification toward functional selectivity analysis.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, multilingual language models, functional selectivity, neuron attribution, neuron intervention, layer-wise relevance propagation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g6fHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the functional selectivity of language-conditioned MLP components in multilingual LLMs. CRANE uses AttnLRP to obtain relevance distributions, selects candidate components using layer-normalized kurtosis, and evaluates their functional influence through zero-masking on target and non-target languages. Experiments cover English, Chinese, and Vietnamese on LLaMA2-7B Base, with fixed-set transfer to LLaMA2-7B Chat. CRANE produces larger target-language performance drops than LAPE in all six Base-model language-by-evaluation settings, but only four satisfy the criterion that target degradation exceeds every non-target degradation. The paper interprets these results as overlapping functional influences rather than isolated language-specific modules.

Critical strengths
  1. A useful distinction between functional influence and functional selectivity, supported by transparent reporting of mixed results. The paper explicitly separates attribution-based candidate discovery from intervention-based validation and distinguishes target-language degradation from preferential target-language degradation. The cautious interpretation of Base-to-Chat transfer as persistence of functional influence is also appropriate.
Critical weaknesses
  1. Limited language coverage and model family coverage. The experimental evaluation is restricted to a single model family (Llama-2-base, which is rather outdated and not very representative in terms of multilingual LLMs), and only three languages. As a methodology paper within its main contribution to introduce a general framework, this scope is relatively narrow. The generalization of the conclusions and findings remains unconvincing.

  2. The experiments do not adequately distinguish language-related functional selectivity from task sensitivity or general capacity degradation. The NLU evaluation uses MMLU for English, C-Eval for Chinese, and Belebele for Vietnamese. Consequently, language varies together with task, content, and difficulty. Although the paper acknowledges this limitation, comparing within-benchmark drops across these different benchmarks does not remove the confound. The resulting selectivity indicator therefore cannot establish whether the observed asymmetry is attributable to language.

Questions for authors
  • What motivated the choice of LLaMA2-7B as the sole model family? Is Llama-2 a good representative of multilingual LLMs? Why not consider other more balanced LLMs in terms of language distribution in the pretraining data?
  • Is there any reasons why Chinese, English, and Vietnamese, these three languages, were selected? Were they chosen to test specific hypotheses about language characteristics or pretraining representation? Why not consider including more languages to improve generalization of the findings?
Reviewer_Byf3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the functional selectivity of language-related neurons in multilingual language models. The authors propose CRANE, which combines AttnLRP-based attribution, kurtosis-based candidate selection, and controlled neuron masking. Experiments on LLaMA2-7B across English, Chinese, and Vietnamese show stronger target-language intervention effects than the LAPE baseline, alongside substantial effects on non-target languages. The paper also transfers Base-identified components to the Chat model and observes that some functional influence persists after instruction tuning.

Critical strengths

1.The paper addresses an interpretability question by distinguishing activation-based language associations from functional selectivity. Evaluating both target- and non-target-language effects provides a more informative analysis than examining target-language degradation alone.

2.The framework clearly separates candidate discovery from functional validation and compares methods under matched masking budgets. This design provides a reasonable basis for examining the behavioral influence of identified components.

3.The evaluation covers both language understanding and open-ended generation, with consistently stronger target-language effects than LAPE across the six Base-model settings. The fixed-set Base-to-Chat transfer provides an additional perspective on whether these effects persist after instruction tuning.

Critical weaknesses

1.The methodological novelty appears limited, as the approach primarily combines existing AttnLRP attribution, and zero-masking interventions. Since prior work already investigates language-related neurons through identification and intervention, the authors should clarify the distinctive explanatory value of this combination and provide ablations demonstrating the necessity of its components.

2.The cross-language evaluation is confounded by differences between benchmarks. English, Chinese, and Vietnamese are evaluated using MMLU, C-Eval, and Belebele, respectively, which differ in content, task format, and difficulty. Consequently, differences in degradation may reflect task-specific dependencies rather than language selectivity.

3.Important implementation details remain insufficient for reproducibility. These include the source and size of the candidate-discovery data, token-level relevance aggregation, and the exact MLP locations being masked. The paper also does not report masking counts or proportions.

4.The evaluation is restricted to LLaMA2-7B model. The observed patterns may depend on characteristics of this particular model family and may not generalize to newer architectures or training approaches.

Questions for authors

n/a

Reviewer_SgjtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

CRANE identifies language-conditioned MLP neurons in LLaMA2-7B by applying AttnLRP, selecting neurons with high within-layer normalized kurtosis of relevance, and zero-masking the selected sets. The authors compare target- and non-target-language effects using English MMLU, Chinese C-Eval, Vietnamese Belebele, and GPT-4o-judged generation. CRANE causes greater target-language degradation than LAPE in all six Base-model settings, but satisfies the stricter target-greater-than-all-non-target criterion in four of six.

Critical strengths

The distinction between activation association and functional importance is valuable. Reporting the full cross-language ablation profile, rather than target degradation alone, is good practice. The manuscript is generally clear; the principal tables are internally consistent, and the authors appropriately acknowledge that language-related effects need not be exclusive.

Critical weaknesses
  1. Novelty relative to prior intervention studies. The framing underplays existing functional analyses: Zhao et al. (NeurIPS 2024; PLND) deactivate language-associated neurons across layers; Wang et al. (Sharing Matters, 2024/2025) study neuron sharing, attribution, and ablation on LLaMA2-7B-Chat; Wendler et al. (ACL 2024) analyze multilingual processing in LLaMA-2. LinguaMap (ICLR 2026) and Lindsey et al. (On the Biology of a Large Language Model, 2025) provide relevant layer- and circuit-level context. These papers are not identical to CRANE, but the authors should more precisely identify what AttnLRP–kurtosis selection and their selectivity protocol establish beyond them; a PLND-style neuron baseline would strengthen this case.
  2. Selectivity is not clearly better than random masking. By the paper's strict criterion, the single random draw is selective in 3/6 Base settings versus 4/6 for CRANE; transferred CRANE masks are selective in only 2/6 Chat settings. In English generation, random masking has a slightly larger selectivity margin (target drop minus worst non-target drop: 0.486 versus 0.457). One random draw cannot establish equivalence, but it also cannot support reliable superiority. Please report multiple layer/budget-matched random trials, confidence intervals, and direct comparisons of selectivity margins rather than only target degradation.
  3. The LAPE comparison needs validation. LAPE causes less target-language degradation than the random draw in all six settings and sometimes improves the target score. This does not prove an implementation error—LAPE was designed around language-related behavior, not necessarily these task scores—but it warrants checks of neuron counts, layer distributions, implementation fidelity, and whether the evaluation measures the behavior LAPE targets. As written, the 6/6 comparison is difficult to interpret.
  4. Candidate discovery and masking are under-specified. The paper does not define the attribution objective (f_\ell(x)), discovery data and potential evaluation overlap, signed token-relevance aggregation, or precise MLP component being zeroed. Set sizes, layer profiles, and overlaps are missing; Algorithm 1 introduces budget (B) but selects using (z>1) without explaining their relationship. Kurtosis is sensitive to rare tails rather than relevance magnitude and selection uses no target-versus-other-language contrast. Figure 3 also makes the apparent selection boundary unclear. Please specify the pipeline and compare against simple relevance-magnitude and contrast-based selectors.
  5. Evaluation confounds and uncertainty. MMLU, C-Eval, and Belebele differ in task, format, and difficulty, so the cross-language maximum-drop criterion can reflect task sensitivity rather than language specificity. In generation, the Vietnamese Base score is only 1.93/10, creating a floor-effect risk. A parallel benchmark across all languages (e.g., Belebele), clear GPT-4o judging details, and uncertainty across examples/judgments are needed to substantiate the cross-language conclusions.
Questions for authors
  1. Can the authors report neuron counts, layerwise locations, set overlaps, and all discovery/masking implementation details?
  2. Why is LAPE weaker than the random draw on every target-language degradation measure, and what happens with a PLND-style or attribution-strength baseline?
  3. Do the central selectivity results hold with parallel language tasks and repeated random masks, including uncertainty estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.