- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the functional selectivity of language-conditioned MLP components in multilingual LLMs. CRANE uses AttnLRP to obtain relevance distributions, selects candidate components using layer-normalized kurtosis, and evaluates their functional influence through zero-masking on target and non-target languages. Experiments cover English, Chinese, and Vietnamese on LLaMA2-7B Base, with fixed-set transfer to LLaMA2-7B Chat. CRANE produces larger target-language performance drops than LAPE in all six Base-model language-by-evaluation settings, but only four satisfy the criterion that target degradation exceeds every non-target degradation. The paper interprets these results as overlapping functional influences rather than isolated language-specific modules.
Critical strengths
- A useful distinction between functional influence and functional selectivity, supported by transparent reporting of mixed results. The paper explicitly separates attribution-based candidate discovery from intervention-based validation and distinguishes target-language degradation from preferential target-language degradation. The cautious interpretation of Base-to-Chat transfer as persistence of functional influence is also appropriate.
Critical weaknesses
-
Limited language coverage and model family coverage. The experimental evaluation is restricted to a single model family (Llama-2-base, which is rather outdated and not very representative in terms of multilingual LLMs), and only three languages. As a methodology paper within its main contribution to introduce a general framework, this scope is relatively narrow. The generalization of the conclusions and findings remains unconvincing.
-
The experiments do not adequately distinguish language-related functional selectivity from task sensitivity or general capacity degradation. The NLU evaluation uses MMLU for English, C-Eval for Chinese, and Belebele for Vietnamese. Consequently, language varies together with task, content, and difficulty. Although the paper acknowledges this limitation, comparing within-benchmark drops across these different benchmarks does not remove the confound. The resulting selectivity indicator therefore cannot establish whether the observed asymmetry is attributable to language.
Questions for authors
- What motivated the choice of LLaMA2-7B as the sole model family? Is Llama-2 a good representative of multilingual LLMs? Why not consider other more balanced LLMs in terms of language distribution in the pretraining data?
- Is there any reasons why Chinese, English, and Vietnamese, these three languages, were selected? Were they chosen to test specific hypotheses about language characteristics or pretraining representation? Why not consider including more languages to improve generalization of the findings?