acceptodds
Under review as a conference paper at ICLR 2027

NLMR: Neuron-Level Locate-then-Merge for Multilingual Reasoning in English-Centric LLMs

Abstract

Large Language Models (LLMs) exhibit strong reasoning capabilities in high-resource languages but struggle to generalize such capabilities to low-resource ones, making effective cross-lingual adaptation essential for multilingual reasoning. Common adaptation approaches, such as Supervised Fine-Tuning (SFT) on task data and Continual Pretraining (CPT) on language-specific text, face a fundamental challenge: SFT strengthens reasoning ability but provides limited target-language adaptation, whereas CPT enhances language-specific adaptation but can catastrophically forget previously acquired reasoning abilities. To address this challenge, we propose **N**euron-**L**evel Locate-then-Merge for **M**ultilingual **R**easoning **(NLMR)**, the first neuron-level locate-then-merge framework for multilingual reasoning that integrates the complementary strengths of task-specific SFT and language-specific CPT models. Specifically, we identify neurons critical to reasoning versus linguistic adaptation and selectively merge them, thus mitigating catastrophic forgetting while enhancing reasoning transfer across languages. To evaluate whether NLMR can jointly improve low-resource language reasoning and preserve language adaptation capability, we conduct extensive experiments across 10 languages, 3 reasoning benchmarks, and 2 backbone models, where NLMR consistently outperforms both single-trained models and strong parameter-level merging baselines. Furthermore, neuron-level analysis reveals distinct yet complementary subspaces for reasoning and language adaptation. By selectively preserving and integrating both subspaces, NLMR improves multilingual reasoning performance while also providing an interpretable view of how language and reasoning capabilities are transferred and combined.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multilingual reasoning, large language models, model merging, neuron-level analysis, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y5nBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a model merging method for multilingual reasoning models to improve their multilingual reasoning ability. Specifically, authors first need train a continue-pretraining model in the target language and a supervised fine-tuning model in English. Then, they locate and merge the most important neurons for each model to obtain the final reasoning model working on the target language. Experimental results show that the proposed method outperforms baseline methods across most of involved languages.

Critical strengths
  • The proposed method is easy to follow and obtains promising results across ten languages and three reasoning datasets.
Critical weaknesses
  • The motivation and findings are not supported from the analyses of this work. Take Figure 3 as an example, the curve is mostly above than the y=x (random sample baseline). It means that the most language-sensitive neurons are more likely to overlap with the most reasoning-sensitive neurons, rather than the claim in this paper (the most language-sensitive neurons induced by CPT are largely distinct from the most reasoning-sensitive neurons emphasized by SFT). For example, the top 10% neurons of CPT have 20% more overlap with the ones of SFT, which are two times higher than random baseline.

  • "Many neurons are highly important for only one training stage" is not supported. If that were the case, we would find data points clustering distinctly near the axes (i.e., high CPT paired with low SFT, or vice versa). Instead, the actual data form a coherent, moderately dispersed cloud-like distribution. To make the claim solid, the authors need to report Spearman correlations per module type, use rank-normalized axes, mark top-k overlap (for example, the fraction of the top 1% CPT neurons that are also in the top 1% SFT neurons), and include a same-stage, different-seed control.

  • Missing multitask learning baseline method that training on both CPT and SFT samples. This is also an important baseline for comparison.

Questions for authors
  • Please refer to the wrong claims and the missing baseline method in the weakness part.
Reviewer_RmuVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Neuron-Level Locate-then-Merge for Multilingual Reasoning (NLMR), a model merging framework designed to improve reasoning capabilities in low-resource languages while preserving the complementary benefits of language adaptation and task-specific reasoning. The central motivation is that supervised fine-tuning (SFT) on English reasoning datasets and continual pretraining (CPT) on language-specific corpora provide complementary capabilities, but directly combining their parameter updates may introduce interference and lead to catastrophic forgetting.

Critical strengths

Strength 1: Consistent improvements in average multilingual reasoning performance across multiple benchmarks and backbone models

Why this strength matters for acceptance: These results suggest that neuron-level selective merging is a promising alternative to conventional parameter-level approaches for multilingual reasoning. The evaluation covers diverse languages, reasoning tasks, and model scales, making the empirical contribution practically relevant.

Strength 2: A simple and practically useful approach to integrating complementary model adaptations

NLMR provides a relatively straightforward strategy for combining separately trained models without requiring additional gradient-based optimization during the merging stage.

By identifying and merging subsets of parameter updates from language-oriented CPT and reasoning-oriented SFT, the method offers a practical way to integrate complementary adaptation objectives.

The formulation is easy to implement, and the authors provide a clear algorithm and implementation details. Furthermore, the distinction between attention and feed-forward components in the neuron-level analysis provides an interesting starting point for investigating the structural effects of different training objectives. Why this strength matters for acceptance: The method addresses an important practical problem: how to leverage separately acquired language and reasoning capabilities without repeating expensive joint training.

Critical weaknesses

Weakness 1: Insufficiently specified hyperparameter selection and limited evidence of experimental robustness n Appendix B.3, the authors state that the neuron selection ratios are chosen from

pCPT,pSFT∈{1%,20%,40%,60%,80%,99%}.p_{\mathrm{CPT}},p_{\mathrm{SFT}} \in\{1\%,20\%,40\%,60\%,80\%,99\%\}.

The optimal ratios are determined separately for each dataset–language combination, as shown in Table 7. This corresponds to a search space of 36 possible ratio combinations for each configuration. However, the manuscript does not clearly explain how these configurations were selected, particularly whether an independent validation set was used.

It is also unclear whether the competing merging baselines were given comparable hyperparameter optimization budgets. This is especially important because NLMR's improvements over the strongest baselines are relatively modest on some benchmarks. For example, the average improvement on mCSQA is only 0.88 percentage points for LLaMA-3 8B and 1.32 percentage points for Gemma-2 2B. Furthermore, the experiments do not provide confidence intervals, repeated-run statistics, or significance tests. Since each language is evaluated on only 250 instances per benchmark, these relatively small differences require careful statistical interpretation. There is also noticeable variation across languages. For instance, on LLaMA-3 8B mCSQA, NLMR performs worse than DELLA for several languages, including German, Spanish, French, Chinese, Thai, and Bengali, despite achieving higher overall average accuracy. Why this weakness matters for rejection: The main empirical contribution of the paper depends on demonstrating that neuron-level merging provides reliable improvements over existing merging techniques. Without a clearly specified validation protocol, comparable tuning budgets, and statistical uncertainty estimates, it is difficult to determine whether the reported advantages arise from the proposed merging mechanism or from differences in model selection and hyperparameter optimization. Clarifying these issues and providing appropriately controlled comparisons would substantially strengthen the paper.

Weakness 2: The claimed functional specialization of neurons is not sufficiently supported by the analysis My second major concern is the interpretation of neuron importance scores and the evidence supporting the proposed mechanism. The central hypothesis is that CPT primarily modifies language-related neurons, whereas SFT primarily modifies reasoning-related neurons, and that the corresponding neuron subsets represent complementary functional subspaces. However, the proposed localization procedure identifies important neurons solely through the average absolute magnitude of their parameter changes.

A large parameter change does not necessarily imply that the corresponding neuron is functionally important for a particular capability.

Questions for authors
  1. Whether the neuron selection ratios were determined using a validation set strictly separated from the reported test data?
  2. Whether the parameter-level merging baselines received comparable hyperparameter search budgets?
  3. Whether the main improvements remain consistent across multiple random seeds or under paired statistical significance tests?
Reviewer_a1DHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes NLMR (Neuron-Level Locate-then-Merge), a parameter-efficient training pipeline for cross-lingual reasoning transfer of LLMs. The core idea is to perform target language continual pretraining (CPT) and supervised finetuning (SFT) respectively, and then merge the parameters belonging to the neurons with largest changes during the two training paths to form a target-language SFT model. Results show that the proposed pipeline surpasses multilple baselines on two base models and three testing datasets. The paper also provides some analysis on the overlap of active neurons during CPT and SFT.

Critical strengths
  1. It proposes a new perspective to train a target-language instruction model by merging CPT and SFT results.
  2. The results show it outperforms all baselines in mean scores on the three datasets and two base models.
Critical weaknesses
  1. The neuron selection method sorts the absolute update amplitudes of all neurons from all layers and modules altogether, neglecting the natually different scales of them. For example, the updating scales between FFN and attention "neurons" may be very different, and sorting them together can result in bias towards certain categories of neurons. Also, averaging the update of different parameters inside one neuron may also suffer from this scales issue.
  2. The definition of attention "neurons", when applied to models with GQA or other modified attention mechanisms, is not clear. In GQA, which is adopted by both LLaMA-3 8B and Gemma-2 2B, the number of Q's is larger than that of K/V's. In this regard, how are the neurons defined "across q, k, v, o" (Line 216)?
  3. There is no direct ablation to support the introduction of neurons in parameter selection. What will the results be if you just select the parameters with the largest updates and merge them, keeping other steps the same with NLMR?
Questions for authors

Several baselines suffer from severe degradation compared with the base in certain low-resource languages. For example, the ARC-easy scores in te of TIES and DELLA on LLaMA-3 8B are largely below those of base, CPT and SFT, dragging down their average scores. Why does this happen to these baselines, and why doesn't it happen to NLMR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.