acceptodds
Under review as a conference paper at ICLR 2027

NAPO: Neuron Activation-based Preference Optimization for Multilingual Reasoning

Abstract

Large language models exhibit uneven reasoning abilities across languages due to imbalances in multilingual training data. Recently, preference optimization has substantially improved multilingual reasoning performance. However, existing methods typically derive preference signals either from external translation models or from coarse-grained internal signals, such as hidden-state representations. These signals may entangle reasoning-relevant information with language-specific or surface-form noise, resulting in inaccurate preference estimation and ultimately limiting improvements in multilingual reasoning. To address this issue, we propose **N**euron **A**ctivation-based **P**reference **O**ptimization (**NAPO**), a framework that derives cross-lingual preference signals from internal fine-grained neuron activations. Specifically, we first identify reasoning-relevant neurons shared across languages. We then introduce a position-weighted alignment reward computed in the identified neuron space to construct pairwise preference data. Finally, we iteratively optimize the policy within the standard direct preference optimization framework. We conduct extensive experiments on three popular multilingual mathematical reasoning benchmarks across 10 languages. The results show that NAPO consistently outperforms multiple strong preference optimization-based baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multilingual LLM, mathematical reasoning, preference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y4m6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets the cross-lingual gap in LLM mathematical reasoning. NAPO (i) selects FFN neurons that are language-balanced (high entropy of activation probability across languages) and strongly active in every language; (ii) represents each response by position-weighted pooled activations of these neurons in intermediate layers; (iii) scores each target-language candidate by layer-weighted cosine similarity to an English reference response; and (iv) trains with DPO plus an NLL term for three iterations. Experiments on MNumGLUESub, MSVAMP and MGSM (10 languages) with MistralMathOctopus report gains over MAPO, CM-Align and LaCoMSA (+4.5 and +2.2 average points over MAPO on MNumGLUESub and MSVAMP), plus ablations, consistency analyses, t-SNE plots, and results on Qwen2.5-3B and Llama3.1-8B.

Critical strengths
  1. Three benchmarks, ten languages and three backbones, with iteration curves and several analyses. If the comparisons are fair, the Mistral results are strong.
  2. Which internal representation space gives the most useful cross-lingual reward is a timely question, and a sparse language-balanced subspace is a reasonable candidate worth testing.
Critical weaknesses

1.Limited novelty. The reward paradigm follows MAPO/LaCoMSA, the neuron selection reuses LAPE (Tang et al., 2024) with the entropy criterion inverted, and the iterative DPO+NLL objective is essentially Iterative RPO (Pang et al., 2024, not cited). The method only changes the representation space in which an existing reward is computed. Claims of "reasoning-relevant neurons" and of independence from hidden-state signals are not supported by the implementation. 2.Insufficient experimental evidence and missing methodological details. The paper does not state whether the reimplemented baselines were trained under the same settings as NAPO. Beyond this, the paper provides no direct evidence that the neuron-activation-based reward is effective. Position-weighted pooling, presented as a main contribution, lacks comparison experiments against alternative pooling strategies in terms of downstream performance. Finally, key parameters and methodological details are insufficiently described, which makes the method difficult to reproduce and its results difficult to verify.

Questions for authors
  1. Some reported values are inconsistent across tables (e.g., Table 3 vs. Tables 1 and 6). Please verify and correct all reported results, and explain any discrepancies.
  2. The manuscript contains broken cross-references, typos, inconsistent terminology and notation, unreadable figures, and claims that are not supported by the reported results. Please revise it carefully.
Reviewer_A6JkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NAPO, a neuron activation-based preference optimization framework for improving multilingual mathematical reasoning. The central idea is to derive preference signals from language-shared neuron activations rather than relying on external translation models or dense hidden-state representations. Specifically, NAPO identifies shared FFN neurons using cross-lingual activation entropy and activation magnitude, computes position-weighted activation similarity between target-language responses and English references, and constructs preference pairs for iterative DPO with an auxiliary NLL objective. Experiments on three multilingual mathematical reasoning benchmarks covering ten languages show improvements over several existing alignment methods. Additional ablations and experiments on different backbone models are provided to evaluate the proposed framework.

Critical strengths

S1. An interesting neuron-level approach to multilingual preference estimation.

The paper explores a meaningful alternative to translation-based and dense-representation-based rewards by constructing preference signals in a selected neuron activation subspace. The combination of cross-lingual neuron filtering and activation similarity provides a potentially useful direction for multilingual alignment without requiring an external translation model during reward computation.

S2. Encouraging empirical improvements across languages and model families.

NAPO achieves the highest average accuracy among the evaluated methods on all three benchmarks. Compared with MAPO, it improves average accuracy by 4.5 points on MNumGLUESub and 2.2 points on the out-of-domain MSVAMP benchmark. The additional results on Qwen2.5-3B-Instruct and Llama3.1-8B-Instruct suggest that the approach is not restricted to a single backbone architecture. The ablations also indicate that neuron filtering and iterative optimization contribute to the final performance.

Critical weaknesses

W1. Insufficient evidence that neuron activation similarity is a reliable indicator of reasoning quality.

The central assumption is that reasoning-consistent responses exhibit more similar activation patterns in the selected language-shared neuron space. However, neurons with high cross-lingual activation entropy and magnitude are not necessarily reasoning-specific, and activation similarity does not inherently imply mathematical correctness.

The analyses in Section 4.5 primarily examine question-level activation similarity and its correlation with accuracy, whereas the proposed preference reward ranks generated responses. These are different settings, and the former does not directly establish the reliability of the latter. Furthermore, the Answer Consistency Ratio measures joint correctness across languages and may increase simply because individual-language accuracy improves.

A direct evaluation of whether the proposed reward correctly ranks mathematically correct and incorrect reasoning trajectories is therefore needed. Without such evidence, the main mechanism underlying the claimed improvement remains insufficiently substantiated.

W2. Insufficient controlled comparisons to isolate the contribution of the proposed reward design.

Prior work, particularly LaCoMSA, already leverages internal representation similarity with iterative DPO and auxiliary NLL regularization. Therefore, the main methodological distinction of NAPO lies in its neuron selection and activation-based reward formulation.

Although the paper includes comparisons with existing methods and several ablations, it does not provide a sufficiently controlled comparison between neuron-level rewards and full hidden-state rewards under otherwise identical candidate sets, preference construction, and optimization settings. Additional controls involving random neuron subsets and alternative pooling strategies would also help determine which components account for the gains.

Moreover, the improvement over MAPO on MGSM is only 0.4 percentage points, and no multi-seed variability or statistical significance analysis is reported. These limitations make it difficult to determine how consistently the proposed reward improves over simpler alternatives.

Questions for authors

Q1. Does the proposed activation-based reward directly correlate with reasoning correctness?

Could the authors evaluate preference-ranking accuracy using candidate responses labeled by mathematical correctness? In particular, how does the proposed reward compare with full hidden-state similarity, unfiltered neuron activations, and randomly selected neuron subsets of the same size?

Q2. Can the authors isolate the benefit of neuron filtering and position-weighted pooling under matched training conditions?

For example, could the authors compare different reward representations using identical sampled candidates, DPO objectives, iteration counts, and optimization budgets? Reporting variability across random seeds, particularly for the relatively small MGSM improvement, would also strengthen the empirical conclusions.

Q3. How are the English reference responses constructed and validated?

Are these references ground-truth reasoning trajectories or model-generated responses? How does their correctness affect the resulting preference rankings? Additionally, clarification of the exact neuron-filtering thresholds, layer-weight computation, and training/evaluation data separation would improve reproducibility.

Reviewer_NiYBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Neuron Activation-based Preference Optimization (NAPO) for multilingual mathematical reasoning. It first identifies language-shared neurons in the feed-forward networks using activation statistics from multilingual MGSM8KInstruct: neurons are selected by cross-lingual activation entropy and a global cutoff on average positive activation. For each target-language question, the current policy samples 20 candidate responses. At selected intermediate layers, NAPO applies position-weighted pooling to the activations of the selected neurons and computes layer-wise cosine similarities between each target-language response and its English reference response; a weighted sum of these similarities is used as the activation-based reward. The rewards define preferred and rejected response pairs, and the policy is updated with the standard DPO objective plus a length-normalized auxiliary NLL term (called “SFT Loss” in Table 3); the updated policy generates new candidates and the preference data are reconstructed for three iterative rounds. At evaluation time, the paper directly uses the updated policy and does not describe reward-based reranking. Experiments cover MNumGLUESub, MSVAMP, and MGSM across ten languages and include two additional backbones. In Tables 1–2, NAPO reaches average accuracies of 74.5% on MNumGLUESub, 75.0% on MSVAMP, and 65.8% on MGSM, compared with 70.0%, 72.8%, and 65.4% for MAPO, respectively.

Critical strengths
  1. The empirical evaluation has broad coverage across tasks and backbones. It evaluates three multilingual mathematical reasoning benchmarks over ten languages, includes MSVAMP as an out-of-domain benchmark excluded from training, and tests two additional backbones.

  2. The paper includes targeted component ablations and iterative-training analyses. Table 3 removes the iterative preference reconstruction, the auxiliary NLL term, and entropy-based neuron filtering, while Tables 6–7 report performance across the three iterations. These experiments provide useful evidence about the role of the listed components in the reported training pipeline.

Critical weaknesses
  1. The paper does not directly validate that the activation-based reward ranks correct responses above incorrect responses for the same question. The language-shared neuron set in Section 3.1 is selected from cross-lingual activation entropy and average positive activation, without using correctness, and the reward in Section 3.2 compares a target-language candidate response with an English reference response. However, Section 4.5, Table 4, and Figure 3 analyze activation similarity between English and target-language questions, rather than the reward assigned to candidate responses. This question-level similarity cannot show which candidate response should be preferred by DPO. The Answer Consistency Ratio (ACR) is an overlap metric that can also change with marginal accuracy, and Appendix C compares representation similarity under different pooling rules without reporting pairwise reward-ranking accuracy or downstream training performance. Thus, the central assumption that the proposed reward provides reliable fine-grained supervision for reasoning remains untested. The paper should directly evaluate the correspondence between the response reward and candidate correctness.

  2. The incremental contribution over existing representation-based preference rewards is not sufficiently isolated. [2] already constructs latent-representation rewards and uses iterative DPO with an auxiliary NLL term; the possible increment of NAPO lies in its neuron selection, position-weighted pooling, and layer selection. The paper reports reimplemented results for [1]–[3] in Tables 1–2, but Section 4.1 and Appendix A do not make it clear whether the baselines use the same candidate pool, English references, number of iterations, auxiliary loss, and update budget. The w/o Neurons Filtering ablation only removes entropy filtering by setting r=1; it does not compare equal-size random neurons, dense FFN activations, or hidden-state rewards. On MGSM, the average margin over MAPO is only 0.4 percentage points, and no multiple-seed result or uncertainty interval is reported. The current evidence supports the complete pipeline, but does not establish that the gain over prior methods comes from the proposed neuron-level reward.

[1] Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization. ACL 2024. ACL Anthology.

[2] Khanh-Tung Tran, Barry O'Sullivan, and Hoang D. Nguyen. LaCoMSA: Language-Consistency Multilingual Self-Alignment with Latent Representation Rewarding. EACL 2026. ACL Anthology.

[3] Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. CM-Align: Consistency-based Multilingual Alignment for Large Language Models. Findings of EMNLP 2025. ACL Anthology.

Questions for authors
  1. On the same questions and model-generated candidate pairs with correctness labels, can you report pairwise ranking accuracy or AUROC for the actual response reward, stratified by language?

  2. Are the English reference responses dataset annotations, model-generated responses, or selected/generated responses? What is their accuracy? If a reference response is incorrect, can NAPO prefer a candidate that is more similar to that incorrect reference?

  3. Holding the candidate set, English references, DPO/NLL configuration, number of iterations, and update budget fixed, can you replace only the reward and compare the NAPO reward using selected shared neurons with rewards based on matched-size random neurons, dense FFN activations, and hidden states? This would directly test the contribution of the new component.

  4. Please report the actual r and b, the number of retained neurons, and how the layer weights are normalized. Please also state whether the neuron set, any reward-related parameters, and π_ref change at each round, and whether the baselines use the same number of iterations and candidates.

  5. Are the main results stable across random seeds, especially the 0.4-point MGSM margin over MAPO?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.