- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a cross-lingual alignment objective for decoder-only MOE language models. Instead of directly aligning hidden states, the method aligns the mean-pooled routing distributions of parallel English and target-language sequences. Specifically, it minimizes a KL-divergence loss over selected middle layers, combining this auxiliary objective with the standard language-modeling loss during continued pretraining. The authors evaluate the approach on four MoE models across seven target languages. The experiments report improved routing alignment, increased cross-lingual hidden-state similarity, and better average downstream performance. The proposed method improves over the continued-pretraining baseline on 13 of 14 model–language pairs, with an average gain of approximately 0.9 benchmark points.
Critical strengths
- I think the paper addresses an interesting problem and offers a relatively simple way to introduce explicit cross-lingual supervision into MoE models without requiring token-level alignment. The paper identifies MoE routing as an alternative interface for cross-lingual supervision. Mean-pooled routing weights provide a fixed-dimensional representation regardless of differences in sequence length and tokenization. This is a practical design choice for decoder-only models, where establishing token-level correspondences across languages can be difficult.
- The evaluation across multiple model architectures and languages is convincing. The experiments cover four MoE models and seven languages, including languages with different scripts and resource levels. The consistent direction of improvement across most model–language pairs is encouraging and suggests that the method is not limited to a single architecture.
Critical weaknesses
-
I think there are some misclaims in the paper: the paper argues that cross-lingual representational alignment improves multilingual transfer, and that routing alignment is an effective way to induce this alignment. However, the evidence does not fully disentangle the effects of the proposed mechanism. The auxiliary loss directly encourages translated sequences to use similar aggregate expert distributions. This can change the model's routing behavior, the hidden states entering the routers, and the resulting computation. Yet the experiments do not establish which of these effects actually explains the downstream improvements.
-
For the experimental results, I found the gains are relatively modest, and their statistical reliability is unclear. The main results show an average improvement of approximately 0.9 benchmark points over continued pretraining, with gains on 13 of 14 model–language pairs. However, the absolute improvements are generally small, and the paper does not establish whether they are robust to random seeds, evaluation variability, or hyperparameter choices. Also the experiments use 200,000 parallel training examples per language and apply continued pretraining to already post-trained models. The authors acknowledge that the method may be more effective during pretraining, but this is not demonstrated experimentally.
Questions for authors
see weakness part
Hello Reviewers and AC,
We have submitted a Rebuttal Revision with the updates promised to reviewers, notably (1) many minutia fixes brought up in iKzY's W3, (2) results on English which affirm no regression from the training targetting other languages as requested by s22X Q3 and mB1e Wq, and (3) expanding discussion of our length filtering in Appendix E (from mB1e Q2).
m1Be and s22X have re-affirmed support for their paper, with s22X finding no notable weaknesses of the work. Meanwhile iKzY and GJdR have not responded to our rebuttal yet, but we believe the most important weakness they raise is about marginal effects and the reliability of our experiments given the hyperparameter tuning required (iKzY W2 and GJdR W2). We discuss this with full transparency in our rebuttal to both, and interpret the empirical results as evidence of a consistent effect across models & languages. Importantly, these results are further complemented by (1) Section 5.1 which shows the method is effective in aligning routing & representational alignment and (2) a large literature advocating for better methods for cross-lingual alignment and its benefits.
We argue our paper merits acceptance for showing that our loss based on MoE routers provide a practical and effective solution to the heavily-studied challenge of cross-lingual alignment in decoder-only LLMs.