acceptodds
Under review as a conference paper at ICLR 2027

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers

Abstract

Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only LLMs because of varying multilingual tokenization. However, a growing amount of research suggests that even in LLMs, higher cross-lingual representational alignment leads to improved cross-lingual transfer. In this paper, we propose a novel approach to reimagine cross-lingual contrastive learning given the architectural constraints of modern LLMs. Rather than applying an auxiliary alignment loss on hidden states, we propose using the outputs of the mixture-of-experts (MoE) routers as the target for alignment. Router outputs lend themselves better to pooling over many tokens, enabling more reliable cross-lingual comparisons at the sequence-level. Controlled continual pre-training experiments on four open-source MoEs show that incorporating this routing loss also aligns the underlying hidden representations across languages. Most importantly, this loss improves multilingual performance on our diverse evaluation suite, demonstrating the potential of cross-lingual MoE router alignment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multilingual, cross-lingual transfer, representation learning, mixture-of-experts, auxiliary loss

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_GJdROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a cross-lingual alignment objective for decoder-only MOE language models. Instead of directly aligning hidden states, the method aligns the mean-pooled routing distributions of parallel English and target-language sequences. Specifically, it minimizes a KL-divergence loss over selected middle layers, combining this auxiliary objective with the standard language-modeling loss during continued pretraining. The authors evaluate the approach on four MoE models across seven target languages. The experiments report improved routing alignment, increased cross-lingual hidden-state similarity, and better average downstream performance. The proposed method improves over the continued-pretraining baseline on 13 of 14 model–language pairs, with an average gain of approximately 0.9 benchmark points.

Critical strengths
  1. I think the paper addresses an interesting problem and offers a relatively simple way to introduce explicit cross-lingual supervision into MoE models without requiring token-level alignment. The paper identifies MoE routing as an alternative interface for cross-lingual supervision. Mean-pooled routing weights provide a fixed-dimensional representation regardless of differences in sequence length and tokenization. This is a practical design choice for decoder-only models, where establishing token-level correspondences across languages can be difficult.
  2. The evaluation across multiple model architectures and languages is convincing. The experiments cover four MoE models and seven languages, including languages with different scripts and resource levels. The consistent direction of improvement across most model–language pairs is encouraging and suggests that the method is not limited to a single architecture.
Critical weaknesses
  1. I think there are some misclaims in the paper: the paper argues that cross-lingual representational alignment improves multilingual transfer, and that routing alignment is an effective way to induce this alignment. However, the evidence does not fully disentangle the effects of the proposed mechanism. The auxiliary loss directly encourages translated sequences to use similar aggregate expert distributions. This can change the model's routing behavior, the hidden states entering the routers, and the resulting computation. Yet the experiments do not establish which of these effects actually explains the downstream improvements.

  2. For the experimental results, I found the gains are relatively modest, and their statistical reliability is unclear. The main results show an average improvement of approximately 0.9 benchmark points over continued pretraining, with gains on 13 of 14 model–language pairs. However, the absolute improvements are generally small, and the paper does not establish whether they are robust to random seeds, evaluation variability, or hyperparameter choices. Also the experiments use 200,000 parallel training examples per language and apply continued pretraining to already post-trained models. The authors acknowledge that the method may be more effective during pretraining, but this is not demonstrated experimentally.

Questions for authors

see weakness part

Reviewer_mB1eOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a hardware-efficient approach to cross-lingual alignment tailored specifically for sparse Mixture-of-Experts (MoE) decoder-only Large Language Models. It directly addresses the long-standing bottleneck of varying multilingual tokenization-where parallel texts are split into vastly different numbers of subwords-by abandoning token-level hidden state alignment. Instead, the authors propose an auxiliary loss that aligns the mean-pooled routing probability distributions across an entire sequence, treating the network’s expert footprint as a high-level proxy for semantic meaning. To optimize training throughput, both source and target sentences are processed simultaneously via sequence packing without padding, applying gradient updates exclusively to the target language tokens

Critical strengths

Elegant Mitigation of Tokenization Mismatch: The proposed method presents a novel solution to the problem of varying multilingual Tokenization by moving away from token-level hidden state alignment to sequence-level pooling of MOE's router probability distribution.

Hardware and System Efficiency via Sequence Packing: Instead of naively using the separate forward passes for the source and target sequences to compute the routing-alignment objectives, they compute both objectives in a shared forward pass by packing the source and target together without padding. Together these optimizations eliminate padding and redundant computation.

Avoidance of Explicit Latent Vector Distortions: By treating the low dimensional router outputs as the alignment target rather than forcing the raw hidden states together, this theoretically shields the model from immediate model collapse or severe distortion to its generation capabilities.

Critical weaknesses

Systemic Negative Transfer/Increased Language Modeling Loss: Cross lingual alignment techniques, generally suffer negative transfer for the source language, when we force the model to align with low-resource language i.e. its generation capability in the primary language degrades. Do the proposed alignment methods impact the model's native generation quality or reasoning capability in the source language (e.g. English Benchmarks)?

Pooling Mismatched Token Sequences: The paper relies on simple mean pooling over number of tokens. When computing the KL divergence between English & the target language, the loss function completely ignores the difference in token fertility. The model is penalized simply because the target language requires more tokens to express the same idea. The loss function forces the target language to match the magnitude of the English expert density, which is structurally impossible given the unequal denominators.

Architectural Generalizability: The proposed approach offers no direct utility or transferable insight for dense decoder-only models, which comprises a vast portion of current open source LLM research.

Questions for authors

Handling Token Fertility in Mismatched Sequence Lengths: Given the high fertility target languages (e.g. Telugu and Kannada), generates far more tokens than English for the same concept. How KL divergence in the current form handles this divergence? Did you experiment with length-normalized KL divergence loss?

Justification of Data Truncation and Length Filtering: In appendix E, you mentioned that you use language specific token cap (e.g., 512 tokens for Sinhala, 640 tokens for Telugu), by filtering out the long sequences, did you limit the evaluation to short, and syntactically simple sentences. Could you provide an ablation study or performance breakdown on longer, document-level texts?

Addressing Validation Language Modeling Loss Degradation: The results in Appendix B, show that validation language modeling loss degrades when the routing loss is applied. To ensure this method does not induce negative transfer, could you provide downstream generation metrics on standard English benchmarks (e.g., MMLU, GSM8K) scores?

Reviewer_s22XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Similar to aligning sequence-level representations in encoder–decoder models, this paper presents a simple way to align MoE router outputs across languages in decoder-only models. The aim is to improve transfer from the source language to the target language.

Critical strengths
  • The paper is well motivated by previous work on cross-lingual alignment in encoder–decoder models and the role of MoE routing in - decoder-only models.
  • The experiments are well controlled, with three useful baselines. These comparisons help assess the contribution of the alignment loss.
  • The overall improvements are consistent across different models and languages. The routing-divergence analysis also makes the effect of the method easier to interpret.
Critical weaknesses

I don’t see any major weaknesses in this paper. However, I see its contribution mainly as a practical and effective method, and I am less convinced by the stronger causal claim. An alternative explanation is that the loss makes routing less language-dependent, without necessarily improving the alignment of meaning across languages. Comparing the similarity of translation pairs with that of cross-language pairs with different meanings could help distinguish these explanations. I would also like to know how performance changes on English. Does it improve as well, or are there any trade-offs?

Questions for authors
  • How many parallel pairs are needed to obtain similar improvements? Would the method still be effective with much less parallel data?
  • The paper uses high-quality translation pairs, but obtaining enough of these when scaling to pretraining could be difficult. How sensitive is the method to translation quality? Would pairs that are similar in topic without being direct translations, also be effective?
  • As the authors point out, no negative examples are used, and the loss only encourages translation pairs to have similar routing distributions. maybe drop the term contrastive and simply describe it as a cross-lingual alignment loss?
Reviewer_iKzYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how parallel text can improve multilingual performance in decoder-only MoE models. At selected middle layers, the authors average the router weights over the tokens in each sentence. They then compute the KL divergence between the resulting distributions for an English sentence and its translation. During continued pretraining, the English branch provides the reference distribution. The target language branch receives gradients from both the language modeling loss and the routing loss. If you can address my concerns below, I will be willing to increase the score.

Critical strengths
  1. A sentence and its translation can have different lengths, yet both yield a vector with one entry per expert. This gives each entry a clear interpretation and avoids the need to match individual tokens. The authors also check whether routing changes on held-out FLORES translations. In Figure 1, the routing curve after standard continued pretraining almost overlaps with that of the original model. Adding the auxiliary loss produces a clear drop in routing divergence in the middle layers.

  2. The downstream gains are fairly consistent. For Qwen3 on Kyrgyz, the average benchmark score rises from 43.5 to 45.6. Most gains are smaller, but the average score matches or exceeds the baseline in every setting in Table 1. Table 2 also compares router alignment with hidden-state alignment directly. The authors appropriately limit their conclusion to the two languages they tested.

Critical weaknesses

1. Disentangling the Routing Signal from Semantic Alignment

  • Distributional vs. Semantic Gains: The baseline is trained solely on the target language, whereas the proposed method receives an auxiliary English routing signal. Even when using shuffled sentence pairs, this signal might systematically shift the overall expert allocation toward the English distribution rather than establishing fine-grained cross-lingual semantic alignment.
  • Gradient Attribution: As acknowledged in the submission, adding auxiliary gradients at intermediate layers can aid optimization. However, the current evaluation measures only the holistic gain of the complete training objective.
  • Suggested Action: To substantiate that performance gains truly derive from semantic matching of parallel sentences rather than generic optimization benefits or prior distribution shifting, the authors should provide a deeper ablation. Specifically, how does performance vary when pairing with unrelated or random English sentences versus true parallel counterparts, and can the authors quantify the exact shift in expert distribution?

2. Marginal Effect Sizes and Confounding Factors

  • Marginal Gains: While the performance trend is generally consistent across settings, several observed margins are quite narrow (e.g., for Granite on Sinhala, the score improves merely from 32.8 to 32.9).
  • Missing Variance / Significance Tests: The paper does not report standard deviations or variance across multiple random seeds or runs. For differences as small as 0.1, it is impossible to assess statistical significance without error bars.
  • Inconsistent Hyperparameters: Cross-configuration comparisons involve varying learning rates across settings (Table 1), making it difficult to isolate the contribution of the routing loss itself from hyperparameter tuning effects.
  • Suggested Action: Please report mean scores and standard deviations across multiple runs (at least 3 seeds) for borderline results, and clarify whether hyperparameter choices (such as learning rates) were held constant or systematically tuned for each baseline.

3. Presentation, Formatting, and Bibliographic Quality

The manuscript appears rushed and exhibits numerous formatting artifacts, unedited template leftovers, citation anomalies, and typographical errors:

A. Template Remnants

  • AI Use Statement (Page 10, Lines 488–496): Raw bracketed instructions from the conference template were inadvertently left in the final PDF:
    • [propose or refine hypotheses, ...]
    • [help develop theoretical models...clean and reformat dataset]

B. LaTeX Compilation Leaks

  • Page 23, Line 1237: A missing backslash renders \emph{divergence} as raw text emphdivergence.

C. BibTeX Author Parsing & Citation Completeness

  • Corporate Author Bracing: Corporate entities are missing BibTeX grouping braces ({}), causing incorrect author-year parsing and alphabetization:
    • Gemini Team (2023) is cited in-text as (Team et al., 2023) (Line 072) and sorted under "T" in the bibliography (Line 827).
    • Omnilingual SONAR Team is rendered as (SONAR Team et al., 2026).
  • Missing Venues & Metadata: Key seminal references lack publication venues, proceedings titles, or arXiv IDs entirely:
    • Shazeer et al. (2017) (Line 787) is missing the venue (ICLR 2017).
    • NLLB Team et al. (2022) (Line 723).
    • Gemini Team (2023) (Line 827).
  • Outdated Preprint Entries: Several cited preprints have been formally published in archival venues and must be updated:
    • Xue et al. (2024) →\rightarrow ICML 2024
    • Sturua et al. (2024) →\rightarrow ECIR 2025
    • Mahmoud et al. (2025) →\rightarrow LREC 2026
    • Jiang et al. (2026) →\rightarrow COLM 2026
    • Martin et al. (2026) →\rightarrow EMNLP 2026
    • Khandelwal et al. (2026) →\rightarrow EMNLP 2026

D. Typographical & Punctuation Errors

LocationCurrent TextRecommended Correction / Note
Line 097However,languagesAdd missing space after comma (However, languages)
Line 105Sentence endingMissing terminal period
Line 228peformsTypo →\rightarrow performs
Line 251Section 3.2,. / intitializeRemove redundant punctuation / Typo →\rightarrow initialize
Line 255...after (LLM)Missing period after closing parenthesis
Line 283In constrastTypo →\rightarrow In contrast
Line 343But GradientsLowercase mid-sentence capitalization →\rightarrow but gradients
Line 363This gainsNumber agreement →\rightarrow These gains or This gain
Line 374fewer than < 0.1% / ocasionallyRemove double comparison operator / Typo →\rightarrow occasionally
Questions for authors
  1. How much would downstream performance change if the English sentences were shuffled among examples of similar length from the same corpus? This would test how much the method depends on correct translation pairs.

  2. In settings where the gain is only a few tenths of a point, does it persist across training seeds when both methods use the same learning rate?

  3. Figure 4 compares the model trained with the routing loss against the original checkpoint. How much does hidden-state similarity improve relative to standard continued pretraining?

  4. Equation 1 uses router weights after top-k selection. How does the implementation compute the KL term when an expert has positive average weight in English but zero weight in the translation?

Comments on the paper

1 comment

Hello Reviewers and AC,

We have submitted a Rebuttal Revision with the updates promised to reviewers, notably (1) many minutia fixes brought up in iKzY's W3, (2) results on English which affirm no regression from the training targetting other languages as requested by s22X Q3 and mB1e Wq, and (3) expanding discussion of our length filtering in Appendix E (from mB1e Q2).

m1Be and s22X have re-affirmed support for their paper, with s22X finding no notable weaknesses of the work. Meanwhile iKzY and GJdR have not responded to our rebuttal yet, but we believe the most important weakness they raise is about marginal effects and the reliability of our experiments given the hyperparameter tuning required (iKzY W2 and GJdR W2). We discuss this with full transparency in our rebuttal to both, and interpret the empirical results as evidence of a consistent effect across models & languages. Importantly, these results are further complemented by (1) Section 5.1 which shows the method is effective in aligning routing & representational alignment and (2) a large literature advocating for better methods for cross-lingual alignment and its benefits.

We argue our paper merits acceptance for showing that our loss based on MoE routers provide a practical and effective solution to the heavily-studied challenge of cross-lingual alignment in decoder-only LLMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.