acceptodds
Under review as a conference paper at ICLR 2027

L-MoL: Mitigating Multilingual Interference via a Language-Guided Mixture of Low-Rank Experts

Abstract

Can a trained dense multilingual translation model benefit from sparse adaptation without acquiring new training data? We propose L-MoL, a language-guided mixture of low-rank residual experts attached to decoder feed-forward networks. The adaptation protocol freezes the dense backbone and trains the expert branches and router. Concatenating a target-language embedding with each token representation gives the router a language-specific logit bias while retaining token-dependent selection. On the 188 available OPUS-100 test directions, L-MoL improves its dense seeds by 2.69 BLEU (Base) and 1.54 BLEU (Large), with additional parameters equivalent to 3.8% and 8.3% of the respective seed sizes. At Base scale, it exceeds the strongest tested language-agnostic routing adaptation by 1.84 BLEU and its own hidden-state-only ablation by 1.95 BLEU. On five selected language pairs, mean routing-overlap reductions are approximately 81% for the three distant pairs and 7.5% for the two related pairs. These results support explicit target-language conditioning as a useful inductive bias for multilingual expert allocation; routing overlap is a diagnostic, not a direct measurement of gradient conflict.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mixture-of-Experts, Multilingual Machine Translation, Parameter-Efficient Fine-Tuning, Language-Aware Routing, Low-Rank Adaptation, Frozen Backbone Adaptation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_VM1aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The presented paper introduces L-MoL, an approach to extend a dense base model with a mixture of language-guided low-rank experts that are placed on the decoder FFNs. The approach appears to improve over both the small and large dense baselines for the machine translation task.

Critical strengths
  • The architecture diagrams and methodology section is very helpful for general understanding of the proposed method
  • The modification to the architecture is very lightweight in terms of additional parameters and trains quickly
  • Nice comparison to different routing approaches (AdaMix, MoLE, MixLoRA, LoRA-Mixer).
Critical weaknesses

Weaknesses

  • Going from 0.180.18B parameters to 0.660.66B parameters reduces the gain from +2.69+2.69 to only +1.54+1.54 which is a concerning data point for the scaling ladder for only 500500M extra parameters. Based on this, the gains for larger models will very quickly diminish.
  • The paper uses reported scores GShard, SCoMoE, and Lingual-SMoE which makes them hard to compare since they likely use a different training / evaluation setup and even minor differences in the codebase can lead to comparison issues.
  • Simple target-language routing as proposed in e.g. [2] isn't properly ablated so it is unclear if the gain is really coming from the combination of token-level routing that is target-language informed or if we can just deterministically route all the tokens to the same target language expert without relying on the embedding at all.
  • L-MoL is only applied on the decoder and not the encoder. From [2], it would make sense to add them to the topmost encoder layers and target-language select el\mathbf{e}_l while bottom encoder layers would likely favor the source-language selected el\mathbf{e}_l. Applying it to the topmost layers in the encoders alone could offset the lack of a target language specific decoder as outlined in [2].
  • Throughout the whole report BLEU is used as a metric which is very far from being a good metric for translation Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help (Lavie et al., WMT 2025), for starters I'd suggest switching to chrF or LLM-as-a-judge.
  • In Table A3 it is mentioned that L-MoL uses an additional 100k training updates and a different learning rate which is likely due to the lower number of trainable parameters. I believe the paper should provide a compute matched baseline with ~600k updates and tuned hyperparameters / learning rate / schedules rather than penalizing the baseline like this.
  • The Appendix, especially Appendix G, feels very heavily LLM generated and would benefit a lot from a broader human re-write to make it easier to parse.
  • There are no zero-shot results which could be an obvious failure pattern of the introduced language embedding and intuitively I wouldn't know how we would select the embedding to use here. Potentially, there could be a "fallback" embedding that we train with a certain %?
  • The paper only uses shared decoder architectures which makes it hard to gauge if the approach would still have benefits for larger per-language capacity in the form of language-specific decoders, see Multi-Task Learning for Multiple Language Translation (Dong et al., ACL-IJCNLP 2015) or [2].

Missing References

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Paper Updates

Dear Reviewer VM1a,

We have posted our response to each of the points you raised, with new chrF++ results over all 188 directions, a matched language-only routing control, a longer dense control, and a full rewrite of Appendix G. We appreciate the thoroughness of your review and hope the new results address your main concerns. Before the discussion period closes, we would be grateful for any comments you may have.

Thank you for your time, The Authors

Reviewer_VM1aOpenReview ↗

Response to Rebuttal

W2 — Reported baselines. We agree the GShard, SCoMoE and Lingual-SMoE rows are literature values from different setups (Table 1 †; §5.1); our quantitative claims stay within our protocol. On the codebase concern, both use the same SCoMoE/Fairseq codebase; we ran our dense baselines with it and discussed the configuration with SCoMoE's first author. Nor is every reported dense seed stronger: Lingual-SMoE reports 22.21 BLEU for its own dense baseline, below our 23.10. The reported MoE rows are context, not matched-compute comparisons.

I think including such values in the paper shouldn't be done as it is not comparable. Either reproduce the values with your own setup or do not report.

W4 — Decoder-only placement. §5.3 locates the asymmetry on the target side: with hidden-state-only routing, X→En Med is 29.61 but En→X Med is 22.56 (a 7.05 gap), since the decoder writes 94 target languages while the frozen encoder already reads 94 sources; the experts are placed where the measured deficit is. We will discuss the encoder-layer alternative from Pires et al. explicitly in the revision.

For shared decoder architectures the translation process already starts at the top encoder layers (see [2]) so it should help there too.

W8 — Zero-shot. is indexed by target language, not by language pair. A zero-shot X→Y direction between seen languages uses , already trained through En→Y, so no new embedding or fallback is needed. For unseen target languages (§7), a fallback embedding trained with target-ID dropout is a natural extension, as you suggest; we will state both in the revision.

I think results on this would strengthen the work.

AuthorsOpenReview ↗

Follow-up to Reviewer VM1a: New Zero-Shot Results (W8) and Revisions on W2 & W4

Dear Reviewer VM1a,

Thank you for the follow-up comments.

W2. We agree: values obtained under different setups should not appear as comparable entries in a results table. In the revision we will remove the literature-reported GShard, SCoMoE and Lingual-SMoE rows from Table 1 and keep any discussion of them qualitative. Our codebase statement was only meant to clarify that our dense baseline can be reproduced within the Fairseq/SCoMoE implementation, not to claim that the published scores are comparable; all quantitative claims will stay strictly within our own protocol.

W4. We agree that in shared-decoder architectures translation already starts at the top encoder layers, and thank you for the pointer. Our decoder-side placement is an empirical choice driven by the measured target-side deficit in §5.3 (a 7.05 BLEU gap between X→En and En→X Med under hidden-state-only routing), not a claim that encoder-side placement cannot help. We will discuss the encoder-top alternative (Pires et al., 2023) explicitly in the revision and frame our placement as an empirical observation rather than a general design rule.

W8. We agree that zero-shot results strengthen the work. Because ele_l is indexed by target language, not by language pair, a zero-shot X→Y direction between seen languages reuses the target-language conditioning of En→Y, so no new embedding or fallback is needed. A direct zero-shot (X→Y) evaluation over 12 directions between seen languages on OPUS-100 gives 8.08 BLEU / 17.60 chrF++ for our L-MoL-Base checkpoint vs 3.89 BLEU / 11.76 chrF++ for Dense-Base (+4.19 BLEU / +5.84 chrF++) under the same scoring protocol, with L-MoL-Base improving all 12 directions:

X→YL-MoL-Base BLEUDense-Base BLEUL-MoL-Base chrF++Dense-Base chrF++
de→fr9.494.6321.8816.15
fr→de7.214.2217.2713.60
es→ru8.123.9616.219.33
ru→es10.954.7625.2817.27
zh→ja9.294.1212.788.56
ja→zh6.281.899.096.79
ar→pt10.503.6224.2315.49
pt→ar3.231.3714.307.79
hi→nl8.826.1917.7813.87
nl→hi11.455.7822.6612.37
ko→tr7.764.0318.8312.39
tr→ko3.842.0710.877.47
Average8.083.8917.6011.76

Each direction decodes X→Y directly on 1,000 OPUS-100 test pairs aligned through shared English sentences. Because 94 non-English languages define 94×93=8,74294 \times 93 = 8,742 possible X→Y directions, we cannot exhaustively evaluate all combinations in this short window and therefore selected these 12 directions to span diverse language families and scripts; if there are specific additional language pairs you are interested in, please let us know and we will gladly evaluate them. We will report the zero-shot study in the revision alongside the §7 limitation on truly unseen target languages and your target-ID-dropout fallback suggestion.

Thank you again for the careful reading and constructive discussion.

The Authors

Reviewer_rJ6ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

L-MoLdemonstrates that a trained dense multilingual NMT checkpoint can achieve significant gains via sparse adaptation without new training data. By freezing the dense backbone and adding parallel, zero-initialized low-rank (LoRA-style) residual experts to each decoder FFN, the method introduces a language-guided router that blends token hidden states with target-language embeddings to inject language-specific biases into token-dependent routing. Evaluated on OPUS-100, L-MoL improves over dense seeds by +2.69 BLEU (Base) and +1.54 BLEU (Large) while adding minimal parameters (3.8%/8.3%), outperforming existing MoE-style PEFT baselines by +1.84 BLEU. Ablations confirm that language conditioning is the primary driver of success (+1.95 BLEU), with gains heavily concentrated on weaker En→X generation and routing patterns naturally exhibiting unsupervised language-family clustering.

Critical strengths
  1. A simple, practical, and well-validated recipe. Freezing the dense checkpoint and training only zero-initialized low-rank residual experts plus a language-conditioned router yields large gains (+2.69 BLEU at Base) at 3.8% added parameters, reusing the original corpus. The language-as-logit-bias mechanism (Eq. 5) is trivial to implement and transferable to other frozen-backbone adaptation settings, making the contribution practically consequential beyond this paper.

  2. Methodologically rigorous evaluation.__ The nested ablation ladder (shared expert → frozen random router → trained router → language-guided router) cleanly separates capacity, routing, and language conditioning, showing language conditioning is the largest single increment (+1.95 BLEU); the same-pool controlled routing comparisons and the En→X asymmetry analysis (7.05 BLEU directional gap, partially recovered) substantiate where and why the method helps. Combined with the honest delimitation of claims, this makes the evidence chain trustworthy.

Critical weaknesses
  • The existence of multilingual interference is assumed, not demonstrated. The paper's framing is "mitigating multilingual interference," yet no single-language (bilingual) baseline is ever reported: we never see whether, or by how much, the dense multilingual model underperforms per-direction bilingual models on OPUS-100, nor how much of that gap L-MoL actually closes. In my own experiments on similar settings, multilingual models in fact match or exceed single-language counterparts, with no significant degradation. If interference is absent or small in this setup, both the motivation and the title-level claim need to be reframed (e.g., as parameter-efficient capacity expansion rather than interference mitigation).

  • Evaluation relies on BLEU alone. All conclusions rest on a single metric, and the authors themselves note that the 13a tokenizer is insensitive for scripts without whitespace word boundaries. No reference-based learned metric (e.g., COMET), no LLM-based scoring (e.g., GPT-as-judge), and no character-level metric (chrF) is reported, so it is unclear whether the +2.69/+1.54 gains — and the +1.95 language-conditioning increment — survive under metrics with different biases.

Questions for authors
  1. The paper never actually shows that interference exists in this setup. How do the dense seeds compare against per-pair bilingual models on OPUS-100, and how much of that gap (if any) does L-MoL close? If multilingual turns out to be on par with bilingual here — which matches my own experience in similar settings — the "mitigating interference" framing doesn't hold; how would you reframe the contribution then?

  2. Everything rides on BLEU, which you yourselves note is insensitive for unsegmented scripts. Could you check the main results with COMET, and ideally an LLM-based judge, on a subset of directions? I'd mainly like to know whether the +1.95 from language conditioning and the En→X recovery survive under a different metric. chrF would be a cheap addition for the whitespace-less languages.

  3. One baseline I kept looking for: plain per-language adapters (Bapna & Firat style, or MAD-X), given the same parameter and training budget. If those already recover most of the gain, what is the routing actually buying?

  4. Relatedly, what about a dense baseline that simply gets the same 3.8% extra parameters — e.g., a wider decoder FFN — trained the same way? Without it I can't tell how much of the improvement is sparsity + routing versus just more parameters and more optimization steps.

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Paper Updates

Dear Reviewer rJ6Z,

We have posted our response to your review, including the chrF++ results over all 188 directions, the equal-budget per-language adapter calculation, and the reframing you suggested for the title and abstract. Thank you for the constructive suggestions. With the discussion period drawing to a close, we would be grateful for any further thoughts you may have on our response.

Thank you for your time, The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Paper Updates

Dear Reviewer 8TBY,

We have posted our response, including the matched language-only router control you asked for, the exact baseline selection rules, and the consolidated resource-group and directional table. We also noted the two wording corrections we will make to Appendix H and Table A8. As the discussion phase nears its end, we would be grateful if you could take a look and let us know if anything remains unclear.

Thank you for your time, The Authors

Thank you for the clarifications. After considering the rebuttal, I will maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer 8TBY

Dear Reviewer 8TBY,

Thank you for taking the time to read our response. We respect your assessment, and would be very grateful if you could briefly let us know which specific concern remains unresolved (for example, whether it is the load-balancing run in Q3, or a remaining detail in W2/W3).

In particular, because Q1 noted that a matched language-only control "would directly affect my assessment of the joint routing design," we ran that exact control and observed that joint [h; e_l] routing outperforms language-only routing by +1.54 BLEU and +1.27 chrF++ under identical settings (25.79 / 42.63 vs. 24.25 / 41.36).

If there is a specific remaining gap—such as a Base-scale comparison with a non-zero λaux\lambda _ {\text{aux}}—and the discussion schedule allows, please let us know so we can clarify or address it directly.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.