- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The presented paper introduces L-MoL, an approach to extend a dense base model with a mixture of language-guided low-rank experts that are placed on the decoder FFNs. The approach appears to improve over both the small and large dense baselines for the machine translation task.
Critical strengths
- The architecture diagrams and methodology section is very helpful for general understanding of the proposed method
- The modification to the architecture is very lightweight in terms of additional parameters and trains quickly
- Nice comparison to different routing approaches (AdaMix, MoLE, MixLoRA, LoRA-Mixer).
Critical weaknesses
Weaknesses
- Going from B parameters to B parameters reduces the gain from to only which is a concerning data point for the scaling ladder for only M extra parameters. Based on this, the gains for larger models will very quickly diminish.
- The paper uses reported scores GShard, SCoMoE, and Lingual-SMoE which makes them hard to compare since they likely use a different training / evaluation setup and even minor differences in the codebase can lead to comparison issues.
- Simple target-language routing as proposed in e.g. [2] isn't properly ablated so it is unclear if the gain is really coming from the combination of token-level routing that is target-language informed or if we can just deterministically route all the tokens to the same target language expert without relying on the embedding at all.
- L-MoL is only applied on the decoder and not the encoder. From [2], it would make sense to add them to the topmost encoder layers and target-language select while bottom encoder layers would likely favor the source-language selected . Applying it to the topmost layers in the encoders alone could offset the lack of a target language specific decoder as outlined in [2].
- Throughout the whole report BLEU is used as a metric which is very far from being a good metric for translation Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help (Lavie et al., WMT 2025), for starters I'd suggest switching to chrF or LLM-as-a-judge.
- In Table A3 it is mentioned that L-MoL uses an additional 100k training updates and a different learning rate which is likely due to the lower number of trainable parameters. I believe the paper should provide a compute matched baseline with ~600k updates and tuned hyperparameters / learning rate / schedules rather than penalizing the baseline like this.
- The Appendix, especially Appendix G, feels very heavily LLM generated and would benefit a lot from a broader human re-write to make it easier to parse.
- There are no zero-shot results which could be an obvious failure pattern of the introduced language embedding and intuitively I wouldn't know how we would select the embedding to use here. Potentially, there could be a "fallback" embedding that we train with a certain %?
- The paper only uses shared decoder architectures which makes it hard to gauge if the approach would still have benefits for larger per-language capacity in the form of language-specific decoders, see Multi-Task Learning for Multiple Language Translation (Dong et al., ACL-IJCNLP 2015) or [2].
Missing References
- [1]: Tricks for Training Sparse Translation Models (Dua et al., NAACL 2022) -> also use "dense pre-training" and then introduce sparsity
- [2]: Learning Language-Specific Layers for Multilingual Machine Translation (Pires et al., ACL 2023) -> also use source/target language sparsity on top of a densely pre-trained checkpoint
Questions for authors
N/A
3 comments
Response to Rebuttal
W2 — Reported baselines. We agree the GShard, SCoMoE and Lingual-SMoE rows are literature values from different setups (Table 1 †; §5.1); our quantitative claims stay within our protocol. On the codebase concern, both use the same SCoMoE/Fairseq codebase; we ran our dense baselines with it and discussed the configuration with SCoMoE's first author. Nor is every reported dense seed stronger: Lingual-SMoE reports 22.21 BLEU for its own dense baseline, below our 23.10. The reported MoE rows are context, not matched-compute comparisons.
I think including such values in the paper shouldn't be done as it is not comparable. Either reproduce the values with your own setup or do not report.
W4 — Decoder-only placement. §5.3 locates the asymmetry on the target side: with hidden-state-only routing, X→En Med is 29.61 but En→X Med is 22.56 (a 7.05 gap), since the decoder writes 94 target languages while the frozen encoder already reads 94 sources; the experts are placed where the measured deficit is. We will discuss the encoder-layer alternative from Pires et al. explicitly in the revision.
For shared decoder architectures the translation process already starts at the top encoder layers (see [2]) so it should help there too.
W8 — Zero-shot. is indexed by target language, not by language pair. A zero-shot X→Y direction between seen languages uses , already trained through En→Y, so no new embedding or fallback is needed. For unseen target languages (§7), a fallback embedding trained with target-ID dropout is a natural extension, as you suggest; we will state both in the revision.
I think results on this would strengthen the work.
Follow-up to Reviewer VM1a: New Zero-Shot Results (W8) and Revisions on W2 & W4
Dear Reviewer VM1a,
Thank you for the follow-up comments.
W2. We agree: values obtained under different setups should not appear as comparable entries in a results table. In the revision we will remove the literature-reported GShard, SCoMoE and Lingual-SMoE rows from Table 1 and keep any discussion of them qualitative. Our codebase statement was only meant to clarify that our dense baseline can be reproduced within the Fairseq/SCoMoE implementation, not to claim that the published scores are comparable; all quantitative claims will stay strictly within our own protocol.
W4. We agree that in shared-decoder architectures translation already starts at the top encoder layers, and thank you for the pointer. Our decoder-side placement is an empirical choice driven by the measured target-side deficit in §5.3 (a 7.05 BLEU gap between X→En and En→X Med under hidden-state-only routing), not a claim that encoder-side placement cannot help. We will discuss the encoder-top alternative (Pires et al., 2023) explicitly in the revision and frame our placement as an empirical observation rather than a general design rule.
W8. We agree that zero-shot results strengthen the work. Because is indexed by target language, not by language pair, a zero-shot X→Y direction between seen languages reuses the target-language conditioning of En→Y, so no new embedding or fallback is needed. A direct zero-shot (X→Y) evaluation over 12 directions between seen languages on OPUS-100 gives 8.08 BLEU / 17.60 chrF++ for our L-MoL-Base checkpoint vs 3.89 BLEU / 11.76 chrF++ for Dense-Base (+4.19 BLEU / +5.84 chrF++) under the same scoring protocol, with L-MoL-Base improving all 12 directions:
| X→Y | L-MoL-Base BLEU | Dense-Base BLEU | L-MoL-Base chrF++ | Dense-Base chrF++ |
|---|---|---|---|---|
| de→fr | 9.49 | 4.63 | 21.88 | 16.15 |
| fr→de | 7.21 | 4.22 | 17.27 | 13.60 |
| es→ru | 8.12 | 3.96 | 16.21 | 9.33 |
| ru→es | 10.95 | 4.76 | 25.28 | 17.27 |
| zh→ja | 9.29 | 4.12 | 12.78 | 8.56 |
| ja→zh | 6.28 | 1.89 | 9.09 | 6.79 |
| ar→pt | 10.50 | 3.62 | 24.23 | 15.49 |
| pt→ar | 3.23 | 1.37 | 14.30 | 7.79 |
| hi→nl | 8.82 | 6.19 | 17.78 | 13.87 |
| nl→hi | 11.45 | 5.78 | 22.66 | 12.37 |
| ko→tr | 7.76 | 4.03 | 18.83 | 12.39 |
| tr→ko | 3.84 | 2.07 | 10.87 | 7.47 |
| Average | 8.08 | 3.89 | 17.60 | 11.76 |
Each direction decodes X→Y directly on 1,000 OPUS-100 test pairs aligned through shared English sentences. Because 94 non-English languages define possible X→Y directions, we cannot exhaustively evaluate all combinations in this short window and therefore selected these 12 directions to span diverse language families and scripts; if there are specific additional language pairs you are interested in, please let us know and we will gladly evaluate them. We will report the zero-shot study in the revision alongside the §7 limitation on truly unseen target languages and your target-ID-dropout fallback suggestion.
Thank you again for the careful reading and constructive discussion.
The Authors
Follow-up on Our Rebuttal and Paper Updates
Dear Reviewer VM1a,
We have posted our response to each of the points you raised, with new chrF++ results over all 188 directions, a matched language-only routing control, a longer dense control, and a full rewrite of Appendix G. We appreciate the thoroughness of your review and hope the new results address your main concerns. Before the discussion period closes, we would be grateful for any comments you may have.
Thank you for your time, The Authors