- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- presents a training method to improve cross-lingual MoE agreement for target tasks & languages with strong results compared to baselines
- using an in-domain training set, identifies questions the model gets correct in English but wrong after the question is translated to the target language
- then, identifies experts that the model relies upon in English and trains the model using LoRA to call upon those experts using an auxiliary loss
- Beyond demonstrating strong results on the target task/language, they also demonstrate that MoE routing agreement increases between English and target language. In addition, they show that it also leads to lower cross-entropy on the gold answers.
Critical strengths
- Writing is good and clear, at times quite concise (in a good way). The paper flows well. Numerous good figures (that I suggest making much larger with the extra page allotted for eventual camera-ready).
- Clever multi-stage pipeline with data curation step, expert selection step, and training step.
- Section 5 is really helpful, strong analysis of the impact of the loss
- careful language about claims made, soundness is reasonable
Critical weaknesses
- A concern is the dependence on the parallel data construction part and its scalability. Not only is this an expensive data generation process (requires many inference calls before training even starts) but also requires a funnel (only taking the ci samples) that, for languages&models that are better, would severely limit the amount of training data available. Note that it is also a bit of a shame that this method cannot make use of the tremendous quantity of translation/parallel data already available in open-source.
- Fine-tuning an LLM for a specific task and a specific language is narrow scope and not very common. This is also limited to task/QA eval-style data that already can be judged in a binary way, even if Section 5.5 provides a proxy way for tasks that can’t be easily judged. This limits the applicability of the method to a narrow set of tasks; rather those very close to evaluation benchmarks. This method would be difficult to expand to just language fine-tuning (e.g. improving LLM in Bengali generally) or task fine-tuning (e.g. improving LLM in tasks across many languages).
- Similar to above, Contrived experimental setting where training data is super in-distribution to eval data. This leads to overly clean environment where this method could simply be “better at overfitting” than vanilla SFT. Does not imply generalization to conditions where tasks are much broader (e.g. mathematical reasoning as a whole vs. specifically GSM8K-style questions).
- I had to dig through the appendix to find information about the training and found that it was LoRA ! While I can understand resource constraints, the use of LoRA needs to be made explicit in the main body (including even in the introduction), since this impacts the potential conclusions.
- Related Work is super weak & limited, it does not properly contextualize the paper in the broader landscape of MoE LLMs, multilinguality in LLMs, and language adaptation in LLMs. I think this affects the motivation behind the paper quite a bit. In particular, it’s not properly justified why such a method is not possible in dense models.
- Reproducibility: No recommended reproducibility statement & the link in the abstract leads to an anonymous.4open.science repository with no linked Github “Unavailable. This content can’t be shown. The repository was not found on GitHub. Check the URL and spelling, make sure you are signed in to the account that can see it, and confirm the repo isn't hidden under an org that restricts third-party app access.”
- Similarly, the AI Use Statement is lazy, undetailed, and overall lends very little confidence into the statement that “generative AI tools were used solely to assist with English writing” and at no other parts of the project. Note that the author’s are very clear about it in the main body so it’s not an issue if added, but “Generate synthetic data sets” is a category that requires mandatory disclosure in this statement.
Overall, I think the paper is well written, the method is clever, and the experiments are well-designed. The results are impressive as well. But these only convince me that it works “when targeting a specific eval set in a specific language where there is abundant training data super close to the eval and you’re willing to do inference over the whole training set before starting training”.
Questions for authors
- L50-52 citation ?
- Contribution 1 is just a re-hash of Bandarkar et al., 2026, Chen et al., 2026 ? Unless, I’m missing something, it should be removed (and it doesn’t seem to be particulary expanded upon later anyways)
- It took me a while to understand that “generated token position” G L161 meant the output tokens in the QA format. Why only use these tokens ? In theory, wouldn’t the model already be figuring out how to answer while processing the input. Given the tasks, there are likely very few output tokens to work with.
- Why use OLMoE-1B-7B-0924-SFT ? This is an oddly specific checkpoint.
- Assuming \tau stays fixed, this means the range changes language-to-language, task-to-task ? I would assume that lower-resource languages therefore has a much smaller range ? Would this not be undesired ? How was \tau chosen ?
- Would the use of FLORES English tasks as the “general English distribution” not cause weird bias ? Especially because that’s also unsupervised texts, not chat-formatted QA samples.
- I would suggest a different way of presenting Section 3.4, if I understood it correctly I think it’s a lot simpler than the section makes it out to be. One suggestion could be to ignore the concept of the mini-batch (leave that for the code implementation), unless it matters in a way I don’t understand.
- From practice, I am not concerned with the potential for translation errors in the data generation part because it’s likely small and rare enough to undermine the training operation. It would mostly just add a small bit of noise. However, I believe the potential for errors should be mentioned and argued why it wouldn’t be too much of an issue.
- L400-406, I don’t understand how the set of experts is different across target languages if they’re both defined on the combined cc and ci sets, which should be the same regardless of the target language since it only uses English.
- How are confidence intervals possibly calculated ? Just using binomial distributions ? I would advocate to remove these if so, because it conveys statistical intervals that only cover eval variance in a very narrow way; not training run variance, data variance, etc. and other forms that impact the experiments. As a result I feel like it gives a semblance of statistical soundness of experimental results that isn’t there (for the record, I understand that this isn’t possible and am not docking points for this). I also acknowledge that a different reviewer may have a different opinion.
7 comments
Response to Reviewer YYpq — Part 2: LoRA, related work, and Contribution 1 (W4–W5、Q1/Q2)
This part clarifies the adaptation setup, the architectural scope of RA-MoE, and the relationship between our empirical findings and prior work.
W4. Why was the use of LoRA specified mainly in the appendix?
Appendix C.2 specifies that we apply LoRA to the up/down projections of expert feed-forward layers while freezing the remaining parameters. The training-based comparisons use the same adaptation setup.
We agree that this choice should be visible in the main text. We will explicitly state the use of LoRA in the Experimental Setup. Our fine-tuning results establish the effectiveness of RA-MoE under this parameter-efficient adaptation setup.
W5. How does the related work contextualize RA-MoE, and why does the method require MoE structure?
Section 2 covers four relevant strands: multilingual downstream fine-tuning, MoE-based language expansion, multilingual routing analysis, and routing-aware MoE adaptation. This includes cross-lingual representation alignment in dense models and distinguishes expanding a model for additional languages from adapting an existing multilingual MoE.
Regarding dense models, cross-lingual alignment is certainly possible in both architectures. The architectural requirement concerns the specific alignment target used by RA-MoE:
- RA-MoE identifies a subset of task-relevant experts.
- Its objective aligns both the total routing probability assigned to those experts and the relative allocation within that subset.
- A standard dense feed-forward layer has no corresponding token-dependent distribution over explicit experts.
Thus, the proposed expert-routing objective directly applies to MoE models. A dense-model method could instead align hidden representations or other internal quantities, but that would require a different alignment target.
Q1/Q2. What supports lines 50–52, and how does Contribution 1 differ from prior work?
-
The U-shaped routing pattern is explicitly attributed to prior work.
The Introduction already states: “Prior work has revealed a characteristic U-shaped layer-wise pattern in cross-lingual routing divergence.” We do not claim to have discovered this pattern.
Prior work also includes downstream-task analyses: Bandarkar et al. relate routing divergence to BELEBELE performance, and Chen et al. analyze layer-wise cross-lingual similarity using BELEBELE examples.
-
Figure 1 provides our task-specific evidence.
Building on these findings, our analysis examines mathematical reasoning, logical reasoning, and knowledge understanding. Figure 1 validates the layer-wise pattern in these settings and shows that larger middle-layer divergence is associated with larger target-language performance gaps relative to English. The statements at lines 50–52 refer to these results.
-
Contribution 1 concerns this empirical extension and its relevance to adaptation.
The contribution is phrased as: “We validate the characteristic U-shaped cross-lingual routing divergence pattern in task-specific settings and show that middle-layer divergence is associated with target-language performance gaps.”
This evidence motivates using middle-layer routing as an alignment target during task adaptation. We will make the connection to Figure 1 explicit at lines 50–52 and sharpen the distinction between the previously established pattern and our analyses in the studied task settings.
Response to Reviewer YYpq — Part 3: Response tokens, checkpoint choice, and language-specific configurations (Q3–Q4、Q5/Q9)
This part explains the routing positions we analyze, the OLMoE starting checkpoint, and how middle-layer configurations vary across target languages.
Q3. Why profile only generated-answer token positions?
In §3.3, G denotes positions occupied by the generated response, excluding the prompt. We replay the generated response with teacher forcing and summarize routing over these positions.
Our aim is to characterize routing associated with a particular task response, including the successful reference-language response used for alignment. Response positions provide a consistent scope for this analysis across examples.
We agree that task-relevant computation can already occur while processing the input. Restricting profiling to response positions does not imply that reasoning begins only during response generation. The number of positions also varies by task: mathematical responses can contain multi-token derivations, whereas short-answer tasks provide fewer positions per example, which are aggregated across task examples for expert identification.
Q4. Why use OLMoE-1B-7B-0924-SFT?
The date suffix is part of OLMoE’s official checkpoint naming: 0924 and 0125 distinguish the September 2024 and January 2025 releases. We use the original September release’s official SFT checkpoint.
The choice of the SFT version is intentional. It provides an instruction-tuned starting point for studying cross-lingual task adaptation, without introducing a separate general instruction-tuning stage from the pretrained base model. Our objective is to evaluate task-expert routing alignment, rather than the acquisition of general instruction-following ability. RA-MoE and its corresponding OLMoE baselines use the same starting checkpoint.
Q5/Q9. How is the middle-layer range chosen, and why can expert configurations differ across target languages?
-
The selection rule is shared, but the numerical threshold is configuration-dependent.
As specified in Appendix B.2,
τis the median of the layer-wise mean cross-lingual divergences for each model–task–target-language configuration. We select the longest contiguous interval whose divergence is at or below this median, breaking equal-length ties toward the earlier interval.Thus,
τis not a single fixed numerical threshold shared across all languages and tasks. -
Lower-resource languages do not necessarily receive shorter intervals.
A language with higher absolute divergence can also have a higher median threshold. The interval depends on the distribution of relatively low-divergence layers across depth.
For example, Table 9 selects 11 layers for Bengali and 10 for Spanish with Qwen1.5-MoE on GSM8K; the corresponding OLMoE intervals contain 6 and 8 layers. Resource level alone therefore does not determine interval length.
-
Language-specific intervals explain differences in the overall expert configuration.
For fixed English examples and predictions, the reviewer is correct that
cc ∪ ciidentifies the same English-successful examples, regardless of the target-language correctness labels.The target language nevertheless affects the selected middle-layer interval. Since experts are identified within that interval, the resulting overall layer–expert configuration can differ. At shared layers, expert selection is the same when the English task examples and FLORES reference are held fixed.
-
The overlap between intervals motivates configuration reuse.
These intervals are similar across languages in Table 9. For example, Qwen1.5-MoE on GSM8K selects layers 5–13 for Chinese and 5–14 for Arabic. This overlap motivates the configuration-reuse experiment in Table 4, which applies the Chinese middle-layer interval and associated expert sets to Arabic and Bengali.
Response to Reviewer YYpq — Part 1: Data requirements, practical scope, and mathematical generalization (W1–W3)
Thank you for the detailed review and constructive questions. We clarify the use of training data, preparation costs, and the scope of our contributions, and provide additional evidence on independent mathematical generalization and training-seed robustness. We also address the implementation, reproducibility, and disclosure concerns below.
W1. Does the
ciselection restrict training-data availability, and how scalable is parallel-data preparation?All target-language training examples participate in cross-entropy training.
RA-MoE does not train exclusively on
ciexamples. As defined in §3.4, all target-language training examples contribute to the standard cross-entropy objective; only eligibleciexamples additionally contribute to routing alignment. Consequently, a smallercipool reduces the amount of auxiliary alignment supervision, rather than reducing the supervised training dataset.Data-construction costs and RA-MoE-specific costs are distinct.
In our experiments, SFT and RA-MoE use the same translated target-language training data and therefore share the translation cost. Given these parallel task data, RA-MoE additionally performs correctness annotation and routing profiling.
Table 14 reports this additional preparation cost as up to 16.7% of fine-tuning time, reduced to up to 10.0% when reusing the routing configuration. These measurements begin after parallel-data construction and therefore do not include translation. Starting from native target-language data would additionally require constructing and validating reference-language counterparts.
Existing parallel data can be used when they supply the required task information.
RA-MoE does not require newly translated pairs specifically. Existing parallel corpora can supply language pairs and reduce translation work. Applying the current pipeline additionally requires task responses and a signal for selecting alignment examples. Parallel task datasets containing these elements can therefore be used directly; general parallel text alone does not supply all of them.
W2. Why focus on task–language adaptation, and is binary correctness required?
Task-specific fine-tuning has a practical role when a deployed model must perform a particular function reliably. In a multilingual service, unequal task performance across languages can also give users substantially different experiences. Improving a particular task–language setting is therefore a useful adaptation objective.
Our experiments investigate this setting across three task categories and six target languages. They evaluate task-conditioned multilingual adaptation; they do not claim to establish general-purpose language improvement or joint adaptation across all tasks and languages.
The method is also not restricted to tasks with exact binary verification. Section 5.5 uses response perplexity as a selection proxy for open-ended instruction-following data. This variant improves over SFT by 2.1 percentage points on average across six target languages on IFEval. It retains the requirement for paired instruction–response data, while relaxing the need for an exact correctness verifier.
W3. Do the improvements generalize beyond GSM8K-style questions?
We additionally evaluated the existing Qwen1.5-MoE-A2.7B Spanish math checkpoints on an independent 500-question Spanish PolyMath evaluation, containing 125 questions from each difficulty tier. No additional fine-tuning on PolyMath was performed, and all three checkpoints used the same evaluation protocol.
RA-MoE improves over SFT by 1.6 percentage points, with a 95% CI of [0.6, 2.6], corresponding to eight additional correct answers. The improvement is positive in every difficulty tier, with the largest absolute gain in the Low tier.
This independent evaluation supports transfer beyond the original GSM8K evaluation setting, rather than improvements confined to GSM8K-style test questions.