- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper compare hyper-parameter tuning (weight decay and learning rate) against mixing with a high-resource language to improve low-resource question answering. English is used as the high-resource language, and artificially constrained Arabic as low-resource language. The authors report larger gains by mixing training data that HP tuning.
Critical strengths
The paper gives practical advice.
Arabic is under-represented in NLP research and deserves more attention.
Critical weaknesses
I don’t think that the experiments support their central claim (“bilingual data mixing outperforms HP tuning”). It rather shows two independent effects: (a) learning rate and weight decay generalize quite well across the investigated model sizes, and (b) bilingual training helps QA. However, both observations are well known: (a) is the core of standard practices in large-scale training, and (b) has been demonstrated by a range of papers that show how multilingual training can help the long-tail, also for QA (see, among many others, TyDI QA). There are many works on cross-lingual transfer with similar conclusions.
Furthermore, the experimental setup has a couple of questionable aspects: relatively small models, only one language pair, no “real” low-resource language, MT-translated test sets, mixing ratios dependent on the model size.
Questions for authors
How do you set your work in context with prior work on multi-lingual training for long-tail languages, cross-lingual transfer, etc.
We thank the reviewer for their careful reading.
Central claim. We do not claim that either effect is new on its own. Our question is how a team with a fixed target corpus should spend its effort, on hyperparameter tuning or on mixing, and our results show that the two effects are not independent. Mixing changes the optimal hyperparameters: from 380M to 1.43B, the optimal effective weight decay is 1.2–2.5 without mixing and 0.02–0.4 with it (Table 3). It also makes μP transfer work: at 1.43B, μP-transferred hyperparameters are within 0.5% of per-scale tuning with mixing, against 3.1% without. At that scale, mixing with μP-transferred hyperparameters reaches a 10.1% lower Arabic validation loss than monolingual training tuned per scale, and tuning on top adds 0.5%.
Prior work. Cross-lingual transfer to low-resource languages is well established (e.g. XLM-R, mT5, BLOOM), and we will extend our related work accordingly. These works do not address the data-constrained regime, where a small target corpus must be repeated many times and aggressive regularization through hyperparameter tuning is the main alternative to mixing (Kim et al., 2025). Comparing these two levers systematically, over about 1,000 runs and four scales, is our contribution.
Setup. We agree that one language pair and models up to 1.43B are limitations, and we state them in the paper. Constraining Arabic is deliberate: it lets us measure what mixing is worth against up to 10× more real Arabic data, which a truly low-resource language would not allow. Our primary metric, validation loss, is computed on native Arabic text, and a separate Arabic test split confirms it; only the downstream benchmarks are machine-translated, and a native speaker checked a sample of them. Finally, the dependence of the mixing ratio on model size is a finding rather than a flaw: a fixed corpus becomes a smaller share of a growing training budget, so the optimal Arabic fraction falls from 0.33 at 380M to 0.10–0.21 at 1.43B (Table 3).