acceptodds
Under review as a conference paper at ICLR 2027

Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Abstract

For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularizes the model by shrinking weights to restrict network capacity, auxiliary data mixing uses a tunable mixing ratio to expand the training distribution and diversify the training signal with new knowledge. Both offer a principled way to improve training in a data-constrained domain. We compare these levers systematically across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Three findings emerge. First, mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Second, we quantify how much mixing helps: it boosts performance by an amount equivalent to 2–3 the unique target data on validation loss and 2–13 on downstream task accuracy, with the gain scaling steeply with model size. Third, this divergence reveals that target-language validation loss systematically underestimates mixing's value. Mixing regularizes by diversifying the training signal and contributes knowledge the repeated target corpus cannot supply; validation loss captures only the first effect. Our practical recommendations are: mix in a high-resource language, prioritize the mixing ratio over hyperparameter tuning, and transfer hyperparameters from a small proxy model via P.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data repetition, LLM Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vxt4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper compare hyper-parameter tuning (weight decay and learning rate) against mixing with a high-resource language to improve low-resource question answering. English is used as the high-resource language, and artificially constrained Arabic as low-resource language. The authors report larger gains by mixing training data that HP tuning.

Critical strengths

The paper gives practical advice.

Arabic is under-represented in NLP research and deserves more attention.

Critical weaknesses

I don’t think that the experiments support their central claim (“bilingual data mixing outperforms HP tuning”). It rather shows two independent effects: (a) learning rate and weight decay generalize quite well across the investigated model sizes, and (b) bilingual training helps QA. However, both observations are well known: (a) is the core of standard practices in large-scale training, and (b) has been demonstrated by a range of papers that show how multilingual training can help the long-tail, also for QA (see, among many others, TyDI QA). There are many works on cross-lingual transfer with similar conclusions.

Furthermore, the experimental setup has a couple of questionable aspects: relatively small models, only one language pair, no “real” low-resource language, MT-translated test sets, mixing ratios dependent on the model size.

Questions for authors

How do you set your work in context with prior work on multi-lingual training for long-tail languages, cross-lingual transfer, etc.

1 comment

We thank the reviewer for their careful reading.

  1. Central claim. We do not claim that either effect is new on its own. Our question is how a team with a fixed target corpus should spend its effort, on hyperparameter tuning or on mixing, and our results show that the two effects are not independent. Mixing changes the optimal hyperparameters: from 380M to 1.43B, the optimal effective weight decay is 1.2–2.5 without mixing and 0.02–0.4 with it (Table 3). It also makes μP transfer work: at 1.43B, μP-transferred hyperparameters are within 0.5% of per-scale tuning with mixing, against 3.1% without. At that scale, mixing with μP-transferred hyperparameters reaches a 10.1% lower Arabic validation loss than monolingual training tuned per scale, and tuning on top adds 0.5%.

  2. Prior work. Cross-lingual transfer to low-resource languages is well established (e.g. XLM-R, mT5, BLOOM), and we will extend our related work accordingly. These works do not address the data-constrained regime, where a small target corpus must be repeated many times and aggressive regularization through hyperparameter tuning is the main alternative to mixing (Kim et al., 2025). Comparing these two levers systematically, over about 1,000 runs and four scales, is our contribution.

  3. Setup. We agree that one language pair and models up to 1.43B are limitations, and we state them in the paper. Constraining Arabic is deliberate: it lets us measure what mixing is worth against up to 10× more real Arabic data, which a truly low-resource language would not allow. Our primary metric, validation loss, is computed on native Arabic text, and a separate Arabic test split confirms it; only the downstream benchmarks are machine-translated, and a native speaker checked a sample of them. Finally, the dependence of the mixing ratio on model size is a finding rather than a flaw: a fixed corpus becomes a smaller share of a growing training budget, so the optimal Arabic fraction falls from 0.33 at 380M to 0.10–0.21 at 1.43B (Table 3).

Reviewer_3V1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to pretrain a model to perform well on a low-resource language. The experiments use a single target language, Arabic, subsampled to 200M tokens, and compare two pretraining options: tuning the pretraining hyperparameters, or mixing in high-resource language data (English in their setting) and tuning the mixing ratio. They find that mixing is much more effective, and that its benefit grows with model size. Furthermore, when mixing in English, they show that hyperparameters transfer with µP to larger models and that further tuning at each model size adds little on top of this. They estimate that mixing can give the benefit of 2–3× more unique Arabic data on validation loss and much more on downstream benchmarks (machine-translated into Arabic), arguing that validation loss cannot fully capture the benefit of additional English data. The experiments are on four model sizes, up to 1.43B parameters.

Critical strengths

S1: A careful comparison of mixing and hyperparameter tuning. The paper performs about 1k runs across 4 model sizes up to 1.43B parameters and shows that, once English is mixed in, tuning at each size only slightly improves validation loss, and settings transferred from the smallest model with µP are within 0.5% of the best. This supports a simple rule for practitioners: mix in English and only search for a good mixing ratio.

Critical weaknesses

W1: The benefit of mixing is expected and largely known. Mixing adds new data, which tuning cannot, and the bilingual runs also train on more tokens (97B at 1.43B vs 20B for monolingual), so it is not surprising that mixing wins. Close prior work (Seto et al., 2025) uses almost the same setup with several target languages, including ones far from English, and already measures how much target-language data English is worth (for German). What remains new is mainly a careful comparison with hyperparameter tuning across model sizes, for a single target language far from English (Arabic) (see Q4, Q5).

W2: The claim that validation loss undervalues mixing is not strongly supported. With the µP settings that the paper recommends, the advertised gap only appears at 1.43B (3.2× vs 8.1×, Fig. 7), in single-seed runs and on translated benchmarks. It cannot be seen on closed-book SciQ, which more directly tests the paper's argument that benchmarks capture the knowledge English adds and validation loss misses, since the model must answer science questions from memory. There, mixing is worth about 3× (both models are well above chance), the same as on validation loss. Appendix mentions these caveats, but the abstract and conclusion present the claim as a finding. Also, both metrics pick the same mixing ratio for the two larger model sizes (the ones where accuracy is clearly above chance, Tables 7–8), so for choosing the mixing ratio in practice these results don’t give enough reason to use accuracy over validation loss (see Q1, Q2).

Questions for authors

Q1: The native Arabic benchmarks (e.g. ArabicMMLU, Belebele or AlGhafa) are at chance, but could you report the task loss on them (log-likelihood of the correct answer)? This can be more informative at small scale (Heineman et al., 2025), and would give an idea of the effect on benchmarks written in Arabic (where there are no artifacts that potentially arise from machine translation).

Q2: Could you report the standard error of accuracy on the main downstream benchmark (ARC-Easy), and how much one standard error changes the multiplier, as you do for SciQ and PIQA? For the benchmarks you report this, one standard error can change the multiplier by a factor of 1.3-6 and since the ARC-Easy results also come from single runs, this would give a better idea of how reliable the 2–13× multipliers are.

Q3: The 13.4× in the abstract is referred to in the appendix as an empirical lower bound (App. D.3). But the largest Arabic model (10x more Arabic, 2B tokens) already slightly outperforms the bilingual model (41.9% vs 41.4%). Doesn’t this mean the multiplier is about 10× at most? Could you clarify how the 13.4 x number should be read?

Q4: Sec. 4.3 reports that the µP settings are within 0.5% of the best validation loss at 1.43B, which matches Table 3. However, Fig. 37 (left), which the text cites, seems to show 4.4% at 1.43B for bilingual. Is this perhaps a small error in the figure, or does it come from a different setting?

Q5: At 1.43B, the bilingual models are trained on about 5× more tokens than the Arabic-only ones (97B vs 20B). I am wondering, how do the two compare at 20B tokens (since the LR is constant, this should be possible with the existing checkpoints)?

1 comment

We thank the reviewer for their careful reading.

W1, Q5 (contribution and token count). We agree that the comparison with tuning is the point of the paper. Seto et al. (2025) paved the way for bilingual pre-training under data constraints, with one fixed hyperparameter setting; they do not compare mixing with hyperparameter tuning or study how sensitive the outcome is to hyperparameters. Our contribution is this systematic comparison: about 1,000 runs over four scales that quantify the value of each lever. On token count, we compared the two at matched total tokens (20B) using the existing checkpoints. At 1.43B, within the first 20B tokens and relative to the monolingual model with μP-transferred hyperparameters, mixing lowers Arabic validation loss by 11.2%, against 3.1% for per-scale tuning, although the bilingual model has seen only 4B Arabic tokens at that point. On ARC Easy, the bilingual μP model reaches 38.0% within 20B tokens, against 35.1% for the best monolingual run. Extra Arabic-only tokens do not help: the best monolingual checkpoint comes after 23 of its 100 passes, and validation loss rises afterwards.

W2, Q2, Q3 (multiplier and the validation-loss claim). Our reply to Reviewer gMrY covers the multiplier in detail; in short:

  • Q3: At 1.43B the measured 2B monolingual accuracy (41.9%) lies above the log-linear fit (40.6%), and the bilingual accuracy (41.4%) falls between the two, so inverting the fit gives 2.7B tokens (13.4×). The measured points bound the multiplier between 5× and 10× (the 1B model reaches 38.4%); interpolating between them gives 9.0×. 13.4× is therefore the upper end, not a lower bound. We will correct App. D.3 and report the multiplier with these bounds instead of 13.4×.
  • Q2: the ARC Easy standard error is about 1.0 point; one standard error changes the multiplier by a factor of 1.4 to 1.8 (3 to 6 on PIQA). Propagating accuracy noise through the fit gives 90% intervals at 1.43B of 7.6–25.3× (bilingual, tuned), 4.8–14.7× (bilingual, μP), and 0.8–2.3× (monolingual tuning): the exact value moves, the ranking does not. We are preparing two additional seeds of these runs and will report tighter bounds as soon as we have the results.
  • W2: We agree that the claim is stated too strongly. The claim compares two multipliers: how much Arabic data mixing is worth when measured on validation loss, and when measured on ARC Easy accuracy. Under μP, the two agree at 600M; with tuned hyperparameters, a divergence appears at 600M (6.4× vs 2.7×) but is within accuracy noise. At 1.43B it exceeds noise for both: the 90% intervals of the ARC Easy multiplier (4.8–14.7× for μP, 7.6–25.3× for tuned) lie entirely above the validation-loss values (3.2×, 3.3×). However, closed-book SciQ shows no divergence, and both metrics select the same mixing ratio, so validation loss suffices for choosing the ratio. We will restrict the claim in the abstract and conclusion to 1.43B and to the benchmarks where it holds (ARC Easy, PIQA, open-book SciQ).

Q1 (native benchmarks). We are preparing task-loss evaluations on ArabicMMLU, Belebele, and AlGhafa and will report them as soon as they are available.

Q4 (Fig. 37). The figure is wrong; the text and Table 3 (0.5%) are right. The plotting script measured the μP gap at R_max = 50, the best mixing ratio for the tuned hyperparameters, where μP is 4.4% from the optimum. With R_max chosen for the μP hyperparameters, as in Sec. 4.3 and Table 3, the gap is 0.5%. This also shows that the mixing ratio must be searched at each scale, even when the other hyperparameters transfer with μP. We corrected Fig. 37, and Table 18, which reported the wrong hyperparameter cell.

Reviewer_gMrYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The problem addressed in this paper is of practical significance. When target language data is limited, determining whether to further tune training hyperparameters or incorporate data from high resource languages ​​is a question that warrants careful comparison. The authors tune the base hyperparameters on smaller models and transfer them to larger models. This makes the comparisons more credible. The experimental results support the value of mixed language training under the current setup and further demonstrate that validation loss and downstream task performance do not always yield the same conclusions. However, I have reservations regarding some of the conclusions. The concept of "data equivalence multiplier" is a key finding, yet its robustness requires further substantiation. Furthermore, authors’ explanations for the sources of performance gains are more definitive than the current evidence warrants. It is not yet possible to clearly disentangle the respective contributions of regularization, knowledge transfer, and the evaluation methodology. I acknowledge the experimental value of this work, the aforementioned issues complicate the assessment of its primary contributions.

Critical strengths

The experimental comparisons are straightforward. The authors examine data mixing and tuning at each model scale separately. They use base hyperparameters selected on a smaller model and transferred through μP. This design addresses a practical training choice. Figure 1 shows that bilingual training with base hyperparameters can outperform monolingual training with tuned hyperparameters. The gap is larger at greater model scales, which supports prioritizing auxiliary language data. I also appreciate the evaluation of both validation loss and downstream performance under different model selection rules. Figure 2 shows that the benefits of mixing do not depend entirely on picking the highest benchmark score. The paper also finds that further tuning brings different gains on the two metrics. This is useful because validation loss often guides training decisions.

Critical weaknesses

I still have doubts about the robustness of the data multipliers. The 13.4 times estimate in Section 4.4 comes from fitting and inverting a monolingual performance curve. The paper also reports that the monolingual model trained on 2B Arabic tokens slightly outperforms the bilingual model. Yet the fitted equivalent is about 2.7B tokens. This discrepancy is possible with a fitted curve, but it shows that the estimate depends on the curve’s shape. The main text does not show how much the estimate changes with the fitting range or interpolation between nearby points. The effect of fluctuations in accuracy also remains unclear. In addition, the main benchmark multipliers use checkpoints with the highest accuracy. The training strategies search different numbers of configurations, so selection may affect the results. I find the evidence for a benefit from mixing more convincing than the precise multipliers or their growth with model size.

The practical recommendations and the explanation of the gains also need some qualification. Section 4.2 reports results selected by Arabic validation loss. Further tuning raises the Arabic accuracy of the 1.43B bilingual model from 37.4% to 41.1%. This gain is not small. Don’t Tune may therefore lead readers to underestimate the value of tuning for downstream tasks. The search in Section 3 also covers only a specific grid of learning rates and weight decay values. It cannot establish an upper bound on what tuning could achieve. The proposed mechanism needs more evidence as well. Different multipliers across the two metrics do not show that validation loss captures only regularization, or that the extra benchmark gains come from knowledge learned from English data. Reading comprehension, the ability to use information in context, and translation artifacts may also contribute.

Questions for authors
  1. The monolingual model trained on 2B Arabic tokens already exceeds the bilingual target. Why does the fit still give an equivalent of about 2.7B tokens? Do the 13.4 times estimate and its growth with model size hold under different fitting ranges or interpolation between nearby points? I would like to see this sensitivity analysis, including how uncertainty in accuracy affects the estimates.

  2. Could you clarify whether model selection and final evaluation use separate data?

  3. Are the training recommendations intended to optimize validation loss or downstream performance? Further tuning after mixing still adds 3.7 percentage points in accuracy. Do the results therefore support prioritizing data mixing first, then deciding whether to tune based on the final task?

  4. Is there direct evidence that separates regularization from knowledge transfer, beyond the difference between validation loss and downstream gains?

1 comment

We thank the reviewer for their careful reading and comments. We agree that the exact multiplier depends on the shape of the fitted curve, and that the evidence for a benefit from mixing is stronger than the evidence for the value 13.4×.

  1. For our 1.43B model, the measured monolingual accuracy at 2B tokens (41.9%) lies above the log-linear fit (40.6%). The bilingual accuracy (41.4%) falls between these two values, so the fit only reaches it past 2B tokens, and inverting it yields an equivalent of 2.7B tokens. Hence the 13.4x multiplier is an artefact of the fit, which is misleading since 2B monolingual outperforms the bilingual run. To quantify how accuracy noise affects the multiplier, we estimate the checkpoint-to-checkpoint noise of accuracy from the ±3 checkpoints around each peak, excluding the peak itself, which is biased upward as a maximum. Pooled over the runs that enter the multiplier, this gives a standard deviation of about 0.8 points at 1.43B. We then model each peak accuracy (the seven monolingual DLRD_{LR} points and the bilingual target) as normally distributed around its measured value with this standard deviation, and repeat the procedure 10,000 times: draw the accuracies, fit the log-linear curve, and invert it at the bilingual target. The table reports the 90% interval of the multiplier.
Reference curve1.43B tuned1.43B μP1.43B mono tuned
Log Linear fit7.6–25.3×4.8–14.7×0.8–2.3×

We see that the exact multiplier fluctuates however the ranking of the intervals and the order of magnitude hold. We are currently running two additional seeds to provide empirical confidence intervals for the accuracies, and will hopefully be able to report those results before the end of the rebuttal.

  1. Yes. Hyperparameters, the mixing ratio and checkpoints are selected on Arabic validation loss, and we also measured loss on the separate FineWeb-2 Arabic test split at those checkpoints. It reaches the same conclusions: at 1.43B, mixing lowers test loss by 13.3% and per-scale tuning by 3.4% (12.9% and 3.1% on validation loss). Peak-accuracy results select the checkpoint on the benchmark split itself, which is optimistic; selecting by validation loss instead keeps the ranking. We add the test-loss figure to the appendix and state both selection rules.

  2. We agree that tuning does provide a benefit on the benchmarks. Our intention is to recommend prioritising mixing additional data over spending effort in tuning the hyper-parameters. We will make that more clear in the paper and soften the "don't tune" claim.

  3. We have no direct evidence that separates the effects, and we now state the mechanism as a hypothesis with the reviewer's alternatives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.