- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper explores how target-language performance changes when scaling English training data while keeping the target-language data fixed. The main experiments focus on Chinese and show that increasing English data in a proportional mix can worsen Chinese perplexity. The authors introduce ScaleX and ScaleX-DR, two-stage training setups where the model is first trained on English with a constant learning rate, then on the target language with a fixed cosine learning-rate schedule. ScaleX-DR additionally repeats the target-language data during the second stage to increase its effective exposure. The approach is also used on Turkish, Hungarian, and Bengali.
Critical strengths
- Well-defined research question: This work investigates how scaling high-resource language data affects target-language performance when the amount of available target-language data remains fixed, addressing a relevant challenge in multilingual pretraining under data constraints.
- Practical relevance: The paper also considers a realistic data-constrained setting and explores practical training strategies to improve target-language performance as high-resource data scale.
Critical weaknesses
- Limited novelty: The proposed methods, ScaleX and ScaleX-DR, primarily combine established techniques, including staged training, learning-rate scheduling, and data repetition, without demonstrating clear methodological novelty or substantial contributions beyond existing multilingual training strategies.
- English performance degradation: While the method improves target-language performance, it clearly degrades English performance compared to proportional mixing, as expected from the target-language-only second stage.
- Insufficient downstream evaluation details: The reported accuracy metric is poorly defined, with no per-benchmark results or clear aggregation procedure. In particular, the downstream benchmarks used for Turkish, Hungarian, and Bengali are not specified.
Questions for authors
Could the authors clarify which benchmarks are used for Turkish, Hungarian, and Bengali, and how downstream accuracy is aggregated across benchmarks?
2 comments
Official Comment by Reviewer o1Lh
Thank you for the detailed response and the per-task results, which clarify the evaluation setup. However, the limited novelty and the English degradation remain. I therefore maintain my original rating.
Thank you for recognizing the clear research question and its practical relevance. Our central result is that a capped target corpus still supports further capability gains through increased English training. Below, we explain the finding, quantify the same-budget language trade-off, and show the individual task scores.
W1 — The empirical contribution beyond the individual components
Without DR, fixing Chinese at 5B tokens and increasing total training from 20B to 160B raises the Chinese four-task mean from 21.13 to 26.26. With 20B Chinese tokens fixed, 80B→320B raises the mean from 30.22 to 35.81. All four Chinese tasks and PPL improve between endpoints in both settings.
The schedule comparison explains a training dependency: Cos/Cos-Cos match the English-stage trajectory at each budget, but restarting Chinese-stage cosine changes the 10B–25B PPL trend from 30.59→32.24 to 26.57→24.62. ScaleX organizes these choices as Const-Cos; ScaleX-DR adds target repetition. The contribution connects fixed-corpus scaling, its training conditions, and downstream gains to guide resource allocation when target text is capped.
W2 — Same-budget retention trade-offs and within-recipe scaling
The same-budget results quantify the allocation choice:
The English task breakdown is:
MMLU improves at both endpoints, while the other three English tasks decline. This makes the target-adaptation/English-retention choice explicit.
Both languages can nevertheless improve when the budget grows within a recipe. Under non-DR ScaleX, 20B→160B with 5B Chinese tokens improves Chinese from 21.13 to 26.26 and English from 14.86 to 21.00. With 20B Chinese tokens, 80B→320B improves Chinese from 30.22 to 35.81 and English from 20.87 to 23.76.
Table 3 uses an 80% Chinese / 20% English target stage and shows gains for both languages from 500B to 1T. It supplies a within-recipe budget comparison; Table 2 quantifies the same-budget allocation choice.
W3 / Q1 — Task scores and aggregation
The Chinese mean equally weights C-Eval, AGIEval, CMMLU, and Math23K; the English mean equally weights GSM8K, BBH, MMLU, and HumanEval pass@1, all expressed as percentages. Table 2 decomposes as follows:
Every Chinese task improves at both endpoints.
For Figure 8, the task names in the evaluation records are
m_mmluandm_hellaswag. The plotted percentage is their equal-weight mean. ScaleX-DR gives:Both recorded components improve between endpoints in each language, with different gain sizes. These identifiers preserve the evaluation-record names; the breakdown shows what drives the reported means.