acceptodds
Under review as a conference paper at ICLR 2027

Cross-Lingual Data Scaling for Large Language Models

Abstract

Many target languages have limited pretraining text even as English corpora continue to grow. We study a controlled cross-lingual scaling question: when the available target-language corpus is fixed, how does target-language performance change as English data increases? In English–Chinese experiments, proportional mixing dilutes Chinese exposure and can worsen Chinese perplexity. Under staged training, a single cosine schedule couples the Chinese-stage learning rate to the length of English training; holding the Chinese-stage schedule fixed instead produces positive trends over the evaluated scales. Token-category ablations are consistent with shared lexical content contributing to this transfer, although they do not establish a general mechanism. Repeating the fixed target-language corpus preserves its effective training share, with diminishing returns at high repetition. We instantiate these interventions in ScaleX and ScaleX-DR and evaluate them across English data scales. In multilingual experiments that scale English while holding Chinese, Turkish, Hungarian, and Bengali corpora fixed, ScaleX-DR attains lower perplexity and higher downstream accuracy at the largest than at the smallest English scale for all four target languages. A separate 2.5B-parameter English–Chinese comparison at 500B and 1T tokens improves both languages. These results characterize how schedule and target-language exposure shape cross-lingual scaling in the evaluated settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
cross-lingual pretraining, data scaling, low-resource languages

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o1LhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores how target-language performance changes when scaling English training data while keeping the target-language data fixed. The main experiments focus on Chinese and show that increasing English data in a proportional mix can worsen Chinese perplexity. The authors introduce ScaleX and ScaleX-DR, two-stage training setups where the model is first trained on English with a constant learning rate, then on the target language with a fixed cosine learning-rate schedule. ScaleX-DR additionally repeats the target-language data during the second stage to increase its effective exposure. The approach is also used on Turkish, Hungarian, and Bengali.

Critical strengths
  1. Well-defined research question: This work investigates how scaling high-resource language data affects target-language performance when the amount of available target-language data remains fixed, addressing a relevant challenge in multilingual pretraining under data constraints.
  2. Practical relevance: The paper also considers a realistic data-constrained setting and explores practical training strategies to improve target-language performance as high-resource data scale.
Critical weaknesses
  1. Limited novelty: The proposed methods, ScaleX and ScaleX-DR, primarily combine established techniques, including staged training, learning-rate scheduling, and data repetition, without demonstrating clear methodological novelty or substantial contributions beyond existing multilingual training strategies.
  2. English performance degradation: While the method improves target-language performance, it clearly degrades English performance compared to proportional mixing, as expected from the target-language-only second stage.
  3. Insufficient downstream evaluation details: The reported accuracy metric is poorly defined, with no per-benchmark results or clear aggregation procedure. In particular, the downstream benchmarks used for Turkish, Hungarian, and Bengali are not specified.
Questions for authors

Could the authors clarify which benchmarks are used for Turkish, Hungarian, and Bengali, and how downstream accuracy is aggregated across benchmarks?

2 comments

Thank you for recognizing the clear research question and its practical relevance. Our central result is that a capped target corpus still supports further capability gains through increased English training. Below, we explain the finding, quantify the same-budget language trade-off, and show the individual task scores.

W1 — The empirical contribution beyond the individual components

Without DR, fixing Chinese at 5B tokens and increasing total training from 20B to 160B raises the Chinese four-task mean from 21.13 to 26.26. With 20B Chinese tokens fixed, 80B→320B raises the mean from 30.22 to 35.81. All four Chinese tasks and PPL improve between endpoints in both settings.

The schedule comparison explains a training dependency: Cos/Cos-Cos match the English-stage trajectory at each budget, but restarting Chinese-stage cosine changes the 10B–25B PPL trend from 30.59→32.24 to 26.57→24.62. ScaleX organizes these choices as Const-Cos; ScaleX-DR adds target repetition. The contribution connects fixed-corpus scaling, its training conditions, and downstream gains to guide resource allocation when target text is capped.

W2 — Same-budget retention trade-offs and within-recipe scaling

The same-budget results quantify the allocation choice:

SettingChinese mean: mixing→DR (%)English mean: mixing→DR (%)
5B Chinese / 160B total18.96→29.1423.85→20.85
20B Chinese / 200B total24.96→36.6229.37→22.70

The English task breakdown is:

English task5B/160B Mixing5B/160B DR20B/200B Mixing20B/200B DR
GSM8K19.1114.3323.2818.73
BBH41.5429.8147.2627.62
MMLU26.5736.8837.7639.27
HumanEval pass@18.172.389.215.18

MMLU improves at both endpoints, while the other three English tasks decline. This makes the target-adaptation/English-retention choice explicit.

Both languages can nevertheless improve when the budget grows within a recipe. Under non-DR ScaleX, 20B→160B with 5B Chinese tokens improves Chinese from 21.13 to 26.26 and English from 14.86 to 21.00. With 20B Chinese tokens, 80B→320B improves Chinese from 30.22 to 35.81 and English from 20.87 to 23.76.

Table 3 uses an 80% Chinese / 20% English target stage and shows gains for both languages from 500B to 1T. It supplies a within-recipe budget comparison; Table 2 quantifies the same-budget allocation choice.

W3 / Q1 — Task scores and aggregation

The Chinese mean equally weights C-Eval, AGIEval, CMMLU, and Math23K; the English mean equally weights GSM8K, BBH, MMLU, and HumanEval pass@1, all expressed as percentages. Table 2 decomposes as follows:

Chinese task5B/160B Mixing5B/160B DR20B/200B Mixing20B/200B DR
C-Eval25.4740.6834.5151.11
AGIEval22.1523.5424.2726.51
CMMLU25.7241.7135.8350.90
Math23K2.5110.635.2217.95

Every Chinese task improves at both endpoints.

For Figure 8, the task names in the evaluation records are m_mmlu and m_hellaswag. The plotted percentage is their equal-weight mean. ScaleX-DR gives:

Target languagem_mmlu, 50B→450B (%)m_hellaswag, 50B→450B (%)Mean (%)
Turkish27.15→31.6130.70→32.4928.93→32.05
Hungarian25.15→31.7629.52→31.3927.34→31.57
Bengali26.04→28.0926.08→26.3926.06→27.24

Both recorded components improve between endpoints in each language, with different gain sizes. These identifiers preserve the evaluation-record names; the breakdown shows what drives the reported means.

Reviewer_o1LhOpenReview ↗

Official Comment by Reviewer o1Lh

Thank you for the detailed response and the per-task results, which clarify the evaluation setup. However, the limited novelty and the English degradation remain. I therefore maintain my original rating.

Reviewer_UUvQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies cross-lingual data scaling: how continuously increasing the amount of pretraining data in a high-resource language (English) affects performance in a second language (Chinese). Through systematic experiments, the authors analyze how the language data share, learning-rate schedules, and data repetition influence target-language perplexity. A central finding is that the learning-rate schedule applied during the target-language stage can determine whether additional English data helps or hurts. Based on this, the authors propose ScaleX, which uses a constant learning rate for English pretraining and a fixed cosine schedule for the Chinese stage. Experiments cover models up to 2.5B parameters.

Critical strengths
  • The paper addresses a practically relevant question: how should pretraining be configured when data for a high-resource language keeps growing while data for other languages remains limited? This matters for anyone training multilingual models over successive data releases.
  • The experiments are well designed and clearly described, and the paper is well structured overall (with the exception of the introduction, see below). The findings are easy to trace back to the experimental setup.
Critical weaknesses
  • Major: Limited scope and significance. In essence, the paper is an extensive investigation of how known factors (language data share, learning rate, schedule, data repetition) influence target-language perplexity. These results are useful, but the overall contribution feels too narrow for ICLR. Related to this, framing cross-lingual data scaling as a “new problem” overstates the novelty: language mixing during pretraining is an established research area, as the authors’ own related work in Section 2 shows.
  • Major: Generalizability to larger models. While I don't like to raise this point usually (I understand that compute is scarce), I think this is a paper where the size of the models really matters. The paper describes its 2.5B-parameter setting as a “large-scale case study”, which I find difficult to accept. Model size is a key factor for how to set up pretraining data, and capacity constraints likely shape how languages compete or transfer based on different language mixes. Without evidence at larger scales, it remains unclear whether the findings, and ScaleX in particular, transfer to models of practically more relevant size (at least 7 or 13B).
  • Minor: Clarity in writing.
    • The introduction is very dense. The contributions are listed, but it is hard to identify which ones are new. Formulations such as “the learning-rate trajectory experienced by the target stage can determine the sign” are hard to parse (what is the “target stage”, what is the “trajectory experienced”?).
    • The paper should separate new findings from confirmations of prior work more clearly. For example, cosine scheduling is well established, and after the introduction it remains unclear whether the “coupling” of stages is presented as a contribution or as a consequence of known behavior.

Small Fry:

  • Section 3.2 states that “As Figure 1 shows, this naive two-stage alternative degrades more sharply”, but I cannot find this comparison in Figure 1.
  • Plot sizes and table font sizes vary considerably across the paper, which makes it harder to read.
Questions for authors
  • How exactly does ScaleX differ from the const-cos setting? From the description (constant learning rate for English, fixed cosine schedule for Chinese), the two seem identical.
  • Which result supports the claim in Section 3.2 about the naive two-stage alternative, given that Figure 1 does not appear to show it?
  • Broader question/suggestion: How realistic is the assumption that English data will keep growing in quantity? Future gains seem more likely to come from higher data quality. Did the authors consider a setting where English data quality improves at fixed quantity, and would they expect their findings to hold there?
2 comments

Thank you for recognizing the practical relevance of the question and the soundness of the experimental design. The study connects a concrete resource constraint to an actionable finding: target-language capability can keep improving with more English training even after the target corpus stops growing. We first clarify the recipe, then explain what the results teach us.

Q1 / W3 — ScaleX is the Const-Cos recipe

Yes. ScaleX is Const-Cos in the controlled bilingual setting: a constant-learning-rate English stage followed by a fixed cosine Chinese stage. ScaleX-DR adds target repetition. Section 3 analyzes these choices; Sections 4–5 organize and evaluate the recipe.

Cos versus Cos-Cos is the local schedule comparison: the English-stage trajectory is matched at each budget, while the Chinese cosine is restarted. Over 10B–25B, the Chinese-PPL trend changes from 30.59→32.24 to 26.57→24.62. Const-Cos additionally changes the English schedule and is the complete recipe used for the main sweeps.

W1 — The new knowledge: continued target gains with a fixed corpus

The study identifies a practical opportunity: target-language capability can keep improving after the target corpus stops growing.

Without DR, fixing Chinese at 5B tokens and increasing total training from 20B to 160B raises the Chinese four-task mean from 21.13 to 26.26 and reduces PPL from 24.83 to 22.03. With 20B Chinese tokens fixed, 80B→320B raises the mean from 30.22 to 35.81 and reduces PPL from 19.40 to 18.43. All four Chinese tasks improve between endpoints in both settings, without extra Chinese repetition.

The schedule comparison explains a training dependency: under a global cosine, more English training also changes the learning conditions of the fixed Chinese stage. Restarting that stage changes the measured PPL trend. The contribution connects this fixed-corpus opportunity, its training conditions, and downstream gains. It gives practitioners a concrete way to assess additional English training when target text is capped.

W2 — Model size and token budget

The two model scales serve complementary roles. The 1.3B models enable comparisons across schedules, budgets, two Chinese-data settings, and multiple target languages. The 2.5B case extends training to 500B and 1T tokens with Chinese data fixed: Chinese improves from 45.35 to 48.41 and English from 37.47 to 42.39.

“Large-budget case study” precisely describes this second experiment. It demonstrates continued gains along the data-scaling axis, while the 1.3B comparisons characterize the training choices. The reported parameter range is these two model scales.

Q2 / W3 / M1–M2 — Locating the naive two-stage curve and clarifying the terminology

The naive two-stage result is the orange “Cross-lingual Parameter Initialization” curve in Figure 1 (p. 3), corresponding to Cos in Figure 3. Its Chinese PPL is 30.59, 30.84, 31.90, and 32.24, versus 27.71, 27.79, 27.88, and 28.04 for the blue proportional-mixing baseline. “Naive two-stage (Cos)” directly describes this correspondence.

The “target-stage learning-rate trajectory” means the learning rates used for Chinese-stage updates. Under a global cosine, they depend on the preceding English-stage length. The finding is the measured effect on the cross-lingual scaling curve. The contribution follows a simple sequence: fixed-corpus gains, the schedule comparison, and the resulting recipe. Consistent figure and table typography would also help readability.

Q3 — English data quantity and quality

Quality and quantity offer complementary routes to better pretraining. Our experiments ask whether improving a target language requires new target-language text. The fixed-corpus results show that more English training can provide another route.

English quantity is varied here; quality is not independently varied at fixed quantity. Better English initialization could plausibly help target adaptation through quality too, with target scheduling still relevant. That extension is a hypothesis; the measured result establishes the quantity-based route.

Thanks for the reply! As my criticism is on a very fundamental level, I don't feel like this can be alleviated with additional numbers; the paper would have to be rewritten and rephrased significantly. I thus maintain my score.

Reviewer_o9CiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how target-language performance changes as English training data increases while the target-language corpus remains fixed. Through English–Chinese experiments, it examines the effects of data mixing, learning-rate schedules, and repeated exposure. It proposes ScaleX, which combines a constant-learning-rate English stage with a fixed cosine schedule for the target-language stage, and ScaleX-DR, which additionally repeats target-language data. The methods are evaluated using perplexity and downstream accuracy at two Chinese-data budgets and across four target languages.

Critical strengths
  • The paper shows that controlling the target-language learning-rate schedule can change how performance responds to additional English data. Keeping the English-stage schedule unchanged while restarting a fixed schedule for the Chinese stage turns a worsening perplexity trend into an improving one. This highlights the importance of controlling optimization conditions when interpreting data-scaling effects.

  • ScaleX and ScaleX-DR improve Chinese performance at two fixed data budgets. For Chinese, Turkish, Hungarian, and Bengali, ScaleX-DR also achieves lower perplexity and higher downstream accuracy at the largest English scale than at the smallest. This provides evidence that the observed benefit is not limited to a single language or data budget.

Critical weaknesses
  • A comparison with fixed-ratio mixing of English and target-language samples within each batch would strengthen the analysis. The paper does not specify batch composition or sampling procedures for its mixing baselines. Matching per-language token counts and compute while controlling learning-rate conditions would help isolate the benefit of staged training.

  • The analyses motivating the method rely mostly on perplexity. Appendix E shows that the best settings for perplexity and downstream accuracy differ, so key comparisons involving data transformation and repetition should also be checked on downstream tasks.

  • Training and evaluation details are insufficient for reproducibility. Corpus sources, composition, and benchmark decontamination procedures are not specified. The paper also does not state whether evaluation involves additional SFT or describe prompting, scoring, and metric aggregation. The downstream benchmarks for Turkish, Hungarian, and Bengali in Figure 8 are not identified.

  • Table 3 lacks a baseline at the same training budget and uses a different second-stage mixture from the main experiments. While useful as a large-scale case study, it does not establish the method’s relative advantage without a matched comparison.

Questions for authors
  • How are batches constructed in the proportional-mixing and balanced-repetition baselines? Do either maintain a fixed English/target-language ratio within each batch throughout training? If so, are per-language token counts, total compute, and learning-rate conditions matched to ScaleX or ScaleX-DR?

  • What are the sources and composition of the pretraining corpora, and how is benchmark overlap handled? Are downstream results obtained directly from pretrained checkpoints or after additional SFT? Please specify the benchmarks used for Turkish, Hungarian, and Bengali, along with the prompting, scoring, and aggregation procedures.

2 comments

Thank you for highlighting the schedule comparison. Our results show that a fixed target corpus can support further downstream gains as English training grows, even without additional target repetition. The comparisons below connect this finding to the training controls and task-level evidence.

W1 / Q1 — Training controls and the fixed-corpus result

The controlled schedule comparison is Cos versus Cos-Cos (Section 3.3). At each budget, they share the English-stage learning-rate trajectory; Cos-Cos restarts a fixed cosine for Chinese. Over 10B–25B total tokens, Chinese PPL changes from 30.59 to 32.24 under Cos and from 26.57 to 24.62 under Cos-Cos. Target-stage scheduling thus changes the measured trend.

The main capability result also holds without DR:

Fixed Chinese corpusTotal training tokensChinese task mean (%)Chinese PPL
5B20B→160B21.13→26.2624.83→22.03
20B80B→320B30.22→35.8119.40→18.43

All four Chinese tasks improve between the endpoints in both rows. Batch composition and stage scheduling are separate controls. Cos/Cos-Cos matches the English-stage schedule when testing the Chinese-stage restart. We use this local comparison for the schedule finding and the within-recipe budget sweeps for the fixed-corpus capability finding. Mixing and repetition comparisons evaluate their respective complete recipes.

W2 — Downstream evidence for the diagnostic comparisons

Existing downstream scores complement Figure 4. Each cell gives Chinese PPL / four-task mean (%).

Method20B40B80B160B
Translation26.34 / 18.7426.07 / 19.9426.44 / 20.1727.67 / 19.86
Code-switching27.29 / 18.7727.26 / 18.9728.38 / 19.6430.26 / 20.25
Target-data repetition23.51 / 21.7022.60 / 24.4023.79 / 26.3925.78 / 27.77

Repetition improves the task mean after its best PPL point; code-switching improves the mean from 20B to 160B despite worsening PPL. Translation shows modest gains with fluctuations. These results distinguish likelihood from downstream capability.

For the main result, non-DR ScaleX improves both PPL and all four Chinese tasks between endpoints in both corpus settings. At the two Table 2 endpoints, ScaleX-DR improves every Chinese task and all five Chinese PPL domains relative to proportional mixing.

W3 / Q2 — Benchmark components and score aggregation

The Chinese mean equally weights C-Eval, AGIEval, CMMLU, and Math23K; the English mean equally weights GSM8K, BBH, MMLU, and HumanEval pass@1. Task scores use percentages. PPL is averaged over five domains.

Figure 8 averages two task columns named m_mmlu and m_hellaswag in the evaluation records. ScaleX-DR gives:

Target languagem_mmlu, 50B→450B (%)m_hellaswag, 50B→450B (%)Mean (%)
Chinese30.08→35.1132.45→34.5531.27→34.83
Turkish27.15→31.6130.70→32.4928.93→32.05
Hungarian25.15→31.7629.52→31.3927.34→31.57
Bengali26.04→28.0926.08→26.3926.06→27.24

Both components improve between the English-budget endpoints in every language. The gains differ: Bengali improves more on m_mmlu than m_hellaswag. The names above preserve the evaluation-record identifiers.

W4 — The role of the larger-token-budget case study

Table 3 extends the fixed-corpus finding to larger token budgets and a mixed target stage. The 2.5B model processes 100B unique Chinese tokens twice, with 300B or 800B English tokens, totaling 500B or 1T tokens. Stage two uses 80% Chinese / 20% English.

Increasing the budget raises the Chinese mean from 45.35 to 48.41 and the English mean from 37.47 to 42.39; both PPLs improve. Table 2 provides the recipe comparisons; Table 3 provides this complementary within-recipe budget comparison.

Thank you for the clarifications and additional results. Some responses do not fully address my concerns, particularly regarding the mixing baselines, training controls, and reproducibility details. I am keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.