- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how different compositions of multilingual pre-training data influence cross-lingual knowledge transfer (XLT) in language models. The authors focus on three data compositions: knowledge overlap, where factual information is shared across languages; knowledge dispersion, where complementary information is distributed across languages; and language mixing, where multilingual elements are incorporated into documents or question-answer pairs.
To systematically evaluate these factors, the authors establish two experimental settings. The first is a controlled environment in which 135M-parameter language models are trained from scratch on synthetic multilingual biographies in English, French, and Hindi. The second is a more realistic setting involving continual pre-training of Gemma 3 4B on approximately 70K Wikipedia articles across English, Hindi, and Kannada. Cross-lingual factual recall is evaluated using question-answering tasks, with additional experiments on two-hop question answering.
The experimental results demonstrate that introducing parallel data substantially improves cross-lingual knowledge transfer, although the improvements can be asymmetric across linguistically dissimilar languages. The authors further show that distributing parallel information across more documents is more beneficial than concentrating the same amount of parallel information within fewer documents. Interestingly, knowledge dispersion and language mixing also facilitate transfer, and combining different data compositions generally leads to further improvements.
Overall, the paper provides a systematic empirical investigation of multilingual data composition and offers potentially useful guidelines for constructing multilingual pre-training corpora.
Critical strengths
Strength 1: A systematic and well-controlled framework for investigating cross-lingual factual knowledge transfer The paper introduces a systematic experimental framework that allows the effects of different multilingual data compositions to be examined separately.
Unlike evaluations that measure general multilingual task performance, the proposed setting explicitly investigates whether factual knowledge acquired in one language can be recalled in another language. The use of synthetic biographies enables greater control over the distribution of factual information and reduces the risk of evaluation contamination.
Furthermore, the authors investigate three distinct data compositions and provide detailed analyses of language-specific transfer patterns, including the observed asymmetry between English/French and Hindi.
The additional experiments examining tokenizer fertility, language imbalance, and training-batch composition further strengthen the investigation.
Why this strength matters:
This controlled framework is important because identifying the factors responsible for cross-lingual factual knowledge transfer is challenging in existing multilingual models, whose pre-training data is often insufficiently documented.
By isolating different data compositions and examining their effects on factual recall, the paper provides useful empirical evidence that goes beyond conventional multilingual performance comparisons.
Strength 2: Actionable findings on data composition, supported by complementary experiments
The paper provides several practically relevant findings about multilingual data curation.
One particularly interesting observation is that, under the same translation budget, introducing smaller amounts of parallel information across a larger number of documents yields better cross-lingual transfer than concentrating more parallel information within fewer documents (Figure 6).
Another valuable finding is that knowledge dispersion can improve cross-lingual recall even without explicit parallel information. The authors provide additional evidence for this phenomenon through the language-diversity ablation in Appendix D.6, where restricting training batches to a single language substantially reduces the benefits of dispersion.
The experiments also show that different data compositions can complement one another. For example, combining knowledge overlap and QA mixing increases cross-lingual recall from 9.2% to 52.8% in the controlled setting, while corresponding improvements are also observed in the real-world experiments (Figure 2). Why this strength matters: These findings offer concrete guidance for multilingual data construction rather than merely documenting the existence of cross-lingual transfer.
Critical weaknesses
Weakness 1: Insufficient control over training-data size and computational budgets
A central concern is that different data compositions introduce substantially different amounts of training data, making it difficult to isolate the contribution of data composition itself.
According to Appendix A.3, Table 1, the controlled experiments use approximately 13M training tokens for the baseline, 26M for knowledge overlap, 40M for QA mixing, and 48M for the combination of overlap and mixing. Similarly, in the real-world setting, the baseline contains 1.6B tokens, while the overlap and combined configurations contain 2.4B and 2.5B tokens, respectively. These differences are substantial.
Although the authors maintain consistent model architectures and training hyperparameters, the additional data may increase factual exposure, provide more opportunities for memorization, or change the effective amount of optimization.
Therefore, some improvements could arise from increased training data rather than the proposed compositional properties alone. This concern is particularly relevant to QA mixing, which introduces additional multilingual question-answer supervision. It is unclear to what extent the observed improvements arise from language mixing itself rather than increased exposure to answer-generation patterns in different languages.
I acknowledge that the authors provide more carefully controlled experiments, including comparisons with fixed translation budgets in Figure 6 and the knowledge-dispersion setting, which does not increase the overall token count. These experiments partially mitigate the concern. However, they do not fully resolve it for the main comparisons involving overlap, QA mixing, and their combinations.
Why this weakness matters: The paper's primary objective is to establish how specific data compositions affect cross-lingual knowledge transfer. Without sufficiently matched training-data and computational budgets, it is difficult to determine whether the reported improvements are caused by the proposed compositions or by additional training exposure.
This limits the strength of the causal interpretation and the practical implications of the results. A particularly important additional experiment would compare the proposed recipes against baselines that receive an equivalent number of training tokens and optimization steps through repeated or paraphrased monolingual data.
Weakness 2: Limited robustness and generalizability of the real-world evaluation
Although the real-world experiments are a valuable component of the paper, their robustness remains insufficiently established. First, as stated in Appendix B.2, the real-world experiments use only a single training run per data recipe because of computational constraints. The reported uncertainty estimates are obtained through bootstrapping evaluation samples. While bootstrapping captures evaluation-sample variability, it does not account for variability caused by model initialization, training-data ordering, or other stochastic aspects of continual pre-training. Consequently, the stability of the reported improvements across independent training runs remains unclear. Second, the real-world experiments are restricted to one model family and scale, Gemma 3 4B, and three languages. Furthermore, the data consists primarily of Wikipedia articles and translated question-answer pairs. Although this setting provides a useful intermediate evaluation between synthetic experiments and practical multilingual pre-training, it does not fully establish whether the findings generalize to larger models, other multilingual architectures, or naturally occurring multilingual corpora. Third, some reported real-world results require clarification. For example, Figure 2 reports cross-lingual recall values of approximately 12.9% for the baseline and 22.1% for overlap, whereas Table 3 in Appendix B.2 reports 6.3% and 15.3%, respectively, for the default evaluation judge. The manuscript does not clearly explain whether these differences arise from different evaluation subsets, aggregation procedures, or other experimental settings.
Why this weakness matters: The practical significance of the paper relies heavily on demonstrating that the findings extend beyond synthetic experiments. Without multiple independent runs and a clearer explanation of the real-world evaluation protocol, it is difficult to assess the reliability and general applicability of the reported improvements. These issues are especially relevant because the absolute cross-lingual recall performance in the real-world setting remains relatively modest.
Questions for authors
- How much of the observed improvement can be attributed specifically to data composition rather than increased training exposure?
- Could the authors clarify the differences between the real-world XLR results reported in Figure 2 and Table 3?
We thank the reviewer for their positive feedback that helps improve our work. We are glad to know that they appreciate the systematic and well-controlled approach of our study and find our insights actionable. We address their concerns and comments below:
We agree with the reviewer. Based on their suggestion, we additionally trained models from all recipes with an equal number of steps to fix the training budget. Cross-lingual recall shows the same trends as in our original experiments across both settings, with minor differences in absolute values. This confirms that the improvements in transfer can be attributed solely to differences in data compositions. More details are available in the general comment, and we will include this analysis in the paper.
We agree, and will state this a limitation of our work in the paper. We currently avoid replicating results across different seeds in the real-world setting due to compute constraints. We note that the current experiments take up to 29 hours per recipe on 2x H200 GPUs. Reproducing the results in figure 2 of the paper with three seeded runs would cost approximately $1.6K and take up to 12 days to complete.
We agree that our coverage of languages, model architectures and model sizes is limited, and we acknowledge this as a limitation of our work in the paper.
The primary contribution of our work is a controlled setup where we pre-train language models from scratch, to study the effects of various data compositions on cross-lingual knowledge transfer. Subsequent real-world experiments with the Gemma models show that the insights we derive transfer well to a realistic setting, which we view as a validation of our findings.
As trends in recall hold consistently across both settings, we believe our findings reflect the properties of the data compositions instead of a single model, and we see no clear reasons why these will not hold for other models or sizes. We will try to make this framing more explicit throughout the paper, and present our results in this light.
We thank the reviewer for pointing this out. When comparing results across judges, we filter out examples that the base model (prior to training) also gets right based on exact match, to compare variation in evaluation only over newly-transferred knowledge. Below, we provide a comparison of our judges across the full set of examples from the evaluation set:
We find that judges continue to show high agreement across evaluations, confirming the robustness of our judge. We will update the manuscript with the table above.