acceptodds
Under review as a conference paper at ICLR 2027

Read in English, Answer in Hindi? Exploring Data Compositions for Cross-Lingual Knowledge Transfer

Abstract

Recent evidence suggests that multilingual language models (LMs) can inexplicably transfer knowledge acquired in one language to other languages without explicit incentives for such behavior. Factors that govern such transfer aren’t well understood, with past studies disagreeing on what constitutes knowledge transfer and what factors are important. Our work addresses this gap, and studies the role of patterns in multilingual pre-training data in unlocking cross-lingual knowledge transfer. We devise a controlled setup wherein we train language models from scratch on a synthetic corpus comprising biographies across multiple languages. Our experiments highlight that introducing parallel data improves transfer asymmetrically for dissimilar languages. Additionally, how parallel data is introduced also matters: smaller amounts of parallel data across many documents is more beneficial than higher parallelism across fewer documents. In addition to parallel data, the presence of code mixing across documents or complementary information across languages also improves transfer. We find that these findings also hold in real-world settings, where we continually pre-train off-the-shelf models on new knowledge, albeit transfer is moderate. The trends generalize similarly for a two-hop question-answering task, which requires a mix of cross lingual recall and reasoning. Overall, our findings offer data-curation strategies for pre-training that enable knowledge transfer in multilingual language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multilingual training, cross-lingual transfer, data compositions

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5Dt9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how different compositions of multilingual pre-training data influence cross-lingual knowledge transfer (XLT) in language models. The authors focus on three data compositions: knowledge overlap, where factual information is shared across languages; knowledge dispersion, where complementary information is distributed across languages; and language mixing, where multilingual elements are incorporated into documents or question-answer pairs.

To systematically evaluate these factors, the authors establish two experimental settings. The first is a controlled environment in which 135M-parameter language models are trained from scratch on synthetic multilingual biographies in English, French, and Hindi. The second is a more realistic setting involving continual pre-training of Gemma 3 4B on approximately 70K Wikipedia articles across English, Hindi, and Kannada. Cross-lingual factual recall is evaluated using question-answering tasks, with additional experiments on two-hop question answering.

The experimental results demonstrate that introducing parallel data substantially improves cross-lingual knowledge transfer, although the improvements can be asymmetric across linguistically dissimilar languages. The authors further show that distributing parallel information across more documents is more beneficial than concentrating the same amount of parallel information within fewer documents. Interestingly, knowledge dispersion and language mixing also facilitate transfer, and combining different data compositions generally leads to further improvements.

Overall, the paper provides a systematic empirical investigation of multilingual data composition and offers potentially useful guidelines for constructing multilingual pre-training corpora.

Critical strengths

Strength 1: A systematic and well-controlled framework for investigating cross-lingual factual knowledge transfer The paper introduces a systematic experimental framework that allows the effects of different multilingual data compositions to be examined separately.

Unlike evaluations that measure general multilingual task performance, the proposed setting explicitly investigates whether factual knowledge acquired in one language can be recalled in another language. The use of synthetic biographies enables greater control over the distribution of factual information and reduces the risk of evaluation contamination.

Furthermore, the authors investigate three distinct data compositions and provide detailed analyses of language-specific transfer patterns, including the observed asymmetry between English/French and Hindi.

The additional experiments examining tokenizer fertility, language imbalance, and training-batch composition further strengthen the investigation.

Why this strength matters:

This controlled framework is important because identifying the factors responsible for cross-lingual factual knowledge transfer is challenging in existing multilingual models, whose pre-training data is often insufficiently documented.

By isolating different data compositions and examining their effects on factual recall, the paper provides useful empirical evidence that goes beyond conventional multilingual performance comparisons.

Strength 2: Actionable findings on data composition, supported by complementary experiments

The paper provides several practically relevant findings about multilingual data curation.

One particularly interesting observation is that, under the same translation budget, introducing smaller amounts of parallel information across a larger number of documents yields better cross-lingual transfer than concentrating more parallel information within fewer documents (Figure 6).

Another valuable finding is that knowledge dispersion can improve cross-lingual recall even without explicit parallel information. The authors provide additional evidence for this phenomenon through the language-diversity ablation in Appendix D.6, where restricting training batches to a single language substantially reduces the benefits of dispersion.

The experiments also show that different data compositions can complement one another. For example, combining knowledge overlap and QA mixing increases cross-lingual recall from 9.2% to 52.8% in the controlled setting, while corresponding improvements are also observed in the real-world experiments (Figure 2). Why this strength matters: These findings offer concrete guidance for multilingual data construction rather than merely documenting the existence of cross-lingual transfer.

Critical weaknesses

Weakness 1: Insufficient control over training-data size and computational budgets

A central concern is that different data compositions introduce substantially different amounts of training data, making it difficult to isolate the contribution of data composition itself.

According to Appendix A.3, Table 1, the controlled experiments use approximately 13M training tokens for the baseline, 26M for knowledge overlap, 40M for QA mixing, and 48M for the combination of overlap and mixing. Similarly, in the real-world setting, the baseline contains 1.6B tokens, while the overlap and combined configurations contain 2.4B and 2.5B tokens, respectively. These differences are substantial.

Although the authors maintain consistent model architectures and training hyperparameters, the additional data may increase factual exposure, provide more opportunities for memorization, or change the effective amount of optimization.

Therefore, some improvements could arise from increased training data rather than the proposed compositional properties alone. This concern is particularly relevant to QA mixing, which introduces additional multilingual question-answer supervision. It is unclear to what extent the observed improvements arise from language mixing itself rather than increased exposure to answer-generation patterns in different languages.

I acknowledge that the authors provide more carefully controlled experiments, including comparisons with fixed translation budgets in Figure 6 and the knowledge-dispersion setting, which does not increase the overall token count. These experiments partially mitigate the concern. However, they do not fully resolve it for the main comparisons involving overlap, QA mixing, and their combinations.

Why this weakness matters: The paper's primary objective is to establish how specific data compositions affect cross-lingual knowledge transfer. Without sufficiently matched training-data and computational budgets, it is difficult to determine whether the reported improvements are caused by the proposed compositions or by additional training exposure.

This limits the strength of the causal interpretation and the practical implications of the results. A particularly important additional experiment would compare the proposed recipes against baselines that receive an equivalent number of training tokens and optimization steps through repeated or paraphrased monolingual data.

Weakness 2: Limited robustness and generalizability of the real-world evaluation

Although the real-world experiments are a valuable component of the paper, their robustness remains insufficiently established. First, as stated in Appendix B.2, the real-world experiments use only a single training run per data recipe because of computational constraints. The reported uncertainty estimates are obtained through bootstrapping evaluation samples. While bootstrapping captures evaluation-sample variability, it does not account for variability caused by model initialization, training-data ordering, or other stochastic aspects of continual pre-training. Consequently, the stability of the reported improvements across independent training runs remains unclear. Second, the real-world experiments are restricted to one model family and scale, Gemma 3 4B, and three languages. Furthermore, the data consists primarily of Wikipedia articles and translated question-answer pairs. Although this setting provides a useful intermediate evaluation between synthetic experiments and practical multilingual pre-training, it does not fully establish whether the findings generalize to larger models, other multilingual architectures, or naturally occurring multilingual corpora. Third, some reported real-world results require clarification. For example, Figure 2 reports cross-lingual recall values of approximately 12.9% for the baseline and 22.1% for overlap, whereas Table 3 in Appendix B.2 reports 6.3% and 15.3%, respectively, for the default evaluation judge. The manuscript does not clearly explain whether these differences arise from different evaluation subsets, aggregation procedures, or other experimental settings.

Why this weakness matters: The practical significance of the paper relies heavily on demonstrating that the findings extend beyond synthetic experiments. Without multiple independent runs and a clearer explanation of the real-world evaluation protocol, it is difficult to assess the reliability and general applicability of the reported improvements. These issues are especially relevant because the absolute cross-lingual recall performance in the real-world setting remains relatively modest.

Questions for authors
  1. How much of the observed improvement can be attributed specifically to data composition rather than increased training exposure?
  2. Could the authors clarify the differences between the real-world XLR results reported in Figure 2 and Table 3?
1 comment

We thank the reviewer for their positive feedback that helps improve our work. We are glad to know that they appreciate the systematic and well-controlled approach of our study and find our insights actionable. We address their concerns and comments below:

A central concern is that different data compositions introduce substantially different amounts of training data, making it difficult to isolate the contribution of data composition itself.

How much of the observed improvement can be attributed specifically to data composition rather than increased training exposure?

A particularly important additional experiment would compare the proposed recipes against baselines that receive an equivalent number of training tokens and optimization steps through repeated or paraphrased monolingual data.

We agree with the reviewer. Based on their suggestion, we additionally trained models from all recipes with an equal number of steps to fix the training budget. Cross-lingual recall shows the same trends as in our original experiments across both settings, with minor differences in absolute values. This confirms that the improvements in transfer can be attributed solely to differences in data compositions. More details are available in the general comment, and we will include this analysis in the paper.

The reported uncertainty estimates are obtained through bootstrapping evaluation samples. While bootstrapping captures evaluation-sample variability, it does not account for variability caused by model initialization, training-data ordering, or other stochastic aspects of continual pre-training. Consequently, the stability of the reported improvements across independent training runs remains unclear.

We agree, and will state this a limitation of our work in the paper. We currently avoid replicating results across different seeds in the real-world setting due to compute constraints. We note that the current experiments take up to 29 hours per recipe on 2x H200 GPUs. Reproducing the results in figure 2 of the paper with three seeded runs would cost approximately $1.6K and take up to 12 days to complete.

The real-world experiments are restricted to one model family and scale, Gemma 3 4B, and three languages. Furthermore, the data consists primarily of Wikipedia articles and translated question-answer pairs. Although this setting provides a useful intermediate evaluation between synthetic experiments and practical multilingual pre-training, it does not fully establish whether the findings generalize to larger models, other multilingual architectures, or naturally occurring multilingual corpora. [...] it is difficult to assess the reliability and general applicability of the reported improvements.

We agree that our coverage of languages, model architectures and model sizes is limited, and we acknowledge this as a limitation of our work in the paper.

The primary contribution of our work is a controlled setup where we pre-train language models from scratch, to study the effects of various data compositions on cross-lingual knowledge transfer. Subsequent real-world experiments with the Gemma models show that the insights we derive transfer well to a realistic setting, which we view as a validation of our findings.

As trends in recall hold consistently across both settings, we believe our findings reflect the properties of the data compositions instead of a single model, and we see no clear reasons why these will not hold for other models or sizes. We will try to make this framing more explicit throughout the paper, and present our results in this light.

Could the authors clarify the differences between the real-world XLR results reported in Figure 2 and Table 3?

We thank the reviewer for pointing this out. When comparing results across judges, we filter out examples that the base model (prior to training) also gets right based on exact match, to compare variation in evaluation only over newly-transferred knowledge. Below, we provide a comparison of our judges across the full set of examples from the evaluation set:

Evaluation JudgeAgreement
RecipeGemma 4 MoELlama 4 ScoutQwen 3.6Fleiss’ Kappa
Base Model (before training)7.46.28.10.9
Baseline12.911.613.50.9
Mixing (QA)17.314.918.10.9
Dispersion18.717.519.40.9
Overlap22.120.822.90.9
Combination24.823.525.40.9

We find that judges continue to show high agreement across evaluations, confirming the robustness of our judge. We will update the manuscript with the table above.

Reviewer_KaNLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the phenomenon of Cross-Lingual Knowledge Transfer (XLT) in multilingual language models, with a focus on how the composition of multilingual pre-training data affects the model's ability to transfer factual knowledge acquired in one language to other languages. This paper studies factual knowledge transfer, i.e., whether the model can correctly answer questions posed in another language using facts learned only in the source language. The authors further propose three representative multilingual data composition strategies: Knowledge Overlap, Knowledge Dispersion, and Language Mixing. The authors conduct experiments on synthetic data and real data across different languages respectively. The experimental results show that knowledge overlap is an effective strategy for improving cross-lingual factual recall. In addition, knowledge dispersion and language mixing can also promote transfer, and different strategies have a certain degree of complementarity. Overall, by combining controlled experiments with real-world experiments, this paper provides valuable empirical insights into the knowledge transfer in multilingual models and offers valuable data construction suggestions.

Critical strengths
  • This paper investigates a valuable and interesting question in multilingualism: whether factual knowledge acquired by models in high-resource languages can be effectively utilized in low-resource languages, and what kinds of data composition can facilitate this process. At the same time, the article also provides a clear and explicit definition of XLT, making the research more targeted and easier to follow.
  • The paper simultaneously adopts two complementary experimental designs: controlled experiments based on synthetic fictitious data and real-world experiments based on real data, further strengthening the reliability of the conclusions.
  • This paper is simple, well-motivated, and easy to follow. Detailed analyses and results on a variety of experimental settings and datasets further strengthen the conclusions.
  • This paper provides a clear and systematic classification of data composition, dividing data composition methods into Knowledge Overlap, Knowledge Dispersion, and Language Mixing, and controlling them through a variety of different parameters. This enables the authors to effectively study the effects of different strategies and their interactions, and to provide valuable insights for different strategies.
Critical weaknesses
  • There are significant differences in the number of training tokens across different data strategies in Table 1, which weakens the attribution of effectiveness to data composition strategies. The paper aims to study how different data composition methods affect XLT, but different strategies not only change the way data is composed but also simultaneously change the total amount of training data. Further controlled experiments may be helpful.
  • The XLR metric has not yet sufficiently distinguished between knowledge acquisition, cross-lingual transfer, and target language output capabilities. This paper defines XLR as the accuracy with which a model uses facts learned in the source language to answer questions in the target language. However, besides the model not mastering the target knowledge, there may be multiple reasons for the model's failure to answer in the target language. The current XLR reflects all possible reasons together in a single accuracy metric. For example, in Figure 4, the low XLR when Hindi is the target language may partly stem from insufficient generation capability in the target language, and does not necessarily fully represent an inability to transfer cross-lingual knowledge.
  • The author use Wikipedia article data in real-world experiments and then used TranslateGemma 27B to translate the content into Hindi and Kannada. However, the data obtained in this way is not entirely consistent with the distribution of naturally occurring multilingual data. The complementary knowledge in real multilingual Wikipedia may come from independent edits by different language communities, involving different cultural backgrounds and narrative conventions. In contrast, this paper essentially artificially partitions information from the same set of facts from a single source.
  • The article's coverage of languages, model sizes, and task types remains limited, and the generalizability of its conclusions across more diverse experimental settings is still unclear.
Questions for authors

Please refer to the weaknesses above.

2 comments

We thank the reviewer for their extensive and positive feedback on our work. We are pleased to know that they find the paper well-motivated and easy-to-follow, appreciate the study design and the experiment settings, and find our conclusions well grounded in evidence. We address some of their concerns below:

  • Differences in training tokens weakens attribution to data compositions

We agree with the reviewer’s concern. To isolate the effect of training tokens, we extend all our training runs to the same number of steps, fixing the number of total training tokens for all recipes. Cross-lingual recall at the final checkpoint shows the same trends as our original results, as all data compositions improve transfer over the baseline. This confirms that the improvements can be attributed to the data compositions. Further details are provided in the general comment, and we will include these details in the paper.

  • Cross-lingual recall (XLR) is not a measure of cross-lingual transfer in isolation

While cross-lingual recall (XLR) is defined as an end-to-end measure, and it may appear to conflate errors from other failure modes such as poor generation quality or knowledge acquisition, our setup and evaluation criteria ensures that these failures are progressively ruled out. As part of our setup, we train models until convergence, i.e., until the models can successfully complete facts that are part of the training biographies. This ensures that knowledge in the source language is acquired, ruling out failures due to lack of knowledge acquisition. We then select the best checkpoint across the course of training, based on a held-out validation set. Over this checkpoint, multilingual recall (MLR) over the facts seen in training is extremely high in all languages, further ruling out issues with generation quality, as shown in the table below.

RecipeBiography Recall (avg, %)Biography Recall (hi, %)MLR on train set (avg, %)MLR on train set (hi, %)
Baseline99.999.999.699.6
Mixing (QA)95.494.697.295.8
Dispersion94.888.799.899.9
Overlap99.699.299.599.0

With other failure modes ruled out, XLR measures the extent of cross-lingual transfer, and a low XLR reflects a failure to transfer knowledge. We thank the reviewer for pointing this out, and will include this explanation in the paper to clarify what XLR measures.

  • Use of translations and relevance to naturally occurring multilingual data

We agree that translated data may not reflect the distribution of naturally-occurring multilingual data. We use translations because partitioning facts from a single source language provides us with precise control of the composition of the training corpus, allowing us to isolate its effect on transfer. With this control, our experiments progress from a fully controlled synthetic setting to a real-world setting, where we observe consistent findings. The use of translations is also relevant in practice, as prior work finds that a non-trivial fraction of web-content is machine-translated [1]. Our work focuses specifically on language-agnostic knowledge, and we do not claim our findings extend to knowledge that differs across cultures. We will clarify our scope more clearly in the paper.


[1] Brian Thompson, Mehak Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. 2024. A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1763–1775, Bangkok, Thailand. Association for Computational Linguistics.

Reviewer_KaNLOpenReview ↗

Official Comment by Reviewer KaNL

Thanks for your reply. The newly added experimental results and explanations have basically addressed my main concerns. So I decide to maintain my positive score. The authors should include these new details in the next version and further clarify the boundaries of the current work.

Reviewer_4Bm4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the role of data patterns in pre-training that enables cross lingual knowledge transfer. They observe that models can often recall and apply knowledge learnt in other languages without any explicit training. To study the effects of cross-lingual transfer, they use a synthetic multilingual dataset of fictitious biographies and train small language models from scratch. They test three settings (1) Knowledge Overlap (2) Knowledge Dispersion and (3) Codemixing. They use three languages for their experiments: English, French, Hindi.

To simulate the real world setting, they also continually pre-train models on a corpus of M = 70K documents from Wikipedia. Overlap yields the highest improvements in cross lingual recall and additionally shows improvements in two-hop questions. The use of code-mixing within a document or a QA pair improves the balance of transfer between languages.

Critical strengths
  1. The paper shows controlled as well as real world setting for data curation recipes. The controlled setup helps study transfer across languages and gains can be contributed to the data recipe directly. The real world setting where they continually train on Wikipedia documents shows transferability of results to real world settings. The paper demonstrates several results that can be adopted for data curation to enable better cross lingual transfer: (1) usefulness of overlap upto 50%. (2) greater coverage is more useful than higher extent of overlap.

  2. Strong evaluation protocol: The authors add controls for test set contamination. Each test fact lives in exactly one language and the real world setup uses Wikipedia articles post a cutoff.

Critical weaknesses
  1. Data tokens seems to be a confound in experiments: As per the experiment setup, each of the data mixing experiments use different no. of tokens for training. Overlap would use 2x the tokens and also more compute. It is not clear how much of the gains come from training on more tokens vs cross-lingual training,

  2. The baseline performance of Hindi on in Fig 4 (a) is 4%, which is much lower than English (47.5%) and French (43.3%). Asymmetric transfer in Sec 5.2 might be due to the model not being able to recall facts in Hindi in the baseline. I would recommend adding the cause of the low performance on baseline model while discussing results.

Questions for authors
  1. For QA mixing, some of the gains could be coming from the model being introduced to the prompt format. Do you have any experiments that demonstrate how much of the gain comes from learning the format?
1 comment

We thank the reviewer for their positive assessment of our work. We are grateful to know they appreciate our evaluation protocol for measuring transfer and the extension of our experiments to real-world settings, and find our insights actionable for data curation. We address their comments and suggestions below:

Data tokens seem to be a confound in experiments: [...] It is not clear how much of the gains come from training on more tokens vs cross-lingual training.

We agree this is a fair concern. To rule out the effects of differences in training tokens, we extend the training runs of all recipes to an equal number of steps. As the tokens per step is roughly the same, this equalizes the total training tokens across all recipes.

Results at both the best and final checkpoint show consistent trends for cross-lingual recall as in our original experiments. This confirms that the improvements in transfer are due to data compositions and not differences in data tokens. We provide more details in the general comment, and will include these results in the paper.

The baseline performance of Hindi in Fig 4 (a) is 4%, which is much lower than English (47.5%) and French (43.3%). Asymmetric transfer in Sec 5.2 might be due to the model not being able to recall facts in Hindi in the baseline. I would recommend adding the cause of the low performance on the baseline model while discussing results.

Thank you for this observation. We agree that the low baseline recall for Hindi could be relevant to the asymmetry across a few other recipes, and we will discuss this in the revision. However, we would suggest treating this as an association and not a cause. Poor recall in Hindi and transfer to other languages may itself originate from a deeper underlying cause, instead of one explaining the other.

For QA mixing, some of the gains could be coming from the model being introduced to the prompt format. Do you have any experiments that demonstrate how much of the gain comes from learning the format?

This is an interesting question. In general, models for all recipes are trained on both documents and QA pairs, so the general prompt format is already observed during training. In language mixing we additionally introduce a marker in the prompt to indicate the answer language. This is included to make sure that the predictions are unambiguous, otherwise for the same question there are multiple answers observed during training.

To test whether the format in language mixing provides any advantage, we train models for some recipes in the controlled setting with the language mixing prompt format. As shown in the table below, cross-lingual recall does not improve over the original values. This suggests that the differences in prompt format do not explain the improvements from language mixing.

RecipeXLR (%)XLR, with mixing prompt (%)
Baseline9.29.0
Dispersion23.721.5
Overlap41.439.7

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We are grateful to all the reviewers for their careful consideration and positive assessment of our work. We are pleased to know they found the paper clear and easy to follow, appreciated the experiment setup (4Bm4, KaNL) and the evaluation protocol (4Bm4), and found our insights actionable (KaNL, 5Dt9).

All reviewers asked whether differences in training tokens across data compositions explain our results, in addition to the compositions themselves. We agree this is a fair concern. Compositions such as overlap and mixing change the token count of the training corpus, and recipes that see more tokens may have an advantage.

We would like to highlight that while the training data across compositions varies, the number of facts in each corpus remains exactly the same. Compositions do not add new information to the data. Across a single epoch, all models are trained over the same amount of information.

To rule out budget effects, we additionally train all recipes for the same number of steps (150K in the controlled setting and 7.5K in the real-world setting, i.e., the maximum across recipes). Since tokens per step are roughly the same across recipes, this fixes the total budget. However, we note that recipes with smaller datasets, such as our baseline, now run for more epochs. The table below reports cross-lingual recall (XLR) at the final step and at the best checkpoint, as in our original protocol.

RecipeXLR, Controlled Setting (%)XLR, Real-World Setting (%)
CheckpointFinal StepBest CheckpointFinal StepBest Checkpoint
Baseline9.19.212.512.9
Mixing (QA)15.015.416.717.3
Dispersion20.823.717.918.7
Overlap41.441.421.622.1
Combination (Overlap + Mixing)52.152.824.124.8

From the table, we observe consistent trends for recall as in our original experiments. All recipes improve cross-lingual transfer in both settings. Additionally, the best checkpoints for recipes like baseline and dispersion occur early in training, suggesting no benefit from a higher training budget. Together, this confirms that the improvements in cross-lingual transfer are due to differences in data compositions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.