acceptodds
Under review as a conference paper at ICLR 2027

Why Pretraining Fails to Share Cross-Lingual Knowledge

Abstract

Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6% of native-language learning efficiency — 14 the baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cross-lingual, pretraining, generalization, transfer, multilingual, tokenizer, multi-lignual LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BUWQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how cross-lingual transfer emerges in LLMs during pretraining, evaluated through cross-lingual factual QA. The authors pretrain models from scratch under controlled conditions: (i) multilingually, showing that poor cross-lingual alignment already arises during pretraining, and (ii) monolingually with disjoint token sets. They identify disjoint tokens as a main driver of poor cross-lingual alignment and derive simple remedies to improve the cross-lingual abilities of LLMs.

Critical strengths
  • The paper addresses an important problem with an original and well-controlled experimental approach.
  • The focus on tokenization as a driver of cross-lingual misalignment is valuable, since this factor receives too little attention in the literature.
  • The analyses build on each other naturally, and the paper is well written overall (exceptions below).
  • The finding that a method as simple as WWT resolves a large part of the problem is convincing and opens a promising direction for future work.
Critical weaknesses
  • Scale and undertraining (major weakness). It is unclear whether the findings transfer to models trained at realistic scale. The 7B model is pretrained on 140B tokens, roughly an order of magnitude less than comparable models (e.g., 2T tokens for LLaMA 2). Some cross-lingual abilities may only emerge later in training, so the weak representational alignment observed here may partly reflect undertraining. The approach remains interesting, but this limitation narrows the scope of the conclusions and should be stated far more prominently.
  • Definition of Cross-lingual Knowledge Generalization (major weakness). The computation of this metric is unintuitive. The obvious formulation would be the accuracy on a fact in one language for a model trained on that fact in another language. The paper should motivate why the more complex formulation is needed.
  • Presentation of FKD (minor weakness). The FKD descriptions are currently confined to the appendix fully, I think they should be moved to the main paper, together with examples, since the setup is currently somewhat hard to follow without them.
Questions for authors

Questions:

  • Why is Cross-lingual Knowledge Generalization computed in its current form instead of as plain cross-lingual accuracy? What does the more complex formulation capture in addition?
  • Do the authors have evidence (e.g., alignment trends across checkpoints) that the observed misalignment persists with longer training?

Suggestions:

  • Investigating pretraining data mixes beyond 50/50 would strengthen the paper. A more realistic setting, such as 80% English and 20% in a second language, would allow stronger conclusions about state-of-the-art models.
6 comments

scale and undertraining

Scale is a natural problem in modern LLM research, while industry scales keep increasing, academic scales remain bound by financial constraints. We aim to provide a close-to-realistic scale analysis with our 7B experiments, focusing on key findings, yet increasing the token budget to LLaMA 2 scales remains unrealistic at our budget (we estimate a single pretraining run of that configuration at 200K\givena1 given a 1\\/h GPU cost). To solidify our claims across scale, however, we turn towards adding more scale configurations to our experiments. We repeat our 3 key experiments with models of 140M, 220M, and 580M parameters, and find consistent results.

Regarding undertraining, while we generally followed compute-optimal 20x Chinchilla-based scaling, following your suggestion, we experiment with a closer to modern realistic data ratio — 100x. Due to the previously mentioned compute limitations, we run these experiments at the 220M parameter scale. Our findings hold in this setting.

We add a detailed report of these new scales and token budget to the paper in a new appendix M referenced in the main text. We also add a clause to our limitations appendix stating our scale limitations more clearly. For your convenience, we report the new results below:

ScaleTraining TokensEnglish1–English2English–ArabicEnglish–WWT-Ar
140m2.9B0.05%0.95%12.39%
220m4.4B-1.43%1.22%12.78%
580m11.7B-2.24%-1.07%11.65%
220m22B-1.18%1.02%12.59%

CLeqCL_{eq} measured for our 3 main bilingual setups across various scale and training budget configurations.

motivation for the CLeqCL_{eq} formulation

Thank you for this comment, we agree that a clear motivation paragraph for choosing our more complex metric would strengthen the paper's clarity. We add the following paragraph to our metric definition section 2.2: "The obvious formulation is accuracy in language A on facts learned in language B. While this approach makes sense when each fact is learned in only one language, in practice, a model may see the same fact in both languages, at different frequencies. FKD allows modeling precise injection in both A and B at different rates, covering a wider range of cross-lingual dynamics. Trying to capture these dynamics through raw accuracy would conflate cross-lingual knowledge generalization with the learning curves of the specific model configuration tested, e.g., larger models acquire knowledge from less exposures and reach higher accuracies (Chang et al., 2024a), which would synthetically inflate their score in low-frequency native exposure scenarios. Therefore, we introduce a metric that normalizes for these differences in learning curves and natively captures complex bilingual injection dynamics — the Cross-Lingual equivalence (CLeqCL_{eq}) score."

FKD presentation in the main text

Thank you for pointing this out, we move some of the FKD description from the appendix to section 2.1. we also add a new Figure2 to our main text to help clarify FKD.

Reviewer_T7RKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates limited cross-lingual factual knowledge transfer during pretraining. It controls language-specific exposure to fictitious facts and introduces CLeq, a metric that measures the effect of cross-lingual exposure relative to direct exposure in the evaluation language. Experiments with identical language copies represented in disjoint token spaces show that token-space separation alone can hinder transfer. The paper also proposes word-wise translation (WWT) to enable token sharing and evaluates factual knowledge transfer and language modeling across two language pairs and two model scales.

Critical strengths
  • Controlling language-specific exposure to fictitious facts provides a useful framework for studying knowledge transfer. The setup explicitly varies where and how often each fact appears, quantities that are difficult to track in natural pretraining corpora. Measuring transfer relative to direct learning makes the analysis more interpretable than a comparison of raw accuracy across languages.

  • The cloned-language experiments systematically compare disjoint token spaces, different degrees of embedding sharing, and identical initialization. By removing differences in linguistic structure, these experiments isolate how representation sharing affects transfer. They provide evidence that learning the same underlying language distribution does not automatically produce knowledge sharing across separate token systems.

Critical weaknesses
  • Prior work has demonstrated cross-lingual transfer without shared vocabularies. Although the paper cites this literature, it does not sufficiently explain which conditions, such as the pretraining objective or available alignment signals, account for the differing findings beyond the distinction between task transfer and factual knowledge transfer. The cloned-English experiments show that token-space separation can hinder transfer, but provide limited evidence that it is the primary cause in natural multilingual settings.

Emerging Cross-lingual Structure in Pretrained Language Models. ACL 2020.

  • Despite pretraining on substantial amounts of general text and transforming the entire input representation, evaluation focuses on injected fictitious facts through MCQA and on language modeling metrics. Without external evaluations, such as multilingual MMLU or reading comprehension in both English and the target languages, it remains unestablished whether CLeq gains translate into broader knowledge and reasoning improvements or come at the expense of other capabilities.

  • The paper reports corpus-level dictionary coverage and manual inspection of some generated data, but provides limited quantitative analysis of mapping coverage and semantic alignment within FKD training documents, evaluation questions, and answer choices. In particular, transfer results may depend on how closely entity names and key answer expressions match their English counterparts after mapping. Performance breakdowns by alignment quality and mapping errors are needed to establish the conditions and robustness of WWT’s gains.

  • The paper does not adequately describe how word boundaries, affixes, inflected forms, and punctuation are handled in WWT. These choices directly affect dictionary size, coverage, and mapping accuracy. The distinction between mapping whole surface forms and segmenting them into morphemes may be particularly consequential for agglutinative languages. With natural-language experiments limited to English–Arabic and English–Russian, applicability to languages with other morphological characteristics remains untested.

Questions for authors

Please refer to the Weaknesses section

10 comments

Prior work findings on transfer without shared vocabularies

Similar to previous works (including the one mentioned), we find naturally emerging structural similarity between language representations in the model (Section 5.2 and Appendix I). That is, some form of cross-lingual generalization does occur. We measure how much this translates into the factual knowledge domain, and find it to be very weak. This phenomenon is documented (Cross-lingual consistency of factual knowledge in multilingual language models, Qi et al., 2023; Beneath the surface of consistency: Exploring cross-lingual knowledge representation sharing in LLMs, Ifergan et al., 2025; Eclektic: a novel challenge set for evaluation of cross-lingual knowledge transfer, Goldman et al., 2025); our contribution is the mechanism. What accounts for the differing findings between task transfer and factual knowledge transfer is outside the scope of this paper. We attempt to address this tension more clearly in the revised Related Work section lines 489-493.

The cloned-English experiments show that token-space separation can hinder transfer, but provide limited evidence that it is the primary cause in natural multilingual settings.

We do not argue that token-space separation is the only or primary cause for the failure in natural multilingual settings, instead, we argue that it is a fundamental blocker to knowledge generalization.

Section 6.3:

"We conclude that while linguistic differences might dampen cross-lingual knowledge generalization in LLMs, disjoint token spaces gate it."

Conclusions:

"Tokenization is a substantial part of the story, but not all of it. While token space merging provides significant gains in knowledge generalization, it is far from resolving compartmentalization completely."

The cloned-English experiments show that token-space separation alone precludes generalization, establishing separate token spaces as a fundamental blocker. Our English WWT-Ar experiments show that even partially resolving disjoint token spaces provides substantial improvements to knowledge generalization.

external benchmarks beyond CLeq

We appreciate the feedback and provide results of common benchmarks across reading comprehension, commonsense reasoning and knowledge, for our trained 360M parameter models (Global MMLU is not included as it is too difficult at this scale; results are around chance). We add these results to our paper, referencing them in sections 5.2 and 6.2. We include bounds to help interpret the results: lower bound trained on the English half of the data and an upper bound where the second language is replaced with more English data. Notice that the English1–English2 and English–Arabic results are close to the lower bound, demonstrating our measured heavy compartmentalization on downstream tasks, English–WWT-Ar performs closer to the upper bound:

BenchmarknnRandomabsolute compartment.English1–English2English–ArabicEnglish–WWT-Arperfect generalization
RACE104525.027.7 ±1.428.0 ±1.428.7 ±1.429.8 ±1.430.2 ±1.4
PIQA183850.063.2 ±1.163.1 ±1.162.8 ±1.163.4 ±1.164.6 ±1.1
ARC-Easy235925.048.8 ±1.048.5 ±1.049.9 ±1.051.6 ±1.052.9 ±1.0
SciQ100025.076.9 ±1.376.1 ±1.377.9 ±1.379.8 ±1.380.5 ±1.3
BMLAMA601610.747.6 ±0.651.4 ±0.649.8 ±0.655.6 ±0.655.2 ±0.6
Average52.8 ±0.553.4 ±0.553.8 ±0.556.0 ±0.556.7 ±0.5

Zero-shot accuracy (%) on English benchmarks of 360M parameter models. Absolute compartmentalization (a monolingual English model trained on 3.6B tokens) and perfect generalization (a monolingual English model trained on 7.2B tokens) serve as lower and upper bounds; bold marks the best result excluding them.

Arabic performance is not directly comparable, as the WWT-Ar model is at a data disadvantage due to higher token fertility. Despite this disadvantage, WWT-Ar gains are measurable here as well:

BenchmarknnRandomEnglish–ArabicEnglish–WWT-Ar
PIQA183850.052.9 ±1.253.3 ±1.2
ARC-Easy235925.040.4 ±1.043.1 ±1.0
SciQ99525.055.1 ±1.659.0 ±1.6
BMLAMA601610.744.8 ±0.647.2 ±0.6
Average48.3 ±0.650.6 ±0.6

Zero-shot accuracy (%) on Arabic benchmarks of 360M parameter models. The benchmarks are machine-translated from English (Almazrouei et al., 2023; Han et al., 2025). WWT-Ar model is evaluated by applying WWT to the benchmark text before tokenization.

Reviewer_rfArOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the cross-lingual transfer of factual knowledge during bilingual pre-training. By constructing a Fictional Knowledge Dataset (FKD), proposing a novel metric Cross-Lingual equivalence score (CLeqCL_{eq}), and comparing different tokenizer / embedding settings (including English clone, embedding sharing, word-wise translation or WWT, etc.), it finds that: (1) disjoint token spaces are responsible for the low transfer efficacy; (2) sharing tokens or embeddings can effectively raise the transfer efficacy.

Critical strengths
  1. The metric CLeqCL_{eq} is interesting and inspiring.

  2. WWT provides a novel setting for cross-lingual transfer experiments.

Critical weaknesses
  1. The experimental settings are inconsistent in the paper and the code repository. For example, the paper says there are 2048 seed facts, but in the repo there are 2304 in gemini_seeds.txt; The paper reports the size of FKD is 2048, but in the repo multiple settings write range(2080), indicating 2080 seed facts are used during training; The paper says 200-300 documents are generated for each entity, but in the repo DATA_COUNT is by default 400; The paper says the generation model is GPT-5.5-mini, but the repo has gpt-5-mini-2025-08-07. Also, there seems to be no mechanism to prevent duplication of entity names or documents when expanding seed facts to training samples.

  2. The conclusion, "disjoint tokens space is a root cause of compartmentalization", is contradicted by the results of the WWT experiments. In the WWT experiments, English and Arabic-WWT (Arabic grammar, English words) nearly share all their tokens, but the CLeqCL_{eq} score is only 12.5% for the 7B model, which is a lot lower than that between English and English clone even with no shared updates, which is 78.6%. This result indicates that shared vocabulary can only activate a small portion of transfer efficacy, leaving a lot of room for other factors such as grammar.

  3. The CLeqCL_{eq} metric uses linear regression on the QA accuracy vs. language-wise fact exposure. However, in Figure C.1 we can see the Acc curve is highly non-linear. Also, Table E.2 shows that curve-aware fitting can reach much higher R2R^2 than linear ones. As a result, keep using linear fitting makes the results to some extent misleading (for example, based on Figure C.1, English accuracy grows for about 48pp with 100 exposure, much higher than the fitting β\beta of 4-5pp per 100 exposure), and using derivatives with curve-aware fitting, or just use an interval (e.g. 40%-80% accuracy) in the linear fitting can be better.

  4. The paper talks about WWT in the abstract and introduction as if it is a new training mechanism that can improve multilingual pretraining. However, it is only a extreme experimental setting and cannot be applied to real training. Who will put samples with English words and Arabic grammar in the pretraining corpus to improve this artificial language?

  5. There are lots of redundant experiments in the paper, making the logic unclear and the reading difficult. Please remove the results that cannot support your main idea for clarity.

Questions for authors

None

17 comments

FKD inconsistencies and diversity concerns:

Thank you for bringing these inconsistencies with FKD to our attention, we fix them, the paper stands as is and we will better document the code to explain the inconsistencies below. We fixed the actual number of entities used to 2080, the generator model to gpt-5-mini, and documents per entity to 200-400 in the relevant places. Regarding the 2304 entities in our released dataset, our experiments used only the first 2080 entities as our code shows (the rest of the entities were generated at a later stage). We decided to upload the full 2304 entities generated as a resource for the community.

Regarding duplicates, your point is well taken -- thank you! Following your comment, we conducted a check of FKD's diversity and report the results in a new Appendix C.5. The bottom line is: we found less than 3% entity name duplicates and 0.22% injection document duplicates, and following further checks like distinct-n score (detailed in C.5) and previous literature, we argue these experiments have a sufficient level of data diversity.

The conclusion… is contradicted by the results … shared vocabulary… leaving room for other factors such as grammar.

We believe your comment is a result of a misunderstanding: The experiment referenced, where CLeqCL_{eq} reached 78.6%, is not the plain English-with-English-clone experiment, it is a variant of it that tests a solution to disjoint token spaces via shared initialization. Plain English-with-English-clone experiments, where disjoint token space is the only difference between languages, show almost no knowledge sharing: CLeqCL_{eq} scores of -2.6% and -1.5% for 360M and 7B parameter models. This trend is consistent across scale: 0.1%, -1.4% and -2.2% for 140M, 220M and 580M parameters.

We clarify in lines 508-516 (in the revised submission) the paper's conclusion and how we arrive at it. "Our results converge on a single claim: cross-lingual knowledge compartmentalization originates in pretraining and is in substantial part a tokenization phenomenon. Two controlled experiments establish this. Pairing English with a token-disjoint clone of itself shows that disjoint token spaces alone suffice to cause compartmentalization (CLeqCL_{eq} = -2.6%), even between languages that differ in nothing but their token inventory. Conversely, resolving the disjoint token spaces of a natural language pair via literal translation (WWT-Ar) raises cross-lingual knowledge generalization from 0.9% to 12.6%, through an intervention that changes nothing but the tokens. Thus, while linguistic differences might account for the remaining large gaps in cross-lingual knowledge generalization in LLMs, disjoint token spaces gate it."

Linear fitting of a non-linear accuracy curve

Regarding the concern about misleading method: Our Appendix E provides a curve-aware metric (see table E.2), supporting our main results. We appreciate your suggestion to use interval linear fitting. We experiment with this and provide results below, finding this metric is mostly consistent with what we report.

EstimatorEnglish–ArabicEnglish–WWT-ArEnglish1–English2
all cells (paper)0.93%12.64%-2.62%
40–80% interval0.20%13.60%-0.30%

CLeqCL_{eq} of our 3 main bilingual setups at 360M parameter scale under the linear estimator and the interval linear estimator restricted to cells with accuracy between 40% and 80%.

We note, however, that focusing the fitting down to an interval means it is based on fewer data points: in this case roughly 50% of points survive. Given these findings, do you think we should replace our metric with the interval-fitted one? or perhaps include it in Appendix E?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.