- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how cross-lingual transfer emerges in LLMs during pretraining, evaluated through cross-lingual factual QA. The authors pretrain models from scratch under controlled conditions: (i) multilingually, showing that poor cross-lingual alignment already arises during pretraining, and (ii) monolingually with disjoint token sets. They identify disjoint tokens as a main driver of poor cross-lingual alignment and derive simple remedies to improve the cross-lingual abilities of LLMs.
Critical strengths
- The paper addresses an important problem with an original and well-controlled experimental approach.
- The focus on tokenization as a driver of cross-lingual misalignment is valuable, since this factor receives too little attention in the literature.
- The analyses build on each other naturally, and the paper is well written overall (exceptions below).
- The finding that a method as simple as WWT resolves a large part of the problem is convincing and opens a promising direction for future work.
Critical weaknesses
- Scale and undertraining (major weakness). It is unclear whether the findings transfer to models trained at realistic scale. The 7B model is pretrained on 140B tokens, roughly an order of magnitude less than comparable models (e.g., 2T tokens for LLaMA 2). Some cross-lingual abilities may only emerge later in training, so the weak representational alignment observed here may partly reflect undertraining. The approach remains interesting, but this limitation narrows the scope of the conclusions and should be stated far more prominently.
- Definition of Cross-lingual Knowledge Generalization (major weakness). The computation of this metric is unintuitive. The obvious formulation would be the accuracy on a fact in one language for a model trained on that fact in another language. The paper should motivate why the more complex formulation is needed.
- Presentation of FKD (minor weakness). The FKD descriptions are currently confined to the appendix fully, I think they should be moved to the main paper, together with examples, since the setup is currently somewhat hard to follow without them.
Questions for authors
Questions:
- Why is Cross-lingual Knowledge Generalization computed in its current form instead of as plain cross-lingual accuracy? What does the more complex formulation capture in addition?
- Do the authors have evidence (e.g., alignment trends across checkpoints) that the observed misalignment persists with longer training?
Suggestions:
- Investigating pretraining data mixes beyond 50/50 would strengthen the paper. A more realistic setting, such as 80% English and 20% in a second language, would allow stronger conclusions about state-of-the-art models.
6 comments
motivation for the formulation
Thank you for this comment, we agree that a clear motivation paragraph for choosing our more complex metric would strengthen the paper's clarity. We add the following paragraph to our metric definition section 2.2: "The obvious formulation is accuracy in language A on facts learned in language B. While this approach makes sense when each fact is learned in only one language, in practice, a model may see the same fact in both languages, at different frequencies. FKD allows modeling precise injection in both A and B at different rates, covering a wider range of cross-lingual dynamics. Trying to capture these dynamics through raw accuracy would conflate cross-lingual knowledge generalization with the learning curves of the specific model configuration tested, e.g., larger models acquire knowledge from less exposures and reach higher accuracies (Chang et al., 2024a), which would synthetically inflate their score in low-frequency native exposure scenarios. Therefore, we introduce a metric that normalizes for these differences in learning curves and natively captures complex bilingual injection dynamics — the Cross-Lingual equivalence () score."
FKD presentation in the main text
Thank you for pointing this out, we move some of the FKD description from the appendix to section 2.1. we also add a new Figure2 to our main text to help clarify FKD.
scale and undertraining
Scale is a natural problem in modern LLM research, while industry scales keep increasing, academic scales remain bound by financial constraints. We aim to provide a close-to-realistic scale analysis with our 7B experiments, focusing on key findings, yet increasing the token budget to LLaMA 2 scales remains unrealistic at our budget (we estimate a single pretraining run of that configuration at 200K\givena1/h GPU cost). To solidify our claims across scale, however, we turn towards adding more scale configurations to our experiments. We repeat our 3 key experiments with models of 140M, 220M, and 580M parameters, and find consistent results.
Regarding undertraining, while we generally followed compute-optimal 20x Chinchilla-based scaling, following your suggestion, we experiment with a closer to modern realistic data ratio — 100x. Due to the previously mentioned compute limitations, we run these experiments at the 220M parameter scale. Our findings hold in this setting.
We add a detailed report of these new scales and token budget to the paper in a new appendix M referenced in the main text. We also add a clause to our limitations appendix stating our scale limitations more clearly. For your convenience, we report the new results below:
CLeq measured for our 3 main bilingual setups across various scale and training budget configurations.