- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies and attempts to mitigate catastrophic forgetting (CF) in multilingual continued pretraining. A small gemma model is trained on different language compositions and exhibits forgetting on a reading comprehension task. The results indicate that this forgetting can be mitigated through different parameter interventions during or after training. The paper further includes a study onboard localizing these effects in the model.
Critical strengths
(S1) Interesting focus and problem. Multilingual CPT is a critical problem that might help with low resource language modeling and could impact communities speaking such languages greatly.
Critical weaknesses
Critical weaknesses:
(CW1) The presentation, in particular the structuring, of this paper is lacking. Details see below.
(CW2) I am not sure what novelty your “new diagnostic method” introduces. It seems like a plain application of parameter interventions across the model layers which has been used and proposed by multiple previous works, some of which you cite. Please specify what you propose when you say “we propose our interpolation-based localization approach as a diagnostic framework”(line 484) and how that is new compared to previous work.
As it stands, these weaknesses (together with the weaknesses I raise below) prevent me from properly reviewing the results and findings. I would suggest a careful revision targeting the above issues raised, potentially extending experiments to strengthen and to resubmit to another round of reviews. As it stands, this work does not meet what I consider ought to be the standard at ICLR.
Detailed feedback:
(W1.1) Your first “primary contribution”(line 094) is “a new diagnostic method that at uses layer interpolation to systematically identify which layer ranges drive performance gains and losses during model adaptation”, yet your paper does not have a dedicated methodology section defining what exactly that method is. Besides, there is no discussion of what about this “method” is new and innovating beyond previous work. This ultimately leads to confusion about what your framework is and what of it you claim as your contribution (see CW2).
(W1.2) Several concepts are discussed before they are introduced, either informally or formally, in the main text. In particular, “Layer Range L2-SP”, “ChrF”
(W1.3) The introduction, for my taste, goes into the results in too much details without providing the necessary information to fully understand them. I would advise summarizing them at a higher level and then going into detail in the results section by which the reader has read the necessary context.
(W1.4) Writing could do a better job at incorporating the results into the story. At times, the paper feels more like list of unconnected results, in particular Section 4.
(W1.5) I think this paper would benefit from a dedicated methodology and experiment settings section describing and explaining the experiments. In it’s current form, this is all over the place and one needs to jump between different Sections to follow.
(W2) Presenting the results, the paper at times fails to mention all information necessary to follow/understand them. For instance:
(W2.1) What does Figure 2 show? Is it an aggregated result over all languages?
(W2.2) What is the performance of the baseline model before fine-tuning? (To contextualize results in Section 3.1)
(W2.3) I am not following your choice of layer boundaries for the “first”/“middle”/“last” in Section 3.1. The paper you cite does not study the Gemma model you consider and thus this division seems like a very random choice.
(W2.4) The sliding window results are only presented for a subset of positions. The remaining positions do not even seem to appear in the appendix. Why?
(W2.5) How do you compute the confidence intervals in Table 5 (“All Average cells include percentile 95% bootstrap CIs.”). They do not seem to be consistent with the shown values averaged over.
(W3) I am not sure what novelty your “new diagnostic method” introduces. It seems like a plain application of parameter interventions across the model layers which has been used and proposed by multiple previous works, some of which you cite. Please specify what you propose when you say “we propose our interpolation-based localization approach as a diagnostic framework”(line 484) and how that is new compared to previous work.
(W3) Adding to (W2.3), the choice of the window size and stride for your search in Section 3.2 seems to not be motivated by any theoretical or empirical argument. Why is this a good size? Why not have a more fine-grained stride size? Have you tried other combinations? Could combinations of non-consecutive windows improve your search?
(W4) No code submission. The work of this community is enabled by an open-source culture which allows us to quickly build on and extend previous findings. This includes this very paper you're submitting for review. Besides, some of your claims would be even stronger if the reviewers had access to your code to check how your have implemented the experiments you describe in the paper.
(W5) You only study a single, rather small model, gemma-3-4b. Bandaraktar et al. (2025) which you cite as one of the main influences of your work and which you claim to extend, look at a much larger Llama 8B. Why did you not decide to study a similar model? Would your results generalize to other models like the ones considered in Bandaraktar & Peng (2025), which are also all 7B+ parameters? Besides, extending to more evaluations and tasks would also be welcome, as your current focus on two tasks probably biases your insights.
Typos/smaller comments:
- missing space in line 144
-
“we sample” a “5B token corpus” (line 152)
-
A table in the appendix should be named “Appendix Table”(line 425, 429, 433)
Questions for authors
See above.
2 comments
Thank you for your response. While some of my concerns are addressed, in particular sharing of scientific artifacts, some more critical ones remain open. In particular, I have to agree with reviewer Bi47, who maintains their concerns regarding motivation, framing, and scope. Adding to that, my concerns regarding presentation, which in part follow from these issues, also remain open.
This paper requires some deep revisions which encourage the authors to work on, as I find the topic quite exciting. Despite this excitement, I however have to maintain my rating.
Rebuttal to Reviewer dY4X
Thank you for the thoughtful feedback on our work; we discuss your raised concerns below. Please also see the general response above for a summary of the major additions and revisions to the paper.
[W1.1,W1.5] We agree that introducing the methodology and presenting its results in the same section makes it confusing to distinguish between them, though we intentionally structure our paper so that we first study which layers drive performance change (§3) and then test whether preserving those layers improves CPT (§4). For each set of experiments, we have explicitly separated the proposed methodology and corresponding results into different subsections. We acknowledge that the method is straightforward, which we consider to be a strength of this work.
[W1.2,W1.3,W1.4] We have addressed all three of these concerns in our updated draft [Intro, Section 3].
[W2.1] Yes, they are aggregated results across languages. We have added a sentence in our caption mentioning this.
[W2.2] The baseline numbers are explicitly mentioned in Table 4 and 5. We have added a reference in our section 3.1 results pointing to these tables.
[W2.3] Although prior work provides the motivation to explore an "outer-vs-middle" distinction, it offers no explicit guidance on how to define these specific boundary choices. Furthermore, unlike existing studies, our work adopts a principled approach to systematically executing this sweep. We validate our boundary choice with a coarser sweep where we evaluate the language-acquisition and knowledge preservation tradeoff using perplexity and Belebele as our benchmark . The initial coarser split gives us the ideal performance v/s language learning split, with 9.57 as an average PPL and delta of 7.85 v/s dense. The subsequent equal-width analysis smoothen out these coarse middle-band splits. Appendix A4 and Table 9 have been updated with these results.
[W2.4] We share the complete grid-search with stride as 1 for belebele here which identifies
10,16) and [5,11) for FLORES as the best performing layer consistent with our reported results in Appendix A5 Table 10\. [W2.5The Average interval is for the different quantities: For each bootstrap draw, we resample sentences within each language-direction task, recompute all five family scores, then average those scores. The average is summarizing the uncertainty in the mean across five families and is not meant to cover every family score.
[W3] Prior work has explored layer swapping and reversion primarily in supervised fine-tuning [1,2], as adaptation methods applied to layers identified by other means (e.g., the magnitude of parameter changes); we evaluate these methods in our adaptation experiments in §4. In §3, we also formalize layer interpolation as a diagnostic procedure to find the relevant layers for a given task based on downstream performance, before applying any targeted intervention. The use of interpolation for layer localization is therefore a contribution separate from our test of whether the findings of [1,2] generalize beyond reasoning tasks and supervised fine-tuning.
[W4] We have updated the Supplementary Material with a zip to our anonymized code. Our final paper will also have all the checkpoints released for the community to build on.
We agree window size and position are hyperparameters and other configurations could further boost performance. To address stride, we added a full sweep of six-layer windows at stride 1 (Appendix A5, Table 10), matching our Table 2 results. We skip exhaustive combinatorial searches across all layer subsets as they are unnecessary for our main claim: that layer ranges impact tasks differently. Finding optimal setups in new domains requires our proposed task-specific localization, so we leave broader searches to future work as noted in limitations.
[W5]: We use a 4B model because our CPT setup (25B tokens, ~5000 H200 GPU hours) has vastly different computational demands than prior SFT-focused work [1,2], making an 8B model prohibitive.
While results may vary across architectures (as noted in Appendix A.1), our main claim does not rely on model-wide generalization: holding the model fixed shows middle-layer preservation transfers to reading comprehension but not translation, showing layer sensitivity is task-dependent. Thus, we propose a task-specific layer selection procedure rather than a fixed prescription on which layers to preserve.
[1] Bandarkar, Lucas, and Nanyun Peng. "The unreasonable effectiveness of model merging for cross-lingual transfer in llms." Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025). 2025.
[2] Bandarkar, Lucas, et al. "Layer swapping for zero-shot cross-lingual transfer in large language models." International Conference on Learning Representations. Vol. 2025. 2025.