acceptodds
Under review as a conference paper at ICLR 2027

Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation

Abstract

While continual pretraining (CPT) is a practical way to extend large language models to new languages, naïve finetuning often erodes existing capabilities through catastrophic forgetting. We investigate which model layers drive this trade-off, and whether interventions at these layers can guide knowledge preservation during adaptation. We interpolate gemma-3-4b model states before and after CPT on five language families to localize forgetting on reading comprehension and translation, finding that middle-layer reversion yields the largest comprehension recovery, while translation effects vary by language family and direction. Guided by these findings, we evaluate CPT strategies that leverage this layer information to mitigate forgetting: layer freezing, layer-range L2 regularization, post-hoc layer reversion, and model souping, comparing all strategies against joint multilingual and family-specific vanilla CPT baselines. We find that preserving the layer weights identified via model interpolation substantially reduces comprehension loss relative to joint CPT, with layer freezing exceeding base model performance on average. However, these strategies yield mixed translation results: dense training or post-hoc reversion often outperforms both training-time constraints and family-specific specialization, complicating prior assumptions about how models should be aligned when extended to new tasks. Instead, we argue that multilingual adaptation strategy should be informed by target language, base model knowledge, and downstream task, and propose interpolation-based localization as a diagnostic for identifying candidate layers before committing to a training-time intervention in a new setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multilinguality, continual pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dY4XOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies and attempts to mitigate catastrophic forgetting (CF) in multilingual continued pretraining. A small gemma model is trained on different language compositions and exhibits forgetting on a reading comprehension task. The results indicate that this forgetting can be mitigated through different parameter interventions during or after training. The paper further includes a study onboard localizing these effects in the model.

Critical strengths

(S1) Interesting focus and problem. Multilingual CPT is a critical problem that might help with low resource language modeling and could impact communities speaking such languages greatly.

Critical weaknesses

Critical weaknesses:

(CW1) The presentation, in particular the structuring, of this paper is lacking. Details see below.

(CW2) I am not sure what novelty your “new diagnostic method” introduces. It seems like a plain application of parameter interventions across the model layers which has been used and proposed by multiple previous works, some of which you cite. Please specify what you propose when you say “we propose our interpolation-based localization approach as a diagnostic framework”(line 484) and how that is new compared to previous work.

As it stands, these weaknesses (together with the weaknesses I raise below) prevent me from properly reviewing the results and findings. I would suggest a careful revision targeting the above issues raised, potentially extending experiments to strengthen and to resubmit to another round of reviews. As it stands, this work does not meet what I consider ought to be the standard at ICLR.

Detailed feedback:

(W1.1) Your first “primary contribution”(line 094) is “a new diagnostic method that at uses layer interpolation to systematically identify which layer ranges drive performance gains and losses during model adaptation”, yet your paper does not have a dedicated methodology section defining what exactly that method is. Besides, there is no discussion of what about this “method” is new and innovating beyond previous work. This ultimately leads to confusion about what your framework is and what of it you claim as your contribution (see CW2).

(W1.2) Several concepts are discussed before they are introduced, either informally or formally, in the main text. In particular, “Layer Range L2-SP”, “ChrF”


(W1.3) The introduction, for my taste, goes into the results in too much details without providing the necessary information to fully understand them. I would advise summarizing them at a higher level and then going into detail in the results section by which the reader has read the necessary context.

(W1.4) Writing could do a better job at incorporating the results into the story. At times, the paper feels more like list of unconnected results, in particular Section 4.

(W1.5) I think this paper would benefit from a dedicated methodology and experiment settings section describing and explaining the experiments. In it’s current form, this is all over the place and one needs to jump between different Sections to follow.

(W2) Presenting the results, the paper at times fails to mention all information necessary to follow/understand them. For instance:


(W2.1) What does Figure 2 show? Is it an aggregated result over all languages?

(W2.2) What is the performance of the baseline model before fine-tuning? (To contextualize results in Section 3.1)

(W2.3) I am not following your choice of layer boundaries for the “first”/“middle”/“last” in Section 3.1. The paper you cite does not study the Gemma model you consider and thus this division seems like a very random choice.

(W2.4) The sliding window results are only presented for a subset of positions. The remaining positions do not even seem to appear in the appendix. Why?

(W2.5) How do you compute the confidence intervals in Table 5 (“All Average cells include percentile 95% bootstrap CIs.”). They do not seem to be consistent with the shown values averaged over.


(W3) I am not sure what novelty your “new diagnostic method” introduces. It seems like a plain application of parameter interventions across the model layers which has been used and proposed by multiple previous works, some of which you cite. Please specify what you propose when you say “we propose our interpolation-based localization approach as a diagnostic framework”(line 484) and how that is new compared to previous work.

(W3) Adding to (W2.3), the choice of the window size and stride for your search in Section 3.2 seems to not be motivated by any theoretical or empirical argument. Why is this a good size? Why not have a more fine-grained stride size? Have you tried other combinations? Could combinations of non-consecutive windows improve your search?

(W4) No code submission. The work of this community is enabled by an open-source culture which allows us to quickly build on and extend previous findings. This includes this very paper you're submitting for review. Besides, some of your claims would be even stronger if the reviewers had access to your code to check how your have implemented the experiments you describe in the paper.

(W5) You only study a single, rather small model, gemma-3-4b. Bandaraktar et al. (2025) which you cite as one of the main influences of your work and which you claim to extend, look at a much larger Llama 8B. Why did you not decide to study a similar model? Would your results generalize to other models like the ones considered in Bandaraktar & Peng (2025), which are also all 7B+ parameters? Besides, extending to more evaluations and tasks would also be welcome, as your current focus on two tasks probably biases your insights.

Typos/smaller comments:


- missing space in line 144

  • “we sample” a “5B token corpus” (line 152)

  • A table in the appendix should be named “Appendix Table”(line 425, 429, 433)

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer dY4X

Thank you for the thoughtful feedback on our work; we discuss your raised concerns below. Please also see the general response above for a summary of the major additions and revisions to the paper.

[W1.1,W1.5] We agree that introducing the methodology and presenting its results in the same section makes it confusing to distinguish between them, though we intentionally structure our paper so that we first study which layers drive performance change (§3) and then test whether preserving those layers improves CPT (§4). For each set of experiments, we have explicitly separated the proposed methodology and corresponding results into different subsections. We acknowledge that the method is straightforward, which we consider to be a strength of this work.

[W1.2,W1.3,W1.4] We have addressed all three of these concerns in our updated draft [Intro, Section 3].

[W2.1] Yes, they are aggregated results across languages. We have added a sentence in our caption mentioning this.

[W2.2] The baseline numbers are explicitly mentioned in Table 4 and 5. We have added a reference in our section 3.1 results pointing to these tables.

[W2.3] Although prior work provides the motivation to explore an "outer-vs-middle" distinction, it offers no explicit guidance on how to define these specific boundary choices. Furthermore, unlike existing studies, our work adopts a principled approach to systematically executing this sweep. We validate our boundary choice with a coarser sweep where we evaluate the language-acquisition and knowledge preservation tradeoff using perplexity and Belebele as our benchmark . The initial coarser split gives us the ideal performance v/s language learning split, with 9.57 as an average PPL and delta of 7.85 v/s dense. The subsequent equal-width analysis smoothen out these coarse middle-band splits. Appendix A4 and Table 9 have been updated with these results.

[W2.4] We share the complete grid-search with stride as 1 for belebele here which identifies

10,16) and [5,11) for FLORES as the best performing layer consistent with our reported results in Appendix A5 Table 10\. [W2.5

The Average interval is for the different quantities: For each bootstrap draw, we resample sentences within each language-direction task, recompute all five family scores, then average those scores. The average is summarizing the uncertainty in the mean across five families and is not meant to cover every family score.

[W3] Prior work has explored layer swapping and reversion primarily in supervised fine-tuning [1,2], as adaptation methods applied to layers identified by other means (e.g., the magnitude of parameter changes); we evaluate these methods in our adaptation experiments in §4. In §3, we also formalize layer interpolation as a diagnostic procedure to find the relevant layers for a given task based on downstream performance, before applying any targeted intervention. The use of interpolation for layer localization is therefore a contribution separate from our test of whether the findings of [1,2] generalize beyond reasoning tasks and supervised fine-tuning.

[W4] We have updated the Supplementary Material with a zip to our anonymized code. Our final paper will also have all the checkpoints released for the community to build on.

Adding to (W2.3),

We agree window size and position are hyperparameters and other configurations could further boost performance. To address stride, we added a full sweep of six-layer windows at stride 1 (Appendix A5, Table 10), matching our Table 2 results. We skip exhaustive combinatorial searches across all layer subsets as they are unnecessary for our main claim: that layer ranges impact tasks differently. Finding optimal setups in new domains requires our proposed task-specific localization, so we leave broader searches to future work as noted in limitations.

[W5]: We use a 4B model because our CPT setup (25B tokens, ~5000 H200 GPU hours) has vastly different computational demands than prior SFT-focused work [1,2], making an 8B model prohibitive.

While results may vary across architectures (as noted in Appendix A.1), our main claim does not rely on model-wide generalization: holding the model fixed shows middle-layer preservation transfers to reading comprehension but not translation, showing layer sensitivity is task-dependent. Thus, we propose a task-specific layer selection procedure rather than a fixed prescription on which layers to preserve.

[1] Bandarkar, Lucas, and Nanyun Peng. "The unreasonable effectiveness of model merging for cross-lingual transfer in llms." Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025). 2025.
[2] Bandarkar, Lucas, et al. "Layer swapping for zero-shot cross-lingual transfer in large language models." International Conference on Learning Representations. Vol. 2025. 2025.

Thank you for your response. While some of my concerns are addressed, in particular sharing of scientific artifacts, some more critical ones remain open. In particular, I have to agree with reviewer Bi47, who maintains their concerns regarding motivation, framing, and scope. Adding to that, my concerns regarding presentation, which in part follow from these issues, also remain open.

This paper requires some deep revisions which encourage the authors to work on, as I find the topic quite exciting. Despite this excitement, I however have to maintain my rating.

Reviewer_Bi47OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses catastrophic forgetting in continual pre-training (CPT) of LLMs for multilingual adaptation. In particular, it looks at which model layers drive a forgetting-retention trade-off and uses its analytical insight to examine whether task-specific targeted interventions can mitigate forgetting during CPT. To study these, the paper uses gemma-3-4b and targets 32 languages in total. Experiments show that there is a task-specific and language-specific pattern: middle layers are important for reading comprehension tasks across languages, while machine translation exhibits mixed results depending on languages. The corresponding targeted interventions seem to work well for validation perplexities and reading comprehension, showing better a forgetting-retention trade-off compared to full fine-tuning methods.

Critical strengths
  1. The paper explores different layer freezing and weight merging strategies for specific task types (Belebele: reading comprehension and FLORES: machine translation) to mitigate catastrophic forgetting, while improving target-task performance against a wide range of languages. I think this type of analysis has not been done yet. The paper also introduces held-out languages to test out-of-domain generalization on language modeling, reading comprehension, and machine translation tasks. This is also a nice analysis.

  2. The paper is mostly well written.

Critical weaknesses
  1. The major issue of the paper is its narrow scope. First of all, I do not understand why the paper only looks at and compares layer selection strategies to mitigate catastrophic forgetting. The paper states that “An open question for model adaptation is which regions mediate knowledge retention, …” in Section 3, but immediately after this, the paper specifically focuses on “layers”. There are more diverse ranges of studies in this direction. For instance, selective parameter tuning (e.g., [1,2]) is one of them. The paper needs to clearly justify why it looks at layers. By looking specifically at layer selections, what advantages can practitioners gain compared to other existing forgetting mitigation methods?

    [1] HFT: Half Fine-Tuning for Large Language Models (Hui et al., ACL 2025)
    [2] Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates (Yamaguchi et al., ACL 2026)

    Moreover, the task selection is also quite narrow. Why does the paper focus only on Belebele and FLORES? It should also test a wider range of tasks like GMMLU (general knowledge), XLSUM (summarization), MGSM (math) etc. to give more insights on task-specific trends. I also don’t understand the benefit of conducting task-specific forgetting mitigation for LLMs. In general, LLMs are considered to be general-purpose. While I understand the necessity of adapting LLMs to underrepresented languages, this should try to retain their general-purpose capabilities. Finally, the paper only targets “base” models, not post-trained models.

  2. The paper overgeneralizes its contribution. While the paper targets a diverse range of languages, it just only focuses on gemma-3-4b. Therefore, any findings and the resulting claims might be an artifact of this particular model and scale. To address this, the paper needs to test 1 larger scale and 1 more language model family in my opinion.

  3. The experimental settings seem unfair in terms of training budgets. Specifically, Dense CPT uses 25B tokens, while Family Expert uses 5B tokens. Also, different approaches use a different number of training steps according to Tables 7 and 8. Despite these differences, the paper compares perplexities and task performances of these models in Section 4.2. I think the budget should be matched across settings. If this is not something impossible, the paper should make a note on this whenever comparing results or entirely avoid comparing methods that use different numbers of training steps.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Bi47

We thank the reviewer for giving us thoughtful feedback on our work. Below we engage with your concerns in detail. Please also see the general response above for a summary of the major additions and revisions to the paper.

[ CW1]: We acknowledge that there are multiple approaches to mitigating catastrophic forgetting in LLMs. However, our focus on layers is motivated by prior work on layer-specific contributions to multilingual capabilities [1,2] [L46-L53]. This kind of granularity also enabled us to do a manageable and interpretable analysis. The practical benefit practitioners can take from this approach is identifying where preservation of these layers helps or harms different capabilities; we do not claim this methodology to be better than finer-grained parameter selection.

We selected Belebele and FLORES to run evaluations across broad-range of training and held-out languages in our setup. Our motivation for doing this analysis is to show a trade-off between preserving downstream capabilities and language acquisition while doing pre-training. It does not imply preserving one task is sufficient.
We target the “base” model to investigate the relationship between language acquisition and task retention during continual pretraining, before instruction tuning [Section 1, and Section 2.3]. Prior work [1,2] focuses on fine-tuned models and reports their findings linking middle layers to mathematical reasoning and first/last layers to language-specific capabilities.

[CW2]: We explicitly acknowledge this limitation in Appendix A.1. As discussed earlier, we were constrained by computers and the number of runs (spent ~4000 H200 hours) that we could do with our limited budget. We will explore more architectures and scale this work up in our future work.

[CW3]: The 25B tokens for the dense run are union of the corresponding 5B tokens for the corresponding family experts. Our comparison with dense CPT v/s family experts also measures what helps most – joint multilingual adaptation versus family-specific adaptation. It does not establish which approach performs the best under an equal training budget.
Regarding training steps, we set the maximum training steps to be around ~0.35 epoch for all. We then use early stopping for all our runs (Dense, Family, Layer Range and Freeze), which would terminate the runs based on the described criteria in Table 7 and 8. We have updated this in our Appendix Tables.

[1] Bandarkar, Lucas, and Nanyun Peng. "The unreasonable effectiveness of model merging for cross-lingual transfer in llms." Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025). 2025.
[2] Bandarkar, Lucas, et al. "Layer swapping for zero-shot cross-lingual transfer in large language models." International Conference on Learning Representations. Vol. 2025. 2025.

Thanks for the responses.

After reading the responses, I am afraid my core concerns remain unresolved.

First, on my first point in the weaknesses, reframing the contribution does not fix this. Specifically, positioning the paper as a diagnostic test of whether prior layer-swapping insights generalize to CPT explains why they looked at layers, but it still does not justify why practitioners should prefer coarse layer-window freezing over existing parameter-level forgetting mitigation methods (such as HFT or Source-Shielded Updates), nor does it evaluate general-purpose LLM capabilities beyond Belebele and FLORES.

The paper does not address the second weakness either. It does not include a second model family and/or a different model scale. Acknowledging the single-model limitation in the appendix does not substitute for validating the robustness of the claims with additional experiments.

Finally, beyond mismatched training budgets, different approaches use different learning rates, early-stopping thresholds, and max step caps, according to Tables 7 and 8. These confound direct comparisons across settings.

Given the above, I maintain my rating.

Reviewer_EG7nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates catastrophic forgetting during multilingual Continual Pretraining (CPT) and explores methods for identifying and preserving model layers that are important for retaining previously acquired capabilities.

The authors conduct CPT on Gemma-3-4B across five language families and employ layer-wise weight interpolation between the base and CPT-adapted models to identify layer regions that are particularly sensitive to downstream performance changes. Based on these findings, they evaluate several preservation strategies, including Layer Freezing, L2-SP Regularization, and Post-hoc Reversion, using reading comprehension (Belebele) and machine translation (FLORES-200) benchmarks.

The experimental results show that reverting middle layers [10,16) improves reading comprehension performance but negatively affects translation quality. Furthermore, the effectiveness of different layer preservation strategies varies across tasks and language families, and preserving parameters during training does not necessarily produce the same effects as reverting them after training.

Based on these observations, the authors propose layer interpolation as a diagnostic approach for identifying candidate layers to preserve during multilingual CPT.

Critical strengths

S1. Addresses an important research problem

Preserving existing model capabilities while acquiring new language skills through multilingual CPT is an important challenge in practical LLM development. Investigating the relationship between knowledge retention and acquisition at the layer level is a meaningful research direction.

S2. Interesting findings on task-dependent layer sensitivity

The observation that reverting the same layer region can have opposing effects on reading comprehension and translation is interesting. This finding suggests that uniformly preserving specific layers may not be an appropriate strategy across different downstream tasks.

The comparison of multiple strategies, including Layer Freezing, L2-SP, Post-hoc Reversion, and Expert Soup, is also valuable, as it highlights differences between preserving parameters during training and restoring them after training.

S3. Evaluation across multiple language families

The experiments cover five language families and include evaluations on related languages excluded from CPT. This provides useful insights into the heterogeneous effects of multilingual adaptation across languages.

Critical weaknesses

W1. Insufficient distinction between knowledge preservation and knowledge acquisition

The central motivation of this paper is to identify layers responsible for the loss of previously acquired capabilities during multilingual CPT and to develop strategies for mitigating catastrophic forgetting. However, the current experimental design does not clearly distinguish between knowledge forgetting and knowledge acquisition.

According to Tables 4 and 5, reading comprehension accuracy decreases from 69.9% to 60.2% after Dense CPT, whereas translation performance substantially improves from 33.3 to 53.7 ChrF.

The decline in reading comprehension can be interpreted as evidence of forgetting. In contrast, the improvement in translation primarily reflects the acquisition or enhancement of capabilities through CPT rather than the preservation of existing knowledge.

Consequently, the recovery of reading comprehension performance and the degradation of translation performance following layer reversion may reflect fundamentally different mechanisms.

The proposed layer interpolation method measures how changes to selected layer parameters affect downstream performance. However, it does not directly establish whether these layers are responsible for forgetting previously acquired knowledge or for acquiring new capabilities.

Therefore, a clearer distinction between knowledge preservation and knowledge acquisition is necessary to support the interpretation of layer interpolation as a diagnostic method for catastrophic forgetting.

W2. Limited coverage of downstream capabilities for analyzing knowledge preservation

Although the paper aims to characterize layer-specific knowledge preservation during multilingual CPT, its downstream evaluation is restricted to only two capabilities: reading comprehension and machine translation.

The capabilities of LLMs extend beyond these tasks and include factual knowledge, logical reasoning, mathematical reasoning, and various forms of text generation. These capabilities may depend on different internal representations and computational processes.

Therefore, the layer sensitivity patterns observed in reading comprehension and translation may not adequately characterize knowledge preservation across a broader range of model capabilities.

Importantly, the authors' own results indicate that different layer regions are beneficial for different tasks. This observation raises a further question: whether additional capabilities would also require preserving different, potentially conflicting, layer regions.

Moreover, the two evaluated tasks differ not only in the underlying capabilities being assessed but also in their output formats. Belebele evaluates multiple-choice reading comprehension, whereas FLORES-200 evaluates free-form text generation for translation.

As a result, it is difficult to determine whether the observed differences in layer sensitivity arise from the underlying task requirements or from differences between discriminative answer selection and generative output.

I acknowledge that the authors do not claim the existence of universally important layers across all tasks. Nevertheless, evaluating a broader range of downstream capabilities would be necessary to establish the usefulness of the proposed diagnostic approach across diverse multilingual CPT settings.

W3. Limited generalizability due to evaluation on a single model and one benchmark per task

All experiments are conducted using a single base model, Gemma-3-4B. Although the paper evaluates multiple checkpoints obtained through different CPT strategies, these experiments do not establish whether the findings generalize across different model architectures or scales.

Transformer models can differ substantially in their number of layers, internal representations, and pretraining distributions. Consequently, it remains unclear whether the observed layer sensitivity patterns are specific to Gemma-3-4B or represent more general properties of LLMs.

Furthermore, reading comprehension is evaluated exclusively on Belebele, while translation is evaluated exclusively on FLORES-200. Although both are established multilingual benchmarks, results from a single benchmark per task are insufficient to establish whether the identified layer regions remain important across different datasets measuring the same capability.

This limitation is particularly relevant because the central methodological contribution of the paper is the use of layer interpolation to identify which layers should be preserved.

Demonstrating improvements on the same model and benchmarks used to identify sensitive layers is informative, but it does not establish that the diagnostic procedure will be effective in previously unseen model or evaluation settings.

Additional experiments involving different model families and independent downstream benchmarks would substantially strengthen the generalizability of the conclusions.

W4. Insufficient validation of the layer selection procedure

The authors select layers [10,16) for reading comprehension and [5,11) for translation based on the layer interpolation analysis.

However, the paper does not provide sufficient evidence that these selected regions consistently outperform alternative layer selections, including randomly selected regions or other window configurations.

Furthermore, the benchmarks used to identify the relevant layer regions overlap with those used for the final downstream evaluation. This raises concerns about whether the selected regions reflect generalizable layer importance or benchmark-specific sensitivity.

A stronger evaluation would identify sensitive layers using a separate calibration dataset and then validate the resulting preservation strategies on an independent evaluation set. Comparisons against random layer selection and alternative window configurations would also help establish the effectiveness of the proposed localization procedure.

Additionally, the average reading comprehension performance of L2-SP and Expert Soup is 71.3% and 71.1%, respectively. Given this relatively small difference, further statistical analysis would be helpful to determine whether the proposed layer-aware preservation strategy provides a meaningful advantage over simpler alternatives.

Questions for authors

Q1. How does the proposed approach distinguish knowledge preservation from knowledge acquisition?

Reading comprehension performance decreases after CPT, whereas translation performance improves substantially. What justifies interpreting the layer interpolation results for these two tasks under the same knowledge preservation framework?

Could the authors separately analyze instances where the base model was correct but the CPT model became incorrect (forgetting), and instances where the CPT model became correct after the base model failed (acquisition)?

Q2. Do similar layer sensitivity patterns emerge for other model capabilities?

Would layer interpolation identify similar or different important regions for factual knowledge, logical reasoning, mathematical reasoning, or other downstream capabilities?

If different capabilities require preserving conflicting layer regions, how should the proposed diagnostic approach determine which layers to preserve in practical multilingual CPT?

Q3. Do the findings generalize across model architectures and evaluation benchmarks?

Would the same layer sensitivity patterns and preservation benefits be observed in models other than Gemma-3-4B or on independent reading comprehension and translation benchmarks?

Additionally, is the relevant factor the absolute layer index or the relative depth within the model architecture?

Q4. How robust is the layer selection procedure?

Do the selected layer regions consistently outperform randomly selected regions or alternative window sizes?

More importantly, do the benefits remain when the layers are identified using a separate dataset and subsequently evaluated on an independent benchmark?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer EG7n

We thank the reviewer for giving us thoughtful feedback on our work. Below we engage with your concerns in detail. Please also see the general response above for a summary of the major additions and revisions to the paper.

[W1, Q1]: We would like to clarify that our paper’s focus is on language acquisition which is measured using perplexity and knowledge preservation which is measured using downstream benchmark metrics during multilingual CPT; this is because the goal of CPT to teach a model new languages is for the model to acquire that language without losing the general-purpose knowledge learned from higher-resource languages like English. The translation improvements are additional downstream gains from CPT. Our layer-aware analysis focuses on whether preservation strategies motivated by previous work [1,2] about middle layer v/s outer layer transfer across tasks in the CPT setting. Our results show that reverting a layer region can recover reading comprehension performance but sacrifice translation gains. Thus, an intervention that benefits one task need not benefit another.

SplitBase accuracyCorrect → wrongWrong → correctDense accuracy
Held-out65.2%19.3 pp7.0 pp53.0%
Held-in69.9%17.2 pp7.4 pp60.2%

The above table answers the reviewer's concern where there was forgetting for belebele. For held-out languages, we notice 19.3% of average number of samples flipping from correct to wrong and 17.2% for held-in languages.

[W2,W3, Q2, Q3]: We selected Belebele and FLORES for their broad multilingual coverage and their complementary evaluation of reading comprehension and translation. Regarding other tasks such as mathematical reasoning, logical reasoning etc. we hypothesize that it may follow similar trends as we observed with Belebele. Previous work [1,2] done using fine-tuning uses mathematical reasoning as their target tasks, and show the same trends that we observed i.e. freezing middle-layers does preserve baseline task performance and allow the model to learn the target language.

We also explicitly acknowledge the limitations wrt benchmark selection and single model architecture selection in Appendix A1 (Limitations). Expanding this to other architectures and benchmarks is computationally infeasible because of our current academic budget.

The practical takeaway is that layer interpolation can help analyze the effects of preservation regions on downstream capabilities. What layers to preserve depend on what task performance matters to the model user. If they want a better translation performance, they optimize to identify those layers.

[W4, Q4]: Our belebele localization experiment (updated in Appendix A.4) shows all contiguous six-layer windows with stride 1, and

10,16)achievinghighestaccuracy.Regardinggeneralizability,ourlayer−interpolationanalysiswasdoneonaCPTcheckpointv/sbasecheckpoint.Theanalysiswasvalidatedfurtheronadifferenttrainingsetup(FamilyExpert,L2−SP,Freezing).Table2andourAppendixA.4alsoshowsthattheselectedlayersarethebestfortheirtasks.However,theseexperimentshavethesamearchitectureanddownstreambenchmarks.WepointthatinourlimitationsinAppendixA.1aswell.[110,16) achieving highest accuracy. Regarding generalizability, our layer-interpolation analysis was done on a CPT checkpoint v/s base checkpoint. The analysis was validated further on a different training setup (Family Expert, L2-SP, Freezing). Table 2 and our Appendix A.4 also shows that the selected layers are the best for their tasks. However, these experiments have the same architecture and downstream benchmarks. We point that in our limitations in Appendix A.1 as well. [1

Bandarkar, Lucas, and Nanyun Peng. "The unreasonable effectiveness of model merging for cross-lingual transfer in llms." Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025). 2025.
[2] Bandarkar, Lucas, et al. "Layer swapping for zero-shot cross-lingual transfer in large language models." International Conference on Learning Representations. Vol. 2025. 2025.

Thank you for the detailed response and for addressing my concerns. I appreciate the additional analyses and clarifications, particularly the instance-level accuracy transitions on Belebele and the expanded layer localization experiments. These additions strengthen the empirical observations. However, several of my main concerns remain unresolved.

  • W1/Q1: Knowledge preservation vs. acquisition The correct-to-wrong and wrong-to-correct statistics provide useful evidence of behavioral forgetting during CPT. However, they do not directly establish how the identified layers contribute to knowledge preservation. In particular, it remains unclear whether reverting the selected layers primarily recovers previously correct predictions or introduces other changes in model behavior. An instance-level analysis of prediction transitions before and after layer reversion would help establish a stronger connection between layer localization and forgetting mitigation.

  • W2/Q2 and W3/Q3: Generalizability I appreciate the references to related studies and understand the computational constraints. However, the cited studies investigate somewhat different training settings, and their findings do not directly establish that similar layer sensitivity patterns hold across other capabilities during multilingual CPT. Furthermore, the experiments remain limited to a single model architecture and one benchmark per task. Although these limitations are now acknowledged, the generalizability of the proposed diagnostic approach remains uncertain. I encourage the authors to ensure that their conclusions are appropriately scoped to the evaluated settings.

  • W4/Q4: Validation of the layer selection procedure The expanded analysis of contiguous six-layer windows is a useful addition and partially addresses my concern regarding alternative layer selections. Nevertheless, my primary concern is whether the identified regions generalize beyond the benchmark used for their selection. Evaluating the selected layers on the same benchmark used for localization does not sufficiently establish their generalizable importance. A simple calibration/evaluation split, even within the existing benchmark, would provide stronger evidence without requiring additional CPT training. Additionally, the paper observes that post-hoc layer reversion and training-time preservation strategies do not necessarily produce consistent effects. This raises an important methodological question: to what extent can layer interpolation reliably identify regions that should be preserved during subsequent CPT? I believe this relationship requires further validation, as it is central to the practical usefulness of the proposed approach.

I find the task-dependent layer sensitivity observations interesting and potentially valuable. The rebuttal improves the clarity of the paper and provides additional evidence of forgetting. However, the central methodological concern remains: whether layer interpolation serves as a reliable and generalizable diagnostic tool for selecting layers to preserve, rather than identifying benchmark-specific regions of sensitivity. Therefore, while I appreciate the authors' efforts and acknowledge the improvements, I maintain my original assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to all the Reviewers

We are grateful to the reviewers for their feedback. We agree that the main contribution of our work could have been stated more clearly and would like to reiterate this point here. We examine whether the layer-preservation findings from cross-lingual fine-tuning experiments for reasoning [1,2] (a) extend to multilingual Continual Pre-Training (CPT) rather than finetuning and (b) can benefit other downstream tasks beyond reasoning. We observe that the middle layer intuition from prior work holds for comprehension but not for translation, leading us to propose layer interpolation as a way to identify which layers to preserve per task, rather than recommending a general recipe. We have updated our submission to make this research question and contribution clearer.

The updated version also includes additional investigations we performed during the review period to sharpen some of our claims. We believe these updates have strengthened our submission. Below we summarize the major additions and revisions:

  • Edited the Introduction to focus more on high-level ideas and Research Questions, as outlined above (Reviewer dY4X)
  • Revised the writing in Sections 3 and 4 to ensure that the results are better incorporated into the narrative of the paper (Reviewer dY4X)
  • Added experiments supporting our motivation for our initial coarser split for doing the layer-interpolation Analysis in Appendix A4, Table 9 (Reviewer dY4X,Eg7n)
  • Added the complete sweep for stride =1 for both Belebele and FLORES in Appendix A5, Table 10 ((Reviewer dY4X, Eg7n))
  • We have updated our supplementary material with our codebase to replicate our work. Here’s the anonymous link: https://anonymous.4open.science/r/scaling-multilingual-experts-D354/ (Reviewer dY4X)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.