acceptodds
Withdrawn from ICLR 2027

Preserving General Capabilities without Past Data: Parameter-Importance-Driven Continual Learning for Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

Domain-specific post-training often causes catastrophic forgetting, making foundation models lose their general reasoning capability and limiting their adaptability to dynamic real-world environments. Preserving these capabilities while acquiring domain knowledge remains a key challenge for large language and multimodal models. Traditional continual learning methods, such as regularization, replay and architectural isolation, suffer from poor downstream performance, reliance on inaccessible historical data, or additional parameter overhead. While recent parameter-efficient tuning (PET) methods can alleviate forgetting, their effectiveness strongly depends on the choice of parameters and update strategies. In this paper, we introduce PIECE, a Parameter Importance Estimation-based Continual Enhancement method that preserves general ability while efficiently learning domain knowledge without accessing prior training data or increasing model parameters. PIECE selectively updates only 0.1% of core parameters most relevant to downstream tasks, guided by two importance estimators: PIECE-F based on Fisher Information, and PIECE-S based on a second-order normalization that combines gradient and curvature information. Experiments across three language models and two multimodal models show that PIECE maintains general capabilities and achieves state-of-the-art continual learning performance across diverse downstream tasks. Our results highlight a practical path to scalable, domain-adaptive foundation models without catastrophic forgetting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Large Language Models, Multimodal Large Language Models, Parameter-Efficient Tuning, Parameter Importance Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cCrxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a continual learning method for training LLMs. Similar to previous methods there method proposes to find a small set of parameters to update with new task data. They present based on importance estimation method based on Fisher information and using gradient and curvature information. Unlike previous methods that rely on past task data there method needs new task data to ensure continual learning

Critical strengths
  • It's a continual learning method that shows improvement over existing SOTA methods.
  • Unlike previous data, this only relies on new task data, making it more practical for use
  • They show their improvements on both text LLMs and multimodal LLM families
  • Exhaustive comparison with previous methods, and the datasets cover a lot of domains with consistent improvement across a lot of them.
Critical weaknesses
  • Their method uses a similar idea to the Fish Mask paper (Sung et al. 2021), applied to parameter selection for an LLM. This is still an empirical contribution but with weaker novelty
  • Prior methods like EWC penalize old task important parameters to prevent forgetting. The authors' method relies on new task data to find parameters for updating. This doesn't guarantee that previous tasks are not forgotten. The authors show, using the BWT score for general capability preservation, that their method is among one of the lowest. But the evidence is still insufficient to conclude it preserves general capability. And it seems replay still outperforms their method at times.
Questions for authors
  • Could the authors provide a reason for 1 epoch of training for Qwen3-14B compared to other models?
  • For Table 3, why did the authors select a narrow set of baselines? Even though their method is the lowest BWT, it is still generally higher, so I am curious whether it is still low compared to other baselines (replay, EWC, GEM, etc.)
  • Can the authors clarify the relationship to the Fish Mask paper (Sung et al. 2021)? How does PIECE - F/S distinguish from it?
Reviewer_p3pjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual finetuning of LLMS and MLLMs under a strict no-history setting with no past task data, no stats stored from earlier tasks and no added parameters. Before training on each task, PIECE scores every parameter using current-task data and keeps the top 0.1%. It then trains with a fixed gradient mask so the remaining never changes. There are two scores PIECE-F and PIECE_S and the experiments conducted include TRACE, VQA v2 while measuring general capabilities on HumanEval and Flickr30K

Critical strengths
  1. The no-history, fixed architecture and fixed-budget constraint matches well with how open-weight foundation models are actually post-trained without the original data. PIECE fits this setting with one fixed gradient mask per task while needing no extra modules and no change to training beyond masking
  2. It shows strong retention of general ability in most settings. PIECE-S has the best HumanEval among all the baselines. Overall task performance also stays competitive
Critical weaknesses
  1. It is unclear what drives retention and the novelty is limited
  • The selection score uses only the current-task data so retention may simply come from updating so few parameters, not from choosing the right ones. Fig4 supports this argument showing how the forgetting increases as the budget rises. A random 0.1% mask with the same budget would test this directly.
  • The novelty is limited since PIECE-F (Eq. 4) is essentially FISH Mask (Sung et al., 2021), applied once per task. PIECE-S always lies between 0 and 1 and measures how consistently samples agree on a gradient’s direction. Calling it “second-order” overstates it, since no curvature beyond squared gradients is used. GPS (Zhang et al., 2024), which the paper cites, should be included as a baseline, and the contributions reframed accordingly.
  1. The evaluation is to thin to support the general ability claim
  • All results come from a single seed and since general ability is measured only on HumanEval, which has very less data, some improvements over the baselines would boil down to just a handful of samples.
  • The baseline collapse for smaller models can be attributed to the high training epochs compared to Qwen3-14B as much as the methods
Questions for authors
  1. How does a random 0.1% mask compare on OP, BWT and HumanEval, with the same budget and learning rate?
  2. What are the HumanEval and Flickr30K scores of each base model before continual training? Are the language models base or instruct checkpoints?
  3. Can you report the mean and std over at least 3 seeds for the main Llama3-8B setting?
  4. How does PIECE compare with GPS at the same budget? Is there any practical difference between PIECE-F and FISH Mask?
  5. What do importance estimation and masked training cost in GPU memory and time, compared with LoRA?
Reviewer_uSyeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies continual learning for foundation models when historical training data are unavailable and the model architecture and total parameter count must remain fixed. At the beginning of each new task, PIECE scores parameters using current-task data and constructs a fixed mask that allows approximately 0.1% of the parameters to be updated. Experiments cover three language models, two multimodal models, three TRACE task orders, and a 25-task Super-NI sequence.

Critical strengths
  1. The writing is easy to follow.
Critical weaknesses
  1. The novelty and distinct contribution of the scoring mechanism are not sufficiently established. The contribution of PIECE-F therefore needs a more precise account of what follows existing fixed-mask approaches and what changes through task-wise mask re-estimation or the continual-learning setting. The current related-work discussion does not adequately distinguish the closest methods in terms of scoring, mask recomputation, and optimization. This is a request to establish the differences, rather than a claim that the methods are identical. The experiments of PIECE-S do not yet establish whether its benefits depend on this particular normalization.
  2. The second-order interpretation relies on unvalidated approximations and does not establish preservation of prior capabilities.
  3. The scope of capability evaluation and the task-order controls do not support the broad preservation claims. The separate capability evaluation consists of HumanEval for language models and the mean of ROUGE-L and BLEU on Flickr30K captioning for multimodal models. These support conclusions about code generation and image captioning, respectively, but do not adequately cover the broader reasoning and general-capability claims in the abstract and conclusion.
  4. Figure 4 contains an internal metric inconsistency, and small improvements lack uncertainty estimates. For the eight-task TRACE setting, the metric definitions imply AP = OP − 7/8 × BWT. For PIECE-S (k = 0.1%), Table 1's OP = 49.05 and BWT = −1.78 imply AP ≈ 50.61, whereas Figure 4 reports 51.3. This discrepancy exceeds rounding error. Please clarify whether these metrics use the same runs and definitions, and correct the inconsistent values.
  5. The research problem is limited. The paper studies continual learning for foundation models when historical training data are unavailable, and the model architecture and total parameter count must remain fixed. This problem has too many constraints and is not practical.
  6. The comparison baselines are too old. Besides, why didn't the authors use more common benchmarks for multimodal continual learning?
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.