acceptodds
Under review as a conference paper at ICLR 2027

Not Which Paramters, but Which Updates: Forgetting-Aware Fine-Tuning of Multimodal Large Language Models

Abstract

Fine-tuning multimodal large language models (MLLMs) on specialized tasks can substantially degrade their general capabilities. Existing methods largely make adaptation decisions at the parameter level, either regulating parameters during fine-tuning or revising their accumulated changes after adaptation. Yet these formulations do not explicitly evaluate the individual optimizer proposals that form the adaptation trajectory. We observe that comparable target-task adaptation can be reached through trajectories that incur markedly different amounts of forgetting, indicating that forgetting depends not only on the target performance achieved, but also on how optimization reaches it. This motivates controlling adaptation at a finer granularity: the candidate updates proposed throughout optimization. To this end, we introduce SPUR (Selective Per-step Update Reconciliation), an update-centric fine-tuning framework that treats each candidate optimizer update as the decision object. At every step, SPUR estimates each update coordinate’s plasticity benefit from its contribution to reducing the target loss and its stability cost from its predicted deviation from pre-trained behavior, reconciles the two into an update utility, and applies only the globally highestutility coordinates. The decision is recomputed at every optimization step, allowing the selected update pattern to evolve with the optimization trajectory. Extensive experiments demonstrate that SPUR consistently improves the stability– plasticity trade-off over strong forgetting-mitigation baselines, without external retention data, additional trainable parameters, or inference-time overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
catastrophic forgetting, multimodal large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4PEAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPUR addresses catastrophic forgetting during multimodal large language model fine-tuning by selecting optimizer-update coordinates at every step. It estimates plasticity using gradient–update products and stability using the gradient of KL divergence from the pretrained model together with a static diagonal Fisher approximation. These signals are normalized and combined through a harmonic-mean utility, after which the highest-utility coordinates are applied. Experiments cover three backbones, six target tasks, eight general-capability benchmarks, and multiple fine-tuning depths.

Critical strengths
  1. The method provides a concrete way to evaluate current optimizer proposals using both adaptation benefit and behavioral stability. Its step-dependent selection distinguishes it from the fixed-mask and post-hoc alternatives evaluated in the paper.
  2. Table 1 covers two backbones and two target tasks with eight general-capability benchmarks. SPUR obtains the highest reported Avg, Hscore, and Hscore* in all four configurations. Appendix I extends the evaluation to a third backbone and four additional target tasks.
  3. Table 3 includes fixed-mask selection, random selection under the same budget, utility scaling, and removal of individual stability terms. Full SPUR achieves the highest reported aggregate scores in this ablation setting. Table 9 additionally examines the performance–cost trade-off of curvature estimators.
Critical weaknesses
  1. Lines 321–323 select the update budget ρ by maximizing Avg. Appendix I.1 similarly selects ρ while using Model-Dowser’s official configuration. Although Table 5 specifies evaluation splits, an independent selection split and comparable baseline tuning procedures are not explicitly identified. This is a reporting ambiguity, not evidence of test-set tuning.
  2. Table 1 reports point estimates without training-run variability. For NVILA-Lite-2B/PathVQA, the improvements over Model-Dowser are relatively small: 66.72 to 67.01 in Avg and 85.38 to 85.74 in Hscore*. Repeated-run results would help establish the reliability of these particular margins.
  3. Two minor reference corrections are needed: Appendix I cites Wu et al. (2025a), the Qwen-Image Technical Report, for Qwen2.5-VL-3B; the relevant reference is the Qwen2.5-VL Technical Report. “Decoupled Weight Decay Regularization” is listed as ICLR 2017 rather than ICLR 2019.
Questions for authors
  1. Which datasets and splits determine ρ, and how are baseline hyperparameters selected?
  2. Are repeated-run results available for the smaller Table 1 improvements?
  3. Please correct the Qwen2.5-VL and AdamW references.
  4. Please report matched training-time and peak-memory comparisons against standard fine-tuning, including the overhead of reference-model inference, stability-gradient computation, and global update selection. Table 9 isolates curvature-estimation costs but does not establish the total overhead of SPUR.
Reviewer_bv5YOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly focuses on catastrophic forgetting in MLLMs during fine tuning. The authors point out that the degree of forgetting is determined by the optimization path rather than solely by downstream task performance. They propose the SPUR method, which at each step estimates the Plasticity Benefit and Stability Cost for each task, constructs a Stability Plasticity Reconciliation using the harmonic mean, and then performs masked training.

Critical strengths

The main advantage of this paper is that compared with the baselines it compares against, the performance improvement is relatively large, indicating that the method itself can effectively alleviate catastrophic forgetting.

Critical weaknesses

I think there are two weaknesses in this paper:

  • I think the motivation of the paper itself is incorrect. The authors explain that existing methods have coarse discrimination granularity, but the method in [1] performs gradient mask tuning at every optimization step rather than merely parameter-level updates.

  • This paper discusses catastrophic forgetting in MLLMs, but the proposed method is not specifically designed for MLLMs, nor does it discuss the unique training dynamics of MLLMs compared with LLMs during fine-tuning.

[1] Enhancing Large Language Model Performance with Gradient-Based Parameter Selection

Questions for authors

Why is the paper oriented toward MLLMs when its method is not specifically designed for the problems unique to MLLMs during fine-tuning (compared with LLMs)?

3 comments

Thank you again for your thoughtful review. As the discussion period is coming to an end, we would like to briefly summarize how we have addressed your two main concerns. We have also uploaded a revised PDF in which all changes are highlighted in red.

  • Comparison with GMT (Table 3 and Discussion). We reimplemented GMT following the original paper and evaluated it under the same setting as our ablation (NVILA-Lite-2B / PathVQA, 8 layers). GMT reaches 64.65 / 64.58 / 20.13 (Avg / Hscore / Hscore$^*$), compared with 67.01 / 66.69 / 85.74 for SPUR. In the new Discussion paragraph "Per-Step Masking versus Per-Step Update Evaluation", we explain the key difference: GMT masks gradients by their magnitude alone, while SPUR evaluates each proposed update by weighing its task benefit against its stability cost and then selects updates globally.
  • Specificity to MLLMs (Discussion and Table 4). In the new paragraph "Differences from Existing LLM and MLLM Fine-Tuning", we clarify that existing methods mostly inherit post-hoc strategies from LLMs, whereas SPUR is designed for MLLM fine-tuning. Its stability cost uses an image–text probe on the reference model, so it captures cross-modal drift. Table 4 further shows that SPUR adapts the connector and the LLM at different rates, and it compares this with Model-Dowser under a frozen and a trained connector.

We would be very grateful if you could take a look at the revision and let us know whether these changes address your concerns. If they do, we would appreciate it if you would consider updating your score. We are happy to answer any further questions before the discussion period closes.

I do not find the following statement sufficiently justified: “SPUR is specifically motivated by the stability–plasticity problem in MLLM fine-tuning, where adaptation must be coordinated across the vision–language connector and the language model.”

The stability–plasticity problem also arises in LLM fine-tuning and is therefore not, by itself, a distinctive motivation for studying MLLMs. A central difference is that MLLMs incorporate additional modalities, typically through a vision encoder and a connector that interfaces with the LLM. An MLLM-specific motivation should therefore clarify what distinguishes forgetting in MLLM.

Based on SPUR’s current design, I do not see a clear reason why techniques for mitigating catastrophic forgetting in LLMs would not also be applicable here. If the paper positions MLLMs as its central research focus, it should investigate how forgetting differs between MLLMs and LLMs and explain how the proposed method specifically addresses those differences. The current motivation and method design do not sufficiently establish this connection. I therefore maintain my original score.

Thank you for the follow-up. We would like to clarify our position and point to the evidence from our previous response that directly addresses your concern.

On the scope of the method. SPUR's selection principle (executing updates according to their estimated downstream benefit and stability cost) is general and is not restricted to MLLMs. We respectfully note that this is also the case for the established methods in this line of research. Model-Tailor (ICML 2024), CGM and Model-Dowser all address forgetting in MLLM fine-tuning by constraining updates to the language model, in essentially the same way as forgetting mitigation for LLMs, without treating the vision–language connector separately. Their contribution to MLLM forgetting mitigation has nonetheless been widely recognized. SPUR follows the same setting, benchmarks and baselines, and goes a step further by taking the multi-module structure of MLLMs into account: it also regulates connector updates and coordinates them with language-model updates. Moreover, we have empirically verified that this MLLM-specific design matters, as shown by the controlled experiment below.

What differs in MLLMs, and the evidence. In MLLM fine-tuning, the connector is also updated, and its updates are a distinct source of forgetting: they change the visual representations the language model relies on. Model-Dowser, like forgetting-mitigation techniques developed for LLMs, decides updates within a single model and has no mechanism for coordinating updates across modules. Our controlled experiment in the previous response shows the consequence:

SettingModel-Dowser (connector frozen, official)Model-Dowser (connector trained)SPUR
LLaVA-1.5-7B (8) / PathVQA57.9159.7061.10
LLaVA-1.5-7B (8) / DriveLM59.0059.7859.90
NVILA-Lite-2B (8) / PathVQA66.7265.8167.01
NVILA-Lite-2B (8) / DriveLM62.7559.8963.45

When Model-Dowser is run with the connector trainable, its performance on NVILA-Lite-2B drops below its official frozen-connector setting (PathVQA: 66.72 → 65.81; DriveLM: 62.75 → 59.89). Simply making the connector trainable therefore does not help and can hurt. Under exactly the same trainable-connector setting, SPUR outperforms both Model-Dowser variants in all four settings.

How SPUR addresses it. SPUR places candidate updates from the connector and the language model under one global budget, so how adaptation is split between them is decided by each update's benefit and stability cost, with no ratio fixed in advance. The resulting allocation is non-uniform across modules (e.g., 27.5% vs. 19.8% within-module update ratios for LLaVA-1.5-7B on PathVQA under a 20% global budget).

Reviewer_Zk9pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SPUR, a novel method for steering the fine-tuning of language models at each optimization step which differs from previous approaches in the field. They propose plasticity and stability as two conditions for deciding how to steer the optimization.

Critical strengths
  1. The proposed idea of steering fine-tuning at each optimization step is genuinely interesting and well motivated
  2. Their measures of study (e.g. plasticity and stability) seem well motivated and are explained well
  3. They provide extensive experimentation, ablation studies and comparisons to other methods which very much help ground their method in the larger literature and provide evidence to strengthen their claims.
Critical weaknesses
  1. SPUR seems to use a single, static Fisher matrix from the pretrained model. I can see that you provided experiments to test the sensitivity of the conclusions to choice of a static Fisher model vs a dynamically updated model, however, due to the small amount of epochs used for training, I wonder if the demerits of using a static approximation would only become pronounced at longer timescales. Could the authors try training with a static vs dynamically refreshed estimate for a longer timescale?
  2. Many accuracies in the reported tables seem within extremely small margin of improvements over other reported methods. Could the authors extend the experiments to train over multiple seeds to gain more confidence in the effects of this method over others?
Questions for authors
  1. Because SPUR uses an optimization oracle for the proposed updates, it seems like its selection behavior may depend substantially on the choice of optimizer. I acknowledge that AdamW is the most widely used optimizer for modern training but it would be nice to see comparisons with other optimizers such as SGD with momentum. This also helps know whether the benefits of SPUR generalize to broader optimization methods.
  2. Can the authors clarify the overhead incurred by this method relative to ordinary fine-tuning (e.g. increase in training time and peak GPU memory)?
  3. As more of a conceptual question, can the authors speak to the conceptual difference between directly adding a regularization term during fine-tuning (e.g. with a KL-penalty and a constraint on update sparsity) compared to their proposed method? Could the authors also provide a comparison to such a regularization approach?
  4. When doing the second order Taylor expansion for estimating stability, the authors define their “stability cost” as the maximum of the dynamic term and the static term about the pre-trained solution. Can the authors speak to why a maximum was used rather than just a sum of the two terms? Could the authors provide an experiment to show if there is a meaningful practical difference in whether a maximum or sum is used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.