acceptodds
Withdrawn from ICLR 2027

Where, How Much, and What to Retain: Sparse Optimization for Multimodal LLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) exhibit strong general capabilities across vision-language tasks, yet downstream fine-tuning can degrade pre-trained capabilities due to catastrophic forgetting. Existing approaches mitigate this issue through post-hoc model merging or sparse fine-tuning. While dynamic masking improves upon fixed sparse updates by adapting parameter selection during training, our analysis reveals that step-wise mask reconstruction can incur considerable memory overhead while committing to most of its eventual update region early in optimization, limiting subsequent parameter reallocation. To address these limitations, we propose the Stability-Plasticity-Aware Sparse Optimizer (SPSO), which formulates sparse fine-tuning as an optimizer-level dynamic sparse optimization problem. SPSO introduces a Safe-Update Score (SUS) that combines a lightweight pre-trained weight-based preservation prior with online gradient-based task relevance. Guided by SUS, SPSO dynamically reallocates a fixed-size active set, scales parameter-wise updates, and selectively retains sparse Adam states, jointly controlling where to update, how much to update, and what to retain. Experiments across multiple MLLMs and downstream tasks show that SPSO consistently improves the balance between pre-trained capability retention and task adaptation while maintaining sparse updates and reducing optimization memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x6RgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPSO, which formulates sparse fine-tuning as an optimizer-level dynamic sparse optimization problem. SPSO introduces a Safe-Update Score (SUS) that combines a lightweight pre-trained weight-based preservation prior with online gradient-based task relevance. Experiments on several models and tasks show promising results, but some details and comparisons remain unclear.

Critical strengths
  1. The method addresses a practical problem and jointly considers parameter selection, update size, and optimizer memory.

  2. The experiments cover several models and tasks, with useful ablations and additional results in the appendix.

Critical weaknesses
  1. Alg.1 prunes moment states before updating all active parameters, but it is unclear how parameters whose states were removed are updated. The paper also describes both periodic and per-step selection. Eq.10 uses gradient magnitudes, while Alg.1 uses signed gradients.

  2. The early SPSO coverage in Fig.1(b) is below 0.1 despite the stated 10% active budget. SPIDER’s optimizer memory differs between Fig.1(a) and Tab.5 and exceeds its reported peak GPU memory. FISH Mask updates about 10% of parameters cumulatively, yet only 2.14% remain nearly unchanged.

  3. Fig.1(b–d) omit Adam without explaining why. SPIDER and SPSO have substantially different update budgets in Tab.5, so a matched-budget comparison is needed.

  4. SUS also reduces update sizes, and the paper should better separate the benefit of parameter scoring from that of smaller updates. Training-time results are also missing, despite the extra sampling, top-K selection, and state-management operations.

  5. Fig.2 does not clearly explain the method. Parameter selection scope, sparsity terminology, and metrics for missing target scores need consistent definitions. The writing also needs careful revision.

Questions for authors
  1. How are active parameters updated if their moment states have been removed? Is selection periodic or per-step, and does Eq.10 or Alg.1 reflect the actual implementation?

  2. Can you explain the early coverage below 0.1 in Fig.1(b), SPIDER’s memory discrepancies, and FISH Mask’s cumulative updates versus final sparsity?

  3. Why is Adam omitted from Fig.1(b–d)? Does SPSO still outperform SPIDER under a matched update budget?

  4. Do the gains remain when update magnitudes are matched? What is the training-time overhead of sampling, top-K selection, and state management?

  5. Could you revise Fig.2 and clarify the parameter selection scope, sparsity definitions, and calculation of metrics when target scores are missing?

Reviewer_MMWCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stability-Plasticity-Aware Sparse Optimizer (SPSO). This is to mitigate catastrophic forgetting during multimodal large language model fine-tuning.

Previous work SPIDER has two limitations, one is that it uses a lot of memory and another one is that it has early commitment to a largely fixed update region. This paper effectively tries to improve the SPIDER. SPSO introduces a Safe-Update Score (SUS) that combines pre-trained weight magnitude and online gradient information. This score jointly guides dynamic parameter selection, update scaling, and sparse optimizer-state retention. This set up enables to explore under a bounded sparse-state budget for fine-tuning.

SPSO is showing useful results when fine-tuning a MLLMs, as well as post-hoc merging, and sparse fine-tuning. More importantly, it shows reducing optimization memory compared with dense Adam and SPIDER.

Critical strengths

The empirical results show strong evidences that SPSO keeps existing capablity while maintain the good capabilities on the old tasks. e.g. LLaVA-1.5-7B in IconQA, SPSO got 60.20 on existing abilities and 58.92 on the target tasks, while SPIDER achieves 55.73/57.12, standard fine-tuning with Adam got 31.49/59.02.

The SPSO seems to be useful and have a good design, the scoring rule uses information available during fine-tuning. It does not require the original pre-training examples. The ablation studies show that the design of the SPSO helps improving the model performance on NVILA.

Critical weaknesses

This paper has over-claimed the memory efficiency. It is cool that the proposed SPSO improve the memory efficiency, however i am confused between the claims. If we look at the results, it seems that the SPSO improves over SPIDER a lot, memory peak changes from 70GB->23GB. But if we look at Adam, the SPSO has small changes in terms of peak memory, e.g. 25 GB->23GB. If we look at static mask methods, SPSO is using more peak memory.

The state-pruning procedure in the proposed SPSO is unclear and unspecified. SPSO selects an active set of KK parameters. It combines their states with previously retained states. It then keeps the top KK states. However, Equation 8 updates every parameter in the active set. See Section 4.1 and Equation 6-8. If say, we have parameters selected for update [1,2], and parameters whose states remain [1, 3], how can we decide which state is used to update parameter 2? In addition, the statement that bias correction follows Adam also needs more detail.

The comparison with SPIDER is not complete and fair. The native-SPIDER comparison is useful. However, it does not show what happens when the SPIDER variant has the same actual update budget. In addition, this paper needs stronger control experiments against a smaller learning rate, shorter training, or matched update magnitude.

The original SPIDER already uses pre-trained weight magnitude and downstream gradient information, i am not sure if those two points are the contributions of SPSO in this paper.

Questions for authors

What exactly is included in the memory measurements?

What is the exact order of state updates, pruning, and weight updates in SPSO?

Can the authors add budget-matched and update-size-matched controls?

Does early commitment actually reduce performance? e.g. Figure 1, and equation 6.

How robust and efficient are the results across repeated runs with SPSO?

Reviewer_DaX9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPSO, a sparse optimizer for reducing catastrophic forgetting during multimodal LLM fine-tuning. A score combining pretrained weight magnitude and current gradient magnitude guides parameter selection, update scaling, and sparse Adam-state retention. Experiments across three MLLMs, multiple downstream tasks, and continual instruction tuning report improved retention–adaptation trade-offs and reduced memory usage.

Critical strengths
  • Jointly controlling parameter selection, update magnitude, and optimizer-state retention is a useful approach to sparse adaptation.

  • Evaluation covers multiple architectures and tasks, with component ablations, sensitivity analyses, and three-seed results on IconQA.

Critical weaknesses
  1. Efficiency statistics require reconciliation. Table 5 reports 182.51 GiB of SPIDER optimizer-state memory but only 70.91 GiB peak GPU memory, without explaining placement or offloading. FISH Mask reportedly updates only 10% of parameters cumulatively, yet only 2.14% remain near initialization. These discrepancies weaken confidence in the efficiency comparison.

  2. The implemented optimizer is insufficiently specified. Equation 10 uses gradient magnitudes, whereas Algorithm 1 uses signed gradients. The appendix describes periodic active-set reconstruction, but the algorithm resamples every step and provides no refresh interval. These alternatives change the optimization process, making the reported results difficult to reproduce.

Questions for authors
  • Please reconcile the memory and FISH Mask statistics, including tensor placement, calibration updates, and measurement scope.

  • Which gradient-scoring and active-set refresh rules were actually used? How are active entries updated if their moment states are pruned before the parameter update?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and Reviewers

We thank the Area Chair and Reviewers for their careful evaluation and constructive feedback. During the rebuttal, we conducted additional controlled experiments and uploaded a revised manuscript incorporating the new results and methodological clarifications. The key findings are summarized below.

Although these results strengthen the empirical support for SPSO, we believe the paper would benefit from clearer methodological descriptions and more systematic integration of the new analyses. We have therefore decided to withdraw the submission to further improve its completeness and rigor.

Responses to Key Concerns

1. Methodological Clarification.

We clarified SPSO's optimization procedure and distinctions from existing sparse methods:

  • SUS and contributions. Weight magnitudes and task gradients were previously used by PaFi and FISH Mask. SPSO explicitly integrates retention and adaptation signals into SUS, guiding parameter selection, moment pruning, and update scaling under a fixed per-step budget.
  • Optimizer procedure. At every step, SPSO samples a top-K set, updates moments over the union of sampled and previously retained coordinates, prunes to K states, and updates weights only on the retained set. We clarified the set definitions, gradient magnitudes, bias correction, and pseudocode.

2. Fair Comparisons with SPIDER.

The official SPIDER implementation uses importance-based gating without a fixed update budget. We conducted complementary controls on LLaVA-1.5-7B:

  • Matched per-step budget. At similar per-step update counts (SPIDER: 262.84M; SPSO: 263.93M), SPSO obtains 59.55 H-score versus 56.39 for SPIDER, with greater cumulative coverage (836.08M vs. 384.92M).
  • Matched cumulative coverage. When SPIDER covers 831.67M coordinates (SPSO: 836.08M), its H-score is 56.53, below SPSO's 59.55.
  • Trainable-layer controls. SPIDER with fewer trainable decoder layers achieves H-scores of 57.13/56.52, again below SPSO.

These controls suggest that update-budget differences alone do not explain SPSO's gains.

3. SUS Scaling and Training Controls.

We evaluated whether smaller updates or shorter training explain the improvements on LLaVA-1.5-7B and NVILA-Lite-8B:

  • Without SUS scaling. Retaining selection and moment management, SPSO without SUS Scaling outperforms Adam and SPIDER across three learning rates. At 2e-5, its H-scores are 57.85/81.62, versus 56.42/80.31 for SPIDER.
  • Complete SPSO. At the same learning rate, SPSO achieves 59.55/83.90, improving upon the unscaled variant by 1.70/2.28 points. It ranks first at all tested learning rates.
  • Training duration. Across 1, 3, and 5 epochs, SPSO consistently outperforms Adam and SPIDER. At one epoch, its H-scores are 55.99/79.29, versus 50.58/63.97 for Adam.

These findings support benefits beyond global learning-rate reduction or shorter training; realized update magnitudes were not strictly matched.

4. Early Commitment and Update-Space Reallocation.

We examined fixed-support variants and the role of stochastic sampling:

  • Commitment timing. Fixing the update set after Step 1 yields 56.30 H-score; fixing it after Steps 50 and 100 yields 57.22 and 57.75, versus 59.55 for fully dynamic SPSO.
  • Deterministic selection. Dynamic SUS-based top-K selection without Gumbel sampling achieves 58.12, outperforming all fixed-support variants.
  • Update-set diagnostics. We distinguish early saturation in cumulative coverage from actual turnover, supplementing coverage with consecutive-set overlap analysis.

These results support continued reallocation of the update space beyond stochastic exploration.

5. Efficiency and Robustness.

Efficiency was evaluated with 12-layer fine-tuning on four H20 GPUs using ZeRO-2:

  • Memory. Per-GPU optimizer-state memory is 2.87 GiB for SPSO versus 4.56 GiB for Adam and 45.63 GiB for SPIDER. Peak GPU memory is 23.75/25.84/70.91 GiB, respectively.
  • Training time. SPSO completes 390 steps in 60.3 min, versus 59.0 min for Adam and 64.5 min for SPIDER (2.2% overhead over Adam).
  • Static-mask comparisons. SPSO updates 263.93M parameters per step while cumulatively covering 836.08M, unlike fixed-mask methods. Its IconQA H-score (59.55) exceeds PaFi (55.85) and FISH Mask (55.31); final update sparsity is 83.00%.
  • Robustness and consistency. Across three seeds, SPSO achieves 59.55 ± 0.17 and 83.90 ± 0.20 on LLaVA and NVILA. We standardized memory and sparsity definitions and corrected FISH Mask's sparsity statistic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.