- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SPUR addresses catastrophic forgetting during multimodal large language model fine-tuning by selecting optimizer-update coordinates at every step. It estimates plasticity using gradient–update products and stability using the gradient of KL divergence from the pretrained model together with a static diagonal Fisher approximation. These signals are normalized and combined through a harmonic-mean utility, after which the highest-utility coordinates are applied. Experiments cover three backbones, six target tasks, eight general-capability benchmarks, and multiple fine-tuning depths.
Critical strengths
- The method provides a concrete way to evaluate current optimizer proposals using both adaptation benefit and behavioral stability. Its step-dependent selection distinguishes it from the fixed-mask and post-hoc alternatives evaluated in the paper.
- Table 1 covers two backbones and two target tasks with eight general-capability benchmarks. SPUR obtains the highest reported Avg, Hscore, and Hscore* in all four configurations. Appendix I extends the evaluation to a third backbone and four additional target tasks.
- Table 3 includes fixed-mask selection, random selection under the same budget, utility scaling, and removal of individual stability terms. Full SPUR achieves the highest reported aggregate scores in this ablation setting. Table 9 additionally examines the performance–cost trade-off of curvature estimators.
Critical weaknesses
- Lines 321–323 select the update budget ρ by maximizing Avg. Appendix I.1 similarly selects ρ while using Model-Dowser’s official configuration. Although Table 5 specifies evaluation splits, an independent selection split and comparable baseline tuning procedures are not explicitly identified. This is a reporting ambiguity, not evidence of test-set tuning.
- Table 1 reports point estimates without training-run variability. For NVILA-Lite-2B/PathVQA, the improvements over Model-Dowser are relatively small: 66.72 to 67.01 in Avg and 85.38 to 85.74 in Hscore*. Repeated-run results would help establish the reliability of these particular margins.
- Two minor reference corrections are needed: Appendix I cites Wu et al. (2025a), the Qwen-Image Technical Report, for Qwen2.5-VL-3B; the relevant reference is the Qwen2.5-VL Technical Report. “Decoupled Weight Decay Regularization” is listed as ICLR 2017 rather than ICLR 2019.
Questions for authors
- Which datasets and splits determine ρ, and how are baseline hyperparameters selected?
- Are repeated-run results available for the smaller Table 1 improvements?
- Please correct the Qwen2.5-VL and AdamW references.
- Please report matched training-time and peak-memory comparisons against standard fine-tuning, including the overhead of reference-model inference, stability-gradient computation, and global update selection. Table 9 isolates curvature-estimation costs but does not establish the total overhead of SPUR.
Thank you again for your thoughtful review. As the discussion period is coming to an end, we would like to briefly summarize how we have addressed your two main concerns. We have also uploaded a revised PDF in which all changes are highlighted in red.
We would be very grateful if you could take a look at the revision and let us know whether these changes address your concerns. If they do, we would appreciate it if you would consider updating your score. We are happy to answer any further questions before the discussion period closes.