Not Which Paramters, but Which Updates: Forgetting-Aware Fine-Tuning of Multimodal Large Language Models

Fine-tuning multimodal large language models (MLLMs) on specialized tasks can substantially degrade their general capabilities. Existing methods largely make adaptation decisions at the parameter level, either regulating parameters during fine-tuning or revising their accumulated changes after adaptation. Yet these formulations do not explicitly evaluate the individual optimizer proposals that form the adaptation trajectory. We observe that comparable target-task adaptation can be reached through trajectories that incur markedly different amounts of forgetting, indicating that forgetting depends not only on the target performance achieved, but also on how optimization reaches it. This motivates controlling adaptation at a finer granularity: the candidate updates proposed throughout optimization. To this end, we introduce SPUR (Selective Per-step Update Reconciliation), an update-centric fine-tuning framework that treats each candidate optimizer update as the decision object. At every step, SPUR estimates each update coordinate’s plasticity benefit from its contribution to reducing the target loss and its stability cost from its predicted deviation from pre-trained behavior, reconciles the two into an update utility, and applies only the globally highestutility coordinates. The decision is recomputed at every optimization step, allowing the selected update pattern to evolve with the optimization trajectory. Extensive experiments demonstrate that SPUR consistently improves the stability– plasticity trade-off over strong forgetting-mitigation baselines, without external retention data, additional trainable parameters, or inference-time overhead.