MapSelect: Mapping Preservation and Adapter Selection for Continual Multimodal Instruction Tuning

Continual multimodal instruction tuning must learn new tasks while maintaining performance on earlier ones. Retaining task-wise language adapters alone does not ensure this: their effectiveness also depends on the visual-to-language mapping and on which adapters are selected. We investigate these two factors on UCIT. Restoring an earlier visual projector recovers forgotten answers with the later language model unchanged; selecting the corresponding task-wise language adapter improves performance with the visual mapping unchanged. These observations lead to MapSelect, a framework that combines mapping preservation with adapter selection. History-Guided Mapping Adaptation retains projector adapters and constrains new updates using their learned directions. Selection-Guided Adapter Reuse combines current and historical adapters during learning and maintains consistent adapter selection throughout each answer. Pretrained Subspace Guidance further constrains language adaptation using pretrained weights. Extensive experiments on UCIT and MLLM-DCL demonstrate the effectiveness of our method.