acceptodds
Withdrawn from ICLR 2027

MapSelect: Mapping Preservation and Adapter Selection for Continual Multimodal Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual multimodal instruction tuning must learn new tasks while maintaining performance on earlier ones. Retaining task-wise language adapters alone does not ensure this: their effectiveness also depends on the visual-to-language mapping and on which adapters are selected. We investigate these two factors on UCIT. Restoring an earlier visual projector recovers forgotten answers with the later language model unchanged; selecting the corresponding task-wise language adapter improves performance with the visual mapping unchanged. These observations lead to MapSelect, a framework that combines mapping preservation with adapter selection. History-Guided Mapping Adaptation retains projector adapters and constrains new updates using their learned directions. Selection-Guided Adapter Reuse combines current and historical adapters during learning and maintains consistent adapter selection throughout each answer. Pretrained Subspace Guidance further constrains language adaptation using pretrained weights. Extensive experiments on UCIT and MLLM-DCL demonstrate the effectiveness of our method.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Lifelong Learning, Multi-Modality Model, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sYKPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper lies at the intersection of continual learning and multimodal large language model fine-tuning, addressing catastrophic forgetting during continual multimodal instruction tuning. Through an analysis of prior work and empirical evidence, the authors identify two complementary challenges: mapping preservation and adapter selection. To address these challenges, they introduce History-Guided Mapping Adaptation, Selection-Guided Adapter Reuse, and Pretrained Subspace Guidance.

The main contribution of this paper is that the authors identify and empirically validate two complementary challenges in continual multimodal instruction tuning: mapping preservation and adapter selection. The authors further validate the proposed method through experiments on UCIT and MLLM-DCL and use component ablations to demonstrate the effectiveness of individual components.

Critical strengths

The paper’s main strength is its empirical investigation of mapping preservation and adapter selection. In particular, Table 1 reveals a nontrivial finding: restoring an earlier projector while keeping the later LLM fixed recovers a substantial fraction of forgotten examples. This suggests that changes in the multimodal mapping account for part of the observed forgetting and could inform future approaches to continual multimodal instruction tuning. This insight is central to my positive assessment of the paper, as it advances our understanding of forgetting beyond the proposed method’s performance gains.

Critical weaknesses
  1. The necessity of the method’s complexity is insufficiently established. The method comprises multiple components, each with a relatively complex design. Although the ablation studies demonstrate the contribution of individual components, they do not establish whether simpler alternatives could address the same two challenges. Comparing against such alternatives would help clarify whether the proposed designs are necessary. Without this evidence, it is difficult to assess whether the performance gains justify the added complexity.
  2. There is substantial overlap among the experiments, while some evidence needed to establish practical usefulness is missing. Table 1 and Figure 1 both support the identification of mapping preservation and adapter selection as key challenges. Tables 7 and 8 report detailed MAPSELECT results on UCIT and MLLM-DCL, supplementing the comparisons across methods in Tables 2 and 4, respectively. Similarly, Figure 3(a,b) provides additional detail on the results in Table 3, while Figure 3(c,d) supplements those in Table 2. These experiments largely reinforce the same findings rather than provide distinct evidence, and much of this material would be better suited to the appendix. Meanwhile, the paper introduces several hyperparameters without examining sensitivity to their choices. This matters for MLLM fine-tuning, where each run is expensive and extensive tuning could substantially increase the cost of achieving good performance. The paper also lacks computational cost comparisons with baselines such as HiDe, CL-MoE, ModalPrompt, and DISCO under the same experimental setting, leaving its computational efficiency relative to these methods unclear. Overall, the insufficient experimental evidence limits my confidence in the method’s practical usefulness.
Questions for authors
  1. Clarification and experimental comparison regarding projector freezing. The paper uses instruction-tuned LLaVA-v1.5-7B as its backbone. Its projector has already undergone alignment pretraining and instruction tuning. Have the authors tried freezing the projector and adapting only the LLM parameters? Specifically, how do simple baselines such as LoRA-FT and O-LoRA perform with the projector frozen versus trainable? Under the frozen-projector setting, how would a simplified MAPSELECT without its mapping-preservation mechanism compare with the original method on continual learning and forgetting metrics on UCIT and MLLM-DCL?
  2. Experimental comparison with simpler component designs. Have the authors evaluated simpler alternatives to the proposed components, such as replacing Pretrained Subspace Guidance with a simpler gradient-constraint method from continual learning? How would such a replacement affect continual learning and forgetting metrics on UCIT and MLLM-DCL?
  3. Evaluation on longer task sequences. The task sequences in UCIT and MLLM-DCL are relatively short. Have the authors evaluated the method on longer sequences?

I would consider raising my score if the authors can fully address the concerns above.

Reviewer_GmsqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MapSelect, a framework for mitigating catastrophic forgetting in continual multimodal instruction tuning through History-Guided Mapping Adaptation (HMA), Selection-Guided Adapter Reuse (SAR), and Pretrained Subspace Guidance (PSG). The method achieves promising results on UCIT and MLLM-DCL. However, this work suffers from multiple weaknesses regarding the selection of compared baselines, adaptation parameter analysis, and the mechanisms underlying SAR. Its overall quality requires further enhancement.

Critical strengths

1.The paper investigates the effects of visual mapping changes and adapter selection on catastrophic forgetting, addressing a meaningful research problem. 2.The projector restoration and historical adapter selection experiments provide intuitive empirical insights.

Critical weaknesses

1.Some components of the proposed method are related to existing approaches involving visual projector adaptation, gradient subspace constraints, and historical adapter routing. However, the paper does not sufficiently clarify its specific technical contributions beyond these existing techniques. Furthermore, the main experiments only compare methods published between 2022 and 2025, omitting recent approaches from 2026, such as SAME (ICML 2026), DGG (CVPR 2026), and CRAM (EMNLP 2026). Notably, SAME and CRAM are already cited in the paper but are not included in the experimental comparisons. CRAM also reports a UCIT score of 74.73, slightly higher than MapSelect's 74.52, although results obtained under different experimental protocols are not directly comparable. Given that some of these works are concurrent, I encourage the authors to provide a more thorough discussion and, where feasible, conduct comparisons with recent methods under consistent and comparable experimental settings. 2.MapSelect employs language LoRA adapters with a rank of 96 on UCIT, accumulating approximately 1.55B additional parameters after six tasks. The paper does not systematically report parameter budgets for the baselines or provide a comparison under matched parameter budgets. Moreover, a baseline retaining the same adapter architecture while removing all three proposed modules is missing. Consequently, it is difficult to determine whether the performance improvements primarily result from the proposed mechanisms or the increased model capacity. 3.In Eq. (2), the shared factor As is continuously updated, which may cause functional drift in historical adapters even when their task-specific parameters are frozen. Furthermore, SAR combines training-time adapter reuse with inference-time fixed routing, but the ablation experiments do not evaluate their individual contributions. The projector also uses a single adapter during training but a Top-2 mixture during inference, and the rationale for this design requires further clarification. 4.The notation ‘bold/underline: best/second best’ in Table 2 is inaccurate. For instance, on the three metrics of F30K, MFT and IQ, the proposed method performs substantially worse than some compared approaches, yet the authors still mark their method as best (Bold). The experimental analysis fails to explore potential reasons why the accuracy is inferior to those competing methods. 5.All experiments are conducted using LLaVA-v1.5-7B, without evaluation on other model architectures such as Qwen.

Questions for authors

1.Could the authors clarify the specific technical contributions of HMA, SAR, and PSG over existing approaches and provide comparisons with recent methods such as SAME, DGG, and CRAM under consistent experimental settings? 2.Could the authors provide comparisons under matched parameter budgets and include a baseline that retains the same adapter architecture but removes all three proposed modules? 3.Does the continuous updating of the shared factor As cause functional drift in historical adapters? Could the authors further clarify the individual contributions of SAR's training-time reuse and inference-time routing, as well as the rationale for using different projector adapter configurations during training and inference? 4.Could the authors provide experiments on other MLLM architectures, such as Qwen, to further validate the generalizability of the proposed method?

Reviewer_6GtBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines two problems in multimodal continual instruction tuning: changes in the vision-to-language mapping and the model’s failure to select the appropriate historical language adapter. Through intervention experiments that restore old projectors and specify historical adapters, the authors show that some forgetting can be reversed. They then propose MapSelect: HMA uses historical projector LoRA factors to constrain new gradients; SAR combines the reuse of historical adapters during training with fixed routing during generation; and PSG uses pretrained language weights to provide gradient constraints. The method is evaluated on UCIT and MLLM-DCL using LLaVA-v1.5-7B, yielding improvements in average performance.

Critical strengths
  1. The paper’s diagnostic experiments have practical value. Freezing the language model at a later stage and restoring only the old projector tests the effect of the visual mapping on the output more directly than simply comparing accuracy before and after training.
  2. The authors provide fairly extensive implementation and diagnostic details, including task features, routing rules, training configurations, a proof concerning the gradient operator, storage costs, and an evaluation of pretrained capabilities on MME/MMBench.
Critical weaknesses
  1. The evidence from the ablation experiments is weak. Table 3 is based on only one random seed (42), and the MFN differences among variants are small: from 73.80 to 74.13, a maximum of just 0.33 percentage points. The paper’s own data show that variation across runs is far greater than this.
  2. The existing ablations do not adequately distinguish the effects of the proposed mechanisms from those of adapter retention, routing changes, and shared-parameter updates. Table 3 reports only the results of removing modules one at a time. It lacks a base system with the same capacity and routing settings in which all three new mechanisms are removed simultaneously. It therefore remains unclear how much of the full system’s substantial gain over the baseline comes from retaining task-specific projectors/language adapters.
  3. Comparisons with the closest methods and capacity controls are insufficient, making it difficult to determine the contribution of the new designs. The paper already cites SAME and CRAM, which directly study routing/expert drift and centroid routing, historical subspace constraints, and expert reuse, respectively, yet neither appears in Table 2 or Table 4.
Questions for authors

See Weakness above。

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.