- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper lies at the intersection of continual learning and multimodal large language model fine-tuning, addressing catastrophic forgetting during continual multimodal instruction tuning. Through an analysis of prior work and empirical evidence, the authors identify two complementary challenges: mapping preservation and adapter selection. To address these challenges, they introduce History-Guided Mapping Adaptation, Selection-Guided Adapter Reuse, and Pretrained Subspace Guidance.
The main contribution of this paper is that the authors identify and empirically validate two complementary challenges in continual multimodal instruction tuning: mapping preservation and adapter selection. The authors further validate the proposed method through experiments on UCIT and MLLM-DCL and use component ablations to demonstrate the effectiveness of individual components.
Critical strengths
The paper’s main strength is its empirical investigation of mapping preservation and adapter selection. In particular, Table 1 reveals a nontrivial finding: restoring an earlier projector while keeping the later LLM fixed recovers a substantial fraction of forgotten examples. This suggests that changes in the multimodal mapping account for part of the observed forgetting and could inform future approaches to continual multimodal instruction tuning. This insight is central to my positive assessment of the paper, as it advances our understanding of forgetting beyond the proposed method’s performance gains.
Critical weaknesses
- The necessity of the method’s complexity is insufficiently established. The method comprises multiple components, each with a relatively complex design. Although the ablation studies demonstrate the contribution of individual components, they do not establish whether simpler alternatives could address the same two challenges. Comparing against such alternatives would help clarify whether the proposed designs are necessary. Without this evidence, it is difficult to assess whether the performance gains justify the added complexity.
- There is substantial overlap among the experiments, while some evidence needed to establish practical usefulness is missing. Table 1 and Figure 1 both support the identification of mapping preservation and adapter selection as key challenges. Tables 7 and 8 report detailed MAPSELECT results on UCIT and MLLM-DCL, supplementing the comparisons across methods in Tables 2 and 4, respectively. Similarly, Figure 3(a,b) provides additional detail on the results in Table 3, while Figure 3(c,d) supplements those in Table 2. These experiments largely reinforce the same findings rather than provide distinct evidence, and much of this material would be better suited to the appendix. Meanwhile, the paper introduces several hyperparameters without examining sensitivity to their choices. This matters for MLLM fine-tuning, where each run is expensive and extensive tuning could substantially increase the cost of achieving good performance. The paper also lacks computational cost comparisons with baselines such as HiDe, CL-MoE, ModalPrompt, and DISCO under the same experimental setting, leaving its computational efficiency relative to these methods unclear. Overall, the insufficient experimental evidence limits my confidence in the method’s practical usefulness.
Questions for authors
- Clarification and experimental comparison regarding projector freezing. The paper uses instruction-tuned LLaVA-v1.5-7B as its backbone. Its projector has already undergone alignment pretraining and instruction tuning. Have the authors tried freezing the projector and adapting only the LLM parameters? Specifically, how do simple baselines such as LoRA-FT and O-LoRA perform with the projector frozen versus trainable? Under the frozen-projector setting, how would a simplified MAPSELECT without its mapping-preservation mechanism compare with the original method on continual learning and forgetting metrics on UCIT and MLLM-DCL?
- Experimental comparison with simpler component designs. Have the authors evaluated simpler alternatives to the proposed components, such as replacing Pretrained Subspace Guidance with a simpler gradient-constraint method from continual learning? How would such a replacement affect continual learning and forgetting metrics on UCIT and MLLM-DCL?
- Evaluation on longer task sequences. The task sequences in UCIT and MLLM-DCL are relatively short. Have the authors evaluated the method on longer sequences?
I would consider raising my score if the authors can fully address the concerns above.