- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article points out that forgetting in MCIT occurs not only in LLM, but also in cross modal projectors, namely the dual forgetting of "perceptual drift" and "inference collapse", and demonstrates this through diagnostic experiments of replacing projectors and LoRA. The author proposes a training free merging framework MAny: cross modal projection merging (CPM) saves one projector for each task, and performs soft weighted merging based on the similarity between image CLS features and task visual prototypes during inference; Low rank parameter merging (LPM) aims to minimize the feature drift of each layer, using recursive least squares to merge the LoRA increments of each task into a closed form solution, while maintaining the cumulative feature covariance matrix. On UCIT, MAny's final average accuracy on LLaVA-1.5-7B and InternVL-Chat-7B was 8.57 and 2.85 points higher than the strongest baseline, respectively, with forgetting close to zero. CPM can also plug and play to improve other methods.
Critical strengths
- The diagnostic experiment for projector forgetting is simple and effective, and CPM, as a plug and play module, can significantly reduce forgetting on multiple baselines. This discovery has direct practical value for the MCIT community.
- The method does not require additional gradient training, has a closed form solution during the merging process, and has achieved significant leading results on both backbones, demonstrating strong practicality.
Critical weaknesses
- LPM aims to minimize the feature drift of each layer and uses the cumulative feature covariance to obtain a closed form solution to merge LoRA. This layer by layer least squares merging approach has been commonly used in model merging and continuous learning; The paper claims to be the first to combine the model with MCIT, and does not explain the unique features of LPM at the method level compared to existing merging methods, which weakens the core contribution and is the main reason why I tend to reject it.
- CPM relies on visual prototype recognition tasks and is naturally effective on sequences with significant differences in image domains, such as UCIT. However, it is likely to fail when multiple tasks share similar images; Moreover, each task requires an additional projector to be saved, and current experiments are insufficient to demonstrate its applicability.
Questions for authors
- Compared to directly weighting the LoRA of each task or adding task vectors, which tasks do LPM show the most significant improvement in? Can you explain where the improvement mainly comes from?
- How is the scaling factor λ=3 selected? Is the sensitivity analysis in Figure 4 conducted on the test set or an independent validation set?
- How much video memory and storage is required to maintain a complete d × d covariance matrix at each layer? How does the total storage cost compare to the baseline, including the projectors for each task?