acceptodds
Withdrawn from ICLR 2027

MANY: Merge Anything For Multimodal Continual Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Continual Instruction Tuning (MCIT) is essential for sequential task adaptation of Multimodal Large Language Models (MLLMs) but is severely restricted by catastrophic forgetting. While existing literature focuses on the reasoning language backbone, in this work, we expose a critical yet neglected dual-forgetting phenomenon across both perception drift in Cross-modal Projection Space and reasoning collapse in Low-rank Parameter Space. To resolve this, we present **MAny** (**M**erge **Any**thing), a framework that merges task-specific knowledge through **C**ross-modal **P**rojection **M**erging (**CPM**) and **L**ow-rank **P**arameter **M**erging (**LPM**). Specifically, CPM recovers perceptual alignment by adaptively merging cross-modal visual representations via visual-prototype guidance, ensuring accurate feature recovery during inference. Simultaneously, LPM eliminates mutual interference among task-specific low-rank modules by recursively merging low-rank weight matrices. By leveraging recursive least squares, LPM provides a closed-form solution that mathematically guarantees an optimal fusion trajectory for reasoning stability. Notably, MAny operates as a training-free paradigm that achieves knowledge merging via efficient CPU-based algebraic operations, eliminating additional gradient-based optimization beyond initial tuning. Our extensive evaluations confirm the superior performance and robustness of MAny across multiple MLLMs and benchmarks. Specifically, on the UCIT benchmark, MAny achieves significant leads of up to 8.57% and 2.85% in final average accuracy over state-of-the-art methods across two different MLLMs, respectively.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Continual Instruction Tuning, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YaVZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article points out that forgetting in MCIT occurs not only in LLM, but also in cross modal projectors, namely the dual forgetting of "perceptual drift" and "inference collapse", and demonstrates this through diagnostic experiments of replacing projectors and LoRA. The author proposes a training free merging framework MAny: cross modal projection merging (CPM) saves one projector for each task, and performs soft weighted merging based on the similarity between image CLS features and task visual prototypes during inference; Low rank parameter merging (LPM) aims to minimize the feature drift of each layer, using recursive least squares to merge the LoRA increments of each task into a closed form solution, while maintaining the cumulative feature covariance matrix. On UCIT, MAny's final average accuracy on LLaVA-1.5-7B and InternVL-Chat-7B was 8.57 and 2.85 points higher than the strongest baseline, respectively, with forgetting close to zero. CPM can also plug and play to improve other methods.

Critical strengths
  • The diagnostic experiment for projector forgetting is simple and effective, and CPM, as a plug and play module, can significantly reduce forgetting on multiple baselines. This discovery has direct practical value for the MCIT community.
  • The method does not require additional gradient training, has a closed form solution during the merging process, and has achieved significant leading results on both backbones, demonstrating strong practicality.
Critical weaknesses
  • LPM aims to minimize the feature drift of each layer and uses the cumulative feature covariance to obtain a closed form solution to merge LoRA. This layer by layer least squares merging approach has been commonly used in model merging and continuous learning; The paper claims to be the first to combine the model with MCIT, and does not explain the unique features of LPM at the method level compared to existing merging methods, which weakens the core contribution and is the main reason why I tend to reject it.
  • CPM relies on visual prototype recognition tasks and is naturally effective on sequences with significant differences in image domains, such as UCIT. However, it is likely to fail when multiple tasks share similar images; Moreover, each task requires an additional projector to be saved, and current experiments are insufficient to demonstrate its applicability.
Questions for authors
  • Compared to directly weighting the LoRA of each task or adding task vectors, which tasks do LPM show the most significant improvement in? Can you explain where the improvement mainly comes from?
  • How is the scaling factor λ=3 selected? Is the sensitivity analysis in Figure 4 conducted on the test set or an independent validation set?
  • How much video memory and storage is required to maintain a complete d × d covariance matrix at each layer? How does the total storage cost compare to the baseline, including the projectors for each task?
Reviewer_FdJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies forgetting in both the multimodal projector and the language model’s low-rank adaptations. It proposes MAny, which combines Cross-modal Projection Merging (CPM) with Low-rank Parameter Merging (LPM). CPM retains task-specific projectors and combines their outputs using similarities between the input image and stored visual prototypes. LPM recursively merges task updates by minimizing a layer-wise feature-matching objective using accumulated second-order feature statistics. A compressed variant, MAny*, reduces the storage required for these statistics. The framework is evaluated on UCIT and MLLM-DCL with two model backbones.

Critical strengths
  1. The projector analysis provides useful evidence about an important source of forgetting. The paper goes beyond measuring representation similarity by restoring task-specific components and observing the effect on task performance. These experiments show why retaining a language adapter alone may be insufficient when the visual input mapping has changed. The improvements obtained by adding CPM to several methods also suggest that this component has value beyond the full MAny framework.

  2. The empirical gains are substantial, and both components contribute. On UCIT, MAny improves FAA from 65.93 to 74.50 on LLaVA and from 69.50 to 72.35 on InternVL, relative to the strongest baseline in each main comparison. The component ablation shows improvements from both CPM and LPM, with further gains when they are combined. Results on MLLM-DCL provide additional support across specialized domains.

Critical weaknesses
  1. The optimality result does not support the broader claims made about the merged model. The recursive solution is derived for a fixed-feature objective of the form min⁡τ∑i∥Xi(τ−τi)∥F2.\min_{\tau}\sum_i\|X_i(\tau-\tau_i)\|_F^2. This establishes equivalence to the batch least-squares solution for that objective. It does not establish that interference is eliminated in the complete nonlinear model, where merging earlier layers and changing the projector can also change the inputs to later layers. The distinction between minimizing a layer-wise surrogate and guaranteeing end-task retention is important here.

The deployed model also uses a scaled update, with λ=3\lambda=3 by default. In general, 3τ∗3\tau^* is no longer a minimizer of the stated objective. Likewise, the covariance truncation in MAny* introduces an approximation that is not covered directly by the exact equivalence result. The paper should distinguish these empirically motivated choices from the setting in which optimality is established.

  1. The comparisons do not sufficiently isolate LPM’s contribution or establish its practical cost. The main baselines are continual-learning methods, while the proposed contribution includes a specific model-merging rule. Comparisons using the same task-specific adaptations and CPM, but different merging rules, would be more informative about what LPM adds. In particular, the feature-aware merging approaches discussed in the related work are relevant points of comparison.

The efficiency discussion would also benefit from measured merging time and peak memory. Avoiding additional gradient-based training is attractive, but collecting layer-wise statistics, solving the associated systems, and retaining multiple projectors still incur costs. The reported compression ratio for MAny* is useful, but does not by itself characterize the cost of the complete procedure.

Questions for authors
  1. With CPM and the task-specific adaptations held fixed, how does LPM compare with simple weighted merging and a feature-aware merging method discussed in the paper?
  2. Which model and projector are used to collect XtX_t, and how are singular or ill-conditioned covariance matrices handled? Which guarantees remain applicable after scaling and truncation?
  3. What are the measured merging time and peak memory for MAny and MAny*, including feature-statistic collection?
Reviewer_znyuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MAny, a method for multimodal continual instruction tuning. It preserves task-relevant vision-language mappings through Cross-modal Projection Merging (CPM), which is guided by visual prototypes, and it merges LoRA task vectors via recursive least squares (LPM). The paper reports performance gains on two models across two benchmarks, UCIT and MLLM-DCL. It also provides component ablations and hyperparameter analyses.

Critical strengths

The paper addresses the joint forgetting of the projection module and the language-side parameters and proposes a modular solution, which is of practical research significance. Moreover, the standalone design of CPM, together with the experiments that combine it with different continual learning methods, points to a valuable research direction.

Critical weaknesses
  1. In Table 4 on page 17, the FAA improvements of MAny and MAny* over SEFE should be 5.11 and 5.57 percentage points, respectively, yet they are reported as 5.18 and 5.64; the same error recurs in line 890. In lines 451–453 on page 9, the FFM of LoRA-FT is cited as 18.64, whereas Table 3, which is the subject of that discussion, reports 17.44. Furthermore, the green curve for LLaVA on IconQA in Figure 1 on page 2 lies at approximately 73–74, which differs markedly from the value of 66.37 reported in Table 1 on page 8. Likewise, the dashed "best baseline" line for LLaVA FAA in Figures 4 and 5 (on pages 9 and 18, respectively) lies at approximately 65.2, below the best-baseline value of 65.93 in Table 1. Although values read from the figures are only estimates, the discrepancies are evident, and the authors should clarify the experimental settings and the provenance of these results. These inconsistencies cast serious doubt on the authenticity and accuracy of the reported data.
  2. For the full MAny model on UCIT with the LLaVA backbone, the CAA is reported as 83.42 in Tables 1, 2, and 3, but as 83.24 in Table 5 and Table 6. The other metrics indicate that these entries correspond to the same configuration, yet the paper does not explain the discrepancy.
  3. The objective and closed-form solution in Eqs. (8)–(9) on page 6 are identical in form to those of LOT Merging for linear layers. Likewise, the analysis of orthogonal and collinear features in Eqs. (20)–(22) on page 15–16 corresponds, respectively, to Eqs. (16)–(18) of LOT. The novel contributions therefore need to be clearly distinguished from the existing objective, solution procedure, and mechanism analysis. Although Tables 2–3 on page 9 provide component ablations, the paper does not compare against LOT, RegMean, and simple merging under the same task experts, the same CPM, and the same hyperparameter-tuning budget. It is therefore not yet possible to determine how much additional value the proposed merging mechanism contributes.
  4. Eqs. (8)–(11) on pages 6–7 establish optimality only for the uncompressed least-squares problem with fixed features. This result cannot be directly extended to overall task performance, nor to the actual model after the scaling in Eq. (12). In addition, lines 423–424 on page 8 claim that compression incurs "no performance loss." Yet in Table 1, the FAA and CAA of InternVL drop from 72.35 and 83.35 to 71.84 and 83.10, respectively, while its FFM rises from 1.77 to 3.00. Figure 5 on page 18 likewise does not support the blanket conclusion that γ=0.999 outperforms γ=1.0 for InternVL.
  5. The experiments in lines 404–405 on page 8, Table 1, and Table 4 on page 17 cover only two configurations, LLaVA-1.5-7B and InternVL-Chat-7B. They do not examine larger language backbones, such as LLaVA-1.5-13B from the same model family. In addition, Eqs. (9)–(11) on pages 6–7 involve layer-wise covariance matrices and their inversion, and the cost of these operations grows with both layer width and the number of layers. The authors are encouraged to report performance, degree of forgetting, merging time, and peak memory on larger models. This would show whether the gains hold up, and whether the computational cost stays manageable, as model size increases.
  6. The experiments in lines 404–405 on page 8, Table 1, and Table 4 on page 17 are all confined to LLaVA-1.5-7B and InternVL-Chat-7B. They do not cover more recent models such as LLaVA-OneVision (2024) and InternVL3 (2025). The paper takes "perception drift" and "reasoning collapse" as the core rationale for its method design. However, it remains unverified whether this diagnosis, and the associated gains, still hold under newer visual representations, cross-modal alignment schemes, and training recipes. The authors are encouraged to include at least one more recent backbone of comparable scale and to evaluate performance and forgetting under a unified protocol, so as to substantiate the broad applicability of the method.
Questions for authors

Please See Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.