acceptodds
Under review as a conference paper at ICLR 2027

CORE: Collaborative Orthogonal Residual Experts for Multimodal Continual Instruction Tuning

Abstract

Multimodal Continual Instruction Tuning (MCIT) aims to adapt multimodal large language models to sequentially arriving instruction tasks while preserving previously acquired capabilities. Recent methods address this problem by expanding task-specific LoRA experts and freezing historical experts, thereby reducing direct parameter interference. However, parameter isolation alone does not ensure effective use of newly introduced capacity: when tasks overlap, a new expert may redundantly learn representations already captured by relevant historical experts. We propose Collaborative Orthogonal Residual Experts (CORE), which reframes the newly allocated expert as a residual expert that learns with respect to reusable historical knowledge. CORE uses sample-level dual-modal reference scores to reuse relevant historical experts and select a reference expert for residual learning. Because multimodal hidden states entangle task-dependent visual evidence with reusable non-visual capabilities, CORE introduces Attention-Masked Modality Decoupling (AMMD) to construct visual-oriented representations from image-centric attention heads. Orthogonal Subspace Exploration (OSE) then discourages redundant alignment between the current and reference experts in this space, with its strength adapted by the selected reference score. Experiments on the CoIN benchmark demonstrate that CORE consistently outperforms existing methods in current-task adaptation and previous-task retention.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Multimodal Continual Instruction Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DoXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CORE, a framework for multimodal continual instruction tuning that combines historical expert reuse with orthogonal residual learning. The method progressively expands a pool of LoRA experts while keeping previously learned experts frozen. For each input, it uses multimodal similarity to combine historical experts and select a reference expert. Orthogonal Subspace Exploration (OSE) loss encourages the new expert to learn complementary representations, while Attention-Masked Modality Decoupling (AMMD) focuses the orthogonality constraint on visually relevant features by applying attention head masks.

Critical strengths
  1. Multimodal continual instruction tuning is an important research problem that addresses two challenges of facilitating knowledge transfer and mitigating catastrophic forgetting.
  2. The paper is generally easy to follow.
Critical weaknesses
  1. The related work and experimental comparisons do not sufficiently cover recent research on continual instruction tuning with LLaVA, including [1–6]. A more thorough discussion and comparison would help clarify the novelty and empirical effectiveness of the proposed method.

[1] Enhancing Multimodal Continual Instruction Tuning with BranchLoRA. ACL 2025.

[2] From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning. ACL 2026.

[3] PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning. ICLR 2026.

[4] Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models. arXiv 2024.

[5] SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning. ICML 2026.

[6] Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning. ICML 2026.

  1. The weighted reuse of historical experts in Eqs. (4)–(9) is closely related to ProgLoRA’s task-aware allocation. Although ProgLoRA is cited and included in the appendix, the manuscript would benefit from a more detailed explanation/comparison of how CORE’s weighting differs from its allocation mechanisms.

  2. The main results would benefit from reporting zero-shot, multi-task training, and each task’s performance after learning that task, which are commonly reported by previous methods such as CoIN and ProgLoRA.

  3. AMMD introduces a reference-guided mask for computing the orthogonality objective, together with the hyperparameter ρ\rho. The study of ρ\rho in Table 6 would be more informative with more values. Given that the final performance is sensitive to the value of ρ\rho, is it better to use unmasked OSE with a tuned loss coefficient?

  4. It is unclear what the difference is between the Reuse+OSE_{AMMD} ablation and full CORE in Table 2.

Questions for authors

Please see the weakness.

2 comments

Thank you for the feedback. Our responses follow.

W1 — Recent MCIT work. We will add the six cited papers to the Related Work and compare their approaches with CORE's reference-guided residual training. Our current evaluation includes established MCIT methods such as HiDe-LLaVA and LLaVA-DyMoE. The revision will also report UCIT results and a matched comparison with the non-federated DISCO adaptation on CoIN and UCIT.

W2 — ProgLoRA. ProgLoRA already supports historical reuse and adaptive allocation, including instance-level allocation in its dynamic variant. Our claim is not that expert weighting is new. CORE uses a selected historical expert to train the new expert: the reference determines the AMMD head mask, provides the expert output for OSE, and sets the penalty strength. ProgLoRA's allocation and CORE's reference-guided residual objective therefore serve different purposes.

W3 — Additional evaluation. Figure 3 reports stage-wise adaptation and retention; Table 1 reports final task scores. We will add the absolute immediate post-task scores At,tA_{t,t} and show published CoIN/ProgLoRA zero-shot and multi-task results as clearly attributed reference points. Differences in evaluation protocols will be identified; the published reference results will not be presented as new CORE runs.

W4 — AMMD versus tuned full-space OSE. The Appendix already reports results for ρ=0.6,0.7,0.8,0.9\rho=0.6,0.7,0.8,0.9 (MFN: 58.91, 59.72, 60.15, 59.34). Table 2 gives 54.51 MFN for full-space OSE and 59.25 for AMMD-localized OSE under the reported settings. We will add a loss-coefficient sweep for full-space OSE to check whether tuning its regularization strength closes this gap. Unlike changing a global coefficient, AMMD changes which representation components are constrained.

W5 — OSE_AMMD versus CORE. Both use the AMMD-selected space. Full CORE additionally weights OSE by the selected reference score, ωit=max⁡(0,si⋆,t)\omega_i^t=\max(0,s_i^{\star,t}). This distinction is described in the ablation discussion; Table 2 shows an increase from 59.25 to 60.15 MFN.

Thanks for your additional responses, clarifications, and plans.

Reviewer_7CCTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, Collaborative Orthogonal Residual Experts (CORE) reframes the newly allocated expert as a residual expert that learns with respect to reusable historical knowledge. Under CORE, the Attention-Masked Modality Decoupling (AMMD) is designed to construct visual-oriented representations from image-centric attention heads. Moreover, Orthogonal Subspace Exploration (OSE) then discourages redundant alignment between the current and reference experts in this space, with its strength adapted by the selected reference score . Experiments on the CoIN benchmark demonstrate that CORE consistently outperforms existing methods in current-task adaptation and previous-task retention.

Critical strengths
  1. The formulations for sample level historical expert reuse, reference selection, and attention masked modality decoupling are clearly presented.
  2. An excessive number of SOTA methods, such as LoRA, EWC, MoELoRA, HiDe-LLaVA, LwF, O-LoRA, IncLoRA and LLaVA-DyMoE, are included for comparison.
  3. The overall performance results from CoIN experiments are sufficient.
Critical weaknesses
  1. The introduction requires additional references covering recent studies on the differences between newly allocated experts and the existing expert pool.
  2. In the introduction, the authors state that realizing this residual-learning perspective under MCIT introduces three challenges: historical knowledge reuse and reference expert selection should be sample-specific; new experts require dedicated learning signals; and residual expert learning must account for the multimodal structure of MLLMs. However, the authors do not provide justifications for these challenges.
  3. In the introduction, the authors fail to establish the relationship between AMMD and OSE.
  4. Figure 1 only illustrates the architecture of CORE and does not show its differences from other redundant expert learning architectures.
  5. The related work section should include discussions on redundant expert learning for MCIT, as this topic is highly relevant to the paper.
  6. For Equation (6), the choice of parameter 1/2 appears to be an empirical design.
  7. For Equation (12), the origin of the attention weight from textual query tokens to visual tokens is not specified.
  8. The section covering overall training and inference appears to serve little purpose.
  9. The authors do not explain why LoRA, EWC, MoELoRA, HiDe-LLaVA, LwF, O-LoRA, IncLoRA and LLaVA-DyMoE are selected as comparison baselines.
  10. Based on the results in Table 2, the authors do not justify why IncLoRA is adopted as a key component of CORE.
  11. The experimental results for Inter Task Visual Overlap, Sample Level Reference Selection and Task Dependent Image Centric Head Sets are unconvincing, since they lack proper comparison baselines.
Questions for authors

NA

1 comment

Thank you for the comments. Several points are already addressed in the paper; the relevant details are below.

W1/W2/W5 — Motivation and prior work. The three challenges follow from the proposed residual-learning setting. Different samples can benefit from different historical experts; independent expansion does not explicitly discourage relearning their representations; and applying orthogonality to the entire multimodal hidden space can constrain useful non-visual information. The Related Work already covers ProgLoRA, SMoLoRA, D-MoLE, HiDe-LLaVA, and O-LoRA. We will make the connection to the residual-learning problem more explicit.

W3 — AMMD and OSE. Sections 3.4–3.5 specify this connection: AMMD uses the selected reference expert's image-centric head mask to construct a visual-oriented representation, and OSE penalizes alignment between expert outputs computed on that representation. Table 2 shows 59.25 MFN with AMMD-localized OSE, versus 54.51 with full-space OSE.

W4 — Figure 1. Figure 1 is the conceptual comparison of independent expert expansion and CORE's residual-learning approach. The detailed architecture is shown in Figure 2.

W6 — Equation (6). The factor 1/21/2 takes the equal-weight mean of two cosine similarities. This keeps the score on the cosine-similarity scale without adding a learned fusion parameter. The Appendix reports 60.15 MFN with dual-modal scoring, compared with 58.82 using images alone and 58.44 using instructions alone.

W7 — Equation (12). The weight αi,q,pe,ℓ,h\alpha_{i,q,p}^{e,\ell,h} comes from the model's self-attention map at layer ℓ\ell, head hh: textual token qq attends to visual token pp. We will state the source and token indexing explicitly.

W8 — Training and inference. Section 3.6 is necessary because the two procedures differ. Training optimizes the current expert with allocation-weighted historical reuse and OSE. At inference, all learned experts are frozen and combined using sample-adaptive allocation; there is no designated current expert. The section specifies this inference rule and the complete training objective.

W9/W10/W11 — Baselines and controls. The baselines cover regularization (EWC, LwF), LoRA-based learning (LoRA, O-LoRA), and expert-based MCIT (MoELoRA, HiDe-LLaVA, LLaVA-DyMoE). IncLoRA is an independent-expansion baseline, not a component of CORE; Table 2 uses it to separate expansion, historical reuse, and residual optimization. The cited figures describe visual overlap, sample-level references, and head distributions. Controlled comparisons are reported separately in Table 2 and the Appendix, including Random/Latest/Task-Level/Sample-Level reference selection, image-only/instruction-only scoring, and head-threshold sensitivity. We will add cross-references where these analyses are discussed.

Reviewer_mW1oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multimodal continual instruction tuning method using collaborative orthogonal residual experts, aiming to reduce redundancy between LoRA experts. The core idea is to dynamically reuse historical LoRA experts by calculating sample similarity through bimodal anchors. It also introduces an attention modality mask decoupling mechanism, using image-centric attention heads to build perception-oriented representations, and constrains the exploration of the current expert through orthogonal subspaces. This approach improves continual learning performance and reduces knowledge overlap between current and historical experts.

Critical strengths

This approach focuses on reusing historical knowledge and complementary learning for new experts, which is a perspective with some value. To address this issue, the method framework proposed in the paper has a fairly reasonable design logic, with each module's role being relatively clear. In addition, this method achieved good experimental results on the benchmarks used, with overall performance outperforming several existing methods.

Critical weaknesses
  1. The evaluation across different benchmarks is pretty limited. Even though the authors tested models of various sizes, including a 13B model, they only ran experiments on a single multimodal continual instruction-tuning benchmark. This means there's not much evidence that the method performs well across different tasks and continual learning scenarios.

  2. There isn't enough experimental evidence to support the motivation. The current experimental results don't fully back up the paper's argument about learning redundant visual knowledge across tasks. The authors should run more controlled experiments, using the same images with different questions, to see if the proposed method can effectively distinguish task-specific visual knowledge and reduce redundant learning between experts.

  3. There’s a lack of comparison with a closely related baseline. The proposed method uses a global prototype-based similarity estimation strategy similar to DISCO (Guo et al., 2025) for historical expert selection and knowledge reuse. However, DISCO wasn’t included in the experimental comparisons. Given the methodological similarity, it’s necessary to make a direct comparison with DISCO to better evaluate it.

  4. The method doesn’t offer much in terms of novelty. Several key components of the proposed framework are very similar to existing work. Specifically, previous studies have already looked into separating visual and text modalities, and the image-centric attention head selection mechanism in AMMD mostly follows Kang et al. (2025). Similarly, the bimodal prototype construction and similarity estimation strategies are largely adapted from DISCO (Guo et al., 2025). Additionally, orthogonality-based constraints have already been widely explored in LoRA-based continual learning. While these techniques are integrated into a single framework, the paper doesn’t really explain what’s new about how they’re combined, nor does it show how the proposed method is significantly different from existing approaches.

Questions for authors
  1. Supplement Disco's comparison experiments.
  2. Test your model's performance on MLLM-DCL, UCIT, or other benchmarks.
  3. Add experiments to show that this model really improves accuracy for samples when image features are similar.
2 comments

Thank you for the review. We address the main concerns below.

W1/Q2 — Benchmark coverage. CoIN contains eight different multimodal tasks. We also test three task orders, three instruction-template settings, and 7B/13B models. We will add results on the independent six-task UCIT benchmark in the revised manuscript.

W2/Q3 — Evidence for redundant expert learning. The Appendix (Redundant Expert Learning Analysis) directly measures overlap between current and historical expert outputs. Under +Reuse, reference relevance is positively associated with overlap (Spearman's rs=0.17r_s=0.17, p=2×10−5p=2\times10^{-5}). CORE reduces the measured overlap by 9.3× on average, with a reduction on 97.2% of samples. Table 2 then shows how this relates to performance: +Reuse achieves 55.11 MFN; full-space OSE, 54.51; AMMD-localized OSE, 59.25; and CORE, 60.15. In particular, orthogonality in the full space is not sufficient—the choice of constraint space matters. A same-image/different-question setting tests responses to different instructions, but shared images alone cannot establish redundancy between learned expert representations. We will point more explicitly to the existing overlap analysis and ablations in the revision.

W3/Q1 — DISCO. The original DISCO is designed for federated continual instruction tuning and includes client/server knowledge organization; those results are not directly comparable to single-model sequential MCIT. However, MCITlib provides a non-federated adaptation. We will evaluate that adaptation on both CoIN and UCIT under matched sequential protocols, identifying it separately from the original federated method.

W4 — Novelty. CORE does not claim to invent similarity-based reuse, head selection, or orthogonality. Its contribution is using historical experts to guide the learning of a new residual expert. Sample-level dual-modal scores determine historical reuse and select a reference; the selected reference determines the AMMD head mask, supplies the expert output used by OSE, and sets the constraint weight. This is a training mechanism for deciding what the new expert should learn, rather than another way of routing existing experts. Table 2 and the reference-selection ablations test these choices.

Reviewer_mW1oOpenReview ↗

Comments

Thank you for the detailed response. I appreciate the clarifications. The pointer to the redundancy analysis in the Appendix and the ablations in Table 2 is helpful, and I agree that the original DISCO targets a federated setting, so its non-federated adaptation is the appropriate baseline. The clarified statement of the contribution, namely using historical experts to guide what the new residual expert learns, also makes the positioning of the paper clearer.

However, my main concerns remain. First, the results on a second benchmark such as UCIT and the comparison with the DISCO adaptation are only promised for the revision. Since no results are provided at this stage, I cannot assess whether the conclusions hold beyond CoIN or against the most closely related baseline. Second, the overlap analysis shows that the outputs of the current and historical experts overlap less, but this remains indirect evidence for the claim about redundant visual knowledge, and the controlled experiment I suggested was not conducted. Third, regarding novelty, the response confirms that the individual components follow prior work and that the contribution lies mainly in how they are combined. Given these points, I will keep my current rating. I encourage the authors to include the promised experiments in a future version of the paper.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your efforts so far. As the discussion period is coming to an end, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.