- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes CORE, a framework for multimodal continual instruction tuning that combines historical expert reuse with orthogonal residual learning. The method progressively expands a pool of LoRA experts while keeping previously learned experts frozen. For each input, it uses multimodal similarity to combine historical experts and select a reference expert. Orthogonal Subspace Exploration (OSE) loss encourages the new expert to learn complementary representations, while Attention-Masked Modality Decoupling (AMMD) focuses the orthogonality constraint on visually relevant features by applying attention head masks.
Critical strengths
- Multimodal continual instruction tuning is an important research problem that addresses two challenges of facilitating knowledge transfer and mitigating catastrophic forgetting.
- The paper is generally easy to follow.
Critical weaknesses
- The related work and experimental comparisons do not sufficiently cover recent research on continual instruction tuning with LLaVA, including [1–6]. A more thorough discussion and comparison would help clarify the novelty and empirical effectiveness of the proposed method.
[1] Enhancing Multimodal Continual Instruction Tuning with BranchLoRA. ACL 2025.
[2] From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning. ACL 2026.
[3] PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning. ICLR 2026.
[4] Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models. arXiv 2024.
[5] SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning. ICML 2026.
[6] Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning. ICML 2026.
-
The weighted reuse of historical experts in Eqs. (4)–(9) is closely related to ProgLoRA’s task-aware allocation. Although ProgLoRA is cited and included in the appendix, the manuscript would benefit from a more detailed explanation/comparison of how CORE’s weighting differs from its allocation mechanisms.
-
The main results would benefit from reporting zero-shot, multi-task training, and each task’s performance after learning that task, which are commonly reported by previous methods such as CoIN and ProgLoRA.
-
AMMD introduces a reference-guided mask for computing the orthogonality objective, together with the hyperparameter . The study of in Table 6 would be more informative with more values. Given that the final performance is sensitive to the value of , is it better to use unmasked OSE with a tuned loss coefficient?
-
It is unclear what the difference is between the Reuse+OSE_{AMMD} ablation and full CORE in Table 2.
Questions for authors
Please see the weakness.
2 comments
Thanks for your additional responses, clarifications, and plans.
Thank you for the feedback. Our responses follow.
W1 — Recent MCIT work. We will add the six cited papers to the Related Work and compare their approaches with CORE's reference-guided residual training. Our current evaluation includes established MCIT methods such as HiDe-LLaVA and LLaVA-DyMoE. The revision will also report UCIT results and a matched comparison with the non-federated DISCO adaptation on CoIN and UCIT.
W2 — ProgLoRA. ProgLoRA already supports historical reuse and adaptive allocation, including instance-level allocation in its dynamic variant. Our claim is not that expert weighting is new. CORE uses a selected historical expert to train the new expert: the reference determines the AMMD head mask, provides the expert output for OSE, and sets the penalty strength. ProgLoRA's allocation and CORE's reference-guided residual objective therefore serve different purposes.
W3 — Additional evaluation. Figure 3 reports stage-wise adaptation and retention; Table 1 reports final task scores. We will add the absolute immediate post-task scores At,t and show published CoIN/ProgLoRA zero-shot and multi-task results as clearly attributed reference points. Differences in evaluation protocols will be identified; the published reference results will not be presented as new CORE runs.
W4 — AMMD versus tuned full-space OSE. The Appendix already reports results for ρ=0.6,0.7,0.8,0.9 (MFN: 58.91, 59.72, 60.15, 59.34). Table 2 gives 54.51 MFN for full-space OSE and 59.25 for AMMD-localized OSE under the reported settings. We will add a loss-coefficient sweep for full-space OSE to check whether tuning its regularization strength closes this gap. Unlike changing a global coefficient, AMMD changes which representation components are constrained.
W5 — OSE_AMMD versus CORE. Both use the AMMD-selected space. Full CORE additionally weights OSE by the selected reference score, ωit=max(0,si⋆,t). This distinction is described in the ablation discussion; Table 2 shows an increase from 59.25 to 60.15 MFN.