acceptodds
Withdrawn from ICLR 2027

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multimodal continual learning, mixture of experts, parameter-efficient fine-tuning, structured matrices, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_K3u4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets catastrophic forgetting in multimodal continual instruction tuning with mixtures of LoRA experts, arguing that a dense expert lets an input-factor update change many output features. Monarch-MoE builds each expert from two block-diagonal factors joined by a fixed permutation, so every input-output block pair has its own low-rank channel writing to one output block, with batched and fused kernels for single-GPU use. It is evaluated on LLaVA-1.5-7B over MLLM-DCL and UCIT, standalone and as a drop-in expert in CL-MoE, O-LoRA and DISCO, plus capacity, partition, representation and efficiency probes. The authors conclude that structured experts improve first-task retention with fewer adapter parameters, at method-dependent cost to final performance.

Critical strengths
  1. A simple drop-in change with an exact, carefully scoped locality argument (Section 3.3, Eq. (3)). In DISCO on DCL it gives the best MFN, BWT and first-task forgetting with about a quarter of the adapter parameters (Table 1).
  2. Full task matrices (Appendix G), complete scale sweeps including adverse settings (Tables 9, 10), and explicit disclosure of unmatched comparisons (Appendix B.1).
Critical weaknesses
  1. in the DCL standalone pair, LoRA runs at scale 2 and Monarch at 1/16 with the same learning rate (Appendix B.1); Table 2 matches rank or budget, not scale. With budget and scale fixed (Table 3), the m=1 row, a dense rank-8 LoRA mixture, forgets only 3.29 pp vs 1.74 at m=8, so most of the 25.98 vs 3.01 headline gap looks like configuration. Both standalone mixtures also fall below LoRA-FT in MFN on both benchmarks.
  2. F1 isolates the task that benefits. On DCL, the RS gain exceeds the whole standalone MFN gain, and the CL-MoE and O-LoRA replacements end lower on every other task (Appendix G). MFT drops in all six replacements, MFN in five.
  3. The expert is MoRe’s adapter, and Appendix A.2 shows it is an independent rank-r LoRA per weight block, as in GraLoRA, which uses the same gradient-entanglement motivation; MELoRA is the block-diagonal case. Locality holds for any block partition, including the unrun P=I, so the permutation’s role is untested. Neither these nor CL adapters such as InfLoRA are compared.
  4. Systems gains are modest and unablated: 19.5% faster training at one setting, 8.1% slower generation with 3.7x fewer adapter parameters, and an unexplained dense-path crash (Table 12).
Questions for authors
  1. Is Table 3’s m=1 row a dense rank-8 LoRA mixture? What does dense LoRA-MoE give at scale 1/16 and the same learning rate on RS→Med over three seeds?
  2. What do CL-MoE, O-LoRA and DISCO with LoRA experts achieve when rerun in your codebase at about 100M parameters?
  3. What are the P=I results at the same budget and scale?
  4. How was α chosen per host and benchmark, e.g. α=16 for DCL O-LoRA when α=64 gives higher MFN (Table 9)?
Reviewer_T4fROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Monarch-MoE, which constructs structured experts using block-diagonal matrices and fixed permutations to mitigate catastrophic forgetting in multimodal continual instruction tuning. The authors integrate this structure into an existing continual learning framework and optimize training and inference efficiency. Experiments show that the method can reduce adapter parameters and, in some settings, improve knowledge retention on historical tasks.

Critical strengths
  1. The paper attempts to use structured expert parameters to reduce task interference in continual learning, which has certain research value.
  2. The method can effectively reduce adapter parameters and, in some experiments, alleviate forgetting on historical tasks, showing some potential for practical application.
Critical weaknesses
  1. This paper mainly applies an existing Monarch matrix decomposition to a LoRA-MoE continual learning framework. MoRe (2024) has already studied the application of Monarch structures in parameter-efficient fine-tuning, while D-MoLE (ICML 2025), SAME (ICML 2026), and InfLoRA (CVPR 2024) have also investigated task interference in continual learning from the perspectives of expert structure, update constraints, and low-rank subspaces, respectively. Compared with these existing works, this paper has not yet sufficiently demonstrated independent and significant methodological innovation, and it lacks comparison experiments that highlight the unique advantages of the proposed structure.
  2. In addition, the paper lacks a complete model architecture diagram. The existing figures mainly explain Monarch's block-level parameter structure and do not intuitively show the overall model, expert routing, or the relationships among modules, which hinders understanding of how the method specifically differs from existing LoRA-MoE frameworks.
  3. The paper emphasizes that block-level gradient isolation can reduce cross-task interference, but local parameter updates do not necessarily imply reduced gradient interference across different tasks. The current experiments do not sufficiently demonstrate that the performance improvement originates from this mechanism rather than from changes in parameter structure or optimization settings.
  4. Moreover, although the method reduces forgetting on some tasks, overall continual learning performance is not consistently improved. For example, on the UCIT dataset, after introducing Monarch into three continual learning methods, the final average performance (MFN) all decreased, with DISCO's MFN dropping from 69.66 to 67.58. At the same time, the full task sequence lacks multi-random-seed experimental validation, making it difficult to fully demonstrate the stability and effectiveness of the method.
Questions for authors
  1. Compared with existing works such as MoRe, D-MoLE, and SAME, what is the most core technical innovation of this paper? Can the authors provide more targeted comparison experiments?
  2. How can you demonstrate that block-level gradient isolation effectively reduces cross-task interference, rather than merely changing the parameter update scheme?
  3. Why does the forgetting metric improve on the UCIT dataset while the final average performance of all three methods decreases? Can you provide multi-random-seed experimental results on the full task sequence?
Reviewer_88kFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, Monarch-MoE replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. Under CORE, it provides blockwise gradient isolation: individual block-pair updates affect only one output block. Moreover, the batched computation, fused forward operators, and an optimized backward pass are combined to make these compact experts practical on memory limited hardware. Experiments on the benchmark demonstrate that Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively.

Critical strengths
  1. The paper includes comparisons against an excessive number of SOTA methods, including LoRA-FT, CL-MoE, MoELoRA, O-LoRA, SEFE, DISCO, HiDe-LLaVA and LoRA-MoE.
  2. Training and inference are conducted on a single GPU to verify the efficiency of the proposed method.
Critical weaknesses
  1. The introduction requires additional references covering recent studies on the differences between dense low-rank factors and conventional LoRA.
  2. In the introduction, the authors mention the property of blockwise gradient isolation for preserving learning strategies built upon Monarch matrices, yet they do not justify this design choice.
  3. The introduction fails to establish the connection between Monarch-MoE and memory-efficient execution.
  4. The architecture of Monarch-MoE is not presented.
  5. The related work section should include discussions on block-diagonal or MoE-based continual learning, as these topics are highly relevant to this work.
  6. For Equation (2), definitions of R and M are not provided.
  7. The formulation in Equation (3) lacks interpretation.
  8. The origins of mh, mw, di, and do in Equation (5) are not specified.
  9. The authors do not explain why LoRA-FT, CL-MoE, MoELoRA, O-LoRA, SEFE, DISCO, HiDe-LLaVA and LoRA-MoE are selected as comparison baselines.
  10. From the results in Figure 2, regarding retention and representation stability under DCL, please explain the rationale for the hyperparameter setup q=16, ,m=8, r=2.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.