- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets catastrophic forgetting in multimodal continual instruction tuning with mixtures of LoRA experts, arguing that a dense expert lets an input-factor update change many output features. Monarch-MoE builds each expert from two block-diagonal factors joined by a fixed permutation, so every input-output block pair has its own low-rank channel writing to one output block, with batched and fused kernels for single-GPU use. It is evaluated on LLaVA-1.5-7B over MLLM-DCL and UCIT, standalone and as a drop-in expert in CL-MoE, O-LoRA and DISCO, plus capacity, partition, representation and efficiency probes. The authors conclude that structured experts improve first-task retention with fewer adapter parameters, at method-dependent cost to final performance.
Critical strengths
- A simple drop-in change with an exact, carefully scoped locality argument (Section 3.3, Eq. (3)). In DISCO on DCL it gives the best MFN, BWT and first-task forgetting with about a quarter of the adapter parameters (Table 1).
- Full task matrices (Appendix G), complete scale sweeps including adverse settings (Tables 9, 10), and explicit disclosure of unmatched comparisons (Appendix B.1).
Critical weaknesses
- in the DCL standalone pair, LoRA runs at scale 2 and Monarch at 1/16 with the same learning rate (Appendix B.1); Table 2 matches rank or budget, not scale. With budget and scale fixed (Table 3), the m=1 row, a dense rank-8 LoRA mixture, forgets only 3.29 pp vs 1.74 at m=8, so most of the 25.98 vs 3.01 headline gap looks like configuration. Both standalone mixtures also fall below LoRA-FT in MFN on both benchmarks.
- F1 isolates the task that benefits. On DCL, the RS gain exceeds the whole standalone MFN gain, and the CL-MoE and O-LoRA replacements end lower on every other task (Appendix G). MFT drops in all six replacements, MFN in five.
- The expert is MoRe’s adapter, and Appendix A.2 shows it is an independent rank-r LoRA per weight block, as in GraLoRA, which uses the same gradient-entanglement motivation; MELoRA is the block-diagonal case. Locality holds for any block partition, including the unrun P=I, so the permutation’s role is untested. Neither these nor CL adapters such as InfLoRA are compared.
- Systems gains are modest and unablated: 19.5% faster training at one setting, 8.1% slower generation with 3.7x fewer adapter parameters, and an unexplained dense-path crash (Table 12).
Questions for authors
- Is Table 3’s m=1 row a dense rank-8 LoRA mixture? What does dense LoRA-MoE give at scale 1/16 and the same learning rate on RS→Med over three seeds?
- What do CL-MoE, O-LoRA and DISCO with LoRA experts achieve when rerun in your codebase at about 100M parameters?
- What are the P=I results at the same budget and scale?
- How was α chosen per host and benchmark, e.g. α=16 for DCL O-LoRA when α=64 gives higher MFN (Table 9)?