- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies multimodal continual instruction tuning, arguing that MoE-based approaches assume task-level expertise that does not reliably emerge. It proposes Union-of-Capabilities, in which a frozen generalist is augmented only with capability modules whose benefit is certified, and C2R, which implements this through offline expert certification and a training-free routing cascade. The paper reports that learned routers are nearly task-agnostic. On CoIN and UCIT, C2R achieves mean accuracies of 71.06% and 77.74%, respectively.
Critical strengths
-
Strong empirical motivation for replacing task-level MoE specialization. Organizing modules around capabilities rather than strictly around task boundaries is a reasonble problem that remains underexplored in MCIT.
-
Equation (1) in the method admits an expert based on recall-weighted gain minus leak-weighted damage, explicitly accounting for both correctly routed queries and harmful cross-group routing. The routing mechanism is explicit and reproducible: Equation (2) defines a fixed cascade consisting of format, duplication, and margin.
Critical weaknesses
-
Reliability and theoretical justification of certificates: The central design relies on certificates to determine both whether an expert should be installed and whether an inference query should be routed to it. However, the proposed criteria are primarily empirical, and the paper provides limited justification for their reliability beyond the evaluated settings; indeed, the paper explicitly notes that a certificate is not a guarantee. The authors should clarify the statistical interpretation of these certificates and provide stronger analyses of their sensitivity, robustness, and reliability under limited certification data or distribution shifts.
-
Once trained, capability experts are frozen and are not updated by subsequent tasks, while the generalist can be retrained during refreshes. This design prevents parameter overwriting, but it remains unclear how the system handles new tasks that are similar to old tasks and require modifying capabilities already encoded in existing experts. Can the framework reuse or adapt an existing expert when its capability is relevant but insufficient for a new task, and how is its effectiveness on the new task validated without compromising previously acquired capabilities?
Questions for authors
-
The identification of capability gaps relies on a heuristic procedure based on the generalist’s performance across groups. Although this procedure provides a practical way to select candidate experts, the paper does not establish when or why the criterion reliably identifies capabilities that require dedicated modules. It would strengthen the method to provide a more principled justification with quantitative analysis and evaluate the sensitivity of expert selection.
-
All experiments use the same LLaVA-1.5-7B backbone. Since a new backbone or stream requires a fresh refresh and may produce different certified gaps.
-
Some of the strongest routing behavior relies on benchmark-specific regularities. On CoIN, all groups are format-defined, and the deployed patterns include phrases such as “bounding box coordinate,” “Reference OCR token,” and “book”; the manuscript notes that these regularities may not exist in deployment. Moreover, it is unclear how the system handles capabilities that are absent from the stored training and certification data, or queries whose underlying capabilities are not captured by the predefined routing tests. The authors should evaluate generalization to unseen capabilities and more substantial distribution shifts, and clarify how the system detects cases where neither an installed expert nor the generalist is sufficiently reliable.
-
C2R uses stored task data for generalist training, candidate-expert training, and certification, making it difficult to disentangle the benefit of its certification mechanism from the benefit of additional data exposure and retraining. The current experiments do not fully establish whether the observed gains arise primarily from capability-based expert selection and certified routing, or from the broader training pipeline and data usage.
-
The motivation of the paper builds upon current MCIT is not equal to capability continual learning. However, MLLM-CL[1] propose a benchmark for domain and ability continual learning, which can partly reflect the problem and the performance of existibg methods. What would the performance be like if the continual learning procedure continually learns ability?
-
The resource-efficiency results show that C2R improves training and inference efficiency relative to selected baselines, including SAME and a per-task oracle. However, the comparison does not establish whether these gains hold against a broader range of competitive MCIT methods.
-
The results in Table 14 is too impressive. Although freezing installed experts prevents parameter-level overwriting, it does not guarantee that previously acquired capabilities remain accessible at inference time, since this depends on reliable routing. Does the zero-forgetting only work on CoIN? What is the reason behind this? More analysis and experiments should be established to validate this.
[1] MLLM-CL: Continual Learning for Multimodal Large Language Models.