acceptodds
Withdrawn from ICLR 2027

When Experts Are Not Experts: Multimodal Continual Instruction Tuning with Certified Capabilities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Continual Instruction Tuning (MCIT) aims to let Multimodal Large Language Models (MLLMs) acquire new skills after deployment without forgetting previous ones. Modern MCIT methods build on Mixture-of-Experts (MoE), adding lightweight experts and guarding them against interference. We revisit this design and find that the defenses tax plasticity without securing the expertise they protect. Learned routers barely distinguish tasks even when trained jointly, so the defenses guard a separation that never forms. Meanwhile, the difficulty of the stream concentrates in a few capabilities that a shared model fails to hold. Based on these observations, we introduce Union-of-Capabilities (UoC), a paradigm in which a generalist, retrained on the stored data at each refresh and frozen in between, grows only by certified capability modules. Building on UoC, we propose C2R (Certify to Route), which realizes the paradigm in two stages. (1) At training time, CERTIFY admits an expert only on verified gain over the generalist, and (2) at inference time, ROUTE dispatches a query only on certified evidence through a training-free certificate cascade, without a task label. Every installed module is trained once and then frozen, so what it has learned is never overwritten. On CoIN and UCIT, \ours sets a new state of the art, outperforming the strongest previous method with zero forgetting, fewer trainable parameters, up to 1.7 faster training at a single refresh, and 1.8 faster inference, and it keeps this lead when updated after every task, with forgetting within 0.2 points.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Continual Instruction Tuning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wEugOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multimodal continual instruction tuning, arguing that MoE-based approaches assume task-level expertise that does not reliably emerge. It proposes Union-of-Capabilities, in which a frozen generalist is augmented only with capability modules whose benefit is certified, and C2R, which implements this through offline expert certification and a training-free routing cascade. The paper reports that learned routers are nearly task-agnostic. On CoIN and UCIT, C2R achieves mean accuracies of 71.06% and 77.74%, respectively.

Critical strengths
  1. Strong empirical motivation for replacing task-level MoE specialization. Organizing modules around capabilities rather than strictly around task boundaries is a reasonble problem that remains underexplored in MCIT.

  2. Equation (1) in the method admits an expert based on recall-weighted gain minus leak-weighted damage, explicitly accounting for both correctly routed queries and harmful cross-group routing. The routing mechanism is explicit and reproducible: Equation (2) defines a fixed cascade consisting of format, duplication, and margin.

Critical weaknesses
  1. Reliability and theoretical justification of certificates: The central design relies on certificates to determine both whether an expert should be installed and whether an inference query should be routed to it. However, the proposed criteria are primarily empirical, and the paper provides limited justification for their reliability beyond the evaluated settings; indeed, the paper explicitly notes that a certificate is not a guarantee. The authors should clarify the statistical interpretation of these certificates and provide stronger analyses of their sensitivity, robustness, and reliability under limited certification data or distribution shifts.

  2. Once trained, capability experts are frozen and are not updated by subsequent tasks, while the generalist can be retrained during refreshes. This design prevents parameter overwriting, but it remains unclear how the system handles new tasks that are similar to old tasks and require modifying capabilities already encoded in existing experts. Can the framework reuse or adapt an existing expert when its capability is relevant but insufficient for a new task, and how is its effectiveness on the new task validated without compromising previously acquired capabilities?

Questions for authors
  1. The identification of capability gaps relies on a heuristic procedure based on the generalist’s performance across groups. Although this procedure provides a practical way to select candidate experts, the paper does not establish when or why the criterion reliably identifies capabilities that require dedicated modules. It would strengthen the method to provide a more principled justification with quantitative analysis and evaluate the sensitivity of expert selection.

  2. All experiments use the same LLaVA-1.5-7B backbone. Since a new backbone or stream requires a fresh refresh and may produce different certified gaps.

  3. Some of the strongest routing behavior relies on benchmark-specific regularities. On CoIN, all groups are format-defined, and the deployed patterns include phrases such as “bounding box coordinate,” “Reference OCR token,” and “book”; the manuscript notes that these regularities may not exist in deployment. Moreover, it is unclear how the system handles capabilities that are absent from the stored training and certification data, or queries whose underlying capabilities are not captured by the predefined routing tests. The authors should evaluate generalization to unseen capabilities and more substantial distribution shifts, and clarify how the system detects cases where neither an installed expert nor the generalist is sufficiently reliable.

  4. C2R uses stored task data for generalist training, candidate-expert training, and certification, making it difficult to disentangle the benefit of its certification mechanism from the benefit of additional data exposure and retraining. The current experiments do not fully establish whether the observed gains arise primarily from capability-based expert selection and certified routing, or from the broader training pipeline and data usage.

  5. The motivation of the paper builds upon current MCIT is not equal to capability continual learning. However, MLLM-CL[1] propose a benchmark for domain and ability continual learning, which can partly reflect the problem and the performance of existibg methods. What would the performance be like if the continual learning procedure continually learns ability?

  6. The resource-efficiency results show that C2R improves training and inference efficiency relative to selected baselines, including SAME and a per-task oracle. However, the comparison does not establish whether these gains hold against a broader range of competitive MCIT methods.

  7. The results in Table 14 is too impressive. Although freezing installed experts prevents parameter-level overwriting, it does not guarantee that previously acquired capabilities remain accessible at inference time, since this depends on reliable routing. Does the zero-forgetting only work on CoIN? What is the reason behind this? More analysis and experiments should be established to validate this.

[1] MLLM-CL: Continual Learning for Multimodal Large Language Models.

Reviewer_QYmYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes C2R, a method that helps multimodal models learn new tasks without forgetting previous knowledge. It uses a generalist model and a set of fixed modules, selecting them based on their performance and using a certificate-based method to route queries without task labels. However, the novelty of C2R is not clear, and its differences from existing methods using separate adapters and training-free routing need to be explained better.

Critical strengths
  1. The proposed method achieves good results on CoIN and UCIT, with better accuracy and lower inference costs than the compared methods.
  2. The paper provides a useful analysis of expert routing and forgetting. It also offers a different approach by using measured results to select and route capability modules instead of relying on learned routers.
Critical weaknesses
  1. The introduced method is tested on only two benchmarks, so its ability to work well in other settings is unclear.
  2. This method relies on reliable capability certificates, but their performance under changes in data distribution is unclear.
  3. The efficiency claims need to be tested on larger models and more complex tasks.
  4. The comparisons with existing methods may not fully show how well the introduced method performs in real-world continual learning scenarios.
Questions for authors
  1. How does C2R keep capability certificates reliable when the input data distribution changes over time?
  2. How does the certificate cascade work when a query needs multiple capabilities or when different experts give conflicting results?
  3. Can C2R prevent forgetting when the number of installed modules becomes very large?
  4. I would recommend providing more experiments using larger models and more diverse datasets.
  5. Please compare C2R with existing methods under the same training and inference settings.
  6. Also, please provide more analysis of certificate errors and how they affect routing performance.
Reviewer_vwD4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multimodal continual instruction tuning and proposes Union-of-Capabilities and its implementation, C2R. In a setting where historical task data can be retained, the system maintains a generalist adapter and trains a small number of specialists for its capability gaps. Candidate experts are admitted based on held-out measurements of their gains and misrouting costs, then frozen. At inference, a rule-based dispatcher selects the generalist adapter or one installed expert adapter to answer a query.

Experiments on CoIN and UCIT report improvements in performance and efficiency, with additional evaluations of per-task updates, matched replay budgets, refresh schedules, and prompt paraphrases. The paper also analyzes associations between tasks and routing assignments to question whether existing MoE methods develop the intended task-level specialization.

Critical strengths

1. The method retains practical benefits under a more comparable continual-learning setting.

In Table 21, C2R-seq updated after every task achieves 71.42, compared with 68.16 for SAME under the same replay budget, while reducing training cost from 263.15 to 180.76 GPU-hours. This comparison is more persuasive than the terminal-refresh results alone and suggests that the benefits are not solely due to retraining on historical data at the end of the stream.

2. The modular design has practical value and is supported by several supplementary experiments.

Independent frozen adapters and single-module inference reduce coupling between expert updates and help control inference cost. Considering both specialist gains and misrouting damage is a sensible deployment-oriented admission criterion. The screen-only variant, different refresh schedules, three-seed results, and prompt-paraphrase experiments provide additional support for the overall system.

Critical weaknesses

1. The evidential scope of the zero-forgetting result needs clearer qualification.

The main C2R evaluation uses a single terminal refresh, which differs from the operating regime of sequential continual-learning baselines. Under Appendix B's definition, the initial and final accuracies used to calculate forgetting in this setting are measured using the same final deployed instance. The zero values therefore do not independently demonstrate retention after learning subsequent tasks.

Table 21 partly addresses comparability through per-task updates and matched replay budgets. Section 3.4 and Appendix F also explain that freezing experts does not guarantee unchanged system-level performance. Retention claims should therefore rely primarily on the C2R-seq results. Since average forgetting can mask individual task declines, a complete task-accuracy matrix across updates and the corresponding forgetting results would strengthen this evidence.

2. The central interpretations of specialization and capability organization still lack sufficiently direct evidence.

Low task–expert mutual information at individual gates does not rule out useful specialization through soft routing or cross-layer combinations. Likewise, gradient suppression alone does not establish that protection harms net learning performance. Most capability groups still coincide with tasks, while the reading-text grouping is explicitly constructed by the authors and then validated, leaving limited support for the proposed capability organization.

Table 6 already compares routing methods over the same expert library, but the proposed method and the visual NCM baseline use different input information on CoIN. Comparing simpler text-based routing methods with the expert library fixed and the available input information matched, while reporting final task accuracy, would help assess the added benefit of the routing design. Comparing simpler held-out admission rules over the same candidate pool would help assess the independent contribution of certification.

Questions for authors

1. How are retention and candidate selection validated under per-task updates?

Please provide the complete task-accuracy matrix across all eight per-task updates of C2R-seq, together with the corresponding forgetting results. Please also clarify whether candidate selection, allocation of training data to the generalist adapter, and construction of shared modules depend only on information available at each update. If some candidate groups are specified in advance following analysis of the complete task stream, please distinguish these settings.

2. What evidence more directly supports the specialization analysis and the independent contributions of certification and routing?

For the MoE models analyzed in the paper, expert masking, swapping, or routing-replacement controls would help distinguish weak task-aligned top-1 routing from an absence of useful specialization. For C2R, extending Table 6 with text-based routing controls that use matched input information, and comparing simpler admission rules over the same candidate pool, would help assess the added mechanisms' contributions to final task performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.