acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Continual Learning for MLLMs: A Simple Approach Reveals Hidden Limitations in Existing Benchmarks

Abstract

Multimodal large language models (MLLMs) have emerged as a promising foundation for AI systems, and continual learning has attracted increasing research attention to enable MLLMs to adapt to evolving environments. Despite the proliferation of related work, a critical question remains: do current benchmarks adequately capture the task relationships needed for effective knowledge sharing in MLLM continual learning? In this paper, we take an initial step toward answering this question. First, we show that a small modification to one existing MoE-LoRA method-removing cross terms from its fused LoRA readout-is sufficient to substantially outperform state-of-the-art methods. To understand this unexpected improvement, we analyze the routing behavior and find that task identity can be readily inferred from the input, allowing models to exploit task-level shortcuts and revealing a potential benchmark-hacking issue. To systematically examine this issue, we introduce an entropy-parameterized framework that recasts existing benchmarks into variants with controllable task overlap. As task overlap increases, performance can improve even as task identity becomes harder to infer, while cross-task knowledge sharing becomes increasingly beneficial. These findings suggest that existing benchmarks may overstate the benefit of task specialization while underestimating cross-task knowledge transfer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal large language models, continual learning, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_89eJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a potentially important benchmark-design problem and provides a useful empirical observation about LoRA fusion. Their central conclusion is that existing benchmark designs may be inherently biased towards specialization, not cross-task generalization.

Main contributions are:

  1. Simplified expert composition: The authors revisit HiDe-LLaVA's LoRA expert fusion and demonstrate that eliminating cross-expert interaction terms substantially improves performance. Their hard-routing variant increases by +5.74 pts on UCIT and +11.13 pts on CoIN from their reproduced HiDe-LLaVA baseline.
  2. Task separability diagnosis: They examine expert routing and report perfect task identification on UCIT using 100 sampled test instances per task, suggesting that task-specific distributional cues make expert selection relatively easy.
  3. Entropy-controlled task mixing: They introduce a framework that interpolates between the original sequential task distributions and uniform mixtures of all tasks, using normalized mixture entropy h∈[0,1]h\in[0,1]. Their experiments show that hard expert routing is preferable at low entropy, whereas soft expert aggregation becomes preferable at higher entropy.
Critical strengths

S1. Relevant and potentially consequential research question

The paper goes beyond proposing another continual learning architecture and instead examines what current benchmarks actually measure. This is a worthwhile contribution because current methods can achieve strong scores without necessarily demonstrating positive knowledge transfer. Distinguishing successful task retrieval from genuine cross-task generalization is important for the field.

S2. Useful empirical and diagnostic evidence

The simple fix in Table 1-2 and the routing analysis in Figure 3 and Appendix B provide credible evidence that the original expert fusion/readout mechanism may be suboptimal and that benchmark tasks are distinguishable to varying degrees.

Critical weaknesses

W1. The proposed entropy framework controls stage-wise data mixing rather than intrinsic task similarity.

The proposed normalized entropy framework does not explicitly characterize semantic relationships or knowledge overlap between tasks. Instead, it controls how samples from different original tasks are redistributed across sequential training stages. Although the resulting mixtures preserve a stage-specific dominant task when (h<1), increasing entropy essentially makes the training distributions of different stages more similar through data redistribution.

Importantly, this construction does not alter or measure the intrinsic relationships between the original tasks. Two semantically unrelated tasks and two highly related tasks can yield identical mixture entropy under the same sampling proportions. Consequently, the framework measures stage-wise dataset mixture entropy rather than task similarity or transferable knowledge.

W2. The diagnosis of benchmark limitations and cross-task knowledge sharing is insufficiently supported.

The paper argues that existing MLLM continual-learning benchmarks reward task identification and expert specialization rather than genuine knowledge sharing, but the experiments do not establish this causal claim.

High routing accuracy shows that task identities are distinguishable---not that methods exploit superficial shortcuts or that the benchmarks are fundamentally flawed. Likewise, removing cross-expert terms may improve performance because products of independently trained LoRA factors cause interference, regardless of task similarity. The paper does not isolate this architectural effect from benchmark separability. Moreover, the hard-to-soft routing crossover as entropy increases does not demonstrate cross-task knowledge transfer. It could reflect ensembling, robustness to routing errors, or greater similarity between experts’ training distributions.

Overall, the results show correlations among data mixing, routing, and performance, but insufficiently support the claim that current benchmarks systematically underestimate cross-task knowledge sharing.

Questions for authors
  1. Task similarity versus data mixing: How does the proposed entropy parameter capture intrinsic task similarity rather than merely the proportions of datasets mixed within each training stage? Can the authors compare semantically related and unrelated task pairs under identical entropy levels? Can the authors replace distribution illustrations in Fig. 1 with actual embeddings of the datasets used in the experiments?
  2. Comparison to data mixing, selection, or weighting strategies: Current baselines seem unfair due to future-task and seen-task exposure (much more like multi-task training than continual learning). Can the authors compare at least one data-based continual learning method?
  3. More router diagnosis ablation: Fig. 3 established that the learned router can often identify the benchmark dataset associated with an input. They do not, by themselves, establish that a model exploits an unintended shortcut. Can authors provide more ablations (e.g., text-only router, image-only router, or instruction-template-only router) and experiments at a larger scale (e.g., 1k examples per task)?
  4. Evidence of genuine knowledge transfer: The paper frequently uses task overlap, task similarity, and knowledge sharing almost interchangeably. These notions are not equivalent. Can the authors include stronger controls to demonstrate that soft routing benefits from complementary task knowledge rather than ensemble effects?
  5. CL dynamics metrics: The paper only quantifies Last Average, which fails to capture the dynamics of intermediate training stages. Can the authors report: average forgetting across learning stages, backward transfer, forward transfer, intermediate-stage accuracy, expert utilization and interference over time?
1 comment
AuthorsOpenReview ↗

Response to Review 89eJ

W1 / Q1 (entropy ≠ task similarity)

We accept: normalized entropy characterizes the stage-wise data-mixing ratio, not the intrinsic semantic relations between tasks; entropy is only a data-construction tool, not a similarity measure. To address this, we measure the concentration coefficient (the similarity between stage anchors after training) which is defined as:

ρ=2T(T−1)∑1≤i<j≤Tcos⁡(aiimg,ajimg)+cos⁡(aitext,ajtext)2.\rho=\frac{2}{T(T-1)}\sum_{1\le i<j\le T}\frac{\cos(a_i^{img},a_j^{img})+\cos(a_i^{text},a_j^{text})}{2}.

We find that it grows monotonically with h:

hUCIT simCoIN sim
00.5680.663
0.250.6060.715
0.50.7080.776
0.750.8230.865
1.00.9960.954

This quantity captures the routing/representation convergence caused by task overlap, and can serve as a more faithful x-axis than the entropy h.

W2 / Q4 / Q5 (insufficient evidence for knowledge sharing, only Last Average reported)

We agree with the reviewer: the hard→soft crossover alone cannot prove knowledge transfer, which may just be ensembling or multi-task fine-tuning. We therefore shift the perspective to per-stage transfer metrics in the CL sense, instead of Last Average alone.

Specifically, we added per-stage evaluation on the UCIT benchmark: using each intermediate stage's (Task1–5) model to infer on each task's test set, we obtain the complete a[i][j] matrix (a[i][j] denotes the accuracy on task j after learning stage i), from which we report forgetting (BWT), forward transfer (FWT), and intermediate-stage accuracy.

Taking the a[i][j] matrix at h=0 (UCIT, soft readout) as an example:

a[i][j]ImageNet-RArxivQAVizWizIconQACLEVR-MathFlickr30kLast Avg
stage192.0752.8325.0520.1319.1732.3540.27
stage291.5792.2717.9724.5013.7727.8844.66
stage389.8092.3359.3224.5020.5050.8056.21
stage488.0393.1353.4484.4019.7749.4164.70
stage585.6092.6048.4973.6778.7742.2570.23
stage686.2393.0347.8773.4769.8757.7571.37

Based on the accurcy matrix above, the corresponding FWT and BWT are calculated as 31.46 and -7.27 respectively. The full result table of FWT and BWT with different h is as follows. We find that as h rises, FWT increases monotonically (55.6 → 70.6) and BWT turns from negative to positive (−2.3 → +1.6). FWT measures "the forward transfer that preceding stages have already granted task j before it is learned", which is a quantification of knowledge transfer rather than ensembling or multi-task fine-tuning.

hFWT(soft)BWT(soft)FWT(hard)BWT(hard)
031.46−7.2730.82−5.31
0.2555.62−2.3155.32−3.05
0.563.08−2.3362.65−3.12
0.7566.71−1.3266.41−2.73
1.070.63+1.5568.54+0.72

Both readouts (soft and hard) show FWT increasing with h, indicating that knowledge transfer occurs on the training side (data side) rather than being an artifact of the readout.

Q3 (router diagnosis ablation)

We provide text-only / image-only / template-only router ablations (3000 samples/task each). At the h=0 endpoint, the original routing accuracy is 99.7%; using only the image signal it is 93.7%, using only the question text 83.0%, and using only the fixed instruction template it drops to 33.3%. This shows the router relies mainly on image and question-semantic content to recognize tasks, rather than on the fixed template — the template does not constitute an easily exploitable shortcut.

Routing signalcombinedimage-onlytext-onlytemplate-only
Routing accuracy99.7%93.7%83.0%33.3%
Reviewer_WDudOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper discusses continual instruction tuning for multimodal large language models and focuses on expert composition and the limitations of existing benchmarks. They show that removing cross-expert terms from HiDe-LLaVA's LoRA parameter fusion improves performance, with further gains from hard expert selection. They introduce an entropy-parameterized framework that progressively mixes data from different tasks and analyze the performance of hard versus soft expert routing on UCIT and CoIN.

Critical strengths
  • The question raised is relevant, on whether existing multimodal continual-learning benchmarks evaluate cross-task knowledge sharing rather than primarily task identification.
  • They construct an entropy-parameterized framework that mixes data from different tasks, observing that soft expert fusion eventually outperforms hard selection.
Critical weaknesses
  • The proposed task-overlap setting changes the continual-learning setting itself. Mixing data from different tasks into each training stage, includes samples from future tasks, changes when task information becomes available and the degree of non-stationarity. At maximum entropy, every stage has the same task-source distribution, making the setting closer to multitask learning.

  • The paper argues that increasing task overlap makes task identification harder, favoring soft expert combination. However, Figure 3 shows that soft routing starts outperforming hard routing at (h=0.25), while task-identification accuracy on UCIT remains 100%. This suggests that the gains may arise from training experts on overlapping distributions rather than reduced task separability.

  • Need more clarity on lora cross terms. the original HiDe-LLaVA uses factor-level merging in its remaining layers, whereas the proposed Soft variant introduces weighted expert aggregation across all layers.

  • Analysis is limited to one existing method, Hide-llava.

Questions for authors
  • Could the improvements come from increasingly multitask-like training than cross-task knowledge transfer?

  • Why does soft routing outperform hard routing at (h=0.25) on UCIT when task-identification accuracy remains 100%?

  • Does it generalize to other MoE-LoRA methods, parameter-efficient continual-learning approaches, or replay-based methods

  • The reproduced HiDe-LLaVA accuracy on CoIN (57.35%) is considerably below the originally reported 63.95%. ?

  • It would help to get more clarify the on broader contribution and impact of this work

2 comments
AuthorsOpenReview ↗

Response to Reviewer WDud (1/2)

W2 / Q1 / Q2 (soft>hard while routing remains 100%)

We agree with the reviewer. Here we disentangle three easily-confused concepts: routing accuracy / task similarity / knowledge sharing. The variable we manipulate is task overlap; what we hope to change is the similarity between tasks. Routing accuracy cannot reflect our need — even if the routing accuracy is the same, the smoothness of the weighting coefficients differs, and what we focus on is precisely this smoothness (concentration) of the weighting coefficients. Routing accuracy 100% only shows "task identity is still distinguishable", not "the soft>hard gain comes from reduced separability".

The key evidence is the τ-intervention: fixing the final model at each h, sweeping the routing weighting coefficients from sharp (τ→0, i.e., hard) to smooth (τ→10, near-uniform), and observing the accuracy change. At the h=0 endpoint, accuracy monotonically decreases by 16 points (73.51 → 57.26) — the endpoint's high score is built entirely on "task-identity recognition + expert specialization", and any forced cross-expert integration is harmful. For h>0, the curve reverses to rise-then-stabilize (soft > hard), and the soft−hard gain increases monotonically with h (−2.14 → +2.23), showing that with more task overlap, smooth routing borrows more cross-expert knowledge.

hhard(τ→0)0.020.050.1(soft)0.21.0510
073.5173.4372.6671.3768.7359.7557.5557.26
0.2575.8075.8575.9976.0875.6374.6274.4674.41
0.575.5375.4575.5275.8475.9475.6275.5575.52
0.7575.3775.5876.0776.1775.9775.7775.7975.76
1.073.4475.4075.5875.6775.7075.7475.7775.76

W3 (equal-weight fusion vs. weighted fusion in lower layers)

To further isolate the effect of cross-term removal, we conduct an additional ablation study. From the following table, it can be seen that removing the cross terms consistently improves performance on both UCIT (67.77 → 68.77) and CoIN (57.35 → 59.18), motivating our further investigation into expert combination strategies. Building upon this cross-term-free variant, we replace uniform expert fusion with input-dependent routing, considering both softmax-weighted combination (Soft) and top-1 expert selection (Hard). Both strategies yield substantial improvements over uniform fusion, with Hard consistently outperforming Soft on the original benchmarks. These results suggest that the highly disjoint task distributions may favor expert specialization, whereas uniform fusion can introduce interference from less relevant experts.

DatasetHiDeHiDe (w/o cross terms)HiDe (Soft)HiDe (Hard)
UCIT67.7768.7771.3773.51
CoIN57.3559.1864.0768.48

W4 / Q3 (only HiDe analyzed)

We re-trained and evaluated MoELoRA and DISCO, two independent MoE-LoRA methods, under the task' setting; under this setting, the results can be directly compared.

SettingHiDe (Soft)HiDe (Hard)MoELoRADISCO
Original UCIT (h=0)71.3773.5152.0669.66
task' (h=0.25)76.0875.8072.9270.27
task' (h=0.5)75.8475.5373.5373.37
task' (h=0.75)76.1775.3773.7374.47

Notably, MoELoRA achieves only 52.06% under the original UCIT setting ((h=0)), based on the result reported in CoIN. This relatively low baseline may partly reflect implementation differences, and should therefore be interpreted with caution. More importantly, the relative ranking of MoELoRA and DISCO changes across task-overlap settings: while DISCO substantially outperforms MoELoRA at (h=0), MoELoRA surpasses DISCO at (h=0.25), and their performance becomes comparable at higher overlap levels. This observation further highlights that evaluations restricted to disjoint task sequences may provide an incomplete picture of method effectiveness, motivating a broader evaluation across varying degrees of task overlap.

AuthorsOpenReview ↗

Response to Reviewer WDud (2/2)

Q4 (CoIN reproduction 57.35 vs. reported 63.95)

Our reproduced UCIT values are even higher than the reported ones (HiDe-LLaVA 67.77% vs. reported 64.19%); the core claim is "the final accuracy exceeds both the reported and the reproduced values". Regarding the lower CoIN reproduction: this is not a difficulty unique to us — in a public issue of the original repository (Ghy0501/HiDe-LLaVA#8), other practitioners also reported that on CoIN the accuracy collapses to below 10% after incremental stages and that the reported values could not be reached, with no explanation from the authors so far. We conjecture this more likely stems from the original repository's sensitivity to training environment/hardware (e.g., GPU model, mixed-precision implementation) rather than from a defect of the method itself on CoIN. Our CoIN reproduction (57.35%) is below the reported value, but far from the collapse described in the issue.

Q5 (broader contribution)

We have further clarified the broader contributions and potential impact of our work from two perspectives:

1. Towards more realistic continual instruction tuning. Existing CIT benchmarks typically assume disjoint task distributions, whereas real-world data streams may exhibit varying degrees of task overlap. This motivation is consistent with the broader development of continual learning benchmarks. For example, Three Scenarios for Continual Learning [1] established a systematic framework for distinguishing continual learning scenarios, while subsequent studies have relaxed idealized task assumptions through blurry [2], i-Blurry [3], and Si-Blurry [4] settings in the realm of Class Incremental Learning (CIL). The evolution of CIL benchmarks, from disjoint to blurry, i-Blurry, and Si-Blurry settings, illustrates how evaluation protocols can progressively relax idealized assumptions. Following a similar direction, our generalized CIT framework takes an initial step towards more realistic MLLM-CL evaluation by introducing controllable task overlap, while providing a foundation for further extensions.

2. From single-point to spectrum-based evaluation. Beyond introducing task overlap, our framework extends evaluation from a single fixed configuration to a spectrum of overlap levels. Importantly, we observe ranking shifts among competing methods as task overlap varies, suggesting that conclusions drawn from conventional disjoint benchmarks may not generalize across different task distributions. This enables a more comprehensive assessment of continual learning methods and provides a foundation for developing more realistic CIT benchmarks.

We will incorporate these discussions into the Discussion section of the revised manuscript to further clarify the broader contributions and potential implications of our work.

[1] van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. "Three types of incremental learning." Nature Machine Intelligence, 4, 1185–1197, 2022.

[2] Bang, J., Kim, H., Yoo, Y., Ha, J. W., and Choi, J. "Rainbow Memory: Continual Learning with a Memory of Diverse Samples." CVPR, 2021.

[3] Koh, H., Kim, D., Ha, J. W., and Choi, J. "Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference." ICLR, 2022.

[4] Moon, J. Y., Park, K. H., Kim, J. U., and Park, G. M. "Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning." ICCV, 2023.

Reviewer_txN7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits continual instruction tuning for multimodal large language models (MLLMs). The authors find that removing cross-expert interactions from fused LoRA readouts improves performance on existing benchmarks. They argue that current benchmarks allow task identity to be inferred from inputs, favoring expert specialization over knowledge sharing. To investigate this, they introduce an entropy-parameterized task-mixture framework and compare hard and soft expert routing across different overlap levels.

Critical strengths
  • The paper raises an important evaluation issue: existing continual-learning benchmarks may favor task-specific expert selection when task identity is readily recoverable from inputs.

  • The entropy-parameterized framework provides a simple, interpretable way to study how expert specialization and knowledge sharing change as task mixtures become more diverse.

Critical weaknesses
  • Limited novelty and insufficient discussion of related work. The cross-expert interaction analysis is closely related to prior studies on LoRA merging and interference [1, 2]. In particular, [1] explicitly analyzes the cross-terms introduced by factor-level merging and compares alternative merging formulations. The paper should better distinguish its technical contribution from these existing studies and clarify what new insights arise specifically from the MLLM continual-learning setting. These works should also be discussed in the Related Work section.

  • The benchmark reformulation mixes data from multiple tasks into each training stage, including tasks that would otherwise occur later in the sequence. At maximum entropy, all stages use the same task mixture, making the setting closer to repeated multi-task training than conventional continual learning. This substantially complicates the interpretation of the results as evidence about continual learning.

[1] LoRA Merging with SVD: Understanding Interference and Preserving Performance. ICML 2025.

[2] Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging. 2025.

Questions for authors
  • Given prior work on cross-term interference in LoRA merging, can the authors clarify what is technically novel in their cross-term-free readout and demonstrate benefits specific to continual instruction tuning beyond applying an existing merging principle to new benchmarks?

  • How do the authors justify mixing data from future tasks into earlier training stages, and how should the (h=1) setting be interpreted as continual learning rather than multi-task training?

  • Can the authors validate the conclusions under a protocol that preserves sequential data availability and evaluate additional task orders or continual-learning methods?

1 comment
AuthorsOpenReview ↗

Response to Reviewer txN7

W1 / Q1 (limited novelty; cross terms already in literature)

We cite the merging literature (LoRA Merging with SVD [ICML 2025 Workshop], DO-Merging [EMNLP 2026 Findings]) in Related Work, and clarify the key differences in their learning settings. Existing merging studies typically assume independently trained experts, where each expert is trained separately without dependencies on others in a non-incremental setting. In contrast, our work considers an incremental learning setting, where experts are trained sequentially as new tasks arrive, potentially involving continued fine-tuning from previously learned task weights.

Although the underlying mechanisms share certain similarities, the incremental learning setting introduces different considerations, particularly regarding dependencies between sequentially learned experts. To the best of our knowledge, such cross terms have not been systematically investigated in this setting.

W2 / Q2 (mixing future-task data, deviating from CL)

We regard the mixed dataset at each hh as a distinct new task (task′task'), yielding a task sequence task1′,task2′,…task'_1, task'_2, \ldots, where each stage has a well-defined dominant task distribution and the overall sequence follows the continual learning setting.

This formulation better reflects real-world data streams, where tasks are unlikely to appear in strictly disjoint stages (e.g., one stage containing only CLEVR and the next only ArxivQA). Instead, different task types may coexist and recur over time. Our preliminary analysis of VisionArena [1], a dataset of real-world user-VLM interactions, supports this motivation: among 3,942 analyzed interactions, we observe diverse task categories and recurring task types across different dates, rather than strictly isolated task occurrences.

This formulation is also consistent with the generalized incremental learning framework discussed by van de Ven et al. [2], which uses a context space to characterize non-stationarity in the data distribution. Our entropy-parameterized task mixing can be viewed as a concrete instantiation of this broader perspective. Accordingly, task mixing defines the composition of each new task, rather than merely exposing future tasks in advance.

[1] Chou, C., Dunlap, L., Mashita, K., Mandal, K., Darrell, T., Stoica, I., Gonzalez, J. E., and Chiang, W.-L. "VisionArena: 230K Real World User-VLM Conversations with Preference Labels." CVPR, 2025.

[2] van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. "Three types of incremental learning." Nature Machine Intelligence, 4(12), 1185–1197, 2022.

Q3 (more methods / task orders)

We re-trained two independent MoE-LoRA baselines (MoELoRA / DISCO) under the task' setting. Since training is time-consuming, Replay-based methods and more task orders will be added later.

SettingHiDe(Soft)HiDe(Hard)MoELoRADISCO
Original UCIT (h=0)71.3773.5152.0669.66
task' (h=0.25)76.0875.8072.9270.27
task' (h=0.5)75.8475.5373.5373.37
task' (h=0.75)76.1775.3773.7374.47

Notably, MoELoRA achieves only 52.06% under the original UCIT setting ((h=0)), based on the result reported in CoIN. This relatively low baseline may partly reflect implementation differences, and should therefore be interpreted with caution. More importantly, the relative ranking of MoELoRA and DISCO changes across task-overlap settings: while DISCO substantially outperforms MoELoRA at (h=0), MoELoRA surpasses DISCO at (h=0.25), and their performance becomes comparable at higher overlap levels. This observation further highlights that evaluations restricted to disjoint task sequences may provide an incomplete picture of method effectiveness, motivating a broader evaluation across varying degrees of task overlap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.