- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies a potentially important benchmark-design problem and provides a useful empirical observation about LoRA fusion. Their central conclusion is that existing benchmark designs may be inherently biased towards specialization, not cross-task generalization.
Main contributions are:
- Simplified expert composition: The authors revisit HiDe-LLaVA's LoRA expert fusion and demonstrate that eliminating cross-expert interaction terms substantially improves performance. Their hard-routing variant increases by +5.74 pts on UCIT and +11.13 pts on CoIN from their reproduced HiDe-LLaVA baseline.
- Task separability diagnosis: They examine expert routing and report perfect task identification on UCIT using 100 sampled test instances per task, suggesting that task-specific distributional cues make expert selection relatively easy.
- Entropy-controlled task mixing: They introduce a framework that interpolates between the original sequential task distributions and uniform mixtures of all tasks, using normalized mixture entropy . Their experiments show that hard expert routing is preferable at low entropy, whereas soft expert aggregation becomes preferable at higher entropy.
Critical strengths
S1. Relevant and potentially consequential research question
The paper goes beyond proposing another continual learning architecture and instead examines what current benchmarks actually measure. This is a worthwhile contribution because current methods can achieve strong scores without necessarily demonstrating positive knowledge transfer. Distinguishing successful task retrieval from genuine cross-task generalization is important for the field.
S2. Useful empirical and diagnostic evidence
The simple fix in Table 1-2 and the routing analysis in Figure 3 and Appendix B provide credible evidence that the original expert fusion/readout mechanism may be suboptimal and that benchmark tasks are distinguishable to varying degrees.
Critical weaknesses
W1. The proposed entropy framework controls stage-wise data mixing rather than intrinsic task similarity.
The proposed normalized entropy framework does not explicitly characterize semantic relationships or knowledge overlap between tasks. Instead, it controls how samples from different original tasks are redistributed across sequential training stages. Although the resulting mixtures preserve a stage-specific dominant task when (h<1), increasing entropy essentially makes the training distributions of different stages more similar through data redistribution.
Importantly, this construction does not alter or measure the intrinsic relationships between the original tasks. Two semantically unrelated tasks and two highly related tasks can yield identical mixture entropy under the same sampling proportions. Consequently, the framework measures stage-wise dataset mixture entropy rather than task similarity or transferable knowledge.
W2. The diagnosis of benchmark limitations and cross-task knowledge sharing is insufficiently supported.
The paper argues that existing MLLM continual-learning benchmarks reward task identification and expert specialization rather than genuine knowledge sharing, but the experiments do not establish this causal claim.
High routing accuracy shows that task identities are distinguishable---not that methods exploit superficial shortcuts or that the benchmarks are fundamentally flawed. Likewise, removing cross-expert terms may improve performance because products of independently trained LoRA factors cause interference, regardless of task similarity. The paper does not isolate this architectural effect from benchmark separability. Moreover, the hard-to-soft routing crossover as entropy increases does not demonstrate cross-task knowledge transfer. It could reflect ensembling, robustness to routing errors, or greater similarity between experts’ training distributions.
Overall, the results show correlations among data mixing, routing, and performance, but insufficiently support the claim that current benchmarks systematically underestimate cross-task knowledge sharing.
Questions for authors
- Task similarity versus data mixing: How does the proposed entropy parameter capture intrinsic task similarity rather than merely the proportions of datasets mixed within each training stage? Can the authors compare semantically related and unrelated task pairs under identical entropy levels? Can the authors replace distribution illustrations in Fig. 1 with actual embeddings of the datasets used in the experiments?
- Comparison to data mixing, selection, or weighting strategies: Current baselines seem unfair due to future-task and seen-task exposure (much more like multi-task training than continual learning). Can the authors compare at least one data-based continual learning method?
- More router diagnosis ablation: Fig. 3 established that the learned router can often identify the benchmark dataset associated with an input. They do not, by themselves, establish that a model exploits an unintended shortcut. Can authors provide more ablations (e.g., text-only router, image-only router, or instruction-template-only router) and experiments at a larger scale (e.g., 1k examples per task)?
- Evidence of genuine knowledge transfer: The paper frequently uses task overlap, task similarity, and knowledge sharing almost interchangeably. These notions are not equivalent. Can the authors include stronger controls to demonstrate that soft routing benefits from complementary task knowledge rather than ensemble effects?
- CL dynamics metrics: The paper only quantifies Last Average, which fails to capture the dynamics of intermediate training stages. Can the authors report: average forgetting across learning stages, backward transfer, forward transfer, intermediate-stage accuracy, expert utilization and interference over time?
Response to Review 89eJ
W1 / Q1 (entropy ≠ task similarity)
We accept: normalized entropy characterizes the stage-wise data-mixing ratio, not the intrinsic semantic relations between tasks; entropy is only a data-construction tool, not a similarity measure. To address this, we measure the concentration coefficient (the similarity between stage anchors after training) which is defined as:
ρ=T(T−1)2∑1≤i<j≤T2cos(aiimg,ajimg)+cos(aitext,ajtext).
We find that it grows monotonically with h:
This quantity captures the routing/representation convergence caused by task overlap, and can serve as a more faithful x-axis than the entropy h.
W2 / Q4 / Q5 (insufficient evidence for knowledge sharing, only Last Average reported)
We agree with the reviewer: the hard→soft crossover alone cannot prove knowledge transfer, which may just be ensembling or multi-task fine-tuning. We therefore shift the perspective to per-stage transfer metrics in the CL sense, instead of Last Average alone.
Specifically, we added per-stage evaluation on the UCIT benchmark: using each intermediate stage's (Task1–5) model to infer on each task's test set, we obtain the complete a[i][j] matrix (a[i][j] denotes the accuracy on task j after learning stage i), from which we report forgetting (BWT), forward transfer (FWT), and intermediate-stage accuracy.
Taking the a[i][j] matrix at h=0 (UCIT, soft readout) as an example:
Based on the accurcy matrix above, the corresponding FWT and BWT are calculated as 31.46 and -7.27 respectively. The full result table of FWT and BWT with different h is as follows. We find that as h rises, FWT increases monotonically (55.6 → 70.6) and BWT turns from negative to positive (−2.3 → +1.6). FWT measures "the forward transfer that preceding stages have already granted task j before it is learned", which is a quantification of knowledge transfer rather than ensembling or multi-task fine-tuning.
Both readouts (soft and hard) show FWT increasing with h, indicating that knowledge transfer occurs on the training side (data side) rather than being an artifact of the readout.
Q3 (router diagnosis ablation)
We provide text-only / image-only / template-only router ablations (3000 samples/task each). At the h=0 endpoint, the original routing accuracy is 99.7%; using only the image signal it is 93.7%, using only the question text 83.0%, and using only the fixed instruction template it drops to 33.3%. This shows the router relies mainly on image and question-semantic content to recognize tasks, rather than on the fixed template — the template does not constitute an easily exploitable shortcut.