- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper introduces Streaming Continual Visual Instruction Tuning (StrCVIT), where multimodal models learn from a sequence of data chunks containing changing mixtures of tasks without explicit task identities. To address interference and forgetting in this setting, the authors propose StrLoRA, which combines instruction-guided expert selection, token-wise expert weighting, and an EMA-based routing regularizer. Experiments across multiple model backbones and task distributions show promising improvements in average performance and reduced forgetting compared with existing continual learning baselines.
Critical strengths
Strengths
- The paper is generally well written and clearly organized. The illustrations and formulation make the proposed setting and the three components of the method easy to follow.
- The experimental results are promising, particularly in reducing forgetting. Evaluation across multiple backbones, stream lengths, and task compositions provides reasonably broad empirical coverage.
- The combination of architectural analysis and data-stream analysis is useful. The routing visualizations, component ablations, and experiments with different stream compositions help explain how the design interacts with heterogeneous, evolving data.
Critical weaknesses
Weaknesses
- The characterization of prior work appears overly broad. The claim that existing CVIT methods assume a single task per training stage is supported by several representative benchmarks, but the paper does not sufficiently establish that this characterization applies to the broader literature. A clearer distinction between task-incremental CVIT, mixed-task instruction tuning, and task-free continual learning would strengthen the positioning and clarify the scope of the proposed setting’s novelty.
- The performance–latency trade-off needs stronger justification. Table 14 reports an increase in inference time from 7.62 seconds per sample for MoELoRA to 9.83 seconds for StrLoRA, approximately 29%. Although the reduction in forgetting is encouraging, this is a substantial inference overhead. It remains unclear whether the gains justify this cost, particularly relative to simpler alternatives under comparable latency or resource budgets.
- The evidence for more demanding reasoning tasks is limited. The evaluation primarily covers recognition, grounding, captioning, and established visual question-answering tasks. Although some datasets involve reasoning, the experiments do not establish whether the approach remains effective for complex, multi-step multimodal reasoning. Such tasks may also be less easily separated by instruction-level cues, making the generality of the routing strategy uncertain.
Questions for authors
How does StrLoRA compare with joint training on the same mixed-task data under a comparable training budget? This comparison would help clarify the benefits and costs of continual adaptation relative to learning these tasks together from the outset.
7 comments
Response to Reviewer wb89 (Part II)
Response to Question 1
Different learning settings. Joint training and continual adaptation operate under fundamentally different learning settings. In joint training, data from all tasks are available from the outset and pooled for training. In contrast, continual adaptation requires the model to continually improve its capabilities as new data arrive while retaining previously learned knowledge, without access to past or future data.
Comparison with joint training. With similar numbers of trainable parameters, we compare the final performance of StrLoRA after continual adaptation with that of MoELoRA jointly trained on the pooled and reshuffled data from all 25 mixed-task chunks. StrLoRA still achieves a competitive average accuracy of 65.09, compared with 65.17 for jointly trained MoELoRA.
| Method | Trainable Parameters | Places365 | RS | AD | VQAv2 | Fin | Grounding | OCRVQA | TextCaps | ImageNet | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MoELoRA, joint training | 258.43M | 38.30 | 70.10 | 36.20 | 70.90 | 77.40 | 60.60 | 57.60 | 87.66 | 87.80 | 65.17 |
| StrLoRA, continual adaptation | 250.04M | 34.00 | 73.20 | 37.40 | 71.20 | 80.10 | 61.80 | 53.70 | 86.05 | 88.40 | 65.09 |
Look forward to further discussions
Dear reviewer,
We sincerely appreciate your time and effort in reviewing our paper and providing valuable feedback. We have carefully considered all the questions and concerns raised in your review and provided detailed responses. We would greatly appreciate it if you could take a look at our responses and let us know whether they effectively address your concerns or if there are any additional questions or points you would like to discuss.
Thank you again for your time and consideration.
Best regards,
Authors
Response to Reviewer wb89 (Part I)
We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We address your questions point by point below.
Response to Weakness 1
We clarify the distinctions between StrCVIT and related settings as follows:
Mixed-task instruction tuning. In joint mixed-task instruction tuning, the model is trained in a single phase on a pooled dataset containing mixed tasks. StrCVIT instead studies learning from a continuously arriving data stream, requiring the model to simultaneously acquire new abilities from emerging tasks, reinforce recurring abilities, and mitigate forgetting of absent abilities.
Task-free continual learning. As discussed in Sec. 5.1, task-free continual learning focus on learning from continuously arriving data without explicit task boundaries for homogeneous visual tasks. StrCVIT focuses on a stream of heterogeneous vision-language instruction data for MLLMs, where each chunk contains samples from multiple tasks, and both the number of tasks and their proportions change over time.
Task-incremental CVIT. As discussed in Sec. 5.2, task-incremental CVIT settings aim to enable MLLMs to continually acquire new capabilities, typically assuming that each training phase corresponds to a single predefined task. In real-world applications, data may arrive as a continuous stream of interleaved and dynamically evolving tasks. We therefore introduce StrCVIT, a more general and realistic continual learning setting for MLLMs, in which models learn from a stream of data chunks containing a dynamic mixture of tasks.
Response to Weakness 2
Computational cost comparison. In Table 14, MoELoRA and StrLoRA use the same number of experts with the same expert size. StrLoRA introduces additional routing modules, thereby increasing inference time. However, this comes with an improvement in MAP from 60.19 to 61.85 and a 59.3% reduction in MAF compared with MoELoRA. We additionally increase the number of MoELoRA experts from four to five. As shown below, although MoELoRA has more trainable and active parameters, StrLoRA still achieves a higher MAP of 61.85 versus 60.23 and reduces MAF by 65.0%. This substantial reduction in overall forgetting is more meaningful for continual adaptation, where MLLMs continuously acquire new capabilities while preserving previously learned knowledge.
Response to Weakness 3
Multi-step reasoning. We extend our evaluation to OmniCoT [1], a panoramic spatial reasoning benchmark that tests MLLMs’ ability to use global visual evidence and perform multi-step inference across viewpoints. It covers multi-hop viewpoint transformation, inter-object spatial reasoning, and embodied action simulation. After training on the initial 25 data chunks, we train MoELoRA and StrLoRA on the same additional chunk of 2,000 examples randomly sampled from OmniCoT’s training split and evaluate them on its official test split. StrLoRA achieves 27.90, compared with 26.73 for MoELoRA. After learning the new multi-step reasoning task, MoELoRA’s average accuracy on the original tasks decreases from 63.27 to 54.51, while StrLoRA’s decreases only slightly, from 65.09 to 64.64. These results show that StrLoRA performs better on this multi-step reasoning task while also better preserving prior capabilities under a substantial shift in the task distribution of a non-stationary data stream.
“Before” and “After” denote the model checkpoints obtained before and after training on the additional OmniCoT chunk, respectively.
[1] OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning. ECCV 2026.