acceptodds
Under review as a conference paper at ICLR 2027

StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs

Abstract

Continual Visual Instruction Tuning (CVIT) enables Multimodal Large Language Models to incrementally acquire new abilities. However, existing CVIT methods operate under a restrictive task-incremental setting, where each training phase corresponds to a single, predefined task. This does not reflect real-world conditions, where data arrives as a continuous stream of interleaved and dynamically evolving tasks. To bridge this gap, we introduce Streaming CVIT (StrCVIT), a more general and realistic setting where models learn from a stream of data chunks containing a dynamic mixture of tasks. In StrCVIT, a model must simultaneously acquire new abilities, reinforce recurring abilities, and mitigate forgetting, which requires it to identify the ability underlying each sample. Existing CVIT methods fail here as they cannot reliably distinguish or adapt to the heterogeneous task samples within each chunk. We therefore propose StrLoRA, a regularized two-stage expert routing framework. StrLoRA first performs task-aware expert selection using the textual instruction to activate a sparse subset of relevant experts, reducing cross-task interference. It then applies token-wise expert weighting within this subset, where contribution weights are computed via cross-modal attention between local visual tokens and the global instruction representation. To maintain stability across the non-stationary stream, a routing-stability regularization aligns current routing distributions with a historical exponential moving average reference. Extensive experiments on a newly developed StrCVIT benchmark show that StrLoRA consistently outperforms existing methods, effectively enhancing the model's abilities from continuously evolving data streams.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
MLLM, Continual Learning, Continual Visual Instruction Tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wb89OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper introduces Streaming Continual Visual Instruction Tuning (StrCVIT), where multimodal models learn from a sequence of data chunks containing changing mixtures of tasks without explicit task identities. To address interference and forgetting in this setting, the authors propose StrLoRA, which combines instruction-guided expert selection, token-wise expert weighting, and an EMA-based routing regularizer. Experiments across multiple model backbones and task distributions show promising improvements in average performance and reduced forgetting compared with existing continual learning baselines.

Critical strengths

Strengths

  • The paper is generally well written and clearly organized. The illustrations and formulation make the proposed setting and the three components of the method easy to follow.
  • The experimental results are promising, particularly in reducing forgetting. Evaluation across multiple backbones, stream lengths, and task compositions provides reasonably broad empirical coverage.
  • The combination of architectural analysis and data-stream analysis is useful. The routing visualizations, component ablations, and experiments with different stream compositions help explain how the design interacts with heterogeneous, evolving data.
Critical weaknesses

Weaknesses

  • The characterization of prior work appears overly broad. The claim that existing CVIT methods assume a single task per training stage is supported by several representative benchmarks, but the paper does not sufficiently establish that this characterization applies to the broader literature. A clearer distinction between task-incremental CVIT, mixed-task instruction tuning, and task-free continual learning would strengthen the positioning and clarify the scope of the proposed setting’s novelty.
  • The performance–latency trade-off needs stronger justification. Table 14 reports an increase in inference time from 7.62 seconds per sample for MoELoRA to 9.83 seconds for StrLoRA, approximately 29%. Although the reduction in forgetting is encouraging, this is a substantial inference overhead. It remains unclear whether the gains justify this cost, particularly relative to simpler alternatives under comparable latency or resource budgets.
  • The evidence for more demanding reasoning tasks is limited. The evaluation primarily covers recognition, grounding, captioning, and established visual question-answering tasks. Although some datasets involve reasoning, the experiments do not establish whether the approach remains effective for complex, multi-step multimodal reasoning. Such tasks may also be less easily separated by instruction-level cues, making the generality of the routing strategy uncertain.
Questions for authors

How does StrLoRA compare with joint training on the same mixed-task data under a comparable training budget? This comparison would help clarify the benefits and costs of continual adaptation relative to learning these tasks together from the outset.

7 comments
AuthorsOpenReview ↗

Response to Reviewer wb89 (Part I)

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We address your questions point by point below.

Response to Weakness 1

We clarify the distinctions between StrCVIT and related settings as follows:

Mixed-task instruction tuning. In joint mixed-task instruction tuning, the model is trained in a single phase on a pooled dataset containing mixed tasks. StrCVIT instead studies learning from a continuously arriving data stream, requiring the model to simultaneously acquire new abilities from emerging tasks, reinforce recurring abilities, and mitigate forgetting of absent abilities.

Task-free continual learning. As discussed in Sec. 5.1, task-free continual learning focus on learning from continuously arriving data without explicit task boundaries for homogeneous visual tasks. StrCVIT focuses on a stream of heterogeneous vision-language instruction data for MLLMs, where each chunk contains samples from multiple tasks, and both the number of tasks and their proportions change over time.

Task-incremental CVIT. As discussed in Sec. 5.2, task-incremental CVIT settings aim to enable MLLMs to continually acquire new capabilities, typically assuming that each training phase corresponds to a single predefined task. In real-world applications, data may arrive as a continuous stream of interleaved and dynamically evolving tasks. We therefore introduce StrCVIT, a more general and realistic continual learning setting for MLLMs, in which models learn from a stream of data chunks containing a dynamic mixture of tasks.

Response to Weakness 2

Computational cost comparison. In Table 14, MoELoRA and StrLoRA use the same number of experts with the same expert size. StrLoRA introduces additional routing modules, thereby increasing inference time. However, this comes with an improvement in MAP from 60.19 to 61.85 and a 59.3% reduction in MAF compared with MoELoRA. We additionally increase the number of MoELoRA experts from four to five. As shown below, although MoELoRA has more trainable and active parameters, StrLoRA still achieves a higher MAP of 61.85 versus 60.23 and reduces MAF by 65.0%. This substantial reduction in overall forgetting is more meaningful for continual adaptation, where MLLMs continuously acquire new capabilities while preserving previously learned knowledge.

MethodTrainable ParametersActive ParametersTraining Time (s/step)Inference Time (s/sample)MAP ↑MAF ↓
MoELoRA258.43M258.43M42.819.4260.235.46
StrLoRA250.04M162.75M41.119.8361.851.91

Response to Weakness 3

Multi-step reasoning. We extend our evaluation to OmniCoT [1], a panoramic spatial reasoning benchmark that tests MLLMs’ ability to use global visual evidence and perform multi-step inference across viewpoints. It covers multi-hop viewpoint transformation, inter-object spatial reasoning, and embodied action simulation. After training on the initial 25 data chunks, we train MoELoRA and StrLoRA on the same additional chunk of 2,000 examples randomly sampled from OmniCoT’s training split and evaluate them on its official test split. StrLoRA achieves 27.90, compared with 26.73 for MoELoRA. After learning the new multi-step reasoning task, MoELoRA’s average accuracy on the original tasks decreases from 63.27 to 54.51, while StrLoRA’s decreases only slightly, from 65.09 to 64.64. These results show that StrLoRA performs better on this multi-step reasoning task while also better preserving prior capabilities under a substantial shift in the task distribution of a non-stationary data stream.

“Before” and “After” denote the model checkpoints obtained before and after training on the additional OmniCoT chunk, respectively.

MethodStageAD ↑Fin ↑RS ↑ImageNet ↑VQAv2 ↑Places365 ↑TextCaps ↑OCRVQA ↑Grounding ↑Average ↑
MoELoRABefore33.277.772.987.770.422.385.7258.960.663.27
StrLoRABefore37.480.173.288.471.234.086.0553.761.865.09
MoELoRAAfter32.5 (−0.7)77.2 (−0.5)67.4 (−5.5)72.7 (−15.0)62.6 (−7.8)19.2 (−3.1)74.67 (−11.05)50.2 (−8.7)34.1 (−26.5)54.51 (−8.76)
StrLoRAAfter37.8 (+0.4)79.9 (−0.2)72.8 (−0.4)89.5 (+1.1)70.7 (−0.5)33.0 (−1.0)86.00 (−0.05)54.1 (+0.4)58.0 (−3.8)64.64 (−0.45)

[1] OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning. ECCV 2026.

AuthorsOpenReview ↗

Response to Reviewer wb89 (Part II)

Response to Question 1

Different learning settings. Joint training and continual adaptation operate under fundamentally different learning settings. In joint training, data from all tasks are available from the outset and pooled for training. In contrast, continual adaptation requires the model to continually improve its capabilities as new data arrive while retaining previously learned knowledge, without access to past or future data.

Comparison with joint training. With similar numbers of trainable parameters, we compare the final performance of StrLoRA after continual adaptation with that of MoELoRA jointly trained on the pooled and reshuffled data from all 25 mixed-task chunks. StrLoRA still achieves a competitive average accuracy of 65.09, compared with 65.17 for jointly trained MoELoRA.

MethodTrainable ParametersPlaces365RSADVQAv2FinGroundingOCRVQATextCapsImageNetAverage
MoELoRA, joint training258.43M38.3070.1036.2070.9077.4060.6057.6087.6687.8065.17
StrLoRA, continual adaptation250.04M34.0073.2037.4071.2080.1061.8053.7086.0588.4065.09
AuthorsOpenReview ↗

Look forward to further discussions

Dear reviewer,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable feedback. We have carefully considered all the questions and concerns raised in your review and provided detailed responses. We would greatly appreciate it if you could take a look at our responses and let us know whether they effectively address your concerns or if there are any additional questions or points you would like to discuss.

Thank you again for your time and consideration.

Best regards,

Authors

Reviewer_5vGROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Streaming Continual Visual Instruction Tuning (StrCVIT), where models learn from a single-pass stream of data chunks containing dynamically mixed tasks. StrLoRA uses textual instructions to select relevant LoRA experts, followed by fine-grained expert weighting based on visual tokens and global instructions, with EMA-based routing regularization for stability. Experiments across multiple base models and a standard CVIT benchmark demonstrate improved overall performance and reduced forgetting.

Critical strengths
  • A practically relevant problem setting. Compared with settings where each phase corresponds to a single task, streams with dynamically mixed and recurring tasks better reflect continual model updates and provide a more challenging evaluation of existing CVIT methods.

  • Consistent results and useful component analysis. The method improves overall performance on both InternVL and Gemma and also shows gains in standard CVIT. Ablations support the complementary roles of sample-level expert selection, token-level weighting, and routing regularization.

Critical weaknesses
  • The methodological novelty and necessity of the two-stage design are not sufficiently established. The method combines text-guided Top-K routing, token-level expert weighting, and EMA consistency regularization. While the ablations demonstrate their usefulness, they do not establish the specific advantage of the proposed cross-modal weighting over simpler text-based or standard token-level routing. Comparisons should match the number of experts, active experts, and parameter budgets.

  • Text-based routing may rely on dataset-specific instruction templates rather than task semantics. The t-SNE clusters demonstrate separability in the current textual inputs, but this could arise from fixed wording across datasets. It remains unclear whether expert selection is reliable under paraphrased instructions or similar instructions requiring different visual capabilities. This directly affects the central claim that text provides a stable signal for identifying tasks.

  • Evaluation across different data streams and analysis of result reliability are limited. The presented results primarily concern a single sequence of 25 data chunks, without reporting variability across repeated runs. Evaluations should vary task order, the rate of changes in task mixtures, and task absence duration, using multiple randomized streams. This is particularly important for assessing the smaller MAP gains. Trainable parameter counts, active expert counts, and actual training costs should also be matched or explicitly reported.

Questions for authors
  • Under matched parameter and active-expert budgets, how does StrLoRA compare with simpler text-only routing and standard multimodal token-level routing? What benefit specifically comes from the cross-modal weighting in Equation (5)?

  • Does routing remain reliable when instructions are paraphrased or templates are standardized across datasets? Can you distinguish task-semantic specialization from reliance on dataset-specific wording?

  • How robust are the gains across different task orders, mixture-change rates, and task absence durations? Please report means and standard deviations across multiple randomized streams and training seeds.

  • Are the baselines matched in trainable parameters and active expert counts? What are their training time and peak memory costs, and do StrLoRA’s gains persist under comparable budgets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5vGR (Part I)

We sincerely appreciate the time you spent reviewing our paper and your valuable comments on our experiments. We address your questions point by point below.

Response to Weakness 1 and Question 1

(1) Novelty of StrLoRA. Existing CVIT methods employ routers for joint expert selection and weighting and typically use phase-specific routers that assign samples within the same phase to the same expert or expert subset. Such routers produce highly similar routing weights for samples from different tasks, leading to homogeneous expert utilization (Fig. 2(a)), which undermines parameter isolation and causes cross-task interference. StrLoRA decouples expert selection from expert weighting to handle heterogeneous samples within each chunk. It first selects a sparse expert subset for each sample based on its instruction, allowing samples with different task semantics to activate different subsets, thereby promoting parameter isolation and reducing cross-task interference. To adapt expert contributions to the local visual content under the given instruction, a cross-modal token-wise router then computes instruction-conditioned weights over the selected experts for each visual token. To mitigate router instability as the data stream evolves, routing-stability regularization encourages consistent routing.

(2) Comparison with routing baselines We clarify that Table 3 (Section 4.4) compares StrLoRA with simpler text-only routing and standard token-level soft routing. We additionally evaluate standard token-level Top-2 routing. All methods employ four rank-16 LoRA experts with the same number of expert parameters. As shown below, StrLoRA achieves a higher MAP and a lower MAF than the simpler routing baselines.

MethodLoRA Expert Params.Active Expert Params.MAP ↑MAF ↓
Standard Token-level Soft Routing174.588M174.588M60.194.69
Standard Token-level Top-2 Routing174.588M87.294M55.505.12
Simpler Text-only Top-2 Routing174.588M87.294M60.773.54
StrLoRA w/o EMA Top-2 Routing (ours)174.588M87.294M61.582.97
StrLoRA Top-2 Routing (ours)174.588M87.294M61.851.91

(3) Effectiveness of cross-modal weighting. We clarify that Table 10 (Appendix D.4) provides a dedicated ablation of cross-modal token-wise weighting. Specifically, we retain the same task-aware expert selection stage while directly comparing standard token-wise weighting with our proposed cross-modal token-wise weighting. As shown below, our cross-modal weighting consistently achieves higher final task accuracy than standard token-wise weighting across all datasets.

MethodPlaces365 ↑RS ↑AD ↑VQAv2 ↑Fin ↑Grounding ↑OCRVQA ↑TextCaps ↑ImageNet ↑Average ↑
Task-aware Selection + Standard Token-wise Weighting30.5071.5033.3070.3079.7059.5047.9085.4283.7062.42
Task-aware Selection + Cross-modal Token-wise Weighting (Ours)34.0073.2037.4071.2080.1061.8053.7086.0588.4065.09
AuthorsOpenReview ↗

Response to Reviewer 5vGR (Part II)

Response to Weakness 2 and Question 2

We clarify that StrLoRA's robustness to instruction variation was evaluated in Appendix D.6 of the original submission. We summarize the relevant experimental results below.

(1) Distinguishing tasks with a unified template. Across textual datasets, instructions follow a unified template structure consisting of a task question and an output-format constraint, as detailed in Table 5 (Appendix B.1). We examine tasks whose textual instructions share a nearly identical instruction style but differ in task semantics: ImageNet (“What is the object in the image? Answer the question using a single word or phrase.”) and Places365 (“What is the background in the image? Answer the question using a single word or phrase.”). As shown below, the two tasks activate different subsets of experts and yield different expert activation proportions. Detailed expert activation proportions across tasks are shown in Figure 5.

Each vector denotes the activation proportions (%) in the order [Expert 0, Expert 1, Expert 2, Expert 3].

LayerImageNetPlaces365
L0[0.0,75.8,0.0,24.2][0.0,0.0,69.1,30.9]
L4[51.3,48.7,0.0,0.0][0.0,47.6,0.0,52.4]
L12[47.5,0.0,52.5,0.0][0.0,0.0,51.6,48.4]
L28[73.1,0.0,0.0,26.9][0.0,0.0,54.2,45.8]

(2) Generalization to unseen instructions. We also vary each dataset’s instruction style while preserving the original semantics. For each sample, we construct five unseen instructions in different styles. We report the evaluation results of SMoLoRA, MoELoRA, and StrLoRA under these instructions in Table 15 (Appendix D.6), while Table 16 (Appendix D.6) compares StrLoRA’s expert selections under the unseen instructions with those obtained using the original instructions. As shown below, StrLoRA maintains high routing consistency and achieves the highest average score among the compared methods under unseen instruction styles.

Table 15. Accuracy (%) evaluated under unseen instructions.

MethodAD ↑Fin ↑Grounding ↑ImageNet ↑OCRVQA ↑Places365 ↑RS ↑TextCaps ↑VQAv2 ↑Average ↑
SMoLoRA28.0072.0065.0041.0050.0018.0070.0073.6565.0053.63
MoELoRA28.0078.0060.0055.0056.0016.0068.0077.3669.0056.37
StrLoRA35.8078.6059.3073.2051.5028.4070.9078.3869.1060.58

Table 16. StrLoRA’s routing consistency (%) under unseen instructions.

MetricADFinGroundingImageNetOCRVQAPlaces365RSTextCapsVQAv2Average
Top-1 consistency100.0095.4389.5798.6497.7190.1492.4389.3590.7193.78
Top-2 consistency94.1292.4388.7189.5793.2989.8686.8682.7187.5789.46
AuthorsOpenReview ↗

Response to Reviewer 5vGR (Part III)

Response to Weakness 3 and Question 3

(1) Results across different data streams. We clarify that we evaluated more than one data stream and that Table 6 in Appendix D.1 presents results on an additional randomly generated stream with different rates of change in task mixtures and different task-absence durations. As shown below, although the absolute performance varies with stream composition, StrLoRA consistently achieves the highest MAP and lowest MAF among the compared methods on both streams, demonstrating its robustness under variations in stream composition.

Stream 1

MethodPlaces365 AP↑Places365 AF↓RS AP↑RS AF↓AD AP↑AD AF↓VQAv2 AP↑VQAv2 AF↓Fin AP↑Fin AF↓Grounding AP↑Grounding AF↓OCRVQA AP↑OCRVQA AF↓TextCaps AP↑TextCaps AF↓ImageNet AP↑ImageNet AF↓MAP↑MAF↓
SMoLoRA17.3420.2164.292.9729.274.3169.271.5571.521.0443.938.2340.714.0075.835.5449.983.8051.355.74
MoELoRA26.1421.2769.101.6233.944.5170.140.7274.230.8455.306.9053.671.8684.410.6574.783.8160.194.69
StrLoRA33.372.9969.650.7536.273.9270.111.1375.950.3857.362.5353.003.2984.480.7476.491.4661.851.91

Stream 2

MethodPlaces365 AP↑Places365 AF↓RS AP↑RS AF↓AD AP↑AD AF↓VQAv2 AP↑VQAv2 AF↓Fin AP↑Fin AF↓Grounding AP↑Grounding AF↓OCRVQA AP↑OCRVQA AF↓TextCaps AP↑TextCaps AF↓ImageNet AP↑ImageNet AF↓MAP↑MAF↓
SMoLoRA20.0023.2567.481.2530.343.2670.202.2980.060.7756.532.9346.953.3276.283.4153.163.2355.674.86
MoELoRA30.8517.6668.511.2433.351.5269.702.1081.110.5956.976.5347.295.2478.081.6459.162.7358.344.36
StrLoRA33.603.2769.450.8536.681.8270.581.1681.420.4957.963.5948.673.4781.610.8562.041.4360.231.88

Moreover, Table 7 in Appendix D.2 of the original submission presents results after 15 and 20 chunks across three base models. StrLoRA achieves the highest MAP and lowest MAF among the compared methods in all settings.

(2) Results across training seeds. We additionally evaluated MoELoRA and StrLoRA on Stream 1 using three training seeds, 42, 100, and 2026, with InternVL3.5-8B-Pretrained. The means and standard deviations are presented in the table below. StrLoRA achieves 61.99 ± 0.19 MAP and 2.31 ± 0.37 MAF, compared with 60.02 ± 0.19 MAP and 5.25 ± 0.58 MAF for MoELoRA, supporting the reliability of its improvements across multiple random seeds.

MethodPlaces365 AP↑Places365 AF↓RS AP↑RS AF↓AD AP↑AD AF↓VQAv2 AP↑VQAv2 AF↓Fin AP↑Fin AF↓Grounding AP↑Grounding AF↓OCRVQA AP↑OCRVQA AF↓TextCaps AP↑TextCaps AF↓ImageNet AP↑ImageNet AF↓MAP↑MAF↓
MoELoRA24.53±2.1426.64±6.4769.13±0.831.83±0.5332.67±1.143.05±1.2970.28±0.121.07±0.3774.52±0.260.82±0.2755.79±0.586.78±1.3054.13±0.712.26±0.4784.12±0.261.24±0.5175.03±0.373.50±0.4660.02±0.195.25±0.58
StrLoRA31.75±1.568.24±5.0969.70±0.620.82±0.0638.50±2.052.30±1.4170.42±0.310.88±0.2576.54±0.660.55±0.2558.73±2.172.84±0.4852.99±2.242.44±0.8784.67±0.210.97±0.2774.63±1.641.77±0.2861.99±0.192.31±0.37

Response to Weakness 3 and Question 4

Computational cost comparison. We clarify that the computational comparison was reported in Table 14 in Appendix D.5. Compared with MoELoRA using the same number of experts, StrLoRA has more trainable parameters due to its additional routing modules, but it activates fewer parameters than MoELoRA and achieves a higher MAP of 61.85 versus 60.19 while reducing MAF by 59.3%. We additionally evaluate MoELoRA with five experts. As shown below, although MoELoRA has more trainable and active parameters, StrLoRA still achieves a higher MAP of 61.85 versus 60.23 and reduces MAF by 65.0%, from 5.46 to 1.91.

MethodTrainable ParametersActive ParametersTraining Time (s/step)MAP↑MAF↓
MoELoRA258.43M258.43M42.8160.235.46
StrLoRA250.04M162.75M41.1161.851.91
Reviewer_DhhtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Streaming Continual Visual Instruction Tuning (StrCVIT), where single-pass data chunks contain changing mixtures of multimodal tasks without explicit task labels. StrLoRA first selects a sparse subset of LoRA experts from the instruction embedding, then computes cross-modal token-wise weights within that subset. An EMA-based routing regularizer constrains the conditional expert-weight distribution. The evaluation includes different backbone families and scales, two stream compositions, component and instruction-robustness analyses, and the standard task-incremental CoIN benchmark.

Critical strengths
  1. StrCVIT extends task-incremental CVIT to mixed, recurring and disappearing tasks. StrLoRA addresses this setting through instruction-based expert selection and conditional token-wise weighting without explicit task labels.

  2. StrLoRA improves aggregate performance and reduces average forgetting across backbones and two data streams in Table 1 and Table 6. And component ablations and unseen-instruction tests support the routing design and its robustness to instruction changes as in Table 3 and Table 10.

Critical weaknesses
  1. Figure 6 validates the average accuracy on the proxy set, but it does not validate the "forgetting" metric, which is based on historical peak accuracy. Similar average accuracy does not imply that the estimates of the degree of "forgetting" are also consistent.

  2. High CKA alone does not establish homogeneous expert allocation, and Figure 5 does not provide the corresponding baseline expert distributions.

Questions for authors
  1. Can proxy-based evaluation and full-scale testing yield consistent estimates of the degree of forgetting, and do they preserve the same ranking order between StrLoRA and MoELoRA?

  2. How was CKA computed, and does the homogeneity conclusion hold when expert identities are preserved ?

  3. Please specify the STE backward rule, routed token positions and layer/batch loss aggregation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Dhht

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We address your questions point by point below.

Response to Weakness 1 and Question 1

(1) Consistency of forgetting estimates. Thank you for this helpful suggestion. We compare mean average forgetting (MAF) and per-task average forgetting (AF) on the full and proxy test sets over the full 25-chunk data stream, using LoRA on InternVL3.5-8B-Pretrained. As shown below, the MAF differ by only 0.02, suggesting that the proxy set also provides a similar estimate of the degree of overall forgetting while substantially reducing evaluation costs. We have updated Figure 6 in the revised paper to include this comparison.

MetricMAFAF: FinAF: OCRVQAAF: GroundingAF: ADAF: ImageNetAF: GQAAF: Places365AF: RSAF: VQAv2
Full − Proxy-0.020.020.14-0.15-0.61-0.20-0.591.330.06-0.17

(2) Full-scale evaluation of MoELoRA and StrLoRA. We would also like to compare MoELoRA and StrLoRA using the full test set. Each full evaluation involves approximately 230,000 samples, so evaluating both methods after each of the 25 data chunks is infeasible given the available time and computational resources. We are doing our best to complete as much of this evaluation as possible and will report the results in a subsequent response.

Response to Weakness 2 and Question 2

(1) CKA computation. For each task, we construct a module-by-expert matrix by aggregating routing weights across task samples, center each expert column across modules, and compute linear CKA between each pair of task matrices.

(2) Expert identity preserving analysis. Thank you for this insightful suggestion. We compute base-2 Jensen–Shannon divergence (JSD) between the expert routing distributions of each task pair within the same module. This directly compares routing probabilities associated with the same experts across tasks. For each task pair, we average the JSD over all modules. Across the 55 task pairs, the mean JSD is 0.001712, with the pairwise means ranging from 0.000021 to 0.004141. These results further support our observation that standard MoE routing can produce highly similar expert activation distributions across tasks in StrCVIT. We have replaced the CKA heatmap in Fig. 2(a) with a heatmap of JSD-based similarity scores, defined as 1−JSD1-\mathrm{JSD}, in the revised paper. The pairwise JSD values are shown below.

TaskADFinGroundingImageNetOCRVQAPlaces365RSTextCapsVQAv2ChartQAGQA
AD-0.0033340.0019620.0015700.0017810.0027040.0029910.0010470.0018120.0041410.001846
Fin--0.0017390.0018010.0024250.0029270.0025910.0022440.0015280.0009250.001720
Grounding---0.0006910.0016280.0014320.0017790.0009180.0000530.0023820.000026
ImageNet----0.0008900.0017770.0024640.0004910.0004330.0027100.000546
OCRVQA-----0.0016370.0028070.0003090.0013990.0025590.001465
Places365------0.0005840.0014140.0013900.0025800.001375
RS-------0.0023970.0017720.0024240.001822
TextCaps--------0.0007640.0026150.000782
VQAv2---------0.0022890.000021
ChartQA----------0.002423
GQA-----------

(3) Expert activation distributions of MoELoRA. We appreciate the reviewer’s suggestion. We have added a visualization of MoELoRA’s expert activation distributions across tasks and layers for comparison in Figure 5 of the revised paper. MoELoRA exhibits highly similar expert activation distributions across tasks and layers. In contrast, StrLoRA activates distinct subsets of experts in varying proportions across tasks and layers.

Response to Question 3

Supplementary implementation details. In each routed module, the first-stage task-aware router uses mean-pooled question embeddings to compute routing probabilities and select a sparse expert subset S\mathcal{S}. We apply a binary mask mj=1[j∈S]m_j=\mathbf{1}[j\in\mathcal{S}] to the expert outputs in the forward pass and use an STE that passes the gradient through each selected expert's mask mjm_j directly to its routing probability pjp_j, with ∂mj/∂pj=1\partial m_j/\partial p_j=1 in the backward pass. The second-stage token-wise router then computes contribution weights over the selected experts for each token in the hidden states. The task loss L\mathcal{L} is computed from the final output logits and averaged over the supervised tokens in the batch. For Lreg\mathcal{L}_{\mathrm{reg}}, we average the per-token KL divergence over all routed tokens in the batch, and then average across the routed layers.

AuthorsOpenReview ↗

Further Response to Question 1

Full-scale evaluation of MoELoRA and StrLoRA. Given the available time and computational resources, we compare MoELoRA and StrLoRA on InternVL3.5-8B-Pretrained across 15 chunks using both the proxy and full-scale test sets. As shown below, both test sets yield identical method rankings across all task's AP and AF metrics, as well as the overall MAP and MAF. The full-scale test set generally yields higher overall performance, the forgetting measurements remain highly consistent, with MAF of 3.90 vs. 3.91 for MoELoRA and 1.46 vs. 1.45 for StrLoRA on the proxy and full-scale test sets, respectively.

MethodPlaces365 AP↑Places365 AF↓RS AP↑RS AF↓AD AP↑AD AF↓VQAv2 AP↑VQAv2 AF↓Fin AP↑Fin AF↓Grounding AP↑Grounding AF↓OCRVQA AP↑OCRVQA AF↓TextCaps AP↑TextCaps AF↓ImageNet AP↑ImageNet AF↓MAP↑MAF↓
MoELoRA (Proxy)27.7012.7265.832.2033.903.8169.530.7272.540.7552.177.5151.942.4282.880.6367.954.2958.273.90
StrLoRA (Proxy)32.421.3767.450.9334.874.2069.351.4073.520.2754.781.8253.071.4682.970.6370.231.0459.851.46
MoELoRA (Full)27.2214.1266.111.3733.963.5370.211.0673.760.7255.817.1453.142.4682.880.6369.664.1859.193.91
StrLoRA (Full*)31.900.8968.691.0534.813.6569.921.5374.600.2758.012.9554.290.9782.970.6371.671.1160.761.45

Ranking Consistency. We further evaluate the ranking consistency between MoELoRA and StrLoRA at each chunk in terms of overall performance. As shown below, both MAP and MAF exhibit 100% ranking consistency across all chunks. These results suggest that the proxy test set can provide a reliable evaluation of both accuracy and forgetting.

Δ denotes the difference in metric values between StrLoRA and MoELoRA (StrLoRA − MoELoRA).

MetricChunk 1Chunk 2Chunk 3Chunk 4Chunk 5Chunk 6Chunk 7Chunk 8Chunk 9Chunk 10Chunk 11Chunk 12Chunk 13Chunk 14Chunk 15
ΔMAP (Full)+0.47+1.79+1.20+1.20+1.33+1.05+0.99+1.02+1.00+0.94+1.02+1.22+1.37+1.49+1.57
ΔMAP (Proxy)+0.51+1.69+0.91+0.96+1.09+0.90+0.85+0.91+0.97+1.05+1.16+1.36+1.44+1.53+1.58
MAP Consistency✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
ΔMAF (Full)—-1.77-1.57-1.08-1.53-1.09-1.29-1.43-1.40-1.29-1.39-1.79-1.98-2.30-2.46
ΔMAF (Proxy)—-1.90-1.48-0.87-1.24-0.87-1.00-1.14-1.29-1.38-1.53-1.94-2.02-2.35-2.44
MAF Consistency—✓✓✓✓✓✓✓✓✓✓✓✓✓✓

Thank you again for your time and effort in reviewing our paper and providing insightful comments.

Comments on the paper

3 comments

Dear Reviewers,

Thank you for your efforts so far. As we are halfway through the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. We have revised our paper according to the suggestions. The major revisions are summarized as follows:

  • We refined our analysis of expert routing distribution similarity and updated Fig. 2(a), added the expert activation proportions of the compared method in Fig. 5, provided additional analyses of the differences in forgetting evaluations in Fig. 6 of Appendix B.2, provided additional implementation details in Appendix C.2, and added a full-scale evaluation of MoELoRA and StrLoRA in Appendix D.8, as suggested by Reviewer Dhht.

  • We moved the training setup from the appendix to Section 4.2 of the main paper, added descriptions of supplementary experiments, including additional hyperparameter ablation studies, computational efficiency analysis, and instruction robustness evaluation in Sections 4.3 and 4.4, and added an evaluation across multiple training seeds in Appendix D.7, as suggested by Reviewer 5vGR.

  • We expanded the Related Work section to include a discussion of Mixed Instruction Tuning (Section 5.3) and added additional experiments with a similar number of parameters in Appendix D.5, as suggested by Reviewer wb89.

All modifications have been highlighted in orange in our revised submission.

Best regards,

Authors

AuthorsOpenReview ↗

Summary of Author-Reviewer Discussion

Dear Area Chair and Reviewers,

We sincerely appreciate the time you have devoted to discussing our paper. As the discussion draws to a close, we summarize our core contributions and the main points of our rebuttal to facilitate subsequent review.

Core Contributions

  • We introduce Streaming Continual Visual Instruction Tuning (StrCVIT), a timely and realistic continual learning setting for MLLMs, where each incoming data chunk contains interleaved samples from multiple tasks with dynamically changing compositions and proportions across chunks. In this setting, the model faces cross-task interference not only across chunks but also among samples from different tasks within the same chunk.
  • We diagnose the homogeneous expert utilization problem of conventional CVIT routing strategies in StrCVIT and propose StrLoRA, a regularized two-stage expert routing framework that first uses each sample’s textual instruction to select a sparse subset of experts and then incorporates cross-modal information to assign token-wise weights to each selected expert, with routing stabilized by a regularization loss.
  • We construct a StrCVIT benchmark covering diverse vision-language tasks, and extensive experiments across multiple backbones, baselines, and stream compositions, together with multi-seed evaluations, detailed ablations, and evaluation in the conventional CVIT setting, demonstrate the effectiveness and robustness of StrLoRA.

Rebuttal Summary

  • For Reviewer Dhht, we added a comparison of overall and per-task forgetting for LoRA on the full-scale and proxy test sets across all 25 chunks. We further evaluated whether the two test sets rank MoELoRA and StrLoRA consistently. For each of the two methods, mean average forgetting (MAF) differs by only 0.01 between the two test sets, and the two test sets yield identical rankings in both performance and forgetting at each chunk. These results suggest that the proxy test set can be used to reliably evaluate both accuracy and forgetting. We refined our analysis of expert routing distribution similarity by preserving expert identities across tasks, further supporting our observation that standard MoE routing can produce highly similar expert activation distributions across tasks in StrCVIT. We also added a visualization comparing MoELoRA and StrLoRA. MoELoRA exhibits highly similar activation distributions across tasks, while StrLoRA activates distinct expert subsets in varying proportions. We expanded the training details for StrLoRA to specify the STE backward rule, the token positions involved in routing, and how losses are aggregated across tokens and layers.
  • For Reviewer 5vGR, we summarized our existing comparisons of StrLoRA with simpler routing baselines and added an additional comparison. StrLoRA consistently outperforms all routing baselines. We also summarized our instruction-robustness evaluation, showing that StrLoRA selects different expert subsets for tasks with nearly identical instruction templates but different semantics and maintains high routing consistency between the original instructions and their paraphrases in five unseen styles. We summarized our evaluations on different data streams and added a three-seed evaluation that confirms the reliability of StrLoRA’s gains. We evaluated a MoELoRA variant with more trainable and active parameters than StrLoRA. StrLoRA still achieves higher overall performance while substantially reducing forgetting.
  • For Reviewer wb89, we clarified how StrCVIT differs from mixed-task instruction tuning, task-free continual learning, and task-incremental CVIT. We evaluated a MoELoRA variant with more trainable and active parameters than StrLoRA. StrLoRA still achieves higher overall performance while substantially reducing forgetting. We also extended our evaluation to multi-step panoramic reasoning on OmniCoT. StrLoRA achieves higher accuracy than MoELoRA on the new task and better maintains accuracy on prior tasks (65.09 to 64.64 for StrLoRA, compared with 63.27 to 54.51 for MoELoRA).

Finally, we appreciate the reviewers’ valuable comments and have revised our paper in response. During the discussion period, we addressed Reviewer wb89’s concerns and received positive feedback. We also responded in detail to the concerns raised by Reviewers Dhht and 5vGR and invited further discussion. We hope that our work offers the continual learning community new insights into how MLLMs adapt to evolving streams and supports further research in this setting.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.