acceptodds
Under review as a conference paper at ICLR 2027

COVE: Concept Vocabulary Evolution for Streaming Continual Visual Instruction Tuning

Abstract

Continual Visual Instruction Tuning (CVIT) aims to enable multimodal large language models to continuously acquire new vision-language knowledge while retaining previously learned knowledge. Recent studies move toward more realistic streaming settings, where heterogeneous data arrive as interleaved and dynamically evolving distributions. In such streams, both the reuse of accumulated knowledge and the demand for additional adaptation capacity vary over time, making predefined expert structures difficult to match to the current adaptation demand. To address this challenge, we propose Concept Vocabulary Evolution (COVE), which organizes continual adaptation as an evolving vocabulary of reusable units discovered directly from incoming data. Specifically, COVE decomposes each incoming sample over the accumulated vocabulary, selectively refining reusable historical units when they sufficiently represent the sample, while reconstruction residuals drive the discovery of additional units when existing capacity becomes insufficient. Extensive experiments demonstrate that COVE improves continual performance and substantially reduces catastrophic forgetting, while further analyses show that the learned vocabulary dynamically evolves with the changing demands of the stream.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Lh7TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COVE addresses streaming continual visual instruction tuning, where each incoming data chunk may contain a changing mixture of tasks without explicit task identities. It maintains a sparse dictionary over frozen multimodal features, with each dictionary atom associated with a LoRA expert. Sparse coefficients determine how experts are combined, while reconstruction residuals guide the choice between refining existing experts and introducing new ones. Historical experts are updated selectively on reusable samples, and new experts are trained on samples identified as novel. The paper evaluates this approach across several backbones and streaming configurations.

Critical strengths
  1. The method is well motivated for interleaved data streams. COVE makes reuse and expansion decisions from the relationship between individual samples and the accumulated dictionary, rather than assuming that an entire chunk belongs to a single task. Sparse coefficients and reconstruction residuals provide a coherent basis for these decisions. This is a useful alternative to expanding the model at predefined task boundaries.

  2. The results improve performance and retention together. On InternVL3.5-4B, COVE achieves 63.87 MAP and 0.66 MAF, compared with 62.15 and 2.39 for StrLoRA. On the 8B model, the corresponding results are 64.73/0.81 versus 63.83/2.36. The gains therefore do not appear to come simply from sacrificing adaptation to reduce forgetting. The paper also states that all methods are reimplemented and evaluated on the same constructed streams, which supports the internal consistency of the comparison.

Critical weaknesses
  1. The experiments do not yet establish that residual-guided expansion allocates capacity more effectively than simpler alternatives. A large reconstruction residual indicates that the current dictionary represents a sample poorly, but it does not necessarily mean that the existing experts lack the capacity to learn the required task. Figure 3 shows an association between residuals and emerging distributions, followed by improved performance after expansion. However, the paper describes evaluating the same incoming chunk samples before and after vocabulary evolution, leaving unclear how much of the improvement reflects better generalization rather than additional capacity or training on those samples.

The ablations make this distinction important. Removing expansion leaves MAP almost unchanged, at 63.84 versus 63.87, while increasing MAF from 0.66 to 1.99. This supports a retention benefit, but it does not isolate the value of the residual-based expansion decision. Figure 4 includes matched expert budgets, but only for the first chunk. A comparison over the full stream, with the same expert-growth budget, would more directly test the proposed mechanism.

  1. The stop-gradient formulation appears to change more than the intended expert parameter updates. Equation 12 describes freezing historical AA factors and scaling the gradients of historical BB factors. Appendix H.3 instead applies stop-gradient to the entire expert contribution:
=\gamma\Delta y+(1-\gamma)\operatorname{sg}(\Delta y).$$ As written, this also scales the gradient passed through the expert to its input. It is therefore not equivalent to modifying only that expert’s parameter gradients, and it can affect learning in earlier trainable layers and the multimodal projector. This distinction matters for understanding the method: the reported results may reflect both selective expert updates and a broader change to gradient propagation.
Questions for authors
  1. Does the implementation apply stop-gradient to the full expert output, as in Appendix H.3? Is the resulting change to upstream gradients intentional, and how does it relate to Equation 12?
  2. How many experts are retained at each stage, and how does residual-guided expansion compare with a simpler expansion schedule under the same growth budget?
  3. Does Figure 3(b) evaluate training samples or held-out samples? Under matched training and capacity, do high-residual samples benefit more from new experts than low-residual samples?
3 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions.

W1 and Q2: Effectiveness of residual-guided expert expansion under matched expert budgets

COVE effectively identifies samples that benefit from additional adaptation and outperforms alternative expansion strategies under the same final expert budget. The sample-level identification results on the three expansion chunks are reported below.

ChunkGround-truth NovelResidual-guided NovelPrecision (%)Recall (%)F1 (%)
326524094.5885.6689.90
5507265.2894.0077.05
719819291.1588.3889.74

Furthermore, high-residual samples consistently achieve larger AP gains than low-residual samples after expert expansion.

Data ChunkΔH\Delta_HΔL\Delta_LΔH−ΔL\Delta_H-\Delta_L
3+2.92+0.41+2.51
5+1.39+0.00+1.39
7+7.29+0.17+7.12

Here, ΔH\Delta_H and ΔL\Delta_L denote the AP gains of high- and low-residual groups, respectively.

We further compare COVE with Fixed Expansion and Random Expansion over the full data stream. Both baselines reach the same final expert budget as COVE, while removing its residual-guided expansion decisions.

MethodMAP ↑MAF ↓Final AP ↑Final Experts
Fixed Expansion61.733.0962.756
Random Expansion61.892.7162.366
COVE63.870.6665.736

COVE achieves better MAP, MAF, and Final AP than both alternatives under the same expert-growth budget. Its parameter and computational costs are also lower than those of StrLoRA in terms of trainable parameters and training time, while maintaining a comparable total model size.

MethodAvg. Trainable Params (M)Final Total Params (M)Avg. Training Time per Chunk (s)
MoELoRA53.914769.352951.66
StrLoRA96.024977.323846.93
COVE22.654782.073444.96

Overall, these results demonstrate that COVE effectively identifies samples requiring additional adaptation and allocates expert capacity more effectively without relying on a larger model or more trainable parameters.

W2 and Q1: Clarification of the Stop-Gradient Implementation

We thank the reviewer for identifying this ambiguity in our stop-gradient formulation. Our intention is to selectively control expert parameter updates rather than modify gradient propagation through the entire expert contribution.

Specifically, for historical experts, LoRA-A matrices are frozen, while LoRA-B gradients for reusable samples are scaled according to their expert responsibilities:

\leftarrow \rho_{ij} \nabla_{B\\ _j^{(\ell)}} \mathcal{L}\\ _i, \qquad \nabla_{A\\ _j^{(\ell)}} \mathcal{L}\\ _i = 0, \quad i \in \mathcal{R}\\ _t.$$ Novel samples do not update historical experts, while newly introduced experts are updated using novel samples only. These operations directly control expert parameter gradients without blocking gradient propagation through expert inputs. Upon re-examining our implementation, we confirmed that all reported experiments follow the intended parameter-level update strategy. Specifically, the stop-gradient formulation in Appendix H.3 was retained from an earlier version of our implementation, whereas the code used for the reported experiments had already adopted the parameter-level gradient control described above. We inadvertently overlooked updating this part of the appendix when revising the implementation. We have now corrected Appendix H.3 to align its formulation with the actual implementation and the gradient update mechanism described in Eq. 12. **Q3: Evaluation on held-out samples and benefits of expert expansion** Figure 3(b) evaluates held-out test samples, with 500 samples per task, rather than the training samples used to update newly introduced experts. Thus, the observed improvements reflect performance on unseen evaluation samples. Furthermore, as reported in table above, high-residual samples consistently achieve larger AP gains than low-residual samples after expert expansion. These results further support the effectiveness of reconstruction residuals in identifying samples that benefit from additional adaptation capacity.

Thank you for running the full-stream comparison. All three policies end with six experts, yet COVE achieves 63.87 MAP, compared with 61.73 for fixed expansion and 61.89 for random expansion, and also has lower forgetting. This addresses my main concern: the benefit depends on how COVE allocates capacity, not simply on how many experts it retains. The clarification that Figure 3(b) uses held-out samples also makes the larger gains for high-residual samples more convincing.

The revised Appendix H.3 resolves my concern about the gradient rule. It now distinguishes scaling the LoRA parameter gradients from scaling the gradient passed back through the expert input. I would encourage two small reporting changes: state in Section 5.3 that Figure 3(b) uses 500 held-out samples per task, and include the expert counts at each stage alongside the final counts so readers can see how the three expansion schedules differ.

These results and clarifications address my main reservations. I will raise my rating from 2 to 3 (weak acceptance).

We thank the reviewer for the positive assessment and constructive suggestions. We have incorporated both recommendations into the revised manuscript. We greatly appreciate your time and carefully consideration, and wish you success with your submissions and future work.

Reviewer_DvQ3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COVE, a continual adaptation framework for streaming visual instruction tuning, where heterogeneous data arrive without explicit task identities or boundaries. The method maintains an evolving vocabulary of dictionary atoms associated with LoRA experts. Sparse reconstruction coefficients guide expert composition, while reconstruction residuals determine whether incoming samples should reuse existing experts or trigger vocabulary expansion. The framework also selectively updates historical experts to mitigate catastrophic forgetting.

Experiments on InternVL3.5-4B/8B and Gemma3-4B across different streaming benchmarks report improvements in continual learning performance and reduced forgetting. Additional analyses investigate vocabulary evolution, expert budgets, component ablations, and hyperparameter sensitivity.

I currently lean toward weak rejection, but I would be open to raising my rating if the key concerns outlined below are adequately addressed.

Critical strengths
  1. The paper addresses a meaningful problem in streaming continual visual instruction tuning, where changing data distributions create challenges for knowledge reuse and adaptation.

  2. The proposed framework combines sparse dictionary learning with dynamic LoRA expert expansion and selective refinement, providing an interesting approach to managing adaptation capacity.

  3. The experiments cover multiple model scales, different backbones, and streaming settings with varying numbers of tasks and data chunks. The reported results show promising improvements in continual performance and forgetting.

  4. The paper includes analyses of vocabulary evolution, expert budgets, and component ablations, providing useful empirical insights into the behavior of the proposed framework.

Critical weaknesses
  1. The paper uses reconstruction residuals to decide whether new LoRA experts are needed. However, a high reconstruction error in the feature space does not necessarily mean that the existing experts lack the ability to learn the sample. Figure 3(a) only shows that the proportions of novel samples and high-residual samples are similar, but this does not tell us whether the method actually identifies the correct samples. More direct evidence is needed to support this key assumption.

  2. Although Figure 4 compares methods with the same number of experts, the comparison is only conducted on the first data chunk. Since COVE keeps adding experts throughout training, it is unclear whether its improvements come from better expansion decisions or simply from having more parameters. A comparison under matched parameter budgets across the full streaming process would make the results more convincing.

  3. The paper constructs its own streaming benchmarks, but some important evaluation details are missing. In particular, it is unclear how performance is measured for different tasks such as image captioning and visual grounding, which do not naturally share the same accuracy metric. The authors should also clarify the data splits and stream construction procedures to make the experiments easier to reproduce.

  4. There are several typos or presentation issues in the paper. For example, the method is referred to as both "Concept Vocabulary Evolution" and "Continual Vocabulary Evolution", Appendix F contains an unresolved "Fig. ??" reference, and the captions of Figures 8 and 9 appear to have their chunk counts swapped. There are also several grammatical errors and awkward sentences throughout the paper. A careful proofreading would improve its readability.

Questions for authors
  1. Could the authors report sample-level precision and recall for identifying newly introduced distributions? More importantly, could they compare the benefits of expert expansion on high-residual versus low-residual samples to verify that reconstruction error reflects the actual need for additional adaptation capacity?

  2. Figure 4 compares methods with matched expert counts, but only on the first data chunk. Could the authors extend this comparison to the full streaming process? In particular, how does COVE compare with fixed-capacity experts or fixed/random expansion strategies under the same total parameter budget? Reporting the number of experts and parameter counts at each chunk would help clarify whether the gains come from better expansion decisions or simply increased model capacity.

  3. Could the authors clarify the evaluation protocols used for different tasks, particularly image captioning and visual grounding? How are their outputs evaluated and converted into the reported accuracy scores? Since the streaming benchmarks are constructed by the authors, more details on training/test splits, sampling procedures, and whether the total training data remains fixed across different chunk settings would also be helpful.

4 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions.

W1 and Q1: Reliability of reconstruction-residual-guided expert expansion

Following the reviewer's Q1, the additional results support reconstruction residuals as a useful signal for identifying samples that require additional adaptation capacity. We report sample-level identification performance on Data Chunks 3, 5, and 7, where COVE triggers expert expansion.

ChunkGround-truth NovelResidual-guided NovelPrecision (%)Recall (%)F1 (%)
326524094.5885.6689.90
5507265.2894.0077.05
719819291.1588.3889.74

More importantly, high-residual samples consistently benefit more from expert expansion than low-residual samples, as shown by their AP gains below.

Data ChunkΔH\Delta_HΔL\Delta_LΔH−ΔL\Delta_H-\Delta_L
3+2.92+0.41+2.51
5+1.39+0.00+1.39
7+7.29+0.17+7.12

Here, ΔH\Delta_H and ΔL\Delta_L denote the AP gains of high- and low-residual groups, respectively. The consistent differences indicate that reconstruction residuals reflect the need for additional adaptation capacity and provide a useful criterion for guiding expert expansion.

W2 and Q2: Effectiveness of expert expansion under matched capacity budgets

COVE outperforms alternative expansion strategies over the full streaming process under the same final expert budget. Fixed Expansion adds one expert at each expansion stage, while Random Expansion follows COVE's expansion schedule but randomly selects the same number of samples to train the new experts.

MethodMAP ↑MAF ↓Final AP ↑Final Experts
Fixed Expansion61.733.0962.756
Random Expansion61.892.7162.366
MoELoRA (6E)60.582.1662.076
StrLoRA (6E)62.443.1264.366
COVE63.870.6665.736

COVE achieves the best continual learning performance among these methods while all methods use the same final expert budget. We further compare their parameter and computational costs.

MethodAvg. Trainable Params (M)Final Total Params (M)Avg. Training Time per Chunk (s)
MoELoRA53.914769.352951.66
StrLoRA96.024977.323846.93
COVE22.654782.073444.96

COVE requires substantially fewer trainable parameters than MoELoRA and StrLoRA, while maintaining a comparable total model size. Its sparse-coefficient-based expert selection also avoids the additional sample-router and EMA teacher parameters used by StrLoRA. Although COVE incurs a modest runtime overhead over MoELoRA, it remains faster than StrLoRA.

Together, these comparisons show that COVE's gains come from more effective expansion decisions rather than simply increased model capacity.

W3 and Q3: Evaluation protocols and streaming benchmark construction

We follow the StrCVIT setting introduced in StrLoRA [1] and construct our own data streams using the task-arrival and sampling procedures described in that work. We clarify the task-specific evaluation protocols and data construction below.

Task-specific evaluation protocols. We evaluate COVE on 13 visual tasks with heterogeneous output formats and adopt task-specific metrics according to their output types.

TaskEvaluation Metric
Flickr30kCIDEr
TextCapsCIDEr
Grounding[email protected]
VQAv2VQA Accuracy
TextVQAVQA Accuracy
GQAExact-match Accuracy
RSVQA-HRExact-match Accuracy
OCRVQAExact-match Accuracy
CLEVRExact-match Accuracy
ChartQARelaxed Accuracy
ImageNet-RClassification Accuracy
Places365Relaxed Classification Accuracy
IconQAExact-match Accuracy

For image captioning, Flickr30k and TextCaps are evaluated using corpus-level CIDEr scores against their reference captions, with five official reference captions per image for TextCaps. For cross-task aggregation, raw CIDEr scores are mapped to a 0--100 scale using Scaption=clip(10×CIDEr,0,100)S_{\mathrm{caption}}=\mathrm{clip}(10\times\mathrm{CIDEr},0,100). For visual grounding, the model generates a bounding box in the format [x1,y1,x2,y2][x_1,y_1,x_2,y_2]. We calculate its IoU with the ground-truth box and consider predictions with IoU≥0.5IoU\geq 0.5 correct; malformed predictions are counted as incorrect. For VQAv2 and TextVQA, we use normalized answer matching with consensus-based scoring. The remaining short-answer tasks use normalized exact matching. ChartQA allows a 5% relative tolerance for numerical answers, while Places365 adopts the relaxed label-matching criterion implemented in our evaluator. All task-specific scores are mapped to a 0--100 scale before aggregation. The overall result is therefore the macro-average of normalized task-specific scores, which we refer to as Mean Normalized Task Score rather than conventional mean accuracy.

Dataset splits and stream construction. In the 10-chunk setting, the test set contains 500 samples per task (4,000 samples across eight tasks), while the training stream consists of 10,000 samples. Training samples are randomly drawn from the corresponding task datasets without replacement. Following the task-arrival and sampling procedures in StrLoRA [1], samples are allocated across chunks according to the specified task composition, with 1,000 samples per chunk. The resulting task composition of each chunk is reported in Appendix C. The total training data are not fixed across chunk settings: the 10-, 15-, and 20-chunk configurations contain 10,000, 15,000, and 20,000 training samples, respectively, while maintaining 1,000 samples per chunk. The training and test sets remain separate.

W4: Presentation issues

We thank the reviewer for pointing out these issues. We have proofread the manuscript and corrected the terminology inconsistencies, unresolved references, figure caption errors, grammatical mistakes, and awkward expressions to improve clarity and readability.

Reference

[1] Che, Chang, et al. "StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs." (2026).

Thank you for the detailed rebuttal and the additional experiments. The new sample-level results and capacity-matched comparisons substantially address my two main concerns.

Overall, I find the rebuttal convincing and am inclined to raise my rating to weak accept

We sincerely thank the reviewer for the thoughtful feedback and positive assessment of our rebuttal. We greatly appreciate your time and carefully consideration and wish you success with your submissions and future work.

Reviewer_RLxpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses `streaming continual visual instruction tuning' which learns a model on interleaved data stream with different kinds of data. The proposed method, named COVE, consists of re-usable "units". Either refining or expanding unit based on a threshold value. Empirical validations are done on a streamed task set of 8 tasks. The proposed COVE achieves better results than the prior arts.

Critical strengths
  • The problem set-up of interleaved/dynamically evolving data stream is closer to real-world setup.
  • Forgetting prevention (based on MAF) is superior than the prior arts.
Critical weaknesses
  • Novelty is limited: The COVE is a quite typical network expansion method for continual learning (De Lange et al., A continual learning survey: Defying forgetting in classification tasks, IEEE TPAMI 2021) and determining expansion is an online dictionary learning. It is a combination of existing components without a new insight for addressing the interleaved streaming data.
  • No parameter budget comparison. As the COVE may use more parameters by adding units than the other methods and the performance of the CL methods are heavily affected by the model budgets (Seo et al., Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling, ICLR 2025), the size of the final models should be compared with other models. If the model is larger than others, the empirical gain may be simply from the size of the models.
  • Empirical gain is marginal. In Table 1, the COVE's performance is slightly better than the second best methods despite the added computation of refining or expanding the units.
  • Presentation needs improvements
    • In Table 1, COVE does not perform the best in many tasks (e.g., in 4B model's TextVQA AP: VariGrow performs 76.93 while COVE performs 76.85. 4B model's ChartQA AF: MoELoRA (0.07) is much better than COVE (0.72).)
    • Figure 3 has no explanation of values, color legend, and number for each data chunk.
Questions for authors
  • Minor issues
    • Typo: L389: "A"nalysis.
4 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions.

W1: The methodological novelty of COVE is unclear.

We position COVE within the broader family of network expansion methods. However, COVE addresses an important yet insufficiently studied challenge in interleaved streaming data, where dynamically balancing knowledge reuse and capacity expansion as the proportions of multiple task distributions change across incoming chunks.

Unlike conventional task-incremental learning cited by reviewer, existing network expansion methods adopt different task-oriented expansion strategies, including PNN [1], which adds new network columns, Expert Gate [3], which introduces task-specific experts, and RCL [2], which uses reinforcement learning to determine filter expansion. However, these methods do not explicitly account for varying reuse and expansion demands within interleaved chunks, but primarily trigger expansion at task boundaries. To further evaluate their applicability to our setting, we adapt PNN and Expert Gate by treating each incoming chunk as an incremental stage and compare their performance with COVE. We also include VariGrow, a dynamic expansion method that triggers expert addition upon detecting novel samples, from our original experiments.

MethodMAP ↑MAF ↓Final AP ↑
PNN61.823.7163.54
Expert Gate62.112.0763.89
VariGrow59.707.1261.96
COVE63.870.6665.73

COVE achieves the best performance across all three metrics compared with these expansion baselines. In particular, although VariGrow triggers expert addition upon detecting novel samples, it does not adapt the number of new experts to the varying adaptation demands. In contrast, COVE uses reconstruction residuals to identify samples requiring additional adaptation capacity, while online dictionary learning determines the appropriate expansion scale for each incoming chunk. Furthermore, sparse coefficients enable expert selection without introducing an additional trainable router while maintaining competitive training efficiency.

Overall, our contribution lies not in following the existing expansion idea and technique combination, but in developing a demand-adaptive expansion mechanism that enables expert reuse and expansion to respond to the evolving requirements of interleaved streams, beyond predefined or fixed-step expansion strategies.

W2: Performance gain may result from a larger model capacity.

COVE achieves better performance without relying on a larger model capacity. Following the resource-budget perspective of Seo et al. (2025) cited by reviewer, which highlights the importance of controlling model capacity in continual learning comparisons, we report the parameter and computational costs of COVE, MoELoRA, and StrLoRA below.

MethodAvg. Trainable Params (M)Final Total Params (M)Avg. Training Time per Chunk (s)
MoELoRA53.914769.352951.66
StrLoRA96.024977.323846.93
COVE22.654782.073444.96

COVE uses substantially fewer trainable parameters than both baselines while maintaining a comparable final model size. Its sparse-coefficient-based expert selection avoids the additional sample-router and EMA teacher parameters used by StrLoRA. Although online dictionary learning introduces a modest runtime overhead over MoELoRA, COVE achieve better continual learning performance and still remains faster than StrLoRA.

Thus, its performance gains are not attributable to increased model capacity.

W3: The empirical gain is marginal despite the additional computation for refining and expanding the units.

COVE's additional computation yields consistent improvements in continual learning performance under the same final expert budget. The full-stream comparison is reported below.

MethodMAP ↑MAF ↓Final AP ↑Final Experts
Fixed Expansion61.733.0962.756
Random Expansion61.892.7162.366
MoELoRA (6E)60.582.1662.076
StrLoRA (6E)62.443.1264.366
COVE63.870.6665.736

COVE achieves the best MAP and Final AP and the lowest MAF among these methods under the same final expert budget. Moreover, across different backbones, COVE reduces MAF by 72.4% and 65.7% compared with StrLoRA on InternVL3.5-4B and 8B, respectively.

These results show that the performance gains are consistent across different streaming settings and model architectures, making the additional refinement and expansion worthwhile.

W4 and Q1: Presentation needs improvement.

We thank the reviewer for pointing out these presentation issues. We have revised the descriptions of Table 1, updated Figure 3 with clearer value explanations, color legends, and chunk indices, and corrected the typo at Line 389. All corresponding modifications are highlighted in the revised PDF.

References

[1] Rusu, Andrei A., et al. "Progressive neural networks." (2016).

[2] Xu, Ju, and Zhanxing Zhu. "Reinforced continual learning.'' (NeurIPS 2018).

[3] Aljundi, Rahaf, Punarjay Chakravarty, and Tinne Tuytelaars. "Expert gate: Lifelong learning with a network of experts.'' (CVPR 2017).

Reviewer_RLxpOpenReview ↗

Thank you for the clarification.

The reviewer appreciates the authors' efforts to clarify the points I raised as concerns. I verify that the size of the backbone model is not large for better performance. However, though I understand the application of network expansion to interleaved task setup may be new, the method itself is not novel. And the presented empirical gain is still marginal as the accuracy of CL methods are sensitive to many factors such as task configurations, and this marginal gain usually does not mandate the effectiveness of the method. So, I raise my score from 1 to 2.

We sincerely thank the reviewer for taking the time to reconsider our work.

We acknowledge that network expansion is a relatively established research direction and that many factors may affect continual learning (CL) performance. However, to further evaluate the robustness of COVE and examine whether its performance depends on specific experimental configurations, we have conducted diverse evaluations across different benchmark scales (8, 10, and 13 tasks) and streaming granularities (10, 15, and 20 chunks) in our original experiments, as reported in the main paper and Appendix E. The results are summarized in the following tables:

Performance comparison across different benchmark scales.

Method8 Tasks10 Tasks13 Tasks
MAP ↑MAF ↓MAP ↑MAF ↓MAP ↑MAF ↓
MoELoRA59.452.2755.653.2157.583.95
StrLoRA62.152.3957.733.2859.433.60
COVE (Ours)63.870.6658.741.5460.801.78

Performance comparison under different streaming granularities.

Method10 Chunks15 Chunks20 Chunks
MAP ↑MAF ↓MAP ↑MAF ↓MAP ↑MAF ↓
MoELoRA59.452.2758.743.1859.162.73
StrLoRA62.152.3961.623.1861.922.81
COVE (Ours)63.870.6663.470.9163.700.77

As shown in the tables, COVE consistently achieves the highest MAP in all settings,with more noticible gains when data has finer granularity (15/20 chunks, demonstrating that its performance advantage is maintained under different task compositions and streaming configurations rather than being sensitive to a particular setting. More importantly, COVE substantially reduces MAF compared with StrLoRA across all settings, with reductions ranging from 50.6% to 72.4% across different benchmark scales and from 71.4% to 72.6% across different streaming granularities. These substantial reductions in forgetting indicate that the improvement is far from marginal. Overall, these results demonstrate that COVE consistently improves continual learning performance while substantially mitigating catastrophic forgetting.

Finally, we greatly appreciate your time and careful consideration, and wish you success with your submissions and future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chair

We sincerely thank the reviewers and the Area Chair for their time, constructive feedback, and thoughtful discussions. Our work presents COVE, a framework that dynamically coordinates expert reuse and expansion according to the evolving adaptation demands of interleaved streaming data scenario, enabling effective continual adaptation while mitigating catastrophic forgetting.

During the rebuttal period, we carefully addressed the reviewers' main concerns through additional experiments and revise our manuscript. Specifically:

  • Validating the Effectiveness of Residual-Guided Expert Expansion. To verify whether reconstruction residuals can identify samples that genuinely benefit from additional adaptation capacity (DvQ3's W1, Q1 and Lh7T's W1), we conducted sample-level identification evaluations across three expansion chunks and compared the AP improvements of high- and low-residual samples after expert expansion. The results show that residual-guided identification achieves high F1 scores, while high-residual samples consistently obtain 1.39-7.12 points larger AP gains than low-residual samples among the expanded data chunk, validating the effectiveness and necessity of allocating additional expert capacity according to the varying adaptation demands of incoming samples.

  • Achieving Strong Performance with Reduced Forgetting under Matched Expert Budgets. To examine whether COVE's performance gains stem from its expansion strategy rather than increased model capacity (RLxp's W2, DvQ3's W2, Q2 and Lh7T's W1, Q2), we conducted full-stream comparisons under matched expert budgets and further evaluated parameter and computational costs against strong baselines. The results show that COVE achieves a higher MAP, outperforming fixed and random expansion by 2.14 and 1.98 points, respectively, while reducing MAF by 78.6% and 75.6%. Moreover, COVE requires 58.0% and 76.4% fewer trainable parameters than MoELoRA and StrLoRA, respectively, while maintaining a comparable total model size. These findings demonstrate that COVE achieves stronger average performance with substantially less forgetting, without relying on increased model capacity, and that its gains stem from its demand-adaptive expert reuse and expansion mechanism.

  • Clarifying the Methodological Contributions of COVE. We clarified the distinction between COVE and existing network expansion methods. Specifically, COVE identifies novel samples through reconstruction residuals, discovers new adaptation units via online dictionary learning, and determines the expansion scale directly from incoming data, while coordinating the reuse and selective refinement of historical experts through sparse coefficients. This allows COVE to adapt both expert selection and model capacity to varying data demands, achieving consistently strong adaptation performance with substantially reduced forgetting across different backbones and streaming configurations.

  • Improvements in Presentation and Clarity. We carefully revise the manuscript and corrected terminology inconsistencies, unresolved references, figure caption errors, grammatical mistakes, and unclear expressions. We also incorporated additional evaluation details and reporting information to improve the clarity and completeness of the manuscript.

We believe that we have addressed the majority of the reviewers' concerns during rebuttal period. Encouragingly, all three reviewers increased their ratings. Even though one reviewer retained some reservations regarding the methodological novelty and the magnitude of empirical gains, we have further clarified COVE's methodological contributions and demonstrated its consistent performance advantages across different benchmark scales and streaming granularities, with substantial reductions in catastrophic forgetting.

We have incorporated the corresponding clarifications and revisions into the manuscript. We sincerely appreciate the reviewers' constructive feedback and the Area Chair's consideration, which have helped us strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.