- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
COVE addresses streaming continual visual instruction tuning, where each incoming data chunk may contain a changing mixture of tasks without explicit task identities. It maintains a sparse dictionary over frozen multimodal features, with each dictionary atom associated with a LoRA expert. Sparse coefficients determine how experts are combined, while reconstruction residuals guide the choice between refining existing experts and introducing new ones. Historical experts are updated selectively on reusable samples, and new experts are trained on samples identified as novel. The paper evaluates this approach across several backbones and streaming configurations.
Critical strengths
-
The method is well motivated for interleaved data streams. COVE makes reuse and expansion decisions from the relationship between individual samples and the accumulated dictionary, rather than assuming that an entire chunk belongs to a single task. Sparse coefficients and reconstruction residuals provide a coherent basis for these decisions. This is a useful alternative to expanding the model at predefined task boundaries.
-
The results improve performance and retention together. On InternVL3.5-4B, COVE achieves 63.87 MAP and 0.66 MAF, compared with 62.15 and 2.39 for StrLoRA. On the 8B model, the corresponding results are 64.73/0.81 versus 63.83/2.36. The gains therefore do not appear to come simply from sacrificing adaptation to reduce forgetting. The paper also states that all methods are reimplemented and evaluated on the same constructed streams, which supports the internal consistency of the comparison.
Critical weaknesses
- The experiments do not yet establish that residual-guided expansion allocates capacity more effectively than simpler alternatives. A large reconstruction residual indicates that the current dictionary represents a sample poorly, but it does not necessarily mean that the existing experts lack the capacity to learn the required task. Figure 3 shows an association between residuals and emerging distributions, followed by improved performance after expansion. However, the paper describes evaluating the same incoming chunk samples before and after vocabulary evolution, leaving unclear how much of the improvement reflects better generalization rather than additional capacity or training on those samples.
The ablations make this distinction important. Removing expansion leaves MAP almost unchanged, at 63.84 versus 63.87, while increasing MAF from 0.66 to 1.99. This supports a retention benefit, but it does not isolate the value of the residual-based expansion decision. Figure 4 includes matched expert budgets, but only for the first chunk. A comparison over the full stream, with the same expert-growth budget, would more directly test the proposed mechanism.
- The stop-gradient formulation appears to change more than the intended expert parameter updates. Equation 12 describes freezing historical factors and scaling the gradients of historical factors. Appendix H.3 instead applies stop-gradient to the entire expert contribution:
Questions for authors
- Does the implementation apply stop-gradient to the full expert output, as in Appendix H.3? Is the resulting change to upstream gradients intentional, and how does it relate to Equation 12?
- How many experts are retained at each stage, and how does residual-guided expansion compare with a simpler expansion schedule under the same growth budget?
- Does Figure 3(b) evaluate training samples or held-out samples? Under matched training and capacity, do high-residual samples benefit more from new experts than low-residual samples?
3 comments
Thank you for running the full-stream comparison. All three policies end with six experts, yet COVE achieves 63.87 MAP, compared with 61.73 for fixed expansion and 61.89 for random expansion, and also has lower forgetting. This addresses my main concern: the benefit depends on how COVE allocates capacity, not simply on how many experts it retains. The clarification that Figure 3(b) uses held-out samples also makes the larger gains for high-residual samples more convincing.
The revised Appendix H.3 resolves my concern about the gradient rule. It now distinguishes scaling the LoRA parameter gradients from scaling the gradient passed back through the expert input. I would encourage two small reporting changes: state in Section 5.3 that Figure 3(b) uses 500 held-out samples per task, and include the expert counts at each stage alongside the final counts so readers can see how the three expansion schedules differ.
These results and clarifications address my main reservations. I will raise my rating from 2 to 3 (weak acceptance).
We thank the reviewer for the positive assessment and constructive suggestions. We have incorporated both recommendations into the revised manuscript. We greatly appreciate your time and carefully consideration, and wish you success with your submissions and future work.
We sincerely thank the reviewer for the constructive comments and valuable suggestions.
W1 and Q2: Effectiveness of residual-guided expert expansion under matched expert budgets
COVE effectively identifies samples that benefit from additional adaptation and outperforms alternative expansion strategies under the same final expert budget. The sample-level identification results on the three expansion chunks are reported below.
Furthermore, high-residual samples consistently achieve larger AP gains than low-residual samples after expert expansion.
Here, ΔH and ΔL denote the AP gains of high- and low-residual groups, respectively.
We further compare COVE with Fixed Expansion and Random Expansion over the full data stream. Both baselines reach the same final expert budget as COVE, while removing its residual-guided expansion decisions.
COVE achieves better MAP, MAF, and Final AP than both alternatives under the same expert-growth budget. Its parameter and computational costs are also lower than those of StrLoRA in terms of trainable parameters and training time, while maintaining a comparable total model size.
Overall, these results demonstrate that COVE effectively identifies samples requiring additional adaptation and allocates expert capacity more effectively without relying on a larger model or more trainable parameters.
W2 and Q1: Clarification of the Stop-Gradient Implementation
We thank the reviewer for identifying this ambiguity in our stop-gradient formulation. Our intention is to selectively control expert parameter updates rather than modify gradient propagation through the entire expert contribution.
Specifically, for historical experts, LoRA-A matrices are frozen, while LoRA-B gradients for reusable samples are scaled according to their expert responsibilities:
\leftarrow \rho_{ij} \nabla_{B\\ _j^{(\ell)}} \mathcal{L}\\ _i, \qquad \nabla_{A\\ _j^{(\ell)}} \mathcal{L}\\ _i = 0, \quad i \in \mathcal{R}\\ _t.$$ Novel samples do not update historical experts, while newly introduced experts are updated using novel samples only. These operations directly control expert parameter gradients without blocking gradient propagation through expert inputs. Upon re-examining our implementation, we confirmed that all reported experiments follow the intended parameter-level update strategy. Specifically, the stop-gradient formulation in Appendix H.3 was retained from an earlier version of our implementation, whereas the code used for the reported experiments had already adopted the parameter-level gradient control described above. We inadvertently overlooked updating this part of the appendix when revising the implementation. We have now corrected Appendix H.3 to align its formulation with the actual implementation and the gradient update mechanism described in Eq. 12. **Q3: Evaluation on held-out samples and benefits of expert expansion** Figure 3(b) evaluates held-out test samples, with 500 samples per task, rather than the training samples used to update newly introduced experts. Thus, the observed improvements reflect performance on unseen evaluation samples. Furthermore, as reported in table above, high-residual samples consistently achieve larger AP gains than low-residual samples after expert expansion. These results further support the effectiveness of reconstruction residuals in identifying samples that benefit from additional adaptation capacity.