- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes D²C-MMCL, a multimodal continual learning framework for vision-language models. The proposed method separates task-specific parameter updates into visual and textual components, estimates modality reliability using predictive entropy, and consolidates knowledge across tasks through reliability-weighted reconstruction. The method is evaluated on the MTIL benchmark, consisting of 11 downstream tasks, under full-shot and few-shot settings with different task orders.
Critical strengths
-
The paper explores the dynamic and task-dependent contributions of visual and textual modalities in continual learning, providing an interesting perspective on multimodal knowledge preservation.
-
Compared to existing continual learning methods, the proposed approach achieves competitive or improved performance across multiple evaluation settings, particularly in retaining previously learned knowledge while preserving zero-shot generalization.
Critical weaknesses
[W1] The overall readability of the paper is quite low. I frequently had to revisit earlier and later sections to understand the proposed methodology. Several important concepts and operations are introduced without sufficient explanation, making it difficult to follow the main contributions.
In particular, Figure 1 raises several questions:
- How are the visual and textual contributions in the left plot measured? Are they calculated using Eq. (4)?
- What do the blue circles in the left plot represent? Their meaning is not clearly explained.
- The relationship between the left plot and the two performance plots on the right is unclear. The authors should explain how these figures jointly motivate the proposed method.
[W2] The paper claims to decouple knowledge along modality and task dimensions. However, the modality-level decoupling appears to rely primarily on the existing architectural separation between the visual and textual encoders of CLIP. It is unclear what additional decoupling is achieved beyond separating the parameter updates of these two encoders.
More importantly, the definition and measurement (design choices) of modality contribution are not sufficiently justified. In Eq. (1), the authors construct hybrid models by combining the current-task encoder of one modality with the previous-task encoder of the other modality. However, since the predictions depend on both encoders and their interactions, it is unclear whether the resulting predictive uncertainty can be attributed to the updated encoder alone.
[W3] Eq. (9) introduces a Consolidation operation, but its implementation is not sufficiently explained. Although Eq. (10) provides a reconstruction objective, several important details remain unclear, including the dimensionality and structural constraints of the knowledge space , and how the optimization is actually performed. These details are essential for understanding and reproducing the proposed method.
- The memory and computational costs of the proposed framework are insufficiently discussed. In particular, the dimensionality and storage requirements of are unclear, making it difficult to assess the memory overhead relative to the original CLIP encoders.
Questions for authors
Please refer weaknesses.
We thank the reviewer for the detailed feedback. These concerns stem from insufficient explanation of our method. We first present the complete pipeline, then address each point.
Pipeline. For each new task τt: (1) Fine-tune the pretrained VLM θ0 on τt to obtain task expert θt, and compute task delta δt=θt−θ0.(2) Decouple δt into visual and textual components: δvt=θvt−θv0 and δlt=θlt−θl0.(3) Use task triggers cmi(i=1,...,t−1) to reconstruct previous modality-specific deltas δ^mi from knowledge space Um1:t−1.(4) Integrate {δ^m1,…,δ^mt−1,δmt} with modality reliability weights to obtain updated knowledge space Um1:t and new task triggers cmi(i=1,...,t).
To obtain modality reliability weights, we construct two hybrid models: Mvt=(θvt,θ^lt−1) combines the visual encoder from the current model with the text encoder reconstructed from the previous knowledge space, and Mlt=(θ^vt−1,θlt) does the opposite(θ0 for t=1).
For each hybrid model evaluated on τt, per-sample reliability is ρm,it=1/max(hi,10−3), where hi=−(∑cpiclogpic)/logCt. Averaging over samples gives ρmt, which we normalize to πmt=ρmt/(ρvt+ρlt).
[W1] Figure 1. Left is our motivation experiment. The axes are visual and textual contributions Δvt and Δlt, measured with the hybrid models above as the accuracy gain over the previous-stage model θ^t−1: Δmt=Acc(Mmt;τt)−Acc(θ^t−1;τt),m∈{v,l}. Each point is one task: blue circles are visual-dominant (Δvt>Δlt), green squares are text-dominant, and orange triangles are balanced. It is not computed by Eq. (4): it uses the same hybrid models but reads out labeled accuracy, whereas Eq. (4) reads out predictive entropy. The plot shows that modality contributions vary across tasks, motivating D$^2$C-MMCL to estimate them per task via πmt for consolidation and reconstruction. Right reports Average--Transfer results and is independent of the left plot.
[W2] What modality-level decoupling adds. The visual and textual parameters are separated along the two CLIP encoders, which is the starting point. What we add is that the two modalities are then handled differently for each task. Each modality has its own knowledge space and task triggers, each task gets a separate weight for each modality, and at inference the two modality deltas are rescaled separately. Attribution in hybrid models. We agree that predictions depend on both encoders, so we do not claim the entropy reflects one encoder's intrinsic ability. What we measure is which modality's update matters more for the task given the model learned so far. Mvt and Mlt are built from the same task expert and the same previous-stage model, and differ only in which modality is updated. In each hybrid model, the other encoder stays at the accumulated state, so the updated encoder is evaluated together with accumulated knowledge. We then compare the two hybrid models to obtain the relative weight of the two modalities.
[W3] Implementation of Eq. (10). For each modality, we integrate historical and current task deltas by concatenating them column-wise after scaling each by its reliability weight wmk, then extract an orthonormal basis Um∈Rdout×r from this weighted concatenation, where rank r is adaptively selected to retain 99% cumulative energy. Task-specific information is encoded in trigger cmk∈Rr×din, enabling reconstruction δ^mk=Umcmk. After 11 tasks, the consolidated model requires 2.6 GB versus 6.3 GB for 11 independent task experts (58% compression).