acceptodds
Under review as a conference paper at ICLR 2027

D²C-MCL: Dual Decoupling and Consolidation for Multimodal Continual Learning in Vision-Language Models

Abstract

Continual learning for Vision-Language Models (VLMs) aims to acquire new downstream knowledge while preserving previously learned knowledge and pre-trained zero-shot capabilities. However, existing methods largely overlook the evolving roles of different modalities during continual learning. We observe that the relative contributions of visual and textual modalities are inherently dynamic and task-dependent, varying as new tasks are introduced and knowledge accumulates. Motivated by this observation, we propose D²C-MMCL(Dual Decoupling and Consolidation for Multimodal Continual Learning), a dual-decoupling and knowledge-consolidation framework that disentangles continual knowledge along modality and task dimensions. By dynamically modeling modality contributions, D²C-MMCL performs modality-aware cross-task consolidation and selectively exploits modality-specific knowledge for different tasks.Extensive experiments across various continual learning settings demonstrate that our method consistently outperforms strong existing approaches, achieving superior continual learning performance while better preserving zero-shot generalization.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Continual Learning, Vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kV6pOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D²C-MMCL, a multimodal continual learning framework for vision-language models. The proposed method separates task-specific parameter updates into visual and textual components, estimates modality reliability using predictive entropy, and consolidates knowledge across tasks through reliability-weighted reconstruction. The method is evaluated on the MTIL benchmark, consisting of 11 downstream tasks, under full-shot and few-shot settings with different task orders.

Critical strengths
  • The paper explores the dynamic and task-dependent contributions of visual and textual modalities in continual learning, providing an interesting perspective on multimodal knowledge preservation.

  • Compared to existing continual learning methods, the proposed approach achieves competitive or improved performance across multiple evaluation settings, particularly in retaining previously learned knowledge while preserving zero-shot generalization.

Critical weaknesses

[W1] The overall readability of the paper is quite low. I frequently had to revisit earlier and later sections to understand the proposed methodology. Several important concepts and operations are introduced without sufficient explanation, making it difficult to follow the main contributions.

In particular, Figure 1 raises several questions:

  • How are the visual and textual contributions in the left plot measured? Are they calculated using Eq. (4)?
  • What do the blue circles in the left plot represent? Their meaning is not clearly explained.
  • The relationship between the left plot and the two performance plots on the right is unclear. The authors should explain how these figures jointly motivate the proposed method.

[W2] The paper claims to decouple knowledge along modality and task dimensions. However, the modality-level decoupling appears to rely primarily on the existing architectural separation between the visual and textual encoders of CLIP. It is unclear what additional decoupling is achieved beyond separating the parameter updates of these two encoders.

More importantly, the definition and measurement (design choices) of modality contribution are not sufficiently justified. In Eq. (1), the authors construct hybrid models by combining the current-task encoder of one modality with the previous-task encoder of the other modality. However, since the predictions depend on both encoders and their interactions, it is unclear whether the resulting predictive uncertainty can be attributed to the updated encoder alone.

[W3] Eq. (9) introduces a Consolidation operation, but its implementation is not sufficiently explained. Although Eq. (10) provides a reconstruction objective, several important details remain unclear, including the dimensionality and structural constraints of the knowledge space (Um)(U_m), and how the optimization is actually performed. These details are essential for understanding and reproducing the proposed method.

  • The memory and computational costs of the proposed framework are insufficiently discussed. In particular, the dimensionality and storage requirements of (Um)(U_m) are unclear, making it difficult to assess the memory overhead relative to the original CLIP encoders.
Questions for authors

Please refer weaknesses.

1 comment

We thank the reviewer for the detailed feedback. These concerns stem from insufficient explanation of our method. We first present the complete pipeline, then address each point.

Pipeline. For each new task τt\tau^t: (1) Fine-tune the pretrained VLM θ0\theta^0 on τt\tau^t to obtain task expert θt\theta^t, and compute task delta δt=θt−θ0\delta^t=\theta^t-\theta^0.(2) Decouple δt\delta^t into visual and textual components: δvt=θvt−θv0\delta^t_v=\theta^t_v-\theta^0_v and δlt=θlt−θl0\delta^t_l=\theta^t_l-\theta^0_l.(3) Use task triggers cmic^i_m(i=1,...,t−1i=1,...,t-1) to reconstruct previous modality-specific deltas δ^mi\hat{\delta}^i_m from knowledge space Um1:t−1U^{1:t-1}_m.(4) Integrate {δ^m1,…,δ^mt−1,δmt}\{\hat{\delta}^1_m,\ldots,\hat{\delta}^{t-1}_m,\delta^t_m\} with modality reliability weights to obtain updated knowledge space Um1:tU^{1:t}_m and new task triggers cmic^i_m(i=1,...,ti=1,...,t).

To obtain modality reliability weights, we construct two hybrid models: Mvt=(θvt,θ^lt−1)M^t_v=(\theta^t_v,\hat{\theta}^{t-1}_l) combines the visual encoder from the current model with the text encoder reconstructed from the previous knowledge space, and Mlt=(θ^vt−1,θlt)M^t_l=(\hat{\theta}^{t-1}_v,\theta^t_l) does the opposite(θ0\theta^0 for t=1t=1).

For each hybrid model evaluated on τt\tau^t, per-sample reliability is ρm,it=1/max⁡(hi,10−3)\rho^t_{m,i} = 1 / \max(h_i, 10^{-3}), where hi=−(∑cpiclog⁡pic)/log⁡Cth_i = - (\sum_c p_{ic} \log p_{ic}) / \log C^t. Averaging over samples gives ρmt\rho^t_m, which we normalize to πmt=ρmt/(ρvt+ρlt)\pi^t_m = \rho^t_m / (\rho^t_v + \rho^t_l).

[W1] Figure 1. Left is our motivation experiment. The axes are visual and textual contributions Δvt\Delta^t_v and Δlt\Delta^t_l, measured with the hybrid models above as the accuracy gain over the previous-stage model θ^t−1\hat{\theta}^{t-1}: Δmt=Acc(Mmt;τt)−Acc(θ^t−1;τt),m∈{v,l}.\Delta_m^t=\mathrm{Acc}(M_m^t;\tau^t)-\mathrm{Acc}(\hat{\theta}^{t-1};\tau^t),\quad m\in\{v,l\}. Each point is one task: blue circles are visual-dominant (Δvt>Δlt\Delta^t_v>\Delta^t_l), green squares are text-dominant, and orange triangles are balanced. It is not computed by Eq. (4): it uses the same hybrid models but reads out labeled accuracy, whereas Eq. (4) reads out predictive entropy. The plot shows that modality contributions vary across tasks, motivating D$^2$C-MMCL to estimate them per task via πmt\pi^t_m for consolidation and reconstruction. Right reports Average--Transfer results and is independent of the left plot.

[W2] What modality-level decoupling adds. The visual and textual parameters are separated along the two CLIP encoders, which is the starting point. What we add is that the two modalities are then handled differently for each task. Each modality has its own knowledge space and task triggers, each task gets a separate weight for each modality, and at inference the two modality deltas are rescaled separately. Attribution in hybrid models. We agree that predictions depend on both encoders, so we do not claim the entropy reflects one encoder's intrinsic ability. What we measure is which modality's update matters more for the task given the model learned so far. MvtM^t_v and MltM^t_l are built from the same task expert and the same previous-stage model, and differ only in which modality is updated. In each hybrid model, the other encoder stays at the accumulated state, so the updated encoder is evaluated together with accumulated knowledge. We then compare the two hybrid models to obtain the relative weight of the two modalities.

[W3] Implementation of Eq. (10). For each modality, we integrate historical and current task deltas by concatenating them column-wise after scaling each by its reliability weight wmk\sqrt{w^k_m}, then extract an orthonormal basis Um∈Rdout×rU_m\in\mathbb{R}^{d_{out}\times r} from this weighted concatenation, where rank rr is adaptively selected to retain 99% cumulative energy. Task-specific information is encoded in trigger cmk∈Rr×dinc^k_m\in\mathbb{R}^{r\times d_{in}}, enabling reconstruction δ^mk=Umcmk\hat{\delta}^k_m=U_mc^k_m. After 11 tasks, the consolidated model requires 2.6 GB versus 6.3 GB for 11 independent task experts (58% compression).

Reviewer_vYr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes D²C-MMCL, a multimodal continual learning framework for vision-language models. The method independently fine-tunes a pretrained CLIP model for each incoming task, decomposes the resulting parameter updates into visual and textual components, and estimates their relative contributions using the predictive entropy of hybrid models. These modality-specific task updates are then consolidated into shared knowledge spaces, from which task-specific models are reconstructed at inference time. Experiments on the MTIL benchmark under different task orders and few-shot settings show improvements in knowledge retention and zero-shot transfer compared with existing methods.

Critical strengths

The method achieves competitive results on the MTIL benchmark under different task orders and few-shot settings, with particularly noticeable improvements in final-task retention accuracy. These results suggest that the proposed consolidation framework can help preserve previously acquired knowledge.

Critical weaknesses
  1. Unclear necessity and practical significance of task-specific reconstruction: The method independently fine-tunes a VLM for each task, yet subsequently consolidates and reconstructs these task experts using task-specific triggers at inference. The advantages over directly using the original experts remain unclear. If storage efficiency is the motivation, checkpoints can often reside in secondary storage, and no storage–performance trade-off is provided. If knowledge sharing is the motivation, a direct comparison with independent task experts is essential.

  2. Questionable entropy-based modality importance estimation: The proposed reliability measure uses inverse predictive entropy, but confidence does not necessarily reflect accuracy or a modality's actual contribution, particularly when combining encoders from different learning stages. The mutual information and Fano's inequality discussion does not establish this connection for model predictive entropy, leaving the central assumption insufficiently justified.

  3. Insufficient evidence for the contribution of dynamic modality weighting. Removing Selective Reconstruction yields only marginal performance drops, suggesting a limited benefit from inference-time modality-aware scaling. More importantly, the current ablations do not isolate entropy-based weighting during cross-task consolidation. A controlled comparison retaining the same framework but using uniform consolidation weights is needed to determine whether the gains stem from dynamic modality estimation or simply from parameter consolidation.

Questions for authors

Please refer to the Critical Weaknesses.

4 comments

Thank you for your detailed feedback. We address each concern below:

  1. Under the continual learning setting, the system can often only retain a single VLM during training. The purpose of the reconstruction is to retain the behaviour of multiple task specialists with limited auxiliary memory. Our consolidation extracts the shared structure of all task deltas into a modality-specific basis UmU_m and stores the task-specific differences as compact triggers cmkc^k_m, so that a task-specific model is reconstructed on demand instead of being kept in memory. In terms of storage, after 11 tasks in Order-I the consolidated model (bases plus all triggers) requires 2.62.6 GB, compared to 6.36.3 GB for keeping 11 independent task experts, i.e. a 58% reduction. We have also added the direct comparison with independent task experts that the reviewer requested. Independent FT keeps one dedicated expert per task and reaches an average accuracy of 89.2% (62.0, 95.1, 89.6, 79.5, 98.9, 97.5, 92.7, 99.6, 94.7, 89.6, 81.8 on the 11 tasks in order-I), which we report as an oracle upper bound rather than a competing setting, since it requires storing all experts and selecting the correct one at test time.
  2. Our reliability measure is theoretically grounded and addresses exactly what continual consolidation requires. We construct two hybrid models from the same task expert and the same accumulated knowledge: Mvt=(θvt,θ^lt−1)M^t_v = (\theta^t_v, \hat{\theta}^{t-1}_l) updates only the visual encoder, while Mlt=(θ^vt−1,θlt)M^t_l = (\hat{\theta}^{t-1}_v, \theta^t_l) updates only the text encoder. This controlled design isolates the effect of updating each modality while building on the same knowledge base θ^t−1\hat{\theta}^{t-1}. The predictive entropy H(Y∣X;Mmt)H(Y \mid X; M^t_m) measures the residual uncertainty after observing the representation from modality mm. By the mutual information identity I(X;Y)=H(Y)−H(Y∣X)I(X; Y) = H(Y) - H(Y \mid X), lower entropy directly implies higher mutual information between the learned representation and the task labels, quantifying exactly how much task-relevant discriminative structure the updated modality has acquired. Our normalized inverse entropy \pi^t_m \propto 1/\{h}^t_m is monotonic in this mutual information, making it a principled measure of relative modality contribution. Regarding the concern about "combining encoders from different learning stages", we are not comparing encoders across tasks; we are comparing, for the same task τt\tau^t, whether updating the visual or language modality contributes more when both build on the same accumulated state θ^t−1\hat{\theta}^{t-1}.
  3. SR is an inference-time refinement applied on top of the consolidated model, so its effect is expected to be modest: the consolidation already determines what knowledge the shared subspaces capture, and SR only rescales the reconstruction per task. Its value is therefore complementary rather than additive, and it requires no extra training or storage. Following the reviewer's suggestion, we conducted a controlled ablation that retains the entire framework but replaces the reliability weights in Eq. (10) with uniform weights. Here, RWRW denotes the reliability weight, and the results are shown below.
SettingMethodAircraftCaltech101CIFAR100DTDEuroSATFlowersFoodMNISTOxfordPetCarsSUN397Average
Individual FT62.095.189.679.598.997.592.799.694.789.681.889.2
AverageOurs58.4895.2884.3069.4983.9484.8489.6277.1089.7068.5765.5978.81
-w/o RW57.4694.1882.9567.8882.7483.3288.5775.2988.8767.7963.8777.54
LastOurs58.3695.9188.0579.1098.6596.8392.0499.5394.7489.0181.2388.50
-w/o RW57.4294.7486.8677.0397.5795.0691.1699.5493.8587.2779.6687.29

The consistent gain of +1.27+1.27 Average and +1.21+1.21 Last shows that the gains come from dynamic modality estimation rather than from parameter consolidation alone.When consolidating 11 tasks into shared visual and textual subspaces, uniform weighting treats all modality contributions equally, forcing UvU_v and UlU_l to compromise across heterogeneous tasks. Entropy-based weighting (ρ^t_m) allows the consolidation to prioritize reliable modality signals—for instance, emphasizing visual structure for EuroSAT (satellite imagery). This adaptive prioritization reduces interference and improves the quality of the shared knowledge space.

Thank you for your response. I accept that modality-aware weighting improves performance within the proposed shared framework. Both in my understanding of continual learning and in the manuscript’s own formulation, the goal is to learn new tasks while retaining prior capabilities and preserving zero-shot generalization. Reducing checkpoint storage is not itself a learning objective. The proposed method already uses task identity to select a trigger, so the same information can be used to load an independent expert from cold storage on demand. In the authors’ own comparison, independent experts achieve higher final accuracy (89.2% vs. 88.5%). The reported 58% reduction has little practical meaning without its absolute scale: storage falls from 6.3 GB to 2.6 GB, a saving of only 3.7 GB across 11 tasks. Cold storage is often not a bottleneck in practice, and at the scale evaluated here I see no practical need to sacrifice accuracy for that saving. I therefore judge the practical value of the proposed trade-off to be limited and maintain my original rating.

Thank you for this thoughtful comment. We recognize that storage reduction is not itself a primary objective of continual learning, that independent task experts can be loaded on demand when storage is not a constraint, and that Individual FT achieves higher accuracy than our method in the reported comparison.

We would like to emphasize that storage efficiency is not the primary motivation of D²C-MMCL, but rather an additional benefit of its knowledge consolidation mechanism. Our main objective is to dynamically model task-dependent modality contributions and consolidate visual and textual knowledge across tasks, while preserving previously acquired knowledge and zero-shot generalization. Unlike independently maintaining a separate expert for each task, D²C-MMCL organizes task knowledge within shared modality-specific spaces and uses task-specific triggers for reconstruction.

The practical significance of storage reduction depends on the application scenario, and independent task experts remain a viable alternative when storage is not a concern. We do not intend to suggest that storage efficiency outweighs accuracy, but rather view it as an additional benefit of the proposed knowledge consolidation mechanism.We appreciate your feedback, which has helped us better articulate the intended contribution and scope of our method.

Thank you for your response. The rebuttal still misses my central point. I accept that modality-aware weighting improves performance once the shared framework is in place. My objection is to the case for adopting that framework in the first place. The response simply restates how the method shares parameters and reconstructs experts. By the authors’ own account, storage savings are an ancillary benefit, and their only direct performance comparison favors independent experts (89.2% vs. 88.5%). With the same task identity available at inference, those experts can be loaded on demand. The rebuttal offers no evidence that sharing yields any advantage over this baseline in adaptation, retention, or zero-shot transfer. I therefore maintain my original rating.

Reviewer_YmjvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual learning for VLM. The main focus of this paper is how the relative importance of vision and text varies across tasks. The proposed DC-MMCL separates each task's parameter update into visual and textual components, estimates the reliability of each modality to guide knowledge retention, and uses a task trigger to retrieve the previous task-related expert. At inference, it adjusts the parameter updates according to the modality contributions of the target task. The framework therefore handles knowledge retention during training and modality adjustment at inference. Experiments are conducted on 11 cross-domain tasks. DC-MMCL outperforms the best competitor by a margin. The ablations demonstrate contributions of each proposed component. Also, the t-SNE provides further evidence of better feature clustering and knowledge retention across tasks. The sensitivity analysis also shows that β=0.2\beta=0.2 performs best among the tested settings.

Critical strengths
  1. Existing modality-aware continual learning (CL) methods have already recognized that visual and textual branches behave differently during adaptation. However, most focus on asymmetric updates or compensating for modality gaps. This paper instead asks how much each modality contributes to learning the current task and how these contributions change as learning progresses. It then makes modality as an indictor that can guide knowledge retention during training and modality calibration at inference. It turns modality asymmetry from an observation into a dynamic factor that can be modeled and acted upon. I believe this is a meaningful insight in the CL community.

  2. The separation between training and inference is supported by experiments across relevant settings. Training focuses on consolidating and retaining modality-specific knowledge across tasks, while inference adjusts the relative influence of each modality according to the target task. The experiments also cover a broad range of settings and multiple datasets. The ablations show that both components contribute to performance. Overall, the experiments have demonstrated the effectiveness of the proposed method.

Critical weaknesses
  1. Modality reliability is the central idea in this paper. For example, CCTD uses it to control how much of each task’s modality delta is retained, and SR uses it to rescale the reconstructed delta at inference. However, the paper does not explain precisely how πkm\pi_k^m is computed -- which signals are used, whether extra forward passes or task labels are needed, or what the computational cost is. The reference to Fano’s inequality does not justify the proposed estimator. The inequality relates conditional label uncertainty to classification error, but the paper does not explain why inverse normalized predictive entropy of the cross-decoupled hybrid models is a reliable cue for H(Y∣Zm)H(Y\mid Z^m), or when it can correctly rank modality contributions. This matters because any error in πkm\pi_k^m affects both CCTD and SR. If the score reflects model confidence rather than actual modality contribution, the reported gains from these components are difficult to interpret.

  2. After modality decoupling, all task deltas for each modality are reconstructed in a shared subspace UmU^m, with task identity represented only by the ckmc_k^m. If task-specific directions cannot be well represented in this shared space, updates from different tasks may still interfere. The paper provides no detailed conditions or error bounds of UmU^m.

Questions for authors

see Critical Weaknesses.

1 comment

We appreciate the detailed comments and address each concern point by point below:

On the computation and justification of πmt\pi_m^t: we first present the complete pipeline, for each new task τt\tau^t: (1) Fine-tune the pretrained VLM θ0\theta^0 on τt\tau^t to obtain task expert θt\theta^t, and compute task delta δt=θt−θ0\delta^t=\theta^t-\theta^0.(2) Decouple δt\delta^t into visual and textual components: δvt=θvt−θv0\delta^t_v=\theta^t_v-\theta^0_v and δlt=θlt−θl0\delta^t_l=\theta^t_l-\theta^0_l.(3) Use task triggers cmic^i_m(i=1,...,t−1i=1,...,t-1) to reconstruct previous modality-specific deltas δ^mi\hat{\delta}^i_m from knowledge space Um1:t−1U^{1:t-1}_m.(4) Integrate {δ^m1,…,δ^mt−1,δmt}\{\hat{\delta}^1_m,\ldots,\hat{\delta}^{t-1}_m,\delta^t_m\} with modality reliability weights to obtain updated knowledge space Um1:tU^{1:t}_m and new task triggers cmic^i_m(i=1,...,ti=1,...,t).

To obtain modality reliability weights, we construct two hybrid models: Mvt=(θvt,θ^lt−1)M^t_v=(\theta^t_v,\hat{\theta}^{t-1}_l) combines the visual encoder from the current model with the text encoder reconstructed from the previous knowledge space, and Mlt=(θ^vt−1,θlt)M^t_l=(\hat{\theta}^{t-1}_v,\theta^t_l) does the opposite(θ0\theta^0 for t=1t=1).

For each hybrid model evaluated on τt\tau^t, per-sample reliability is ρm,it=1/max⁡(hi,10−3)\rho^t_{m,i} = 1 / \max(h_i, 10^{-3}), where hi=−(∑cpiclog⁡pic)/log⁡Cth_i = - (\sum_c p_{ic} \log p_{ic}) / \log C^t. Averaging over samples gives ρmt\rho^t_m, which we normalize to πmt=ρmt/(ρvt+ρlt)\pi^t_m = \rho^t_m / (\rho^t_v + \rho^t_l).

The theoretical justification is that by the mutual information identity I(X;Y)=H(Y)−H(Y∣X)I(X; Y) = H(Y) - H(Y \mid X), lower entropy directly implies higher mutual information between the learned representation and the task labels, quantifying exactly how much task-relevant discriminative structure the updated modality has acquired. The controlled hybrid-model design ensures that πmt\pi^t_m reflects the relative contribution of updating each modality on the same task and knowledge base, making it a principled relative ranking rather than an absolute accuracy predictor.

On the shared subspace UmU_m, For each modality, we integrate historical and current task deltas by concatenating them column-wise after scaling each by its reliability weight, then extract an orthonormal basis Um∈Rdout×rU_m\in\mathbb{R}^{d_{out}\times r} from this weighted concatenation, where rank rr is adaptively selected to retain 99% cumulative energy. Task-specific information is encoded in trigger cmk∈Rr×dinc^k_m\in\mathbb{R}^{r\times d_{in}}, enabling reconstruction δ^mk=Umcmk\hat{\delta}^k_m=U_mc^k_m. Because each task maintains its own trigger cmkc^k_m that independently encodes its task-specific structure in the shared basis, tasks do not interfere through the subspace representation itself. The weighted integration explicitly prioritizes more reliable modality updates during basis construction, and the adaptive rank selection ensures that the subspace grows to accommodate new task directions as long as they contribute meaningful variance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.