acceptodds
Withdrawn from ICLR 2027

Regularizing Modality Contribution Drift in Multimodal Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal continual learning (MMCL) aims to acquire new knowledge from multimodal data while retaining previously learned knowledge. Existing MMCL methods primarily mitigate forgetting by aligning cross-modal representations or preserving feature-level semantic similarity. However, different tasks may rely on different modalities, and learning new tasks can alter how modalities contribute to predictions on previously learned tasks. It remains underexplored how modality contributions evolve across incremental stages and how such changes relate to forgetting in MMCL. We term such changes Modality Contribution Drift (MCD) and introduce an MCD score based on controlled modality-subset interventions. Our theoretical and empirical analyses show how contribution drift can lead to forgetting and reveal that existing MMCL and conventional CL methods do not effectively mitigate MCD. To address this issue, we propose Continual Modality Contribution Drift Regularization (CMCDR) to preserve the modality contribution profiles of previously learned tasks in both replay-based and replay-free settings. In the replay-based setting, CMCDR estimates modality contributions from stored old samples and regularizes their drift relative to a frozen previous model. In the replay-free setting, CMCDR uses current-task samples as probes to match old-class contribution profiles between the current and frozen models without storing old exemplars. Across six benchmarks covering multimodal class-incremental learning and continual multimodal question answering, CMCDR substantially reduces modality contribution drift, improves average accuracy by percentage points, and reduces average forgetting by percentage points over the corresponding baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal,Continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QqusOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Modality Contribution Drift in multimodal continual learning that influences the contribution of individual modalities and cross-modal interactions to predictions previously learned tasks. It proposes an intervention-based contribution estimator using modality-subset masking and a Möbius decomposition, then defines an MCD metric combining absolute contribution-magnitude drift and normalized profile drift. The paper further introduces Continual Modality Contribution Drift Regularization that match current contribution profiles to those of a frozen previous model. Experiments report improvements in accuracy, forgetting across continual recognition and multimodal QA.

Critical strengths
  1. A clear contribution-preservation perspective for MMCL. The paper identifies a distinct failure mode beyond representation drift: modality-specific and interaction contributions can change even when multimodal representations are comparatively stable.

  2. The intervention estimator explicitly separates singleton modality effects from higher-order interactions through Möbius inversion, with the decomposition exactly recovering the full-input benefit relative to the no-modality response.

Critical weaknesses
  1. The theoretical evidence for MCD remains only partially established.
  • The theoretical guarantee does not establish a causal role of contribution drift in forgetting. Proposition 1 provides only a conditional margin-based retention certificate. Prediction can still change due to drift in the no-modality baseline or later-class competition even when the contribution profile is preserved. Thus, the claimed connection between contribution drift and forgetting is weaker than the presentation suggests.

  • The replay-free guarantee relies on strong and unverified distributional assumptions. Proposition 4 assumes that contribution drift is Lipschitz and that current-task probes adequately cover the relevant old-task distribution. The paper does not provide a practical way to bound the Lipschitz constant or the Wasserstein distribution gap, making the resulting guarantee potentially vacuous.

  • Proposition 2 has a formal inconsistency in the stated drift bound. Eq. (10) does not consistently use absolute values for the contribution drift, although the subsequent proof and discussion rely on absolute differences. The inequality therefore requires clarification or an additional assumption on the drift terms to be mathematically valid.

  1. The demonstrated effectiveness and efficiency of CMCDR are narrower than the general formulation suggests.
  • The method requires modality-subset interventions, with the manuscript stating that exact contribution profiles require 2∣M∣2^{|M|} subset evaluations and that computational cost therefore grows exponentially with the number of modalities. The concrete scalability experiment only moves from two to three modalities. On UESTC-MMEA-CL, CMCDR increases training time by 34.4–52.7%, while moving from two to three modalities increases total CMCDR training time by 35.8% for iCaRL and 60.8% for LwF. This demonstrates a meaningful practical overhead while leaving substantially larger modality counts untested.

  • The replay-free formulation relies on current-task samples as probes for every old-class output, making its effectiveness dependent on how informative current-task probes are about old-class contribution profiles. The manuscript does not provide a direct sensitivity analysis over probe distribution or task relatedness.

Questions for authors
  1. What performance and computational behavior should be expected as the number of modalities grows beyond three?

  2. The method may over-penalize beneficial modality adaptation. It assumes that preserving previous contribution profiles is desirable, but contribution drift is not necessarily harmful. New tasks may legitimately change modality reliance and enable positive forward transfer. The paper does not directly evaluate whether CMCDR sacrifices plasticity or beneficial transfer by suppressing such changes. More analyses about this should be incorporated and forward transfer should be evaluated and reported.

  3. The methodological novelty is relatively incremental. The method can be viewed as teacher-student distillation of modality-subset decision responses, with Möbius decomposition providing the target representation. The conceptual advantage of explicitly regularizing Möbius contributions over simpler response-level distillation remains insufficiently established.

  4. The experiments does not fully reflect the broader and more recent multimodal continual-learning landscape. It does not compare against more recent multimodal continual instruction-tuning settings[1], nor does it evaluate a broader range of modern MLLM-based continual-learning methods. This makes it difficult to assess the generalization ability beyond the relatively task-specific MMCL/VQA settings considered in the paper.

[1] MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark

6 comments
AuthorsOpenReview ↗

Response to Reviewer Qqus

Thank you for recognizing the contribution-preservation perspective. The revised appendix adds contribution-restoration analysis in C.2, higher-modality approximation and experiments in C.3, old/new-task comparisons and FWT in C.4, replay-free distribution-shift results in C.5, and modern-MLLM comparisons in C.8. All additions are marked in blue.

Theoretical basis and replay-free probes

Contribution drift acts on quantities directly supporting old-task decisions. Proposition 1 connects contribution changes to old-class decision margins. Appendix A.3, Eqs. (22)–(23), and Appendix A.5, Eq. (35), additionally account for new-class competition and provide retention conditions over the full class space. We further restore reference-stage contributions while fixing the current no-modality response and all competing-class scores. Across iCaRL and LwF on AVE and UESTC-MMEA-CL, this reduces error by 10.2–25.3 percentage points (pp), corresponding to 55.3%–60.8% of the net error increase. With the other decision scores held fixed, this diagnostic directly demonstrates the contribution term's influence on predictions and provides a decision-level basis for preserving it.

Current-task probes provide useful signals for preserving old knowledge. Section 3.3 compares old-class contributions of the frozen and current models on the same current inputs, without storing old examples. Proposition 4, Eq. (38), relates old-task contribution drift to current-probe loss and a distribution-discrepancy term. Appendix C.5, Figure 10, reports current-/old-task probe correlations of 0.785 and 0.724 on AVE and UESTC-MMEA-CL, respectively, showing that current probes identify old classes with greater contribution drift. To further test retention under distribution shift, the additional AVE experiment preserves the task split and progressively increases Gaussian video blur, σt=2(t−1)/(T−1)\sigma_t=2(t-1)/(T-1), with audio unchanged. Each task is tested at its introduction-stage blur strength.

Table R1. Replay-free learning under AVE video distribution shift.

MethodAverage accuracy (%) ↑Average forgetting (%) ↓Probe Spearman ρ ↑Normalized gap (%) ↓
LwF53.837.50.7115.4
LwF + CMCDR58.227.30.7915.2

CMCDR improves accuracy by 4.40 pp and reduces forgetting by 10.20 pp, with probe correlation reaching 0.79. These results connect drift identification to actual retention gains, showing that old-class contribution signals from current data remain effective for replay-free continual learning under the tested distribution shift.

Notation. The definition of Δg\Delta_g in Eq. (9) and the contribution difference in Eq. (10), on page 5 of the current manuscript, already contain absolute values and agree with the proof in Appendix A.4.

Q1. How do performance and computational cost change beyond three modalities?

Thank you for raising the question of multimodal scalability. The original manuscript focused on exact contribution estimation and did not sufficiently address efficient scaling to higher modality counts. To address this, we introduce a new approximate contribution estimator for higher modality counts, reducing subset-evaluation cost through a compressed contribution profile, and evaluate its retention gains and training efficiency with four and five modalities.

New approximate estimator. We retain exact singleton and pairwise contributions, aggregate higher-order contributions, and add sampled interaction constraints. By contribution completeness (original manuscript, Section 2.1, Eq. (4)), the higher-order remainder is

=\sum_{|T|\geq3}C(T).$$ Thus, matching low-order contributions and $R_H$ preserves the complete contribution sum without evaluating every higher-order interaction. **Supplementary sampling.** Since individual higher-order changes may cancel within $R_H$, we sample $K=2$ three-modality coalitions and match their third-order contributions between frozen and current models. Each requires only one additional joint-response evaluation, reusing all lower-order responses. The total budget is at most $2+M+\binom{M}{2}+K$ subset evaluations per input and model, reducing the cost from $2^M$ to **$O(M^2+K)$**. **New experiment.** Using the 37-class cross-subject [MMAct](https://openaccess.thecvf.com/content_ICCV_2019/papers/Kong_MMAct_A_Large-Scale_Dataset_for_Cross_Modal_Human_Action_Understanding_ICCV_2019_paper.pdf) setting, we integrate iCaRL with the [MMTSA](https://pi.cs.tsinghua.edu.cn/wp-content/uploads/2023/10/MMTSA.pdf) encoder and fusion architecture under a nine-stage class-incremental protocol (5+4×8), with a total replay budget of 370. Four modalities include RGB, phone acceleration, watch acceleration, and gyroscope; the fifth adds orientation. All methods share synchronized samples, class order, and training settings. The approximation includes sampled-triple constraints. **Table R2. MMAct performance and complete-stage training time at four and five modalities (new experiment).** | M | Method | Avg. accuracy (%) ↑ | Avg. forgetting (%) ↓ | Minutes/stage ↓ | Overhead (%) ↓ | | :---: | :--- | ---: | ---: | ---: | ---: | | 4 | iCaRL | 70.2 | 24.1 | 48.7 | — | | 4 | + Exact CMCDR | 75.6 | 17.1 | 72.3 | +48.5 | | 4 | + Approx. CMCDR | 74.2 | 17.4 | 61.8 | +26.9 | | 5 | iCaRL | 72.9 | 24.3 | 55.3 | — | | 5 | + Exact CMCDR | 77.2 | 17.1 | 103.5 | +87.2 | | 5 | + Approx. CMCDR | 76.4 | 17.5 | 74.6 | +34.9 | At four/five modalities, approximate CMCDR improves baseline accuracy by **4.00/3.50 pp** and reduces forgetting by **6.70/6.80 pp**, while reducing training time by **14.5%/27.9%** relative to exact CMCDR. At five modalities, accuracy differs by only 0.80 pp, while training time decreases from 103.5 to 74.6 minutes. These results demonstrate effective contribution preservation at reduced cost. **The intervention-ratio experiment already in the original manuscript (Appendix C.3, Table 7)** provides a complementary efficiency control by reducing the number of intervention samples under replay and replay-free protocols. **Table R3. Intervention sample ratio and training cost on AVE (original manuscript, Appendix C.3, Table 7).** | Protocol | Method | Ratio | Avg. accuracy ↑ | Avg. forgetting ↓ | MCD ↓ | Minutes/stage ↓ | Overhead (%) ↓ | | :--- | :--- | :---: | ---: | ---: | ---: | ---: | ---: | | Replay | iCaRL | — | 64.20 | 25.60 | 0.779 | 24.7 | — | | Replay | iCaRL + CMCDR | 25% | 70.46 | 18.14 | 0.452 | 29.0 | +17.4 | | Replay | iCaRL + CMCDR | 50% | 70.89 | 17.41 | 0.421 | 31.2 | +26.3 | | Replay | iCaRL + CMCDR | 75% | 71.03 | 17.72 | 0.402 | 33.4 | +35.2 | | Replay | iCaRL + CMCDR | 100% | 71.86 | 17.36 | 0.389 | 37.3 | +51.0 | | Replay-free | LwF | — | 56.61 | 35.11 | 0.842 | 18.3 | — | | Replay-free | LwF + CMCDR | 25% | 57.91 | 28.99 | 0.474 | 22.6 | +23.5 | | Replay-free | LwF + CMCDR | 50% | 58.27 | 28.01 | 0.468 | 24.0 | +31.1 | | Replay-free | LwF + CMCDR | 75% | 58.73 | 27.88 | 0.433 | 25.7 | +40.4 | | Replay-free | LwF + CMCDR | 100% | 60.61 | 26.35 | 0.401 | 27.4 | +49.7 | On AVE/iCaRL, a 25% intervention ratio achieves a **6.26 pp accuracy gain with 17.4% overhead**, retaining **81.7%** of the full-intervention gain. At the same ratio, LwF reduces forgetting by **6.12 pp** with 23.5% overhead. **Overall, CMCDR maintains clear accuracy and retention gains with four and five modalities.** The approximation reduces computation per probe, while intervention subsampling reduces the number of probes, providing complementary efficiency controls under different training budgets.

Q2. Does preserving old contributions hinder beneficial adaptation or positive transfer? What about forward transfer?

The paired class-wise results already in the original manuscript (Appendix C.2, Figure 7) show that contribution preservation benefits the large majority of old classes. Figure 7 connects each class under the baseline and CMCDR on AVE, showing changes in both MCD and forgetting. CMCDR reduces MCD for all displayed classes and reduces forgetting for 19/21 iCaRL classes and 20/21 LwF classes. Tables R4–R5 further quantify the effect on new-task adaptation.

Figure 7 also contains a few classes with lower MCD but higher forgetting, showing that the effects of contribution changes vary across classes. In the original method (Section 3), CMCDR uses contribution preservation as a soft constraint jointly optimized with the new-task objective, with its strength controlled by λ. Benefit shifts shared across all modality subsets cancel from nonempty Möbius contributions, leaving such output changes unconstrained by contribution matching (already in the original manuscript, Appendix C.6).

Table R4. Regularization strength and new-task learning on UESTC-MMEA-CL (already in the original manuscript, Appendix C.4, Table 8) FWT is additionally reported here, in percentage points.

MethodλAvg. accuracy ↑Avg. forgetting ↓MCD ↓New-task accuracy ↑FWT ↑ (pp)
iCaRL0.173.41820.4640.38782.3691.26
iCaRL0.574.83417.5770.33581.6921.22
iCaRL1.075.12017.1520.28881.2571.13
iCaRL2.074.70716.6630.25479.9761.02
LwF0.120.15844.7240.42832.1790.76
LwF0.522.91339.1160.35331.5680.59
LwF1.022.85138.6480.29930.9120.44
LwF2.022.16737.9790.26229.7150.32

Table R4 identifies an effective λ=0.5–1 range. At λ=0.5, new-task accuracy is higher than at λ=1. At λ=1, iCaRL achieves its highest overall accuracy, while LwF is within 0.062 pp of its best result.

Table R5. Direct comparison against λ=0 on UESTC-MMEA-CL. New-task results at λ=0.5 and λ=1 are from the original Appendix C.4, Table 8. The remaining columns are additional comparisons.

MethodNew, λ=0New, λ=0.5New, λ=1Old, λ=0Old, λ=1
iCaRL83.481.69281.25765.272.6
LwF33.231.56830.9128.318.1

The new comparison (Table R5) shows old-task accuracy gains of 7.40/9.80 pp and new-task accuracy changes of −2.14/−2.29 pp for iCaRL/LwF at λ=1. The original results (Table 12) show overall accuracy gains of 3.29/5.72 pp.

Together, class-wise coverage and old/new-task comparisons show that an appropriate preservation strength balances new-task adaptation and old-knowledge retention, translating retention gains into better overall continual-learning performance.

Forward transfer. Table R4 additionally reports FWT, comparing target-task performance before training with performance at the common initialization, following GEM, Eq. (4). At λ=0.5 and λ=1, iCaRL achieves 1.22 and 1.13 pp FWT, while LwF achieves 0.59 and 0.44 pp. Positive FWT means that learning earlier tasks helps target tasks before they are trained, showing that moderate contribution preservation maintains positive forward transfer while improving overall accuracy.

Reviewer_YCmHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Modality Contribution Drift (MCD) in multimodal continual learning, where the contributions of modalities to old-task predictions change over time. The authors introduce a modality-subset intervention-based metric and propose CMCDR to preserve previous contribution profiles through regularization. The method is applied to both replay-based and replay-free settings, showing improved accuracy and reduced forgetting across six benchmarks.

Critical strengths

The paper offers an interesting perspective on catastrophic forgetting through modality-specific and cross-modal contributions.

The proposed method is applicable to both replay-based and replay-free continual learning

Critical weaknesses
  1. Some methods [1–3] have already investigated changes in modality reliance or decision attribution during continual learning. In particular, [1] preserves evidence-reliance profiles using counterfactual interventions and teacher–student regularization, which is highly similar to CMCDR. The authors should clarify the novelty of their method and provide more direct comparisons.

  2. Preserving previous modality contributions may restrict beneficial adaptation to new tasks. It is unclear whether the proposed regularization can distinguish harmful contribution drift from changes that improve old-task performance.

  3. The replay-free approach estimates old-class contribution profiles from current-task samples. Its reliability under substantial distribution shifts between tasks remains unclear.

  4. The computational cost increases exponentially with the number of modalities. Recent studies [4, 5] consider continual learning across richer modality spaces. For five modalities, CMCDR requires 32 modality-subset evaluations, which may introduce non-negligible computational overhead. Its scalability to a larger number of modalities remains unclear.


[1] Chen et al., Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails, 2026.

[2] Cai et al., Is Our Continual Learner Reliable? Investigating Its Decision Attribution Stability through SHAP Value Consistency, CVPRW, 2024.

[3] Nikandrou et al., Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation, ALVR, 2024.

[4] Liu et al., Continual Multimodal Contrastive Learning, NeurIPS, 2025.

[5] Jin and Kim, Continual Learning for Multiple Modalities, 2025.

Questions for authors

Please refer to the weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer YCmH

Thank you for recognizing the value of studying forgetting through modality and interaction contributions. The revised appendix adds old/new-task comparisons in C.4, the replay-free distribution-shift experiment in C.5, and the higher-modality approximation with four- and five-modality results in C.3. All additions are marked in blue.

W1. What distinguishes CMCDR from prior reliance and attribution preservation?

Our core contribution is to formalize and theoretically analyze drift in singleton and cross-modal interaction contributions, and develop a contribution-preservation method accordingly. Proposition 1 characterizes how contribution drift affects old-task decision margins and prediction retention, while Proposition 2 shows why representation stability does not guarantee contribution stability. CMCDR separately preserves singleton and interaction contributions, with its effectiveness validated through controlled comparisons under matched intervention budgets.

RCL (new literature comparison). RCL shows that evidence attribution can drift even when predictions remain correct, and preserves normalized evidence-reliance profiles estimated through channel suppression. In contrast, CMCDR investigates how drift in singleton and cross-modal interaction contributions affects old-task discrimination and forgetting. While channel suppression captures aggregate response changes, CMCDR uses Möbius decomposition to explicitly separate and constrain singleton and interaction contributions, preserving both their values and relative composition (original manuscript, Sections 2.1 and 3).

SHAPC (new literature comparison). SHAPC evaluates decision-attribution consistency in unimodal image continual learning using SHAPC-Mean and SHAPC-Var. CMCDR instead addresses multimodal continual learning, theoretically connects singleton and interaction contribution drift to forgetting, and explicitly preserves these contributions during training.

MAFED (Appendix D, expanded here). MAFED preserves intermediate visual and textual token features through squared-L2 distillation, using balanced modality losses (MAFED-B) or gradient-based weights (MAFED-A). CMCDR instead preserves individual modality and interaction contributions to old-class decisions. Since downstream fusion and decision maps can change these contributions despite stable features (Proposition 2), CMCDR directly constrains the contribution structure underlying old-task discrimination.

Controlled comparisons with matched intervention budgets further validate this distinction. SubsetKD and CMCDR use identical intervention samples, modality subsets, and teacher/student evaluations, preserving subset-wise output distributions and contribution structure, respectively.

Table R1. Matched-budget comparison with SubsetKD (original manuscript, Appendix C.6, Table 12).

ProtocolMethodAVE Acc. ↑AVE Forg. ↓UESTC Acc. ↑UESTC Forg. ↓
ReplayiCaRL64.2025.6071.8322.86
ReplayiCaRL + SubsetKD66.7423.6372.4621.61
ReplayiCaRL + CMCDR71.8617.3675.1217.15
Replay-freeLwF56.6135.1117.1349.37
Replay-freeLwF + SubsetKD57.8933.7819.8646.57
Replay-freeLwF + CMCDR60.6126.3522.8538.65

Across both datasets and protocols, CMCDR improves accuracy over SubsetKD by 2.66–5.12 percentage points (pp) and reduces forgetting by 4.46–7.92 pp, supporting the contribution-preservation target under matched budgets.

The parameterization comparison (original manuscript, Appendix C.6, Table 11) further compares centered subset responses with the complete Möbius profile. Although related by an invertible linear transform and containing identical subset information, explicit contribution matching improves accuracy by 1.15–1.73 pp and reduces forgetting by 1.53–2.06 pp under matched evaluations, loss form, and tuning budget. This decomposition penalizes individual contribution changes that may cancel between singleton and interaction terms in joint responses, supporting the value of both the preservation target and interaction parameterization.

W2. Does preserving old contributions hinder beneficial adaptation?

The paired class-wise results already in the original manuscript (Appendix C.2, Figure 7) show that contribution preservation benefits the large majority of old classes. Figure 7 connects each class under the baseline and CMCDR on AVE, showing changes in both MCD and forgetting. CMCDR reduces MCD for all displayed classes and reduces forgetting for 19/21 iCaRL classes and 20/21 LwF classes. Tables R2–R3 further quantify the effect on new-task adaptation.

Figure 7 also contains a few classes with lower MCD but higher forgetting, showing that the effects of contribution changes vary across classes. In the original method (Section 3), CMCDR uses contribution preservation as a soft constraint jointly optimized with the new-task objective, with its strength controlled by λ. Benefit shifts shared across all modality subsets cancel from nonempty Möbius contributions, leaving such output changes unconstrained by contribution matching (already in the original manuscript, Appendix C.6).

Table R2. Regularization strength and new-task learning on UESTC-MMEA-CL (already in the original manuscript, Appendix C.4, Table 8)

MethodλAvg. accuracy ↑Avg. forgetting ↓MCD ↓New-task accuracy ↑
iCaRL0.173.41820.4640.38782.369
iCaRL0.574.83417.5770.33581.692
iCaRL1.075.12017.1520.28881.257
iCaRL2.074.70716.6630.25479.976
LwF0.120.15844.7240.42832.179
LwF0.522.91339.1160.35331.568
LwF1.022.85138.6480.29930.912
LwF2.022.16737.9790.26229.715

Table R3. Direct comparison against λ=0 on UESTC-MMEA-CL (original results + new results)

The new-task columns at λ=0.5 and λ=1 are already in the original manuscript (Appendix C.4, Table 8). The new-task column at λ=0 and both old-task columns at λ=0 and λ=1 are new results.

MethodNew, λ=0New, λ=0.5New, λ=1Old, λ=0Old, λ=1
iCaRL83.481.69281.25765.272.6
LwF33.231.56830.9128.318.1

The new comparison (Table R3) shows old-task accuracy gains of 7.40/9.80 pp and new-task accuracy changes of −2.14/−2.29 pp for iCaRL/LwF at λ=1. The original results (Table 12) show overall accuracy gains of 3.29/5.72 pp.

Table R2 also identifies an effective λ=0.5–1 range. At λ=0.5, new-task accuracy is 81.692%/31.568% for iCaRL/LwF. At λ=1, iCaRL achieves its highest overall accuracy, while LwF is within 0.062 pp of its best result on this dataset. Together, class-wise coverage and old/new-task comparisons show that an appropriate preservation strength delivers larger old-task retention gains at a quantified new-task adaptation cost, improving overall continual-learning performance.

W3. Can current-task probes effectively preserve old knowledge under inter-task distribution shift?

CMCDR obtains useful old-class contribution-preservation signals from current-task data and improves accuracy and retention under the tested distribution shift.

The replay-free mechanism already in the original manuscript (Section 3.3) uses current inputs to probe old-class decision responses. We compare the frozen and current models using the same inputs, old-class targets, candidate set, and modality interventions to constrain changes in old-class contribution structure. Training uses current-task data without storing old exemplars.

First, the probe validation already in the original manuscript (Appendix C.5, Figure 10) compares class-wise contribution drift measured on current-task and old-task probes at each incremental stage.

Table R4. Drift agreement between current-task and old-task probes (already in the original manuscript, Appendix C.5, Figure 10)

DatasetSpearman ρ ↑Absolute MCD gap ↓Normalized gap (%) ↓
AVE0.7850.08813.7
UESTC-MMEA-CL0.7240.11415.2

Spearman correlations of 0.785/0.724 on AVE/UESTC-MMEA-CL show that current-task probes identify old classes with larger contribution drift. Old-task inputs serve this offline validation, while current-task data provide the training constraint.

Second, the replay-free LwF results already in the original manuscript (Table 12, reproduced as Table R1) show accuracy gains of 4.00/5.72 pp and forgetting reductions of 8.76/10.72 pp on AVE/UESTC-MMEA-CL.

In a new distribution-shift experiment, we retain the AVE class split and task order while increasing Gaussian blur on video frames across stages, σt=2(t−1)/(T−1)\sigma_t=2(t-1)/(T-1), with the original audio. Task k is tested at its introduction-time blur level, and teacher/student use identical probe transformations.

Table R5. Replay-free learning under AVE video distribution shift (new experiment)

MethodAvg. accuracy (%) ↑Avg. forgetting (%) ↓Probe ρ ↑Normalized gap (%) ↓
LwF53.837.50.7115.4
LwF + CMCDR58.227.30.7915.2

Under this shift, CMCDR increases accuracy from 53.8% to 58.2% and reduces forgetting from 37.5% to 27.3%, improvements of 4.40/10.20 pp, respectively. Probe correlation reaches 0.79, with a normalized gap of 15.2%.

Current-task probes therefore enable CMCDR to improve accuracy and reduce forgetting without storing old exemplars, with retention gains maintained under the tested video distribution shift.

W4. Does exponential subset evaluation prevent practical scaling?

Thank you for raising the question of multimodal scalability. The original manuscript focused on exact contribution estimation and did not sufficiently address efficient scaling to higher modality counts. To address this, we introduce a new approximate contribution estimator for higher modality counts, reducing subset-evaluation cost through a compressed contribution profile, and evaluate its retention gains and training efficiency with four and five modalities.

New approximate estimator. We compress the contribution profile into first- and second-order contributions and an aggregate higher-order remainder:

R_H=B(\mathcal M)-B(\emptyset)-\sum_{1\leq|T|\leq2}C(T).$$ By **contribution completeness already in the original manuscript (Section 2.1, Eq. (4))**, $R_H=\sum_{|T|\geq3}C(T)$, so the compressed profile still exactly reconstructs the full-input benefit relative to the empty-input response. Since higher-order changes may cancel within $R_H$, we additionally sample $K=2$ three-modality coalitions and match their third-order contributions between frozen and current models. Each requires only one additional joint-response evaluation because lower-order responses are already available. The total cost is at most $2+M+\binom{M}{2}+K$ subset responses per input and model, yielding **$O(M^2+K)$** complexity. **New experiment.** Using the 37-class cross-subject setting of [MMAct](https://openaccess.thecvf.com/content_ICCV_2019/papers/Kong_MMAct_A_Large-Scale_Dataset_for_Cross_Modal_Human_Action_Understanding_ICCV_2019_paper.pdf), we integrate iCaRL with the [MMTSA](https://pi.cs.tsinghua.edu.cn/wp-content/uploads/2023/10/MMTSA.pdf) encoder and fusion architecture under a nine-stage class-incremental protocol (5+4×8) and a total replay budget of 370. Four modalities include RGB, phone acceleration, watch acceleration, and gyroscope; the fifth adds orientation. All methods share synchronized samples, class order, and training settings. Approximate CMCDR includes the sampled-triple constraint. **Table R6. MMAct performance and complete-stage training time at four and five modalities (new experiment).** | M | Method | Avg. accuracy (%) ↑ | Avg. forgetting (%) ↓ | Minutes/stage ↓ | Overhead (%) ↓ | | :---: | :--- | ---: | ---: | ---: | ---: | | 4 | iCaRL | 70.2 | 24.1 | 48.7 | — | | 4 | + Exact CMCDR | 75.6 | 17.1 | 72.3 | +48.5 | | 4 | + Approx. CMCDR | 74.2 | 17.4 | 61.8 | +26.9 | | 5 | iCaRL | 72.9 | 24.3 | 55.3 | — | | 5 | + Exact CMCDR | 77.2 | 17.1 | 103.5 | +87.2 | | 5 | + Approx. CMCDR | 76.4 | 17.5 | 74.6 | +34.9 | At four/five modalities, approximate CMCDR improves baseline accuracy by **4.00/3.50 pp** and reduces forgetting by **6.70/6.80 pp**, while reducing training time by **14.5%/27.9%** relative to exact CMCDR. At five modalities, accuracy differs by only 0.80 pp, with training time decreasing from 103.5 to 74.6 minutes. These results demonstrate effective preservation through low-order contributions, aggregate higher-order effects, and sampled interactions at reduced cost. **The intervention-ratio experiment already in the original manuscript (Appendix C.3, Table 7)** provides a second budget control, examining intervention frequency, performance, and training cost under replay and replay-free protocols. **Table R7. Intervention sample ratio and training cost on AVE (original manuscript, Appendix C.3, Table 7).** | Protocol | Method | Ratio | Avg. accuracy ↑ | Avg. forgetting ↓ | MCD ↓ | Minutes/stage ↓ | Overhead (%) ↓ | | :--- | :--- | :---: | ---: | ---: | ---: | ---: | ---: | | Replay | iCaRL | — | 64.20 | 25.60 | 0.779 | 24.7 | — | | Replay | iCaRL + CMCDR | 25% | 70.46 | 18.14 | 0.452 | 29.0 | +17.4 | | Replay | iCaRL + CMCDR | 50% | 70.89 | 17.41 | 0.421 | 31.2 | +26.3 | | Replay | iCaRL + CMCDR | 75% | 71.03 | 17.72 | 0.402 | 33.4 | +35.2 | | Replay | iCaRL + CMCDR | 100% | 71.86 | 17.36 | 0.389 | 37.3 | +51.0 | | Replay-free | LwF | — | 56.61 | 35.11 | 0.842 | 18.3 | — | | Replay-free | LwF + CMCDR | 25% | 57.91 | 28.99 | 0.474 | 22.6 | +23.5 | | Replay-free | LwF + CMCDR | 50% | 58.27 | 28.01 | 0.468 | 24.0 | +31.1 | | Replay-free | LwF + CMCDR | 75% | 58.73 | 27.88 | 0.433 | 25.7 | +40.4 | | Replay-free | LwF + CMCDR | 100% | 60.61 | 26.35 | 0.401 | 27.4 | +49.7 | On AVE/iCaRL, a 25% intervention ratio achieves a **6.26 pp accuracy gain with 17.4% overhead**, retaining **81.7%** of the full-intervention gain. At the same ratio, LwF reduces forgetting by **6.12 pp** with 23.5% overhead. **Overall, CMCDR retains substantial accuracy and forgetting improvements at four and five modalities.** The new approximation reduces computation per probe, while intervention subsampling reduces the number of probes, providing complementary efficiency controls for multimodal continual learning under constrained computing budgets.
Reviewer_h1tKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies changes in the contributions of individual modalities and their interactions during multimodal continual learning. It constructs intervention-based contribution profiles through a Mobius/Harsanyi decomposition of target-versus-competitor log-odds and defines a Modality Contribution Drift (MCD) score. The proposed Continual Modality Contribution Drift Regularization (CMCDR) matches raw and normalized contribution profiles between the current model and a frozen previous model. Replay-based CMCDR uses old exemplars, while replay-free CMCDR uses current-task inputs to probe old-class outputs. Experiments cover six benchmarks spanning class-incremental recognition and continual multimodal question answering. The reported results show improvements in accuracy and forgetting across multiple baselines and complementarity with representation-preservation methods.

Critical strengths

A valuable decision-level preservation target supported by targeted controls. CMCDR jointly preserves individual modality contributions and interaction contributions within a unified regularization framework for replay-based and replay-free learning. It improves accuracy by 1.15–1.73 percentage points over centered-response matching and by 2.66–5.12 points over SubsetKD under matched computational budgets. These comparisons strengthen the empirical case for the selected contribution-preservation objective.

Critical weaknesses
  1. The additional explanatory value of contribution drift for forgetting has not been sufficiently validated. MCD is constructed from log-odds responses and combines changes in absolute contribution magnitude and relative composition. Its value can therefore change with confidence even when predicted labels remain unchanged. For example, positive uniform logit rescaling in a binary classifier changes raw contribution magnitudes without changing predictions. MCD can consequently reflect both confidence changes and changes in modality reliance, whose relationships with forgetting need to be distinguished more clearly. Proposition 1 provides a conditional sufficient retention statement within a fixed candidate set, whereas actual class-incremental predictions are also affected by changes in the no-modality response and competition from newly introduced classes. Although the existing analyses support an association between MCD and forgetting, they do not sufficiently quantify the relative roles of these factors. The coarse stratification used in the analysis also leaves potential differences in initial prediction margins, logit scale, and class characteristics. The paper should establish how much additional explanatory value MCD provides after accounting for these factors. This gap weakens the central motivation and the mechanistic interpretation of the results.

  2. The attribution of gains and implementation details remain unclear for some baseline integrations. The paper presents CMCDR as an added regularizer that preserves the original architectures, losses, and training protocols. However, integrating it with MMAL additionally introduces prediction-head copies and gradient-based refinement after the analytic update. It is therefore unclear how much of MMAL's improvement comes from contribution preservation and how much comes from the extra head optimization. A tightly matched control should use the same head copies, ridge objectives, optimizer, and refinement schedule, while setting the CMCDR coefficient to zero. In addition, the exact per-baseline optimization configurations, training epochs, and update counts for MMAL's additional refinement need explicit specification. Generative VQACL scores candidate answers under multiple modality-subset interventions, while the reported timing experiments mainly cover classification settings. Training-time and peak-memory measurements for this QA implementation are needed to assess its practical cost and the scope of the claimed plug-and-play applicability. These issues concern particular integrations and the breadth of validation; they do not invalidate every baseline comparison.

Questions for authors
  1. Contribution drift and confidence changes: Can the authors separately analyze how changes in contribution magnitude and relative contribution composition relate to forgetting, while accounting for initial prediction margins, logit scale, class, and incremental stage? How much of the observed old-class error increase is associated with contribution drift after considering changes in the no-modality response and later-class competition? These results would directly affect my assessment of the proposed mechanism. 
  2. MMAL attribution control: What results are obtained when MMAL undergoes exactly the same prediction-head copying and gradient-based ridge refinement as MMAL + CMCDR, with only the CMCDR coefficient set to zero? Please specify the refinement objective, optimizer, number of updates, and how the refined heads interact with the next analytic update. 
  3. Reproducibility and QA cost: Can the authors provide complete numerical training configurations for each baseline and precise definitions and aggregation rules for average accuracy and average forgetting? For generative VQACL, please report candidate-answer set sizes across stages, whether all candidates are scored at every intervention update, and the resulting training-time and peak-memory overhead. These details would materially affect my assessment of practical generality and reproducibility.
5 comments
AuthorsOpenReview ↗

# Response to Reviewer h1tK

Thank you for recognizing the value of decision-level contribution preservation. The revised appendix adds controlled contribution analyses in C.2, the matched MMAL refinement control in C.6, training and evaluation details in B.2, QA implementation details in B.3, and QA cost comparisons in C.3. All additions are marked in blue.

Q1. The explanatory value of contribution structure and its basis for prediction retention

MCD captures both evidence strength and its allocation across singleton and interaction contributions, linking contribution drift to prediction retention through decision margins. Unlike full-input confidence, contribution decomposition reveals the evidence structure supporting predictions. Controlled analyses and composition-matching gains support jointly preserving contribution magnitude and composition.

(1) Contribution decomposition reveals evidence allocation and provides a margin-based foundation for retention. By Eq. (4),

Bs(M)−Bs(∅)=1⊤cs.B_s(M)-B_s(\varnothing)=\mathbf1^\top\mathbf c_s.

The full-input log-odds benefit equals the sum of signed contributions, while their vector records evidence allocation. For example, profiles (2,1,1)(2,1,1) and (1,1,2)(1,1,2), representing two singleton terms and their interaction, share a contribution sum and L1L_1 magnitude of 4 but differ in composition. With fixed candidate sets and no-modality responses, they yield identical full-input confidence despite different evidence structures.

Proposition 1 connects contribution drift to prediction retention:

γt≥mr→tkeep−∥ct−cr∥1.\gamma_t\ge m^{\rm keep}_{r\to t}-\|\mathbf c_t-\mathbf c_r\|_1.

If drift is smaller than the retained contribution margin, the old sample remains correctly classified within the fixed candidate set.

We measure magnitude and composition drift on identical old-class samples:

\qquad D_{\rm comp}=\|\widehat{\mathbf c}_t-\widehat{\mathbf c}_r\|_1, \qquad \widehat{\mathbf c}_s=\frac{\mathbf c_s}{\|\mathbf c_s\|_1+\epsilon}.$$ The reference stage and candidate set are fixed at class introduction. **(2) Composition drift remains associated with forgetting after controlling for magnitude and confidence-related factors.** Table R1 reports Pearson partial correlations with old-class error change, controlling for the other drift component, initial prediction margin, initial class accuracy, logit-scale changes, and class/stage effects. Correlations are computed between residuals after regressing each variable on the same controls. **Table R1. Pearson partial correlations between drift and old-class error change.** UESTC denotes UESTC-MMEA-CL. | Dataset | Baseline | Magnitude partial $r$ | Composition partial $r$ | |---|---|---:|---:| | AVE | iCaRL | 0.13 | 0.16 | | AVE | LwF | 0.16 | 0.21 | | UESTC | iCaRL | 0.09 | 0.11 | | UESTC | LwF | 0.13 | 0.12 | Magnitude and composition show positive partial correlations of **0.09–0.16 and 0.11–0.21**, respectively, indicating complementary associations with forgetting. **Composition matching further improves retention (Table 10).** Adding normalized-composition matching to raw contribution preservation improves AVE accuracy by **2.59 pp (iCaRL) and 1.15 pp (LwF)**, while reducing forgetting by **3.02 and 2.83 pp**. These results support preserving both magnitude and relative evidence allocation. **(3) Contribution restoration consistently reduces full-class-space errors.** We restore reference contributions for labeled old-class samples while retaining the current no-modality response and all competing-class scores: $$\widetilde z_{t,y}=z_{t,y}-\mathbf1^\top(\mathbf c_t-\mathbf c_r), \qquad \widetilde z_{t,\ell}=z_{t,\ell}\quad(\ell\ne y).$$ Evaluation includes **all classes, including later classes** (Appendix A.3). Reference error $E_r$, current error $E_t$, and restored error $E_t^{\rm keep}$ use identical samples. We define $$\Delta E=E_t-E_r,\qquad \Delta E_{\rm contrib}=E_t-E_t^{\rm keep},\qquad R=100\,\Delta E_{\rm contrib}/\Delta E.$$ **Table R2. Contribution restoration and prediction errors.** Error changes are in percentage points. | Dataset | Baseline | Net error increase (pp) | Error reduction after restoration (pp) | Recovery ratio (%) | |---|---|---:|---:|---:| | AVE | iCaRL | 22.8 | 12.6 | 55.3 | | AVE | LwF | 32.4 | 19.7 | 60.8 | | UESTC | iCaRL | 18.4 | 10.2 | 55.4 | | UESTC | LwF | 41.9 | 25.3 | 60.4 | Restoration reduces full-class-space error by **10.2–25.3 pp**, recovering **55.3–60.8% of the net error increase**, despite unchanged competing-class scores. Together, the margin bound, controlled drift associations, composition-matching gains, and restoration results establish **contribution structure as an explanatory and practical target for mitigating old-task forgetting**.

Q2. Do the MMAL gains come from CMCDR or extra prediction-head optimization?

The reviewer notes that MMAL originally uses analytic updates, whereas integrating CMCDR introduces prediction-head copying and gradient-based ridge refinement. The specific question is therefore how MMAL performs when all these extra optimization steps are retained and only CMCDR regularization is disabled, and how much additional benefit CMCDR provides over that control.

The matched control shows that CMCDR further improves accuracy and reduces forgetting under the same prediction-head refinement. To answer this question directly, we compare MMAL + CMCDR with a refinement-only control obtained by setting λcmcdr=0\lambda_{\rm cmcdr}=0. Both share the analytic initialization, copied heads, frozen features, normalized ridge objectives, and state and reference construction. The optimizer, learning rate, minibatch order, update count, and checkpoint selection are also matched. Both use current-task data without old exemplars.

Refinement objective and updates. With feature extractors frozen, we minimize the sum of the branch-wise ridge losses normalized by sample count (squared fitting error and L2 regularization, η=1\eta=1) and CMCDR. The fusion and compensation branches retain their respective MMAL fitting targets. Following Table S2, we use Adam with a fixed learning rate of 10−310^{-3}, batch size 256, and weight decay 10−410^{-4}. Each stage uses 200 refinement epochs, giving 200×∣loadert∣200\times|\mathrm{loader}_t| updates, where ∣loadert∣|\mathrm{loader}_t| is the number of current-task minibatches per epoch.

Interaction with the next analytic update. The analytic recursion state and refined heads are stored separately. The refined heads serve inference at the current stage and become the frozen CMCDR reference for the next stage. At the next stage, current-task data first update the retained analytic state. New head copies are then initialized from the resulting analytic solutions and refined. Refined weights are not written back into the analytic recursion state.

Table R3. MMAL under matched head refinement. Accuracy and forgetting are percentages.

DatasetMethodAvg. Acc. ↑Avg. Forget. ↓
AVEMMAL70.2819.64
AVE+ matched refinement70.8218.84
AVE+ CMCDR72.6315.02
Kinetics-SoundsMMAL71.4613.72
Kinetics-Sounds+ matched refinement72.0213.09
Kinetics-Sounds+ CMCDR74.169.86
UESTCMMAL75.5724.83
UESTC+ matched refinement76.0324.14
UESTC+ CMCDR78.1220.94

Refinement alone improves accuracy by 0.46–0.56 pp. Relative to this control, CMCDR adds another 1.81–2.14 pp and reduces forgetting by 3.20–3.82 pp across the three benchmarks. The consistent paired gains isolate the benefit of contribution regularization beyond head refinement.

Q3. Training settings, metric definitions, and QA cost

Numerical training settings. Tables S1 and S2 report all 36 method–dataset configurations and method-specific parameters. They cover the optimizer, learning rate, batch size, initial and incremental epochs, schedule, weight decay, and replay budget. Each row applies to the baseline and its CMCDR counterpart, ensuring matched training settings within each pair.

CMCDR uses λcmcdr=β=1\lambda_{\rm cmcdr}=\beta=1, ϵ=10−6\epsilon=10^{-6}, Smooth-L1L_1 threshold δ=1\delta=1, and all 2M2^M subsets including the empty set. SubsetKD uses temperature 2 and weight 1, selected from {1,2,4,8}\{1,2,4,8\} and {0.1,0.5,1,2}\{0.1,0.5,1,2\}.

Exact metric aggregation. Let Dj\mathcal D_j be task jj's fixed test set and nj=∣Dj∣n_j=|\mathcal D_j|. The per-example score s∈[0,1]s\in[0,1] is the correctness indicator for classification or the official answer score for VQAv2. Predictions use the current full output space. We define task scores, sample-weighted stage accuracy, and task-macro stage accuracy as

A_t^{\rm sample}=\frac{\sum_{j\le t}n_j a_{t,j}}{\sum_{j\le t}n_j},\qquad A_t^{\rm task}=\frac1t\sum_{j\le t}a_{t,j}.$$ Stage forgetting averages the drop from each old task's best previous score. $$F_t=\frac1{t-1}\sum_{j<t}\left[\max_{j\le u<t}a_{u,j}-a_{t,j}\right]\quad(t>1).$$ Recognition and audio-visual QA average across learning stages. VQAv2 follows the VQACL protocol and reports task-macro scores at the final outer stage. All methods on a benchmark use the same rule. $$\begin{aligned} \text{Recognition and audio-visual QA}\quad \operatorname{AvgAcc}&=\frac1T\sum_{t=1}^{T}A_t^{\rm sample},& \operatorname{AvgForget}&=\frac1{T-1}\sum_{t=2}^{T}F_t,\\ \text{VQAv2}\quad \operatorname{AvgAcc}&=A_T^{\rm task},& \operatorname{AvgForget}&=F_T. \end{aligned}$$ The first stage is excluded from forgetting averages, and negative forgetting is retained. Audio-visual QA stages follow the outer-task and inner-increment chronology. VQAv2 visual groups are pooled by test-sample counts within each outer task, and outer tasks receive equal weight. MCD follows Eq. (6), averaging within each old-task probe set and then equally across old tasks. **QA implementation.** A classification-based subset pass returns all answer logits. The 2,135 answers of Split-AVQA and 40 answers of Split-MUSIC-AVQA share these predictions, avoiding a separate encoder pass for every answer. Replay-free QA uses current-task inputs to probe old answer classes. Generative VQACL follows Appendix B.3. Candidates are distinct normalized reference answers from the old-task replay buffer. Both models and all subsets share the stage's set. Answers use length-normalized teacher-forced scores. $$\mathcal A_{<t}=\operatorname{unique}\!\left( \bigcup_{b\in\mathcal B_{<t}^{\rm train}}\operatorname{norm}(\mathcal R(b))\right), \qquad z_{s,a}(S)=\frac1{L_a}\sum_{\ell=1}^{L_a} \log p_{\theta_s}(a_\ell\mid a_{<\ell},x_S).$$ Here $\mathcal R(b)$ contains valid reference answers and $L_a$ is the scored answer length. Replay-based CMCDR matches the ground-truth old-answer profiles. Gradients stop at the frozen model. Both QA formulations retain baseline inference. **QA results and computational cost.** We compare each baseline with its CMCDR counterpart on two NVIDIA GeForce RTX 3090 GPUs. Each pair uses the same batch size and training schedule. We report mean wall-clock training time per incremental stage and peak GPU memory. **Table R4. QA accuracy and training cost.** CMCDR uses all available intervention samples. Timing overhead is relative to the corresponding baseline. | Benchmark | Method | Acc. (%) ↑ | Time (min) | Overhead | Peak GPU memory (GiB) | | --- | --- | --- | --- | --- | --- | | VQAv2 | VQACL | 38.77 | 228.1 | — | 13.6 | | VQAv2 | + CMCDR | 41.36 | 282.7 | +23.9% | 15.5 | | Split-AVQA | AVQACL | 32.05 | 158.1 | — | 14.8 | | Split-AVQA | + CMCDR | 34.18 | 185.2 | +17.1% | 16.2 | | Split-MUSIC-AVQA | AVQACL | 33.64 | 133.9 | — | 14.1 | | Split-MUSIC-AVQA | + CMCDR | 36.12 | 172.4 | +28.8% | 15.9 | CMCDR improves accuracy by 2.59, 2.13, and 2.48 pp on VQAv2, Split-AVQA, and Split-MUSIC-AVQA, with training time increases of 23.9%, 17.1%, and 28.8%. Contribution constraints apply during training, and both QA formulations retain baseline inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.