- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies Modality Contribution Drift in multimodal continual learning that influences the contribution of individual modalities and cross-modal interactions to predictions previously learned tasks. It proposes an intervention-based contribution estimator using modality-subset masking and a Möbius decomposition, then defines an MCD metric combining absolute contribution-magnitude drift and normalized profile drift. The paper further introduces Continual Modality Contribution Drift Regularization that match current contribution profiles to those of a frozen previous model. Experiments report improvements in accuracy, forgetting across continual recognition and multimodal QA.
Critical strengths
-
A clear contribution-preservation perspective for MMCL. The paper identifies a distinct failure mode beyond representation drift: modality-specific and interaction contributions can change even when multimodal representations are comparatively stable.
-
The intervention estimator explicitly separates singleton modality effects from higher-order interactions through Möbius inversion, with the decomposition exactly recovering the full-input benefit relative to the no-modality response.
Critical weaknesses
- The theoretical evidence for MCD remains only partially established.
-
The theoretical guarantee does not establish a causal role of contribution drift in forgetting. Proposition 1 provides only a conditional margin-based retention certificate. Prediction can still change due to drift in the no-modality baseline or later-class competition even when the contribution profile is preserved. Thus, the claimed connection between contribution drift and forgetting is weaker than the presentation suggests.
-
The replay-free guarantee relies on strong and unverified distributional assumptions. Proposition 4 assumes that contribution drift is Lipschitz and that current-task probes adequately cover the relevant old-task distribution. The paper does not provide a practical way to bound the Lipschitz constant or the Wasserstein distribution gap, making the resulting guarantee potentially vacuous.
-
Proposition 2 has a formal inconsistency in the stated drift bound. Eq. (10) does not consistently use absolute values for the contribution drift, although the subsequent proof and discussion rely on absolute differences. The inequality therefore requires clarification or an additional assumption on the drift terms to be mathematically valid.
- The demonstrated effectiveness and efficiency of CMCDR are narrower than the general formulation suggests.
-
The method requires modality-subset interventions, with the manuscript stating that exact contribution profiles require subset evaluations and that computational cost therefore grows exponentially with the number of modalities. The concrete scalability experiment only moves from two to three modalities. On UESTC-MMEA-CL, CMCDR increases training time by 34.4–52.7%, while moving from two to three modalities increases total CMCDR training time by 35.8% for iCaRL and 60.8% for LwF. This demonstrates a meaningful practical overhead while leaving substantially larger modality counts untested.
-
The replay-free formulation relies on current-task samples as probes for every old-class output, making its effectiveness dependent on how informative current-task probes are about old-class contribution profiles. The manuscript does not provide a direct sensitivity analysis over probe distribution or task relatedness.
Questions for authors
-
What performance and computational behavior should be expected as the number of modalities grows beyond three?
-
The method may over-penalize beneficial modality adaptation. It assumes that preserving previous contribution profiles is desirable, but contribution drift is not necessarily harmful. New tasks may legitimately change modality reliance and enable positive forward transfer. The paper does not directly evaluate whether CMCDR sacrifices plasticity or beneficial transfer by suppressing such changes. More analyses about this should be incorporated and forward transfer should be evaluated and reported.
-
The methodological novelty is relatively incremental. The method can be viewed as teacher-student distillation of modality-subset decision responses, with Möbius decomposition providing the target representation. The conceptual advantage of explicitly regularizing Möbius contributions over simpler response-level distillation remains insufficiently established.
-
The experiments does not fully reflect the broader and more recent multimodal continual-learning landscape. It does not compare against more recent multimodal continual instruction-tuning settings[1], nor does it evaluate a broader range of modern MLLM-based continual-learning methods. This makes it difficult to assess the generalization ability beyond the relatively task-specific MMCL/VQA settings considered in the paper.
[1] MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
6 comments
Q1. How do performance and computational cost change beyond three modalities?
Thank you for raising the question of multimodal scalability. The original manuscript focused on exact contribution estimation and did not sufficiently address efficient scaling to higher modality counts. To address this, we introduce a new approximate contribution estimator for higher modality counts, reducing subset-evaluation cost through a compressed contribution profile, and evaluate its retention gains and training efficiency with four and five modalities.
New approximate estimator. We retain exact singleton and pairwise contributions, aggregate higher-order contributions, and add sampled interaction constraints. By contribution completeness (original manuscript, Section 2.1, Eq. (4)), the higher-order remainder is
=\sum_{|T|\geq3}C(T).$$ Thus, matching low-order contributions and $R_H$ preserves the complete contribution sum without evaluating every higher-order interaction. **Supplementary sampling.** Since individual higher-order changes may cancel within $R_H$, we sample $K=2$ three-modality coalitions and match their third-order contributions between frozen and current models. Each requires only one additional joint-response evaluation, reusing all lower-order responses. The total budget is at most $2+M+\binom{M}{2}+K$ subset evaluations per input and model, reducing the cost from $2^M$ to **$O(M^2+K)$**. **New experiment.** Using the 37-class cross-subject [MMAct](https://openaccess.thecvf.com/content_ICCV_2019/papers/Kong_MMAct_A_Large-Scale_Dataset_for_Cross_Modal_Human_Action_Understanding_ICCV_2019_paper.pdf) setting, we integrate iCaRL with the [MMTSA](https://pi.cs.tsinghua.edu.cn/wp-content/uploads/2023/10/MMTSA.pdf) encoder and fusion architecture under a nine-stage class-incremental protocol (5+4×8), with a total replay budget of 370. Four modalities include RGB, phone acceleration, watch acceleration, and gyroscope; the fifth adds orientation. All methods share synchronized samples, class order, and training settings. The approximation includes sampled-triple constraints. **Table R2. MMAct performance and complete-stage training time at four and five modalities (new experiment).** | M | Method | Avg. accuracy (%) ↑ | Avg. forgetting (%) ↓ | Minutes/stage ↓ | Overhead (%) ↓ | | :---: | :--- | ---: | ---: | ---: | ---: | | 4 | iCaRL | 70.2 | 24.1 | 48.7 | — | | 4 | + Exact CMCDR | 75.6 | 17.1 | 72.3 | +48.5 | | 4 | + Approx. CMCDR | 74.2 | 17.4 | 61.8 | +26.9 | | 5 | iCaRL | 72.9 | 24.3 | 55.3 | — | | 5 | + Exact CMCDR | 77.2 | 17.1 | 103.5 | +87.2 | | 5 | + Approx. CMCDR | 76.4 | 17.5 | 74.6 | +34.9 | At four/five modalities, approximate CMCDR improves baseline accuracy by **4.00/3.50 pp** and reduces forgetting by **6.70/6.80 pp**, while reducing training time by **14.5%/27.9%** relative to exact CMCDR. At five modalities, accuracy differs by only 0.80 pp, while training time decreases from 103.5 to 74.6 minutes. These results demonstrate effective contribution preservation at reduced cost. **The intervention-ratio experiment already in the original manuscript (Appendix C.3, Table 7)** provides a complementary efficiency control by reducing the number of intervention samples under replay and replay-free protocols. **Table R3. Intervention sample ratio and training cost on AVE (original manuscript, Appendix C.3, Table 7).** | Protocol | Method | Ratio | Avg. accuracy ↑ | Avg. forgetting ↓ | MCD ↓ | Minutes/stage ↓ | Overhead (%) ↓ | | :--- | :--- | :---: | ---: | ---: | ---: | ---: | ---: | | Replay | iCaRL | — | 64.20 | 25.60 | 0.779 | 24.7 | — | | Replay | iCaRL + CMCDR | 25% | 70.46 | 18.14 | 0.452 | 29.0 | +17.4 | | Replay | iCaRL + CMCDR | 50% | 70.89 | 17.41 | 0.421 | 31.2 | +26.3 | | Replay | iCaRL + CMCDR | 75% | 71.03 | 17.72 | 0.402 | 33.4 | +35.2 | | Replay | iCaRL + CMCDR | 100% | 71.86 | 17.36 | 0.389 | 37.3 | +51.0 | | Replay-free | LwF | — | 56.61 | 35.11 | 0.842 | 18.3 | — | | Replay-free | LwF + CMCDR | 25% | 57.91 | 28.99 | 0.474 | 22.6 | +23.5 | | Replay-free | LwF + CMCDR | 50% | 58.27 | 28.01 | 0.468 | 24.0 | +31.1 | | Replay-free | LwF + CMCDR | 75% | 58.73 | 27.88 | 0.433 | 25.7 | +40.4 | | Replay-free | LwF + CMCDR | 100% | 60.61 | 26.35 | 0.401 | 27.4 | +49.7 | On AVE/iCaRL, a 25% intervention ratio achieves a **6.26 pp accuracy gain with 17.4% overhead**, retaining **81.7%** of the full-intervention gain. At the same ratio, LwF reduces forgetting by **6.12 pp** with 23.5% overhead. **Overall, CMCDR maintains clear accuracy and retention gains with four and five modalities.** The approximation reduces computation per probe, while intervention subsampling reduces the number of probes, providing complementary efficiency controls under different training budgets.Q2. Does preserving old contributions hinder beneficial adaptation or positive transfer? What about forward transfer?
The paired class-wise results already in the original manuscript (Appendix C.2, Figure 7) show that contribution preservation benefits the large majority of old classes. Figure 7 connects each class under the baseline and CMCDR on AVE, showing changes in both MCD and forgetting. CMCDR reduces MCD for all displayed classes and reduces forgetting for 19/21 iCaRL classes and 20/21 LwF classes. Tables R4–R5 further quantify the effect on new-task adaptation.
Figure 7 also contains a few classes with lower MCD but higher forgetting, showing that the effects of contribution changes vary across classes. In the original method (Section 3), CMCDR uses contribution preservation as a soft constraint jointly optimized with the new-task objective, with its strength controlled by λ. Benefit shifts shared across all modality subsets cancel from nonempty Möbius contributions, leaving such output changes unconstrained by contribution matching (already in the original manuscript, Appendix C.6).
Table R4. Regularization strength and new-task learning on UESTC-MMEA-CL (already in the original manuscript, Appendix C.4, Table 8) FWT is additionally reported here, in percentage points.
| Method | λ | Avg. accuracy ↑ | Avg. forgetting ↓ | MCD ↓ | New-task accuracy ↑ | FWT ↑ (pp) |
|---|---|---|---|---|---|---|
| iCaRL | 0.1 | 73.418 | 20.464 | 0.387 | 82.369 | 1.26 |
| iCaRL | 0.5 | 74.834 | 17.577 | 0.335 | 81.692 | 1.22 |
| iCaRL | 1.0 | 75.120 | 17.152 | 0.288 | 81.257 | 1.13 |
| iCaRL | 2.0 | 74.707 | 16.663 | 0.254 | 79.976 | 1.02 |
| LwF | 0.1 | 20.158 | 44.724 | 0.428 | 32.179 | 0.76 |
| LwF | 0.5 | 22.913 | 39.116 | 0.353 | 31.568 | 0.59 |
| LwF | 1.0 | 22.851 | 38.648 | 0.299 | 30.912 | 0.44 |
| LwF | 2.0 | 22.167 | 37.979 | 0.262 | 29.715 | 0.32 |
Table R4 identifies an effective λ=0.5–1 range. At λ=0.5, new-task accuracy is higher than at λ=1. At λ=1, iCaRL achieves its highest overall accuracy, while LwF is within 0.062 pp of its best result.
Table R5. Direct comparison against λ=0 on UESTC-MMEA-CL. New-task results at λ=0.5 and λ=1 are from the original Appendix C.4, Table 8. The remaining columns are additional comparisons.
| Method | New, λ=0 | New, λ=0.5 | New, λ=1 | Old, λ=0 | Old, λ=1 |
|---|---|---|---|---|---|
| iCaRL | 83.4 | 81.692 | 81.257 | 65.2 | 72.6 |
| LwF | 33.2 | 31.568 | 30.912 | 8.3 | 18.1 |
The new comparison (Table R5) shows old-task accuracy gains of 7.40/9.80 pp and new-task accuracy changes of −2.14/−2.29 pp for iCaRL/LwF at λ=1. The original results (Table 12) show overall accuracy gains of 3.29/5.72 pp.
Together, class-wise coverage and old/new-task comparisons show that an appropriate preservation strength balances new-task adaptation and old-knowledge retention, translating retention gains into better overall continual-learning performance.
Forward transfer. Table R4 additionally reports FWT, comparing target-task performance before training with performance at the common initialization, following GEM, Eq. (4). At λ=0.5 and λ=1, iCaRL achieves 1.22 and 1.13 pp FWT, while LwF achieves 0.59 and 0.44 pp. Positive FWT means that learning earlier tasks helps target tasks before they are trained, showing that moderate contribution preservation maintains positive forward transfer while improving overall accuracy.
Response to Reviewer Qqus
Thank you for recognizing the contribution-preservation perspective. The revised appendix adds contribution-restoration analysis in C.2, higher-modality approximation and experiments in C.3, old/new-task comparisons and FWT in C.4, replay-free distribution-shift results in C.5, and modern-MLLM comparisons in C.8. All additions are marked in blue.
Theoretical basis and replay-free probes
Contribution drift acts on quantities directly supporting old-task decisions. Proposition 1 connects contribution changes to old-class decision margins. Appendix A.3, Eqs. (22)–(23), and Appendix A.5, Eq. (35), additionally account for new-class competition and provide retention conditions over the full class space. We further restore reference-stage contributions while fixing the current no-modality response and all competing-class scores. Across iCaRL and LwF on AVE and UESTC-MMEA-CL, this reduces error by 10.2–25.3 percentage points (pp), corresponding to 55.3%–60.8% of the net error increase. With the other decision scores held fixed, this diagnostic directly demonstrates the contribution term's influence on predictions and provides a decision-level basis for preserving it.
Current-task probes provide useful signals for preserving old knowledge. Section 3.3 compares old-class contributions of the frozen and current models on the same current inputs, without storing old examples. Proposition 4, Eq. (38), relates old-task contribution drift to current-probe loss and a distribution-discrepancy term. Appendix C.5, Figure 10, reports current-/old-task probe correlations of 0.785 and 0.724 on AVE and UESTC-MMEA-CL, respectively, showing that current probes identify old classes with greater contribution drift. To further test retention under distribution shift, the additional AVE experiment preserves the task split and progressively increases Gaussian video blur, σt=2(t−1)/(T−1), with audio unchanged. Each task is tested at its introduction-stage blur strength.
Table R1. Replay-free learning under AVE video distribution shift.
CMCDR improves accuracy by 4.40 pp and reduces forgetting by 10.20 pp, with probe correlation reaching 0.79. These results connect drift identification to actual retention gains, showing that old-class contribution signals from current data remain effective for replay-free continual learning under the tested distribution shift.
Notation. The definition of Δg in Eq. (9) and the contribution difference in Eq. (10), on page 5 of the current manuscript, already contain absolute values and agree with the proof in Appendix A.4.