acceptodds
Withdrawn from ICLR 2027

Counterfactual Sensitivity-Guided Intervention for Multi-modal Domian Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal domain generalization (MMDG) aims to learn models from multiple source domains that generalize to unseen target domains without access to target data during training. A key challenge in this setting is that the predictive benefits of individual modalities may not persist across domains. Our analysis shows that a modality that supports fused predictions in source domains can become less useful or even detrimental in a target domain, highlighting the need to draw on alternative evidence when modality contributions change. To develop this capability using source data alone, we propose Counterfactual Sensitivity-Guided Intervention (CSI). CSI uses online, sample-specific logit gaps between complete and modality-removed inputs to identify modalities on which the current prediction strongly depends. Preferentially removing these modalities creates training views that encourage prediction from the remaining evidence. An annealed objective progressively increases the contribution of these views to training, while an exponential moving average teacher guides prediction from reduced evidence using stable distillation targets derived from complete inputs. Experiments on two public MMDG benchmarks demonstrate improved unseen-domain generalization and robustness to modality-specific perturbations, with additional loss-landscape analyses indicating flatter minima.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal domain generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T4BpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multimodal domain generalization. It starts from the observation that a modality's contribution to the fused prediction can shrink or even reverse in an unseen domain; on EPIC-Kitchens D2+D3→D1, for example, audio's net contribution drops from +3.27 to +0.14. The proposed method, CSI, zeroes out each modality's features per sample and measures the resulting change in logits. These changes are turned into a sampling distribution, so modalities the current prediction depends on more are removed more often. Training anneals from complete-input to masked-input supervision, adds unimodal losses, and uses an EMA teacher that distills its complete-input predictions into the full, masked and unimodal outputs. Experiments on EPIC-Kitchens and HAC cover multi-source and single-source settings with four modality combinations, plus ablations, a comparison of selection strategies, loss landscape plots, robustness to noise and missing modalities, and an efficiency comparison.

Critical strengths
  1. The paper is motivated by a concrete measurement rather than a generic claim. Fig. 1 shows class-level contribution reversals across domains and compares removal with other interventions, which makes the story easy to follow. The figures in general are clear and well designed.

  2. The evaluation is complete for the benchmarks used: multi-source and single-source settings, all modality combinations, three seeds, robustness to noise and missing modalities, and efficiency. CSI is also cheaper than MBCD in training time and memory.

Critical weaknesses
  1. CSI is very close to MBCD. MBCD already pairs an EMA teacher, which distills into both the fused and unimodal outputs, with adaptive dropout that favours the dominant modality. CSI keeps this teacher and distillation structure and borrows MBCD's feature augmentation. What it changes is the selection signal, from batch-level confidence to a per-sample logit gap, plus an annealed objective. The gains are small. Against the best baseline, the average change is between +0.16 and +1.47 on EPIC-Kitchens and between −0.68 and +1.35 on HAC. CSI also trails on the Cartoon domain in most settings. This looks fundamental, and the paper needs to show clearly what per-sample selection adds over MBCD.

  2. The evidence for several claims is weak. The motivating analysis comes from a single split and a single model, and zeroing a modality at test time is itself out of distribution for that model. The method removes whichever modality is most influential, not the one whose benefit fails to transfer, so the link back to Fig. 1 is loose. The flatness claim rests on a single 2D landscape plot. The manuscript also contains leftovers from another version: the HAC robustness figure is labelled "FOMP" rather than CSI, Appendix B.2 refers to a "direction-aware regularization", and an undefined threshold δ appears in the implementation details. On top of that, some numbers don't match across figures and tables. This is fixable, but it makes it hard to tell which results belong to this method.

Questions for authors
  1. Is Fig. 6 actually showing CSI, and what are "FOMP" and the "direction-aware regularization" mentioned in Appendix B.2?

  2. What value of τ is used, and what does the threshold δ in the implementation details refer to?

  3. How does CSI compare with MBCD's confidence-based dropout plugged into the same framework, on both datasets and all modality settings? This would most directly show what the per-sample selection adds.

  4. Fig. 1(b) and Fig. 4(d) suggest that removal itself helps a lot, but in Table 4 random removal (61.59) is slightly below no removal (62.31). How should these be reconciled, and what does "Baseline" in Fig. 1(b)/4(d) refer to, given that ERM is 52.04?

  5. Why does Fig. 4(d) report 61.78 for CSI while Table 1 reports 60.24 ± 0.45 for the same setting?

Reviewer_idSFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of multi-modal domain generalization. The authors identify that modalities useful in the source domain may be less useful in the unseen target domain. Build on this observation, they propose a method that measures how removing each modality changes the fused prediction and uses this sensitivity to guide modality masking during training. A cosine annealing scheduler gradually increases the contribution of masked-input modalities, while an EMA teacher guides predictions from the remaining modalities .

Critical strengths
  • The paper is relatively easy to read and understand.
  • The results indicate that the proposed method improves over existing baselines in several settings.
Critical weaknesses
  • The proposed framework is incremental, as it mainly combines ideas already proposed in previous work.
  • On HAC dataset, the video modality appears to be a strong modality in comparison to the others two, the proposed method performs worst that previous baselines in 2 out of 4 cases in average. It would be good to have further ablations study on this dataset, as it would help assess whether the proposed masking strategy is effective when on modality is dominant.
  • Ablation studies from Tables 3 and 4 uses different modality pairs for some reasons. I would recommend to use the same setting in both.
  • It is not clear to me how the motivation of the paper and the sensitivity guided modality removal scales with the number of modalities, especially with three the design of it assume that only one modality is masked at the time.

Small weakness:

  • Figure 2 is not consistent necessarily with the text, as lambda for l_uni and p(z_m) require the multimumodal classifier which the arrow draw directly from student model that is only the embeddings of each modality.
Questions for authors

Please see above.

  • Can your sensitivity mechanism be used during inference to detect and drop unreliable modalities at test-time.
  • The paper shows the loss landscape and is pretty flat, do you have any insight that motivate this emerging behavior from the training strategy.
Reviewer_syCnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper starts from the observation that, in multimodal domain generalization (MMDG), the predictive benefit of a modality in the source domains can diminish or even reverse in an unseen target domain (illustrated by class-wise net contributions of audio on EPIC-Kitchens). Building on this, the authors propose Counterfactual Sensitivity-Guided Intervention (CSI). For each sample, CSI computes online the logit change caused by replacing one modality’s features with zeros, Δ_m (z)=1/C∥g(z)-g(z_(\m) ) ∥_2^2, converts these values into a distribution s_m (z)=softmax(Δ_m (z)/τ), and samples a modality to remove. The full-input and masked-input losses are mixed with a cosine-annealed weight α_t, and an EMA teacher’s full-input predictions are used to distill the unimodal, full-input, and masked-input predictions of the student. On EPIC-Kitchens and HAC, the paper reports improved average accuracy over existing MMDG methods in both multi-source and single-source settings, together with robustness to feature noise and missing modalities and loss-landscape visualizations indicating flatter minima.

Critical strengths
  1. The paper concretely shows that a modality’s net contribution can reverse between source and target domains, which clearly articulates a problem in MMDG.

  2. The method is simple and lightweight: it adds no trainable parameters and requires less training time and memory than MBCD (Table 5).

  3. The evaluation is broad, covering two benchmarks, four modality configurations, and both multi-source and single-source settings, and it includes a principled ablation of the selection criterion (random vs. confidence-based vs. sensitivity-based, Table 4).

  4. The authors openly discuss settings where CSI underperforms MBCD (generalization to Cartoon).

Critical weaknesses

W1. No formal connection between the objective and the target risk.

Eq. (1) defines R_T (f), but it is never used afterwards. The paper states neither an assumption relating the source and target distributions (e.g., a shift under which some modalities become uninformative in the target domain) nor any result (bound, consistency, etc.) showing that masked-view training reduces R_T under such an assumption. As a result, the step from “contributions can reverse” to “sensitivity-guided removal improves generalization” is purely empirical.

W2. The definition of Δ_m may not carry the meaning claimed for it.

(a) Triviality under linear fusion. If g is a linear layer applied to the concatenated features, g(z)=∑(j=1)^M W_j z_j+b, then g(z)-g(z(\m) )=W_m z_m ⇒ Δ_m (z)=1/C∥W_m z_m ∥_2^2. This is simply the logit magnitude contributed by modality m, independent of the other modalities, and not a counterfactual or interaction-dependent quantity.

(b) Lack of shift invariance. If removing modality m shifts all logits uniformly, i.e., g(z_(\m) )=g(z)+c1, the softmax prediction is unchanged, yet Δ_m (z)=c^2>0. A centered difference or a divergence in softmax space would be more natural, e.g., Δ ̃_m (z)=1/C∥(I-1⁄C 11^⊤ )(g(z)-g(z_(\m) )) ∥_2^2.

(c) Label-agnostic criterion. Removing a modality that supports the correct class (a hard view) and removing one that drives a wrong prediction (an easy view) can yield the same Δ_m. Since the paper is motivated precisely by modalities that become harmful, the difference from a label-aware criterion such as Δ_m^label (z,y)=l(g(z_(\m) ),y)-l(g(z),y) should be discussed or compared.

W3. Zero-imputation is not a neutral intervention.

Because the encoder outputs are normalized, z_m=0 lies outside the training distribution. Both Δ_m and the net contribution in Fig. 1a therefore conflate the effect of removing the modality’s information with the response of g to out-of-support inputs. In particular, the model in Fig. 1a was never trained with missing inputs, so the observed “reversal” cannot be ruled out as an artifact of zero-imputation. A comparison with mean imputation or a marginalization-based definition, E_z_m' [g(z_(-m),z_m')], is needed.

W4. The objective actually being optimized is not well defined. The expected masked loss is ∑(m=1)^M sg [s_m (z;θ)]l(g(z(\m);θ),y). The weights depend on θ but are detached, so the update is not gradient descent on any fixed function, and no characterization of stationary points or convergence is given. With loss-based softmax weighting, the objective could be related to entropic DRO, ∑_m^ softmax (l_m/τ) l_m≈τlog∑_m^ exp (l_m/τ); with Δ-based weighting, no such interpretation holds. As the logit norm grows during cross-entropy training, Δ_m grows quadratically with it, so s_m drifts toward an argmax even for a fixed τ. This implicit temperature annealing, which acts on top of the explicit α_t schedule, is not analyzed.

W5. Train–test mismatch when α_end=0.

As α_t→0, both the full-input term in L_task and the D_KL (q ∥ p_full ) term in L_cd vanish, so in late training the student receives no gradient on full inputs. With T=20 and a 0.5→0 schedule, α_t<0.1 for t≥15, while the EMA horizon 1/(1-β)=1000 steps corresponds to roughly 2.5 epochs on EPIC D2+D3. The final EMA model is therefore close to an average of students trained almost exclusively on masked inputs, yet inference is performed on full inputs. The paper does not explain why this mismatch is harmless.

W6. The interpretation of the distillation term is overstated.

Contrary to the claim that the student learns to approximate full-modality predictions from the remaining evidence, for functions p of z_(-m) we have arg min┬p E[D_KL (q(x) ∥ p(z_(-m) ))]=E[q(x)∣z_(-m) ], leaving an irreducible gap of E[D_KL (q(x) ∥ E[q(x)∣z_(-m) ])]. Moreover, for the same p_mask, the minimizer of the hard-label cross-entropy is P(y∣z_(-m) ), whereas that of the distillation term is E[q(x)∣z_(-m) ], with q itself being an EMA of the student. The relation between these two targets is not discussed.

Questions for authors
  1. What is the exact architecture of the fusion classifier g? If g is linear, Δ_m (z)=1/C∥W_m z_m ∥_2^2; in that case, on what basis does Δ_m measure “counterfactual dependence”?

  2. What value of τ is used, and what exactly are the “sensitivity threshold δ” in the appendix and the activation rule implied by “When activated” (line 216)? Could the authors provide the complete algorithm as equations or pseudocode?

  3. How do the results change if the logit MSE is replaced by (i) a centered logit difference, (ii) KL/JS divergence in softmax space, or (iii) a label-aware criterion l(g(z_(\m) ),y)-l(g(z),y)? In particular, can an ablation against (iii) be provided?

  4. Under what assumption on the source–target shift is the CSI objective connected to R_T, or to the worst-case risk over a set of modality corruptions? Can such a statement be given, even in a simplified setting (e.g., linear fusion with Gaussian features)?

  5. At which epoch is the selected checkpoint when α_end=0? Are there results comparing α_end>0?

  6. Can the authors provide class-wise confidence intervals or a paired test for the reversal in Fig. 1a, significance tests for the main comparisons in Tables 1–2, and a quantitative sharpness measure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.