acceptodds
Withdrawn from ICLR 2027

Provably Robust Multimodal Fusion under Conditional Dependence Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal robustness is usually studied by corrupting or degrading individual modalities. We study a different failure mode where the quality distribution of each modality remains unchanged while the dependence among their local quality states changes. We call this conditional dependence shift (CDS). Under CDS, every modality experiences the same corruption marginal, yet fusion can still fail because local failures co occur differently across modalities, changing the temporal overlap of jointly reliable evidence. We characterize exactly when fusion risk is invariant to such quality couplings, derive an exact decomposition of the resulting risk shift, and introduce a coupling-sensitivity modulus that tightly bounds risk variation under every coupling with the same modality marginals and is computable by a small linear program. These results yield StableFuse, which aggregates evidence and reliability within each modality and directly regularizes a differentiable upper bound on this modulus, so that robustness to coupling shifts is explicitly controlled rather than only encouraged. Using controlled within sample protocols that preserve individual modality quality while changing only the coupling of local failures, StableFuse improves average and worst case CDS performance across four multimodal benchmarks while reducing the certified modulus and preserving clean accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Multimodal Fusion, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vZ3pOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conditional dependence shift (CDS), where the clean-content distribution and each modality’s conditional quality marginal remain unchanged, while the dependence among modality quality states varies. The authors characterize coupling-invariant losses through additivity, derive a risk decomposition, and introduce a coupling-sensitivity modulus computable by linear programming. Based on this analysis, StableFuse combines modality-local evidence aggregation, coupling-based training, and quality interaction regularization (QIR). Experiments on four multimodal benchmarks report improvements in average and conditional worst-case performance. The problem is meaningful, but inconsistencies in the reported results and limitations in the empirical validation need to be addressed.

Critical strengths
  1. The paper identifies a meaningful robustness setting that separates changes in cross-modal quality dependence from changes in individual modality quality. The within-sample coupling construction provides a controlled way to investigate this failure mode. 2.The theoretical analysis establishes a clear connection between nonadditive quality effects and coupling sensitivity. Under the stated finite-state and full-support assumptions, the main proofs appear sound. The computable modulus also provides a quantitative diagnostic beyond reporting classification performance alone.
  2. The proposed regularizer directly targets the interaction residual identified by the theory. Evaluation across four benchmarks and five random seeds, together with component ablations and alternative corruption operators, provides a useful starting point for assessing the approach.
Critical weaknesses

1.The empirical evaluation covers a narrower coupling class than the theoretical analysis. For three modalities, permutation-based couplings do not generally characterize all fixed-marginal distributions; the scope of the worst-case claims should therefore be qualified. 2.The connection between certificate reduction and classification robustness remains insufficiently established. Since logit rescaling can reduce cross-entropy sensitivity without changing predictions, calibration-controlled comparisons and decision-level stability metrics are needed. 3.The experimental evidence does not fully isolate the contribution of QIR or establish robustness beyond the prescribed corruption setting. Matched training conditions, full-model-minus-QIR ablations, and varied corruption configurations are necessary to substantiate these claims. 4.If Conditional Range denotes the sample-averaged coupling loss range for the same checkpoints and loss normalization, Theorem 3 and Table 3 imply an upper bound of 0.182 on CREMA-D. The larger value in Figure 3 therefore requires clarification. 5.The consistency of the experimental reporting is unclear. Tables 1 and 5–6 report substantially different PH results despite describing the same evaluation protocol, making the reported improvements difficult to assess reliably. 6.The implementation description is insufficient for independent reproduction. Backbone configurations, training schedules, benchmark-specific hyperparameters, and the precise reliability-ranking objective should be explicitly specified.

Questions for authors

1.For the three-modality benchmarks, how much does the worst expected loss increase when optimizing over all fixed-marginal couplings rather than only permutation couplings? 2. Does the reduction in the certified modulus persist after controlling for logit scale or temperature? Does QIR also reduce coupling-induced prediction changes and classification-error variation? 3.Can the authors report full-model-minus-QIR results, matched corruption-training baselines across all benchmarks, and experiments with different numbers of temporal bins and corruption durations? 4.How is Conditional Range in Figure 3 computed? Does it use the same loss normalization, checkpoints, and samples as Table 3? Can the authors provide a per-sample verification of the bound in Theorem 3? 5.What explains the differences between Table 1 and the PH rows in Tables 5 and 6? Please specify any changes in training or evaluation settings.

Reviewer_k8ZuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies conditional dependence shift in multimodal fusion: the clean example and each modality’s conditional quality distribution remain fixed, while the dependence between modality quality states changes. The authors characterize invariance to such shifts through additivity of the quality-conditioned loss, introduce a computable modulus that bounds variation in expected loss, and propose StableFuse, combining modality-local aggregation, coupling-based training, and quality interaction regularization (QIR).

Critical strengths
  1. The question is new and clearly formulated. Changing the dependence between modality failures while preserving their individual quality distributions isolates a meaningful source of robustness failure.
  2. Attention to both risk level and sensitivity. The method combines interaction regularization with predictive objectives, recognizing that a coupling-invariant predictor can still perform poorly.
  3. The evaluation design is careful in one respect. Every alignment keeps each modality’s damage identical, so differences really come from alignment.
Critical weaknesses

W1. Several central experimental results require reconciliation

First, the definition of “Cond. Gap” is unclear. If it denotes synchronous accuracy minus conditional-worst accuracy, as the entries for several other methods suggest, the QMF and StableFuse values in Table 2 appear exchanged across all four datasets. For example, Table 1 gives gaps of 1.47 and 2.23 percentage points for QMF and StableFuse on AVE, respectively, whereas Table 2 reports 2.23 and 1.47. Under this interpretation, correcting the entries would reverse StableFuse’s comparison with QMF on the synchronous-to-worst accuracy drop.

Second, Figure 3 and Table 3 appear inconsistent. Table 3 reports a mean CREMA-D modulus of approximately 0.091 for StableFuse. Theorem 3 implies that the corresponding average range between admissible coupling risks is at most 2·E[Ω], which approximately 0.182. However, Figure 3 appears to show a conditional range near 1.1.

Third, PH results differ substantially between Table 1 and Tables 5/6. Section 4.1 states that the main experiments use previous-observation holding, and Section 4.5 and Appendix A.1 state that Tables 5/6 use the same protocol. Yet the conditional-worst accuracies differ for every method. For example, StableFuse’s conditional-worst accuracy on CREMA-D is 43.47 in Table 1 but 49.19 under PH in Tables 5/6. The corresponding AVE values are 78.89 and 82.04.

W2. The incremental predictive benefit of QIR is not established QIR is the principal connection between the theoretical analysis and the proposed learning method. Its contribution should therefore be assessed by comparing full StableFuse against the same architecture and coupling-training procedure with QIR disabled. Table 3 appears to include the (\tau=0) checkpoints, but reports their modulus without the corresponding predictive metrics. Figure 4’s component comparisons do not isolate the incremental effect of adding QIR to an otherwise complete model.

W3. Three-modality evaluation covers a restricted coupling family The three-modality evaluation searches only permutation couplings, which may underestimate worst-case loss over the full fixed-marginal coupling class considered by the theory. This limits the empirical coverage without invalidating the certificate. Since all 43=644^3=64 state losses are available, please quantify this gap by comparing the permutation-family maximum with the full marginal-constrained LP optimum.

Questions for authors
  1. Can the authors reconcile the apparent Table 2 gap exchange, the Figure 3/Table 3 range discrepancy, and the differing PH results?
  2. What are the predictive results for the exact (\tau=0) checkpoints used in Table 3?
  3. How much larger is the full fixed-marginal worst-case loss than the permutation-family maximum on the three-modality benchmarks?
Reviewer_stRnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conditional dependence shift (CDS), where individual modality quality distributions remain unchanged but the dependence between their quality states changes. The authors characterize how such shifts affect fusion risk and propose StableFuse, which combines modality-local aggregation, quality interaction regularization, and robust coupling training. Experiments on four multimodal classification benchmarks show improvements in clean and corrupted performance. However, the practical relevance and generalization of the proposed setting remain insufficiently established, and the method's ability to preserve useful cross-modal interactions is unclear.

Critical strengths
  1. The paper identifies an interesting failure mode in multimodal fusion that is distinct from conventional modality corruption.
  2. The theoretical analysis is good, providing a risk decomposition and a computable bound on sensitivity to quality-dependence shifts.
  3. The experiments include several baselines, component ablations, and evaluations of the proposed sensitivity measure.
Critical weaknesses
  1. The motivation emphasizes changes in the temporal overlap of reliable evidence across modalities, yet StableFuse independently aggregates each modality and only combines their logits, without explicitly modeling cross-modal temporal relationships. While this design may reduce sensitivity to quality coupling, it may also discard useful complementary or synergistic information. Moreover, the theoretical analysis concerns quality interactions in the loss, which does not necessarily justify removing cross-modal feature interactions. This trade-off is not adequately investigated.
  2. The evaluation does not convincingly demonstrate generalization to unseen dependence shifts. StableFuse is explicitly trained with different quality couplings, and testing uses the same predefined coupling family. This primarily demonstrates robustness within a controlled corruption setting rather than generalization across distinct environments.
  3. StableFuse uses independent modality encoders followed by weighted logit summation, without explicit cross-modal feature interactions. While this may improve robustness, it also limits the ability to capture complementary or synergistic information across modalities. The experiments do not adequately establish whether the benefits come from the proposed regularization or from restricting cross-modal interactions.
  4. Despite the broad framing of robust multimodal fusion, all experiments are limited to classification tasks. The effectiveness of the method on tasks requiring fine-grained cross-modal interactions, such as retrieval, localization, or multimodal reasoning, remains unexplored. Moreover, the worst-case accuracy improvements are relatively modest on three of the four benchmarks.
Questions for authors
  1. How does the proposed modality-local aggregation address the changes in temporal overlap of reliable evidence emphasized in the motivation? Why is explicit cross-modal alignment unnecessary?

  2. Can the authors evaluate QIR on architectures with explicit cross-modal interactions, such as cross-attention-based fusion, to assess whether robustness can be improved without sacrificing cross-modal modeling capacity?

  3. Can the authors evaluate generalization to unseen quality dependence structures and provide more realistic corruption settings?

  4. Can the authors extend equal-CDS-exposure comparisons to other baselines and datasets, and evaluate tasks beyond classification?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.