acceptodds
Under review as a conference paper at ICLR 2027

Measuring Cross-Condition Generalization Under Missing Modalities: A Closed-Form Bound and a Cautionary Case Study

Abstract

Missing-modality robustness methods are typically evaluated at a single, matched train/test missing rate - a protocol that cannot distinguish a model that has genuinely learned to generalize across missingness patterns from one that has merely overfit to the pattern it was shown. We derive a closed-form, optimal-transport-solver-free upper bound on the risk gap between any two missing-rate conditions, exploiting a structural fact specific to the missing-modality setting: the label is invariant under masking, so no irreducible “distortion” term is required, unlike general conditional-distribution-shift bounds for generative models. We verify the transport inequality numerically on synthetic data and use its closed-form geometry to define descriptive sensitivity diagnostics on real trained multimodal networks. These diagnostics support a lightweight, pre-registered evaluation protocol for cross-condition robustness claims. Applying this protocol to a recent missing-modality defense (BALM) across two backbones and two datasets, and to a second, structurally different method (AMB-DSGDN), we find a heterogeneous pattern that no single evaluation setup reveals: (a) a real, previously undocumented artifact in BALM's evaluation masking (a positional correlation with the fixed test-set ordering); (b) on one backbone, BALM's central sensitivity-reduction claim appears confirmed at a small sample size but does not survive a properly powered, pre-registered multi-seed evaluation, and a second dataset shows the same null pattern; (c) on a second backbone, a distinct, slope-independent absolute-risk-reduction effect looks large and unanimous at 3 seeds, then at the full pre-registered 6-seed batch is significant by paired -test but not by Wilcoxon - the same failure mode caught twice on two different axes; and (d) a structurally unrelated method (adaptive modality dropout) improves both slope and intercept together. No accuracy-at-a-few-rates table, the field's current norm, can distinguish these effects, and no single backbone/dataset/method combination reveals the full picture. We argue this decomposition, and the multi-seed, multi-axis protocol needed to see it, is a general need for the field, and offer both as a contribution independent of any specific verdict on any one method.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
missing modalities, deep learning, theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RStPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the risk of a fixed multimodal model changes when modality-missing rates differ between conditions. It derives a closed-form upper bound using a label-preserving coupling of Bernoulli masks and per-modality second moments. It then introduces empirical slope and envelope diagnostics and an evaluation protocol involving independent re-masking, feature checks, and multiple training seeds. Case studies involving BALM and AMB-DSGDN illustrate an order-dependent masking artifact, variation across seeds, and differences between changes in absolute loss and changes in the proposed slope.

Critical strengths
  1. The evaluation question is relevant. Full train-rate × test-rate matrices reveal behavior that cannot be assessed from matched-rate results alone.
  2. The theoretical argument is clear and honest. The proof retains the same label in the coupling, and the paper carefully distinguishes its population bound from empirical slopes, finite-difference probes, and descriptive envelope coverage.
  3. The reporting is transparent. The paper provides seed-level results, discloses unsuccessful reproductions and test-selection limitations.
Critical weaknesses
  1. The pooled slope does not measure how risk varies with missing rate. In section 4.1, d is symmetric, and the full directed grid is used. If the models trained at all reference rates happen to implement the same predictor, the contributions from each pair of directions cancel because the distance is symmetric. The resulting slope is zero regardless of how strongly that predictor’s risk varies across missing rates. Even the combination of mean diagonal risk and slope can therefore be identical for a sensitive and a condition-invariant predictor. This means section 7.2’s reading of a lower slope as reduced sensitivity is not supported.
  2. The claimed agreement between training and evaluation marginal missing rates is inconsistent with the reported grid. Section 5.1 states that the original training masker never removes all modalities and that its marginal missing rates agree with those of independent evaluation masking. This assumption cannot hold at (0.7, 0.7, 0.7), which is in the grid. If every utterance retains at least one of three modalities, the expected number of missing modalities should be at most 2, but here it is 2.1. Moreover, independent evaluation introduces all-missing utterances with a probability of 0.343 at this configuration. The train-to-evaluation change therefore alters marginal rates and joint mask support, not only positional assignment. Sharing it across variants does not rule out variant-specific responses, particularly to the all-missing pattern BALM never sees in training.
  3. The statistical power claim does not match the decision rule. The paper treats an effect as confirmed only when both the t-test and the exact Wilcoxon test are significant. With six nonzero continuous paired differences, a two-sided exact Wilcoxon test at 0.05 requires all six differences to have the same sign. Using Table 1’s observed variance, a 31% reduction has approximately 80% power for the t-test alone, but only approximately 62.5% for Wilcoxon under the same Gaussian calculation. Consequently, the stated “properly powered” six-seed protocol needs correction, and non-significance should be distinguished carefully from evidence of a negligible effect.
  4. The connection between the theory and the empirical diagnostic remains insufficiently established. The experiments do not establish the required Lipschitz condition or certify LθL_\theta for the evaluated models; instead, the proposed distance is used to construct descriptive slopes and empirical envelopes. Proposition 1 establishes the worst-case sharpness of the Wasserstein bound, but does not show that the resulting risk bound is tight or informative for these models. Although the authors acknowledge these limitations, the theorem does not establish that the fitted slope is a reliable measure of robustness or that the proposed distance offers an advantage over simpler missing-rate distances.
Questions for authors
  1. How should the pooled slope be interpreted given the cancellation example, and do the main conclusions persist when using per-model absolute risk changes or direction-specific summaries?
  2. Can you report the realized marginal rates and joint mask frequencies, particularly at ((0.7, 0.7, 0.7))?
  3. What is the power to detect a 31% reduction under the actual rule requiring both tests to be significant, and how does this affect the interpretation of the non-significant results?
  4. How does the theory support the empirical diagnostic beyond motivating its distance, and do the main findings persist with simpler distances such as weighted absolute differences between missing-rate vectors?
Reviewer_E2v8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the setting where train and test time missingness differ. The paper contributes a bound in risk |R(r)-R(r')|<=L_theta*d where L_theta is some intractable Lipschitz constant and d is a metric, tractable descriptors that replace L_theta's role (descriptive, not estimates of L_theta), a protocol that computes these descriptors, and a case study on existing models and datasets, focusing on multimodal emotion recognition (MER) and 2 recent missing modality robustness methodologies (BALM and AMB-DSGDN).

Critical strengths

I found the BALM audit to be quite interesting and useful. The masking artifact in BALM's methodology is a bug that any evaluation using that codebase would inherit and that undermines the MCAR interpretation of its results (its magnitude on reported metrics is not quantified, see Weaknesses). I also found the 3-seed slope effect not replicating to be the most important methodological result in this paper. This shows the broad difficulty of evaluating whether some module improves robustness to missing modality, which might be true beyond BALM as well.

Critical weaknesses

The following weaknesses stood out: [1] On the significance of the theory. The Lemma 1/Prop 1/Thrm 1 chain is correct as far as I can tell, but its added value for the empirical analysis is unclear. The distance d enters the slope and empirical-envelope diagnostics, and L_theta is not computed or estimated (as the authors mention). The burden is on the authors to show whether the particular geometry provided by (d) offers useful insight into missing-modality robustness beyond simpler summaries, for example, of the full matrix of performances obtained by training at rate r and evaluating at rate r'. This setting of examining cross-rate evaluation performances has, in general, already been explored by previous work. As of now, it is unclear whether replacing (d) with a simpler comparison, such as an unweighted L1 distance, would lead to different conclusions. In fact, in Appendix B.1, the correlation between d and delta_aj stays quite similar after z-scoring. This supports robustness to that rescaling, but does not establish the added value of the second-moment-based d. The closest experiment is in section 5.2 via the envelope-transfer results, but even in this case the finding may survive with unweighted distance.

[2] On limited evaluation scenarios. In MER tasks, the modalities will be largely redundant and noisy. This means masking one modality probably doesn't change the performance too much depending on the modality. Different tasks, like VQA, have different structure (e.g. modalities are synergistic), so masking one can be quite steep. The proposed distance cannot see this structure: a high-variance noise modality inflates d while the change in risk is close to 0. The paper does acknowledge the MER-only scope but there is a structural reason why evaluating beyond MER is important.

[3] Let’s say you train at 0.1 missing and evaluate at 0.9: delta is positive. Train at 0.9 and evaluate at 0.1: delta is negative. Because your slope methodology aggregates signed deltas, these contributions can cancel even though the model’s risk swings substantially with the evaluation rate. Beta is thus measuring the average signed trend across the grid rather than the magnitude of the response to a missing-rate change: a model whose risk swings substantially in both directions can have the same or an even lower slope than one whose risk barely moves. Consequently, Section 7.2’s interpretation that “BALM’s central claim is that FCM and GRM make a model less sensitive to the missing-rate pattern, which in our decomposition means a lower slope” does not follow, and the same ambiguity affects the AMB-DSGDN result (Appendix B.5), which is read off a lower beta. Since the MMGCN null and the AMB-DSGDN improvement are the paper’s main slope-axis conclusions, the slope’s validity as a robustness descriptor needs clearer framing as it is a bit unclear what the slope means when describing cross-rate evaluation.

Questions for authors
  1. In the experiments, the slope, envelope, and transfer results all use the second-moment-weighted distance d. Would the main findings change if d were replaced by a simple unweighted distance (e.g., all B² = 1)? If nothing changes, what does the weighting contribute beyond just reading a cross-rate-evaluation performance matrix which previous works have considered?

  2. Given the above sign-cancellation issue in [3], what justifies reading a lower slope as reduced sensitivity? Would the MMGCN and AMB-DSGDN slope conclusions change if the slope were computed on |Δ| instead?

3 comments

Thank you again for your review. Since our response we have uploaded a revised PDF with two changes that relate to your points [1] and [3].

On [1], the value of the theory and of dd: §5 now opens by stating what the experiments take from Theorem 1, namely the distance dd and the intercept/slope split. We no longer suggest that dd is the most informative geometry. Following another reviewer's comment, App. A.3 adds a linear variant of the bound, and Contribution 1 now limits the sharpness claim to W2W_2. As before, no verdict depends on the choice of distance (App. B.9).

On [3], sign cancellation: §4.1 now opens with the quantity that connects the two parts, the ratio Δaj/daj\Delta_{aj}/d_{aj}. Theorem 1 bounds ∣Δaj∣/daj\vert\Delta_{aj}\vert/d_{aj}. The slope averages the signed ratios, so in your example the two directions cancel, which is why we read the slope as specialization to the training pattern rather than as sensitivity. Sensitivity claims are tested on the ∣Δ∣\vert\Delta\vert summaries, which we report next to the slope (§4.1, Algorithm 1).

We would be glad to answer any remaining questions before the discussion closes.

Reviewer_E2v8OpenReview ↗

Response to rebuttal

Thank you for the response and the effort that went into making it. The slope identity resolves my concern [3]. I found the results in the rebuttal interesting, but I am keeping my score. The new results, while informative, suggest that two of the paper’s headline contributions have limited effect on its conclusions: the findings hold unchanged with B²=1, and the positional artifact doesn’t measurably affect F1.

Thank you for reading the response. Good to know that it solves your concerns We have two short clarifications on the two contributions you mention.

On B2=1B^2=1: we agree that no verdict depends on the weighting, and we report this as a robustness check (Contribution 2, App. B.9). The experiments take two things from Theorem 1: that risk changes can be read against a distance between rates, and the split into intercept and slope. That split, not the specific weights, is what separates BALM's intercept effect on MM-DFN from its null slope effect, and GCNet's gains on both axes. That the verdicts survive other weightings shows they are not caused by our choice of dd. The second-moment weighting matters for the theory, where it gives the sharp upper bound on W2W_2 (Proposition 1).

On the positional artifact: we agree that position alone does not move weighted-F1 (+0.02 points). It raises NLL by 2.4% in 11 of 12 configurations, but it leaves the ranking unchanged. The masker audit as a whole does change the published metric. At the same nominal rate, BALM's quota-fill masks give 3.6 points lower weighted-F1 than i.i.d. masks and flip the sign of the full BALM minus baseline gap. Retraining MM-DFN with matched maskers also revealed a sensitivity gain (a 16% lower ∣Δ∣\vert\Delta\vert slope in all six seeds) that the original masker hid (App. B.10).

Thank you again for the careful review.

Reviewer_aRAuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In multimodal learning (settings in which audio, video, and textual data are integrated together), models frequently encounter missing data, and current research development addresses this problem of "missing modalities" by testing it at specific missingness rates that match the rates used during training. In this paper, relative to this missing modality problem, the authors question whether evaluating models at matched training and test missing rates is sufficient to assess robustness when the missingness conditions change. Specifically, they argue that testing a model on the same patterns it was trained on only measures pattern-fitting (overfitting), not generalization. According to the authors, a robust model should perform well even when the pattern of missingness changes (e.g., training on x% missing audio, but being tested on x% missing video). To this purpose, they derive an upper bound on the risk difference between two missing-rate conditions for a fixed model and use the distance introduced by this analysis to define two descriptive diagnostics, i.e., the matched-rate risk, referred to as the intercept, and the change in risk with distance from the training condition, referred to as the slope. Furthermore, they propose an evaluation protocol combining independent re-masking, checks for embedding collapse and positional masking artifacts, and a multi-seed decision rule fixed in advance. The overall contribution is therefore an evaluation framework intended to distinguish baseline performance from sensitivity to changes in missingness conditions. The experimental analysis considers a recent missing-modality defense (BALM)on two backbones and two datasets, together with an additional investigation of an alternative defensive method, namely AMB-DSGDN. The results identify a positional artifact in BALM’s evaluation masking and show that the conclusions depend on the backbone, metric, and number of seeds.

Critical strengths

The paper identifies a relevant problem in multimodal learning, i.e., whether robustness at matched training and test missing rates also holds when the missingness conditions change.

Furthermore, the authors consider multiple missingness configurations, including settings where more than one modality has a high missing rate.

I appreciate the generalization perspective on this task and its presentation, where the authors encourage an evaluation across different training and test conditions, which provides a more complete analysis. Furthermore, the proposed evaluation protocol is supported by a theoretical analysis.

Critical weaknesses
  • Writing and presentation

While reading the paper, I had the feeling that the research questions and the positioning of this work compared to related work are more complicated to derive than necessary. The abstract is too technical and long, presenting several methodological details and experimental outcomes before giving the reader a clear understanding of the main problem, ideas, and practicality of them.

In the same way, the introduction does not present the paper smoothly. To be specific, the storytelling in the introduction does not clearly guide the reader from the limitations of existing evaluation methods to the need for the proposed protocol. This is possibly a consequence of the short length of the introduction and the longer one in the abstract. Consequently, I found it possibly more complex than necessary at first to understand how the theoretical analysis, the evaluation protocol, and the experimental investigation connect to address the same problem.

I believe the paper is missing a simpler and more intuitive presentation of its contribution. Specifically, the authors should clarify earlier what existing evaluation protocols already provide, what they do not capture, and how the proposed approach addresses this limitation.

  • Evaluation

The last point of weakness is related to the limitation of experiments to multimodal emotion recognition downstream tasks. While I agree it covers a wide setting where audio, text, and visual features are utilised and combined, this setting is useful for an initial investigation, I do not think it is sufficient to demonstrate the general applicability of the proposed evaluation protocol. I do have two considerations in this matter:

First, within this setting, the evaluation could include additional methods specifically designed for incomplete multimodal learning, such as GCNet (Lian et al., 2023, IEEE TPAMI) and Emotion-LLaMA (Cheng et al. 2024, NeurIPs). Second, the authors could consider other downstream tasks and model families (e.g., LLaVA, Qwen 3.8) to assess whether the diagnostics remain informative beyond the MER setting.

Otherwise, if MER is the target and only setting of interest, I believe it should be clear, if not from the title, at least from the abstract.

Questions for authors

Is the proposed protocol intended specifically for MER, or also for other multimodal downstream tasks? Because I believe the theory goes beyond the MER application.

Regarding the related works, can the authors explain more intuitively what these approaches do not capture?

Comments on the paper

2 comments
Area_Chair_HYVcOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

We hope our response has addressed your concerns

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.