- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the risk of a fixed multimodal model changes when modality-missing rates differ between conditions. It derives a closed-form upper bound using a label-preserving coupling of Bernoulli masks and per-modality second moments. It then introduces empirical slope and envelope diagnostics and an evaluation protocol involving independent re-masking, feature checks, and multiple training seeds. Case studies involving BALM and AMB-DSGDN illustrate an order-dependent masking artifact, variation across seeds, and differences between changes in absolute loss and changes in the proposed slope.
Critical strengths
- The evaluation question is relevant. Full train-rate × test-rate matrices reveal behavior that cannot be assessed from matched-rate results alone.
- The theoretical argument is clear and honest. The proof retains the same label in the coupling, and the paper carefully distinguishes its population bound from empirical slopes, finite-difference probes, and descriptive envelope coverage.
- The reporting is transparent. The paper provides seed-level results, discloses unsuccessful reproductions and test-selection limitations.
Critical weaknesses
- The pooled slope does not measure how risk varies with missing rate. In section 4.1, d is symmetric, and the full directed grid is used. If the models trained at all reference rates happen to implement the same predictor, the contributions from each pair of directions cancel because the distance is symmetric. The resulting slope is zero regardless of how strongly that predictor’s risk varies across missing rates. Even the combination of mean diagonal risk and slope can therefore be identical for a sensitive and a condition-invariant predictor. This means section 7.2’s reading of a lower slope as reduced sensitivity is not supported.
- The claimed agreement between training and evaluation marginal missing rates is inconsistent with the reported grid. Section 5.1 states that the original training masker never removes all modalities and that its marginal missing rates agree with those of independent evaluation masking. This assumption cannot hold at (0.7, 0.7, 0.7), which is in the grid. If every utterance retains at least one of three modalities, the expected number of missing modalities should be at most 2, but here it is 2.1. Moreover, independent evaluation introduces all-missing utterances with a probability of 0.343 at this configuration. The train-to-evaluation change therefore alters marginal rates and joint mask support, not only positional assignment. Sharing it across variants does not rule out variant-specific responses, particularly to the all-missing pattern BALM never sees in training.
- The statistical power claim does not match the decision rule. The paper treats an effect as confirmed only when both the t-test and the exact Wilcoxon test are significant. With six nonzero continuous paired differences, a two-sided exact Wilcoxon test at 0.05 requires all six differences to have the same sign. Using Table 1’s observed variance, a 31% reduction has approximately 80% power for the t-test alone, but only approximately 62.5% for Wilcoxon under the same Gaussian calculation. Consequently, the stated “properly powered” six-seed protocol needs correction, and non-significance should be distinguished carefully from evidence of a negligible effect.
- The connection between the theory and the empirical diagnostic remains insufficiently established. The experiments do not establish the required Lipschitz condition or certify for the evaluated models; instead, the proposed distance is used to construct descriptive slopes and empirical envelopes. Proposition 1 establishes the worst-case sharpness of the Wasserstein bound, but does not show that the resulting risk bound is tight or informative for these models. Although the authors acknowledge these limitations, the theorem does not establish that the fitted slope is a reliable measure of robustness or that the proposed distance offers an advantage over simpler missing-rate distances.
Questions for authors
- How should the pooled slope be interpreted given the cancellation example, and do the main conclusions persist when using per-model absolute risk changes or direction-specific summaries?
- Can you report the realized marginal rates and joint mask frequencies, particularly at ((0.7, 0.7, 0.7))?
- What is the power to detect a 31% reduction under the actual rule requiring both tests to be significant, and how does this affect the interpretation of the non-significant results?
- How does the theory support the empirical diagnostic beyond motivating its distance, and do the main findings persist with simpler distances such as weighted absolute differences between missing-rate vectors?
Thank you again for your review. Since our response we have uploaded a revised PDF with two changes that relate to your points [1] and [3].
On [1], the value of the theory and of d: §5 now opens by stating what the experiments take from Theorem 1, namely the distance d and the intercept/slope split. We no longer suggest that d is the most informative geometry. Following another reviewer's comment, App. A.3 adds a linear variant of the bound, and Contribution 1 now limits the sharpness claim to W2. As before, no verdict depends on the choice of distance (App. B.9).
On [3], sign cancellation: §4.1 now opens with the quantity that connects the two parts, the ratio Δaj/daj. Theorem 1 bounds ∣Δaj∣/daj. The slope averages the signed ratios, so in your example the two directions cancel, which is why we read the slope as specialization to the training pattern rather than as sensitivity. Sensitivity claims are tested on the ∣Δ∣ summaries, which we report next to the slope (§4.1, Algorithm 1).
We would be glad to answer any remaining questions before the discussion closes.