- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that fusion accuracy does not predict how multimodal models behave when a modality is missing, because complete-input training constrains only the joint output of the branches. It compares each model’s single-modality score with an affine probe on frozen features and with a unimodal model, and finds on CREMA-D, Food101 and MIMIC that many failures lie in the classifier rather than the features. It proposes GRR, a per-direction affine correction applied only when a modality is missing, which recovers much of the lost performance without changing encoders or complete-input predictions.
Critical strengths
- Highlights a real evaluation gap: balancing methods with similar fusion accuracy fail on different branches.
- Simple, interpretable diagnostic.
- Broad experiments (3 domains, 6 methods, 3 seeds) with honest reporting.
- Clear writing and good reproducibility.
Critical weaknesses
== Theory contribution ==
- The novelty is limited; For the first two contributions
- That multimodal models degrade sharply, sometimes below unimodal models, when a modality is missing is well established (Ma et al. 2022; Hazarika et al. 2022), and the link between modality imbalance and missing-modality robustness has been studied (e.g., GMD, AAAI 2024; recent work on imbalanced missing rates such as MissBench, MCE, BALM). The specific contribution in the paper is only the extending of this to balancing methods. The underdetermination argument is a special case of underspecification (D’Amour et al. 2022), and the affine cancellation example is elementary.
- The representation/readout decomposition reuses the standard probe-versus-model comparison from long-tail and spurious-correlation work (Kang et al. 2020; Kirichenko et al. 2023; Izmailov et al. 2022) and from linear evaluation of multimodal encoders (Du et al. 2023). Eq. 4 is an add-and-subtract identity.
- Regarding GRR, it is indeed cheap but basically a per-direction linear head routed by modality availability.
== Experiment ==
- Zero-padding confound. All failures are measured with zero-padded missing features, which are out-of-distribution for the fusion head. The near single-class collapse (99.15% “Disgust” for MASAM video-only) and the strong recovery from LogitCal suggest much of the “readout failure” may be an imputation artifact. Simple alternatives on frozen encoders are not tested: training-mean imputation (label-free), a learned missing-modality token, and cross-modal feature reconstruction (as in MMIN, SMIL, MAE-based imputation). Without these, the central diagnosis is not established.
- Missing obvious baselines. Concat+Aux and MLB already have unimodal auxiliary heads; their single-modality accuracy is a zero-cost baseline and is not reported. A bias-only correction (c_m alone) to test whether the problem is mainly a class-prior shift. Training with modality dropout combined with balancing methods. A practitioner expecting missingness would train this way; it is unclear whether the problem persists. Closely related logit-level and test-time approaches (e.g., test-time logit prompting, MiDl) are neither discussed nor compared.
- Evidence concentrated on one small dataset. The large effects come from CREMA-D (744 test clips; standard deviations of 2 to 5 points). Food101 effects are moderate, and MIMIC effects are within about 1.5 AUPRC and largely within noise, so the clinical framing in the abstract is not well supported.
Questions for authors
I don't mind simple and useful paper, and am open to update my points. But authors need to answer the following questions:
- (essential) How do mean imputation, a learned missing token and feature reconstruction compare with zero-padding plus GRR?
- What do the auxiliary unimodal heads of Concat+Aux and MLB achieve?
- How much of GRR’s gain comes from the bias term alone?
- Does the problem persist with modality-dropout training?
- When is GRR preferable to a unimodal model or hmix? Can you test repair without the original training data?
- Can you quantify the “predicts in advance” claim across all cells?