acceptodds
Under review as a conference paper at ICLR 2027

What Fusion Accuracy Hides About Missing-Modality Robustness

Abstract

Multimodal models are usually evaluated with all modalities available, yet at deployment a modality is often missing. We show that fusion accuracy says little about this setting: models with nearly identical complete-input accuracy can differ sharply once one modality is removed, because complete-input training constrains only the joint output of the modality branches, not how each branch behaves alone. To locate such failures, we separate what the retained representation supports from what the model's own classifier extracts from it, using probes on frozen features and an independently trained unimodal model as reference. On audio–video, text–image, and clinical EHR–X-ray tasks, low missing-modality accuracy often hides a highly predictive representation; balanced training can even make the weaker modality's representation more predictive than unimodal training, while the classifier fails to use it. We propose Gated Residual Repair (GRR), a small affine correction applied only when a modality is missing, which recovers most of this gap without changing the encoders or complete-input predictions. Adapting the encoders helps only where the retained representation falls well below the unimodal reference, a case the diagnosis identifies in advance. Missing-modality robustness is thus a distinct property of multimodal models that fusion accuracy does not reveal.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal learning, Missing-modality robustness, Balanced multi modal learning, Representation decodability, Classifier adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MTdjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that fusion accuracy does not predict how multimodal models behave when a modality is missing, because complete-input training constrains only the joint output of the branches. It compares each model’s single-modality score with an affine probe on frozen features and with a unimodal model, and finds on CREMA-D, Food101 and MIMIC that many failures lie in the classifier rather than the features. It proposes GRR, a per-direction affine correction applied only when a modality is missing, which recovers much of the lost performance without changing encoders or complete-input predictions.

Critical strengths
  • Highlights a real evaluation gap: balancing methods with similar fusion accuracy fail on different branches.
  • Simple, interpretable diagnostic.
  • Broad experiments (3 domains, 6 methods, 3 seeds) with honest reporting.
  • Clear writing and good reproducibility.
Critical weaknesses

== Theory contribution ==

  • The novelty is limited; For the first two contributions
    • That multimodal models degrade sharply, sometimes below unimodal models, when a modality is missing is well established (Ma et al. 2022; Hazarika et al. 2022), and the link between modality imbalance and missing-modality robustness has been studied (e.g., GMD, AAAI 2024; recent work on imbalanced missing rates such as MissBench, MCE, BALM). The specific contribution in the paper is only the extending of this to balancing methods. The underdetermination argument is a special case of underspecification (D’Amour et al. 2022), and the affine cancellation example is elementary.
    • The representation/readout decomposition reuses the standard probe-versus-model comparison from long-tail and spurious-correlation work (Kang et al. 2020; Kirichenko et al. 2023; Izmailov et al. 2022) and from linear evaluation of multimodal encoders (Du et al. 2023). Eq. 4 is an add-and-subtract identity.
  • Regarding GRR, it is indeed cheap but basically a per-direction linear head routed by modality availability.

== Experiment ==

  • Zero-padding confound. All failures are measured with zero-padded missing features, which are out-of-distribution for the fusion head. The near single-class collapse (99.15% “Disgust” for MASAM video-only) and the strong recovery from LogitCal suggest much of the “readout failure” may be an imputation artifact. Simple alternatives on frozen encoders are not tested: training-mean imputation (label-free), a learned missing-modality token, and cross-modal feature reconstruction (as in MMIN, SMIL, MAE-based imputation). Without these, the central diagnosis is not established.
  • Missing obvious baselines. Concat+Aux and MLB already have unimodal auxiliary heads; their single-modality accuracy is a zero-cost baseline and is not reported. A bias-only correction (c_m alone) to test whether the problem is mainly a class-prior shift. Training with modality dropout combined with balancing methods. A practitioner expecting missingness would train this way; it is unclear whether the problem persists. Closely related logit-level and test-time approaches (e.g., test-time logit prompting, MiDl) are neither discussed nor compared.
  • Evidence concentrated on one small dataset. The large effects come from CREMA-D (744 test clips; standard deviations of 2 to 5 points). Food101 effects are moderate, and MIMIC effects are within about 1.5 AUPRC and largely within noise, so the clinical framing in the abstract is not well supported.
Questions for authors

I don't mind simple and useful paper, and am open to update my points. But authors need to answer the following questions:

  • (essential) How do mean imputation, a learned missing token and feature reconstruction compare with zero-padding plus GRR?
  • What do the auxiliary unimodal heads of Concat+Aux and MLB achieve?
  • How much of GRR’s gain comes from the bias term alone?
  • Does the problem persist with modality-dropout training?
  • When is GRR preferable to a unimodal model or hmix? Can you test repair without the original training data?
  • Can you quantify the “predicts in advance” claim across all cells?
Reviewer_tYx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why multimodal models trained with complete inputs perform poorly when modalities are missing. The authors distinguish representation failures from readout (i.e., classifier) failures, arguing that low missing-modality accuracy can mask useful representations. They propose Gated Residual Repair (GRR), a lightweight affine correction to the readout that keeps the backbone frozen.

Experiments on 3 datasets show that adding GRR improves the evaluated baselines under missing-modality conditions.

Critical strengths

S1: Freezing representations and modifying only the readout is a well-motivated and practically relevant research direction.

S2: The analysis distinguishing representation failures from readout failures in Sec. 3.2 is insightful.

S3: Adding GRR improves the performance of the evaluated baselines.

Critical weaknesses

W1: Although Sec. 3.2 examines different readouts, GRR uses a refined readout when only modality m is observed. This resembles a modality-specific classifier, and its novelty is unclear. Prior work has already explored dedicated sub-architectures for different missing-modality conditions [R1, R2]. The paper should clarify how GRR differs from these approaches.

W2: The comparisons mainly focus on methods addressing modality imbalance, omitting methods specifically designed for missing modalities, such as SimMLM [R2], ShaSpec [R3], and MAP [R4]. Moreover, in Table 3, ModDrop substantially outperforms GRR when only modality B is available. Given that ModDrop simply drops modalities randomly during training, the practical advantage of GRR's frozen-backbone approach remains unclear.

W3: Although Table 2 shows that affine probing improves performance, the resulting accuracy remains below or comparable to that of the unimodal models in Table 1. This limits the evidence that the frozen representations are sufficiently effective under missing-modality conditions.

[R1] Wu, Renjie, et al. "Deep multimodal learning with missing modality: A survey." arXiv preprint arXiv:2409.07825 (2024).

[R2] Li, Sijie, Chen Chen, and Jungong Han. "SimMLM: A simple framework for multi-modal learning with missing modality." ICCV 2025.

[R3] Wang, Hu, et al. "Multi-modal learning with missing modality via shared-specific feature modelling." CVPR 2023.

[R4] Lee, Yi-Lun, et al. "Multimodal prompting with missing modalities for visual recognition." CVPR 2023.

Questions for authors

Q1: What distinguishes GRR from existing modality-specific classifiers or sub-architectures for different missing-modality conditions?

Q2: How does GRR compare with SimMLM, ShaSpec, and MAP? What advantages does freezing the backbone offer, particularly given ModDrop's stronger performance when only modality B is available in Table 3?

Q3: How do you reconcile the claim that useful representations are preserved with the affine-probe results remaining below or comparable to the unimodal baselines in Table 1?

Reviewer_amhUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that complete-input fusion accuracy does not determine how a multimodal model behaves when a modality is removed, since training on complete inputs constrains only the joint output of the branches. It splits missing-modality failure into what the retained representation still supports, read off an affine probe, and what the model's own readout realizes, with an independently trained unimodal model as the reference. Gated Residual Repair then adds a zero-initialized correction that fires only when one modality is observed, so complete-input predictions are untouched. Beyond the measurement, the paper claims that the gap says in advance which of the two failures is present, and so which repair to use. Six source models are used across CREMA-D, Food101 and MIMIC phenotyping, with the repair run on four.

Critical strengths

The central comparison is internal to one protocol: on CREMA-D, MLB, MASAM and AUG sit within 0.41 fusion points of each other while their audio-only accuracies range from 23.70 to 51.75. The recommendation that follows, to report missing-modality behavior separately from complete-input accuracy, stands on its own. The appendix bounds a fitted probe's loss reduction by the readout term it is meant to estimate, and separately notes that the readout term itself can go negative when the source readout falls outside the probe family, as happens for AUG.

Critical weaknesses

The claim that the diagnosis says in advance which repair to use is contradicted by the paper's own cells. The abstract reads "Adapting the encoders helps only where the retained representation falls well below the unimodal reference, a case the diagnosis identifies in advance", and Section 4.4 reaches the same place unhedged: "Changing the representation thus helps only when the representation itself is limited... The readout gap and the probe's distance to the unimodal reference indicate in advance which case applies." Its positive evidence is two branches of a single source model, and both the notion of falling well below and the five-point margin that picks them out are read off the same grid. Against that, MASAM's audio branch on CREMA-D is the paper's textbook readout failure: its probe sits at 57.84 against a 56.05 unimodal reference, with a readout gap of 13.93. In that cell the PEA-SSF adaptation reaches a mean of 58.38 against 56.32 for the frozen-feature repair, over three runs each. The two largest shortfalls in the diagnostic grid were never put through recovery at all.

An affine map on the model's own logits also gets most of what the proposed correction gets. Table C4 puts a logit-recalibration column next to the gated correction, and on CREMA-D and Food101 the two land within a point or two of each other on nearly every cell. They are not the same family, the correction spanning the affine readout family on the retained features while recalibration is affine in the logits alone. That the smaller map keeps up where it does supports the readout-failure account the paper argues for, and it also leaves the case for the larger one resting mostly on the MIMIC settings rather than on the grid.

Rescoping the routing claim to the source models actually tested, and accounting for the MASAM audio cell, would address my main objection and I would be willing to raise the score.

Questions for authors

Q1. On MASAM's CREMA-D audio branch the probe exceeds the unimodal reference and the readout gap is 13.93, yet PEA-SSF reaches a higher mean than GRR there. How should that case be read under the proposed diagnosis, and what is the rule's support beyond MASAM's two large-shortfall branches?

Q2. Outside the MIMIC settings, what does the gated residual add over recalibrating the logits the model already produces?

Q3. For the affine-fusion setting the theory analyses, how would the decomposition and the repair extend to three or more modalities, where the number of availability patterns grows quickly? This is a scope question rather than something I expect answered for this submission, and it does not bear on my rating.

Q4. The stated criterion for the recovery grid is coverage of distinct training mechanisms and missing-input behaviors. Why did you not also require coverage of the largest representation deficits, given that those are the cases the diagnosis is meant to route?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.