acceptodds
Under review as a conference paper at ICLR 2027

When Multimodality Stops Helping: Auditing Representation-Conditioned Residual Modality Utility

Abstract

Whether an auxiliary modality helps a prediction is not a property of the modality: it is residual to what the model’s primary representation has already learned. A strong primary representation can leave a powerful auxiliary modality with nothing to add, whereas cross-modal reconstructability tempts exactly the wrong inference. We formalise this view with two estimands. Incremental modality gain (IMG) is the actual test-risk reduction from adding the auxiliary modality under a matched low-capacity head family, with a separate parameter-count control; residual modality utility (RMU) asks whether the component of the auxiliary signal that cannot be predicted from the primary representation explains the target error the primary model left. We show that cross-modal predictability implies neither conditional sufficiency nor gain. Across representation-level ladders spanning weak/generic to more task-relevant encodings on a clinical glaucoma structure–function task, CMU-MOSI, and AV-MNIST, IMG concentrates at weaker or less adapted levels and collapses toward zero at stronger, more adapted levels, while RMU tracks realised gain on every domain (seed-cluster bootstrap Spearman , 95% CI [0.483, 0.825] pooled; per-domain CIs exclude zero on both public benchmarks) and survives negative, missingness, and capacity controls. An independent sweep over seven clinical backbones shows the same pattern. RMU is a capacity-conditional diagnostic, not conditional mutual information. These results support representation-conditioned residual utility as an operational diagnostic, not a universal sufficiency test.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal learning, representation learning, residual modality utility, modality utility, cross-modal predictability, multimodal fusion, representation probing, healthcare AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P1kMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formalises the view that auxiliary modality utility is not an intrinsic property of the modality but is residual to what the primary representation has already captured. The authors introduce three carefully separated estimands, Incremental Modality Gain (IMG), cross-modal predictability (RS), and Residual Modality Utility (RMU), and design a matched low-capacity, cross-fitted estimation protocol to evaluate them without capacity confounds. Across three deliberately heterogeneous representation-level ladders and an independent seven-backbone clinical sweep with exact Shapley values, the paper demonstrates that (i) realised gain concentrates at weaker or less task-adapted representations and collapses toward zero at stronger ones, (ii) high cross-modal predictability does not imply conditional sufficiency and can even be negatively associated with gain, and (iii) RMU tracks realised IMG across domains (pooled seed-cluster bootstrap Spearman ρ = 0.651, 95% CI [0.483, 0.825]) while surviving shuffle, missingness, and capacity controls.

Critical strengths
  • Novel and well-motivated conceptual framing. The explicit separation of signal redundancy (RS) from target-relevant residual utility (RMU), illustrated by the constructive DGP in Eq. 3 where RS → 1 yet gain remains strictly positive, corrects a widespread conflation in the multimodal literature and provides a genuinely useful diagnostic lens.

  • Exceptional methodological rigour. The matched low-capacity ridge head protocol, K = 5 cross-fitted residuals, validation-only penalty selection, and pre-specified go/no-go gate collectively eliminate the most common confounds in modality-contribution evaluation (capacity leakage, in-sample overfitting, post-hoc cherry-picking). The signed reporting rule (negative cells kept, no truncation) is commendable.

Critical weaknesses
  • Figure 1 provenance concerns. The schematic's visual style, uniformly smooth gradients, perfectly symmetric layout, and a highly polished "generated-diagram" aesthetic raises the question of whether it was produced or heavily assisted by a generative image model. The current Generative-AI use statement (Appendix K) limits AI use to "language polishing, grammatical correction, and improvements to clarity and readability," which would not cover figure generation. I ask the authors to clarify how Figure 1 was created; if any generative tool was involved beyond layout assistance, the disclosure should be updated accordingly, and the authors should verify that no text or notation in the figure contains rendering artefacts typical of such tools.

  • RMU is bounded by the linear probe's expressiveness. Any target-relevant residual structure that is non-linear in the auxiliary residuals will be missed. The small-MLP probe result (pooled ρ = 0.291) and ElasticNet failure (0.033) suggest the diagnostic is fragile outside the linear family, a narrower applicability than the framing implies. A dedicated synthetic experiment quantifying the non-linear failure boundary would strengthen the limitations discussion.

Questions for authors

See the Critical Weaknesses part.

Reviewer_gA9FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies when auxiliary modalities offer genuine predictive utility beyond primary representations, introducing three formal metrics: (1) Incremental Modality Gain (IMG), defined as the reduction in test risk upon augmenting the primary representation with the auxiliary input; (2) Cross-modal Predictability (RS), quantifying the predictability of the auxiliary modality from the primary representation; and (3) Residual Modality Utility (RMU), evaluating whether the auxiliary residual orthogonal to the primary representation can predict the residual target error of the primary predictor.

Critical strengths
  1. Differentiating whether an auxiliary modality is merely reconstructible versus whether it genuinely reduces downstream target risk is a well-motivated and valuable perspective for multimodal evaluation.

  2. The authors make a deliberate effort to mitigate data leakage and capacity confounding through cross-fitting, fixed ridge heads, parameter-matched controls, feature shuffling, missing-data checks, and random noise baselines, while transparently noting the lack of external clinical validation

Critical weaknesses
  1. Conceptual Novelty, Theoretical Soundness, and Diagnostic ValidityLimited novelty relative to classical residualization techniques:

RMU is defined by computing the residualsrY=Y−g^(Z),rA=A−h^(Z)r_Y = Y - \hat{g}(Z), \qquad r_A = A - \hat{h}(Z)and fitting a low-capacity probe to predict rYr_Y from rAr_A, interpreting the resulting risk reduction as residual modality utility. Conceptually and mathematically, this procedure is virtually identical to partial regression, Frisch–Waugh–Lovell (FWL) orthogonalization, partial correlation, and standard residual probing. Furthermore, the paper reports that PLS-2 achieves a pooled Spearman correlation of 0.713 with IMG—outperforming RMU's 0.651. This directly undermines the claim that RMU introduces an intrinsically superior or novel diagnostic metric.

Proposition 1 merely restates the classical equivalence of Bayes risk under conditional independence. It provides no finite-sample guarantees that empirical RMU evaluated under a fixed ridge probe will faithfully track IMG. The authors explicitly acknowledge that RMU is sensitive to probe capacity, regularization strength, and hypothesis class. Consequently, the core findings do not constitute theoretical guarantees; they represent post-hoc empirical correlations observed across a manually constructed representation ladder.Confounding by latent representation quality:Both IMG and RMU are computed on the exact same set of representation cells, across which Spearman rank correlations are evaluated. Constructing a ladder spanning weak to strong representations mechanically induces a co-monotonic downward trend across both metrics. Rather than capturing a genuine diagnostic coupling, RMU and IMG may simply serve as joint proxies for an unobserved confounder: the latent suitability of the representation for the downstream task. Without prospective, out-of-sample forecasts on unseen datasets, held-out representation tiers, or novel model backbones, RMU cannot be validated as a reliable operational diagnostic.

  1. Empirical Rigor, Statistical Validity, and ReproducibilityUnverifiable protocol commitments and lack of external validation: The manuscript frequently invokes terms like "pre-specified gate," "frozen registry" and "independent clinical sweep". However, these reflect internal narrative assertions that external readers cannot verify as having been committed prior to data inspection. Furthermore, the clinical split has documented prior project use, disqualifying it as a truly independent external validation benchmark.

Key empirical and methodological concerns:

  • Limited sample size and cluster bootstrap instability: The glaucoma experiments evaluate only 3 random seeds across 4 representation levels per dataset. While the pooled analysis employs a seed-cluster bootstrap, the number of independent clusters remains too low to ensure statistical stability or reliable confidence intervals.

  • In-sample gate evaluation: The RMU decision gate is evaluated on the same experimental units used during development, rather than being tested prospectively to forecast IMG on held-out datasets or representation tiers.

  • Calibration artifacts on AV-MNIST A0: The anomalous behavior observed on AV-MNIST A0 indicates that RMU may primarily be correcting miscalibration in the base predictor g^(Z)\hat{g}(Z) rather than isolating genuine auxiliary modality signal. Treating this post-hoc as a sensitivity check does not resolve the structural concern.

  • High variance in uninformative controls: In Appendix E, nominally uninformative random capacity controls exhibit unexpectedly high rank stability (RS≈0.84–0.98RS \approx 0.84\text{--}0.98). This indicates that the metric is hypersensitive to ambient dimensionality, probe capacity, or implementation details rather than intuitive signal predictability.

  • Inadequate complexity matching: Matching solely the parameter count of linear heads does not equate the effective degrees of freedom, input covariance spectra, regularization paths, or optimization dynamics between real and synthetic baselines.

  • Non-equivalent clinical backbone sweep: The clinical sweep utilizes an XGBoost head, a Shapley value estimand, and a distinct statistical protocol. It functions at best as an internal sensitivity check on the same cohort, not as an independent replication of the RMU protocol.

  • Absence of prospective forecasting: The paper lacks true cross-dataset predictive validation, external clinical cohorts, or preregistered evaluations on unseen representation granularities.Circular synthetic validation: The synthetic data-generating process directly mirrors the linear formulations in Eq. (3). This verifies algorithmic sanity under matched linear assumptions, but offers no evidence of robustness on complex multimodal distributions.

  • Audit-style narrative vs. verifiable reproducibility: Extensive registry identifiers, high-precision decimals, and audit-log phrasing cannot substitute for end-to-end reproducible feature pipelines. The authors acknowledge that raw clinical representations and proprietary checkpoints cannot be regenerated from the public release.

Overall, the manuscript adopts an audit-registry tone, repeatedly asserting "pre-specified," "frozen" and "committed" protocols, while critical findings remain tied to a narrow set of handcrafted experimental cells, accompanied by dense, poorly formatted figures with illegible fonts and clipped labels. While stylistic framing does not indicate misconduct, prioritizing procedural rhetoric over verifiable methodological novelty significantly weakens the paper's evidentiary weight.

Questions for authors

Q1 - Substantive novelty vs. classical baselines: How does RMU fundamentally differ from partial correlation, FWL residual regression, PLS-1/2, or residual probing?

Q2 - Can the authors provide finite-sample bounds or out-of-sample predictive guarantees for IMG?Confounding and circular evaluation: Was the pooled Spearman correlation computed on the same experimental cells used to design the representation ladder, heads, and penalties? How is the shared monotonic trend of the ladder ruled out as an unobserved confounder?

Q3 - Anomalously high random control RS (0.84–0.980.84\text{--}0.98): Please clarify the exact mathematical and algorithmic implementation of ZZ, AA, scaling, and ridge fitting in the random capacity control. Does this high baseline score not suggest RS is hypersensitive to probe capacity rather than true signal?

Q4 - Preregistration claims: Are there immutable, timestamped external registries for the "pre-specified gate" and "independent sweep"? If not, please reframe these as post-hoc analyses.

Q5 - Prospective out-of-sample forecasting: Please report RMU-to-IMG forecasting performance on unseen datasets, architectures, or representation tiers, with error intervals broken down by seed and tier, to support the claim of cross-domain tracking.

Q6 - Figure quality & artifacts (Figure 2 & Appendix): Clarify the "+a" artifact in Figure 2 (CMU-MOSI), define the signed-log transform, and re-export vector figures with legible font sizes, axis ticks, and formatted appendix tables.

Reviewer_ZK2JOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this manuscript, the authors argue that whether an auxiliary modality helps a prediction depends on the primary representation rather than on the modality itself. They define three quantities: incremental modality gain (IMG), the test-risk reduction from adding the auxiliary modality under matched ridge heads; cross-modal predictability (RS); and residual modality utility (RMU), the risk reduction obtained by probing the primary model's target residual with the cross-fitted auxiliary residual. They give a construction in which RS approaches one while the gain stays positive. On representation ladders for glaucoma (fundus to visual field, with OCT as the auxiliary), CMU-MOSI (text, with audio) and AV-MNIST (image, with audio), IMG shrinks as the representation improves, RS does not track this, and RMU rank-correlates with IMG (pooled Spearman ρ = 0.651), supported by negative, missingness and capacity controls, a comparison with six alternative diagnostics, and an XGBoost/Shapley sweep over seven clinical backbones. Overall, the protocol is careful, but the conceptual contribution and practical utility are limited.

Critical strengths

Strengths

  1. The central conceptual point is clear and correct: modality value is conditional on the representation, and cross-modal reconstructability is not evidence of redundancy with respect to the target. The simple residual construction (Eq. 3) makes this concrete.

  2. The estimation protocol is unusually disciplined: matched head classes, a parameter-count control with Gaussian inputs, shuffle and missingness controls, cross-fitted residuals, a pre-specified evaluation gate, no post hoc exclusions, and per-seed reporting.

  3. Claims are scoped honestly, limitations are stated plainly, and RMU is compared against six alternative diagnostics on identical cells.

Critical weaknesses

Major issues

  1. The novelty is limited and the main validation is close to tautological. With matched linear (ridge) heads, the Frisch–Waugh–Lovell theorem implies that the gain from adding A to a linear regression on Z is the fit of the Y-residual on the A-residual, so RMU is essentially a cross-fitted, regularized restatement of IMG and its rank agreement with IMG is expected by construction (the authors acknowledge it is “not a new residualisation principle”). The main empirical finding, that auxiliary gain shrinks as the primary representation improves, is also largely expected. To constitute a contribution, the paper should show settings where RMU and IMG diverge informatively, use nonlinear fusion heads, or demonstrate prospective use in which RMU predicts the gain before the auxiliary modality is collected at scale.

  2. The practical value and the strength of the evidence are unclear. RMU requires the same paired (Z, A, Y) data as IMG, so it is not obvious when one would compute RMU instead of IMG directly. The evidence rests on four levels per dataset across three datasets (13 seed clusters) and is driven by artificially weakened levels (16- or 32-dimensional Gaussian bottlenecks, and raw pixels with a base model worse than uniform); after dropping the weakest levels, ρ falls to 0.369 [0.023, 0.711]. In glaucoma, ridge IMG is near zero at G1–G3, while the XGBoost sweep shows OCT contributing 0.11–0.71 dB for comparable backbones, which suggests the “collapse” partly reflects the capacity of a linear head (especially on 12,544-dimensional tokens with 1,125 training records) rather than an absence of residual information.

Minor issues

  1. The presentation is very dense and relies on project-specific terminology (gate, contract, cells, ladders), with key results scattered between the main text and appendices; negative RS values and entries reported as “± 0.0000” need explanation. CMU-MOSI and AV-MNIST are small, dated benchmarks, and a modern large-scale multimodal setting would strengthen the claim of generality.

  2. The glaucoma cohort is described only briefly (source, inclusion criteria, imaging devices, ethics approval), and its split was reused from earlier projects, which weakens its role as independent evidence. For classification, the simplex projection in RMU mixes calibration repair with information gain, as seen at AV-MNIST A0; this should be discussed in the main text.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.