acceptodds
Under review as a conference paper at ICLR 2027

A Modality with No New Information Can Improve Multimodal Fusion

Abstract

A fusion model that beats its strongest unimodal component is often credited with using information from the added modality. Adding a branch, however, also changes parameterization and optimization. We propose a rotation-placebo audit that separates the two: beside the real modality, retrain the same fusion architecture with an invertible orthogonal rotation of the raw base input, which adds zero Shannon information, and with parameter-matched copy and Gaussian arms. A gain that the placebo reproduces cannot be attributed to new information. In the knowledge-tracing deployment that motivated this work, the audit overturns the original attribution: the placebo raises area under the curve by an amount comparable to the gain that had justified collecting student feedback, including records from minors. It also raises it in the public Computer Science Education Data Mining (CSEDM) challenge, though there its log-loss gain (+0.005 bits) is below our 0.01-bit floor. The audit is also specific. It stays near zero where a real modality helps substantially: on frozen Contrastive Language–Image Pretraining (CLIP) features, in a registered audit of the official Adaptive Gradient Modulation implementation, and in a second public audit that reproduces a +10.81-point real-modality gain while copy and rotation gain only +1.96 and +0.65 points. Where the placebo fires, registered interventions control it. It reduces held-out cross-entropy in all 12 registered raw-input audio–visual MNIST (AV-MNIST) configurations (60 paired runs). Saturating the task removes it, and rank-256 principal-component truncation and whitening make all six settings fail our prespecified positive-placebo criterion while preserving a positive real-modality gain (largest residual +0.022 bits, positive in 4/5 seeds). A converged probe adds a one-sided certificate that bounds the cost of ignoring a candidate, with no violations among 42 eligible cases. A base-versus-fusion comparison alone cannot attribute a gain to new information; the audit makes that attribution testable.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal fusion, evaluation methodology, placebo control, ablation, attribution, preregistration, optimization, conditional mutual information

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9ugaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that outperforming a unimodal baseline does not demonstrate that the added modality provides new information, since introducing an additional branch also changes the model’s parameterization and optimization dynamics. The authors propose a “rotation-placebo audit,” which applies an invertible orthogonal rotation to the base input to construct a control that contains no incremental information while following the same fusion pathway.

Critical strengths
  • The paper identifies an important yet easily overlooked attribution problem in multimodal research and proposes a simple, theoretically well-grounded, and practically meaningful auditing method.
  • The experiments cover diverse datasets, architectures, and real-world deployment settings. By reporting cases in which the placebo effect both appears and remains absent, the paper provides convincing evidence for the method’s discriminative ability and clarifies its scope of applicability.
Critical weaknesses

The paper presents an interesting perspective and appears to include extensive experimentation, but it lacks a clear research narrative. Despite my careful reading, I found the overall argumentative structure difficult to follow: additional experiments, qualifications, and explanations often seem to be introduced reactively to patch or defend earlier claims rather than to develop a coherent line of inquiry. Take the introduction as an example, the transition between Paragraphs 2 and 3 is abrupt, and the term “registered” appears without sufficient motivation or context. More broadly, the manuscript feels closer to an exhaustive record optimized for agent-based reading than to a narrative designed for human readers.

Questions for authors
  • The paper lacks a coherent narrative, as noted in the Weaknesses section. Some methodological choices are also difficult to understand. Questions:

    • If the proposed approach is intended to be generally applicable to multimodal architectures, why are two BML methods selected for the analysis?
    • Several figures and tables are not properly referenced in the narrative; e.g. only Panels B and C of Figure 1 are cited, and Figure 1C appear more than six pages later.
    • The experimental discussion also feels unusual, as considerable space is devoted to establishing the reliability of the experiments rather than explaining what evidence they provide.
    • The Introduction does not cite any prior work, which is highly unusual. Could the authors better position the research question and proposed method within the existing literature?
  • The paper contains a large number of experiments. Could the authors clearly summarize the conditions under which the proposed method is applicable and the main conclusions supported by these experiments? Highlighting the key findings would substantially improve readability.

  • Could the authors conduct experiments with more carefully controlled preprocessing to disentangle the respective effects of the orthogonal rotation, feature scaling, and changes in optimization conditioning?

  • Potential reference inaccuracy: The cited MultiBench: Multiscale Benchmarks for Multimodal Representation Learning was published at NeurIPS 2021, whereas the related JMLR article is titled MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning. The citation should be corrected so that the title and publication venue refer to the same work.

Reviewer_xkoqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper shows that adding a rotated and standardized copy of an existing input can improve multimodal fusion despite providing no new information. It proposes placebo controls to audit whether fusion gains require additional information and investigates how optimization, regularization, and input representation influence these gains.

Critical strengths
  1. The paper provides strong empirical evidence that adding a rotated and standardized version of an existing input can improve fusion performance without introducing genuinely new information, meaning a completely different modality, in some multimodal learning tasks. This challenges the common attribution of multimodal improvements solely to information supplied by the added modality.

  2. The main result is complemented with theoretical analysis and targeted interventions on input scaling, dimensionality, whitening, and task difficulty. These analyses constrain plausible explanations for the gains and identify conditions that suppress placebo benefits while preserving real-modality improvements.

Critical weaknesses
  1. The paper uses a lot of AI in its writing without much postprocessing / revision. Although using AI is fine since the authors disclose it in their AI use statement, this introduces a style which is at times quite dense and difficult to follow. A lot of unintroduced ideas are suddenly introduced without their context being properly clarified. For example, the abstract and the introduction exhibit this trait to a great extent, but so does the remainder of the paper too. For me readability is one of the primary concerns of this paper. I would suggest the authors to revise the entire paper to improve clarity. Ideally, concepts should first be introduced before they are being referenced and avoid writing out-of-context statements; the narrative of the paper should flow smoothly rather than be entangled as it currently is.

  2. Copy control almost never helps. The main type of control that provides gains is rotation. Rotation is not "no new information"; it is more like data augmentation - it tells the model the label of the input is invariant to rotation. To judge this properly, there should be unimodal experiments where these perturbed (such as rotated) versions are used as data augmentation and that should count as a baseline. Moreover, given that the copy control experiments hardly ever helps, the "no new information" claim cannot fully be accepted with certainty.

  3. There is not much analysis on why certain controls are helping the fusion, i.e., what is the model learning that it otherwise ignores. If the no new information is correct, it means that model should be able to achieve the same performance even without the redundant controls, if the mechanisms learned as a result of introducing such controls are explicitly implemented. Without such analyses, it is difficult to validate the proposed hypotheses.

  4. It is also important to understand whether the gains from controls discussed in this paper are actually orthogonal or redundant to gains from genuinely new modalities. To this end, there should be experiments new modalities alongside transformed versions of existing modalities all features together in multimodal fusion.

Questions for authors

Refer to the Weaknesses section.

Reviewer_QypJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses whether performance improvements in multimodal fusion are due to the contribution of new information or are instead caused by the changes in a model's architecture and optimization dynamics resulting from the addition of an additional modality. To check this, the paper proposes a rotation-placebo audit, where the additional modality is replaced by an information-preserving rotated copy of the base input, and copy and Gaussian-noise controls are included. The results are shown on several datasets and models: AV-MNIST, Knowledge Tracing, CLIP on Hateful Memes, AGM and MLA on CREMA-D.

Critical strengths
  • The paper focuses on an important question regarding current approaches in multimodal fusion research. Identifying whether the gain obtained after adding a new modality is achieved from new information supplied by that modality or is obtained from changes in optimization and regularization introduced by the additional branch.
  • The paper shows an interesting observation that an input carrying zero incremental information can also improve performance. By passing a rotated copy of the base input along with the base input to the model, the paper demonstrates that part of a fusion gain can arise purely from the added branch.
  • The paper identifies and quantifies how per-block standardization distorts the rotation and test it against an exactly orthogonal variant. The paper also shows that the conclusions hold across different significance thresholds.
Critical weaknesses
  • In some scenarios, I agree with the authors on simpler tasks, including AV-MNIST classification, adding the fusion branch results in gain which may arise from optimization effects rather than new information. However, in harder recognition scenarios like person re-identification, additional multiple modality information provides a genuinely different perspective and new information, ex. infrared imagery under low illumination, and depth and body-shape cues in clothing variations. The paper does not evaluate such harder tasks where multiple modalities help, so its practical relevance to hard multimodal problems remains unclear.
  • The paper does not consider the dense prediction tasks, such as super-resolution.The analysis is constrained to classification using cross-entropy/AUC metrics. In multimodal guided super-resolution (ex. RGB-guided depth SR, pan-sharpening, multi-contrast MRI SR), the additional modality contains high-frequency information that is missing from the low-resolution input data, thus requiring spatially aligned fusion. A rotation placebo would destroy this spatial correspondence and it is unclear how the audit applies to such tasks.
  • The paper does not present results for the recent era of multimodal models, such as MLLMs.
  • Can the authors please explain in detail the interpretation of the placebo adjusted gain? This placebo-adjusted gain assumes that adding a real modality and a rotated copy of the base input results in the same optimization effect. However, this assumption needs more justification, because the actual modality may differ from the rotated representation in statistical properties and learning dynamics, which may produce larger or smaller optimization-level gains. As the placebo is shown as a useful diagnostic signal in the paper, the difference grealg_{real}-grotg_{rot} can not be interpreted as an estimate of the modality’s information contribution without any supporting justification.
Questions for authors
  • In some scenarios, I agree with the authors on simpler tasks, including AV-MNIST classification, adding the fusion branch results in gain which may arise from optimization effects rather than new information. However, in harder recognition scenarios like person re-identification, additional multiple modality information provides a genuinely different perspective and new information, ex. infrared imagery under low illumination, and depth and body-shape cues in clothing variations. The paper does not evaluate such harder tasks where multiple modalities help, so its practical relevance to hard multimodal problems remains unclear.
  • The paper does not consider the dense prediction tasks, such as super-resolution.The analysis is constrained to classification using cross-entropy/AUC metrics. In multimodal guided super-resolution (ex. RGB-guided depth SR, pan-sharpening, multi-contrast MRI SR), the additional modality contains high-frequency information that is missing from the low-resolution input data, thus requiring spatially aligned fusion. A rotation placebo would destroy this spatial correspondence and it is unclear how the audit applies to such tasks.
  • The paper does not present results for the recent era of multimodal models, such as MLLMs.
  • Can the authors please explain in detail the interpretation of the placebo adjusted gain? This placebo-adjusted gain assumes that adding a real modality and a rotated copy of the base input results in the same optimization effect. However, this assumption needs more justification, because the actual modality may differ from the rotated representation in statistical properties and learning dynamics, which may produce larger or smaller optimization-level gains. As the placebo is shown as a useful diagnostic signal in the paper, the difference grealg_{real}-grotg_{rot} can not be interpreted as an estimate of the modality’s information contribution without any supporting justification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.