- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that outperforming a unimodal baseline does not demonstrate that the added modality provides new information, since introducing an additional branch also changes the model’s parameterization and optimization dynamics. The authors propose a “rotation-placebo audit,” which applies an invertible orthogonal rotation to the base input to construct a control that contains no incremental information while following the same fusion pathway.
Critical strengths
- The paper identifies an important yet easily overlooked attribution problem in multimodal research and proposes a simple, theoretically well-grounded, and practically meaningful auditing method.
- The experiments cover diverse datasets, architectures, and real-world deployment settings. By reporting cases in which the placebo effect both appears and remains absent, the paper provides convincing evidence for the method’s discriminative ability and clarifies its scope of applicability.
Critical weaknesses
The paper presents an interesting perspective and appears to include extensive experimentation, but it lacks a clear research narrative. Despite my careful reading, I found the overall argumentative structure difficult to follow: additional experiments, qualifications, and explanations often seem to be introduced reactively to patch or defend earlier claims rather than to develop a coherent line of inquiry. Take the introduction as an example, the transition between Paragraphs 2 and 3 is abrupt, and the term “registered” appears without sufficient motivation or context. More broadly, the manuscript feels closer to an exhaustive record optimized for agent-based reading than to a narrative designed for human readers.
Questions for authors
-
The paper lacks a coherent narrative, as noted in the Weaknesses section. Some methodological choices are also difficult to understand. Questions:
- If the proposed approach is intended to be generally applicable to multimodal architectures, why are two BML methods selected for the analysis?
- Several figures and tables are not properly referenced in the narrative; e.g. only Panels B and C of Figure 1 are cited, and Figure 1C appear more than six pages later.
- The experimental discussion also feels unusual, as considerable space is devoted to establishing the reliability of the experiments rather than explaining what evidence they provide.
- The Introduction does not cite any prior work, which is highly unusual. Could the authors better position the research question and proposed method within the existing literature?
-
The paper contains a large number of experiments. Could the authors clearly summarize the conditions under which the proposed method is applicable and the main conclusions supported by these experiments? Highlighting the key findings would substantially improve readability.
-
Could the authors conduct experiments with more carefully controlled preprocessing to disentangle the respective effects of the orthogonal rotation, feature scaling, and changes in optimization conditioning?
-
Potential reference inaccuracy: The cited MultiBench: Multiscale Benchmarks for Multimodal Representation Learning was published at NeurIPS 2021, whereas the related JMLR article is titled MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning. The citation should be corrected so that the title and publication venue refer to the same work.