acceptodds
Withdrawn from ICLR 2027

Baseline-Relative Projection Diagnostics for Multimodal Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal fusion gains depend on the nonlinear baseline used for comparison. We develop a reusable diagnostic protocol that applies empirical multimodally-additive projection (EMAP) to both a fusion head and its baseline. Their original gap separates into a projected gap and excess projection sensitivity. The protocol combines shared references, parameter-matched controls, sealed model selection and paired uncertainty. We characterise finite-reference bias, ranking variance and paired score covariance under explicit assumptions. ANOVA and Shapley relations specify the functional interaction being probed. Experiments cover MM-IMDb, MOSI, MOSEI and Balanced Binary Abstract Scenes VQA. On MM-IMDb, Mamba-style retains projected advantages of 1.412 to 1.533 AUC percentage points over matched concatenation with word2vec features. All projected-gap intervals include zero with sentence-transformer features. On MOSEI, validation selection reduces a source-rate TFN advantage over matched concatenation from 0.753 to −0.001 MAE units. In VQA, MLB-style's original advantage of 2.302 accuracy percentage points becomes a projected gap of −0.005, with interval [−0.432, 0.401]. Four predeclared mechanisms across three new datasets yield distinct patterns of baseline strength and relative projection response. An accompanying library implements projection, paired inference and a common reporting record. The protocol makes these properties directly testable components of a fusion comparison.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal learning, fusion evaluation, additive projection, interaction diagnostics, baseline robustness, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3Ge3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a protocol for evaluating whether a multimodal fusion model’s advantage over a nonlinear baseline persists after both models undergo the same additive projection. It reports the original performance gap, the projected gap, and the difference in projection sensitivity, alongside capacity controls and paired uncertainty estimates. Experiments on synthetic tasks, movie classification, sentiment analysis, and VQA show that conclusions about fusion gains can change with the baseline, its optimization, and the projection reference.

Critical strengths
  1. The four-score comparison is clear and useful: it accounts for interactions that a nonlinear concatenation baseline may already exploit.
  2. The experiments examine several distinct sources of variation, including parameter matching, learning-rate selection, feature representations, and reference sampling.
  3. The paper states the assumptions behind its statistical analysis and separates projection sensitivity from the magnitude of a model’s functional interactions.
Critical weaknesses
  1. The methodological novelty is modest. EMAP and comparisons involving nonlinear concatenation baselines have prior precedent, including work that reports projected performance for both models. The main contribution lies in consolidating these elements into a shared-reference protocol with paired inference and additional finite-reference analysis.
  2. The product reference provides a mathematically well-defined projection and does not assume that the observed modalities are independent. However, it evaluates the models on cross-modal combinations that may be rare or outside the observed joint support. The reference sweeps assess stability within this product-reference estimand, but do not determine whether the main effects reflect behavior on realistic inputs or extrapolation on mismatched pairs.
  3. Parameter-count matching does not always produce a viable comparison. The LMF-matched baselines use hidden widths of only 3/2 or 4/2, and several LMF-related runs collapse to constant predictions. These matched contrasts therefore have limited value as evidence about architectural differences, even though the shared-baseline results partly mitigate this concern.
  4. The reported intervals are conditional on selected fits and fixed references and do not jointly incorporate retraining, model-selection, and reference-sampling variability. In addition, the MM-IMDb analysis was informed by earlier test results and should therefore be interpreted as exploratory rather than confirmatory.
Questions for authors
  1. Relative to the original EMAP work and InterSHAP, which part of the proposed protocol constitutes the main new capability: the shared-reference contrasts, the paired inference, or the finite-reference analysis?
  2. Can the authors quantify how far the constructed cross-pairs lie from the observed joint support? Do the main conclusions, particularly the VQA MLB result, remain under semantically matched, local, or conditional reference construction?
  3. Why is exact parameter-count matching considered informative when it forces the LMF baselines to use 3/2 or 4/2 hidden units? Would the conclusions hold with an architecture-constrained baseline using a reasonable minimum width and no constant-prediction runs?
  4. How would the principal intervals change under joint reference-and-scoring resampling or additional independent training runs? Could the historically informed MM-IMDb findings be verified on an untouched split or presented more explicitly as exploratory?
Reviewer_v1xUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a protocol that applies empirical multimodally-additive projection (EMAP) to both a multimodal predictor and its nonlinear baseline. It decomposes their performance difference into a projected gap and excess projection sensitivity, accompanied by shared references, capacity controls, and paired uncertainty estimates. Experiments cover MM-IMDb, MOSI, MOSEI, and balanced binary VQA.

Critical strengths

The protocol addresses important confounds in fusion comparisons through shared references and validation-based model selection. Extensive experiments across multiple datasets support its practical feasibility.

Critical weaknesses
  1. The figures have readability issues. Specifically, Figure 2 does not provide a clear description of each step in the diagnostic workflow.

  2. The diagnostic objective overlaps with prior EMAP analyses. Although the protocol provides more reporting and statistical analysis, its added value over existing interaction diagnostics is not sufficiently demonstrated.

  3. The presentation is hard to follow, and seems to fail to meet the writing standards expected of a normal academic paper..

Questions for authors

1.What additional conclusions or inferential benefits the proposed protocol provides beyond applying EMAP to both the fusion model and its baseline under the same experimental controls.

2.What practical check would identify the unobserved cross-pair failure illustrated in Appendix R?

Reviewer_oRcZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses empirical multimodally-additive projection (EMAP) to investigate whether performance gains from multimodal fusion models can be attributed to cross-modal interactions. The authors apply the projection to both fusion models and their nonlinear concatenation baselines, decomposing the original performance gap into a projected gap and excess projection sensitivity. Experiments across several multimodal benchmarks show that some gains disappear after projection, while others persist. The paper also highlights the importance of strong baselines and careful optimization. The proposed evaluation protocol is supported by statistical analyses, reference-sensitivity studies, and extensive empirical comparisons.

Critical strengths
  1. The authors make substantial efforts to ensure fair comparisons through parameter-matched baselines, validation-based model selection, shared projection references, and paired statistical inference. These efforts strengthen the reliability and reproducibility of the empirical findings.
  2. The experiments illustrate how conclusions about multimodal fusion can change depending on baseline selection, optimization, and additive projection. These results provide useful practical guidance for evaluating multimodal models.
Critical weaknesses
  1. The central idea of using EMAP to assess cross-modal interactions was introduced by Hessel and Lee (2020). Furthermore, Wenderoth et al. (2025) already examined EMAP-based performance gaps for both multimodal models and nonlinear baselines. While this paper provides a more rigorous and standardized evaluation protocol, the extent of its conceptual and methodological advance over these prior works remains unclear.
  2. Many of the paper's conclusions appear to reinforce existing observations, particularly that performance improvements cannot necessarily be attributed to cross-modal interactions and that strong baselines are essential. It is not sufficiently clear what fundamentally new understanding of multimodal interactions is enabled by the proposed framework. This limits the significance of the contribution beyond improved evaluation practices.
Questions for authors
  1. What are the key conceptual or methodological advances over Hessel and Lee (2020) and Wenderoth et al. (2025)? In particular, what can the proposed framework establish that could not be established using existing EMAP-based approaches?
  2. What are the most important new scientific insights obtained from the proposed framework? Could the authors identify specific conclusions that depend on their methodological extensions, rather than simply confirming findings already established in prior work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.