- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a protocol for evaluating whether a multimodal fusion model’s advantage over a nonlinear baseline persists after both models undergo the same additive projection. It reports the original performance gap, the projected gap, and the difference in projection sensitivity, alongside capacity controls and paired uncertainty estimates. Experiments on synthetic tasks, movie classification, sentiment analysis, and VQA show that conclusions about fusion gains can change with the baseline, its optimization, and the projection reference.
Critical strengths
- The four-score comparison is clear and useful: it accounts for interactions that a nonlinear concatenation baseline may already exploit.
- The experiments examine several distinct sources of variation, including parameter matching, learning-rate selection, feature representations, and reference sampling.
- The paper states the assumptions behind its statistical analysis and separates projection sensitivity from the magnitude of a model’s functional interactions.
Critical weaknesses
- The methodological novelty is modest. EMAP and comparisons involving nonlinear concatenation baselines have prior precedent, including work that reports projected performance for both models. The main contribution lies in consolidating these elements into a shared-reference protocol with paired inference and additional finite-reference analysis.
- The product reference provides a mathematically well-defined projection and does not assume that the observed modalities are independent. However, it evaluates the models on cross-modal combinations that may be rare or outside the observed joint support. The reference sweeps assess stability within this product-reference estimand, but do not determine whether the main effects reflect behavior on realistic inputs or extrapolation on mismatched pairs.
- Parameter-count matching does not always produce a viable comparison. The LMF-matched baselines use hidden widths of only 3/2 or 4/2, and several LMF-related runs collapse to constant predictions. These matched contrasts therefore have limited value as evidence about architectural differences, even though the shared-baseline results partly mitigate this concern.
- The reported intervals are conditional on selected fits and fixed references and do not jointly incorporate retraining, model-selection, and reference-sampling variability. In addition, the MM-IMDb analysis was informed by earlier test results and should therefore be interpreted as exploratory rather than confirmatory.
Questions for authors
- Relative to the original EMAP work and InterSHAP, which part of the proposed protocol constitutes the main new capability: the shared-reference contrasts, the paired inference, or the finite-reference analysis?
- Can the authors quantify how far the constructed cross-pairs lie from the observed joint support? Do the main conclusions, particularly the VQA MLB result, remain under semantically matched, local, or conditional reference construction?
- Why is exact parameter-count matching considered informative when it forces the LMF baselines to use 3/2 or 4/2 hidden units? Would the conclusions hold with an architecture-constrained baseline using a reasonable minimum width and no constant-prediction runs?
- How would the principal intervals change under joint reference-and-scoring resampling or additional independent training runs? Could the historically informed MM-IMDb findings be verified on an untouched split or presented more explicitly as exploratory?