- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes S2MLVM, a supervised latent variable model for multimodal prediction. Each modality goes through its own encoder and normalizing flow. The flow outputs and a target embedding are then modeled jointly by a linear Gaussian factor model with four latent blocks: shared predictive, modality-specific predictive, and a shared nuisance block that the target does not depend on. Training combines a task loss on the posterior means of the predictive factors, the flow plus Gaussian negative log-likelihood, and an auxiliary masked-reconstruction or contrastive loss. A Gaussian analysis links the model to partial information decomposition through a conditional-independence adversary. Experiments cover subspace recovery on synthetic data, TriFeatures, and eight real benchmarks (MultiBench and the MCR setups) with two and three modalities.
Critical strengths
-
The nuisance factor z^c is a reasonable addition. Without it, models in the CCA/JIVE family treat any cross-modal correlation as shared signal, so separating task-irrelevant shared variation from predictive shared variation makes sense for supervised fusion. The model also has a clean interpretation as a generalization of PCCA, JIVE and sJIVE.
-
The evaluation is broad: synthetic recovery, factor-wise probes on TriFeatures, eight real datasets and three-modality settings. Gains over joint training are clear on CREMA-D and UCF101. This breadth is the main thing in the paper's favour.
Critical weaknesses
-
The experiments don't support the claim that S2MLVM "consistently outperforms" prior methods. The FactorCL, CoMM and InfMasking numbers are linear probes on frozen self-supervised features, so they aren't comparable to an end-to-end supervised model. Against MCR the gains are small: 0.1 on CREMA-D, 0.5 on MIMIC and 0.8 on MOSI. S2MLVM is lower than MCR on two-modality MOSEI and lower than every baseline on three-modality V&T Contact. The key control is also missing: the same encoders and auxiliary loss without the LVM, which would show whether the gain comes from the structured model at all. MISA and MFM are standard supervised shared/private baselines on these benchmarks, but they are neither compared nor cited. The ablations are hard to trust as well, since the full model scores 73.6 on MOSI validation in Table 5 but 79.0 in Table 9 under the same setting. This is fixable with comparable protocols, the no-LVM control, and multi-seed ablations.
-
The main story isn't backed by evidence. The paper is motivated by unimodal bias but never measures it. Nothing on real data checks whether the factors are actually disentangled or whether z^c is truly task-irrelevant. The PID section is disconnected from the method: the latent blocks don't map to PID atoms, and no PID quantity is estimated anywhere. Novelty is also limited, since shared/private factorization with a supervised target builds on sJIVE, group factor analysis, MISA and MFM, and Zhao et al. (NeurIPS 2025) already map modalities to latent Gaussians with flows. This is partly fundamental. At minimum, the bias claim needs direct measurement and the decomposition needs to be checked on real data.
Questions for authors
-
How does S2MLVM compare with the same encoders, auxiliary loss and MLP head but without the LVM and flows? This matters most for judging the contribution.
-
Were the CoMM and InfMasking rows on CREMA-D, UCF101 and MOSEI (reproduced, no asterisk) evaluated by linear probing or fine-tuning? And was MCR tuned on the MultiBench datasets with the same budget as S2MLVM?
-
Why does the full model's validation accuracy differ between Table 5 and Table 9 (73.6 vs 79.0 on MOSI) under the same setting?
-
How is y~ obtained for categorical targets? If it is learned, what prevents it from collapsing under the density loss?
-
Is there any direct evidence that S2MLVM reduces unimodal bias, e.g. unimodal accuracy of each encoder inside the trained multimodal model, compared with joint training?
Q1
We have added this comparison in Appendix A.6 (Table 11). The baseline keeps the same source encoders, auxiliary representation loss, and MLP head, but removes both the flows and the LVM; the encoder representations are concatenated and passed directly to the MLP.
Although the LVM and normalizing flows introduce only a small number of additional parameters, they yield a substantial improvement in overall performance.
Q2
CoMM and InfMasking on CREMA-D, UCF101 and MOSEI were reproduced with the official code, evaluated with nonlinear, MLP probing (using the same classification head as our method). For MCR on the MultiBench datasets, we ran the official code with the default hyperparameters on similar datasets. We have now tuned these baselines more exhaustively (We thank the reviewer for pointing this out), with the similar tuning budget as S2MLVM. The updated results are given in Table 3 & 4.
Q3
We have now updated Table 5 (Section 5.3) to use the same hyperparameter settings (batch size, masking ratio, temperature, and loss weights) as Table 9 (Appendix A.4). The relative merits of the methods do not change.
Q4
To handle discrete class labels, we learn continuous label embeddings y~ (Section 2.1). These embeddings avoid collapse due to the joint training objective (Section 2.4). Specifically, the LVM maximizes the likelihood of y~, while the classification loss ensures that y~ remains highly predictive of the true labels. Any degeneracy in the embeddings would immediately degrade the predictive power of the latent variables, which the classification objective penalizes.
Q5
We have now trained MLP classifiers on the unimodal representations of our trained models and report the test results in Appendix A.5 (Table 10).
The better representations of our model leads to consistently better accuracy than joint training, especially for weaker modalities.