acceptodds
Under review as a conference paper at ICLR 2027

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Abstract

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal representation learning, partial information decomposition, structured latent variable models, supervised learning, multimodal interactions, normalizing flows

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RhNuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes S2MLVM, a supervised latent variable model for multimodal prediction. Each modality goes through its own encoder and normalizing flow. The flow outputs and a target embedding are then modeled jointly by a linear Gaussian factor model with four latent blocks: shared predictive, modality-specific predictive, and a shared nuisance block that the target does not depend on. Training combines a task loss on the posterior means of the predictive factors, the flow plus Gaussian negative log-likelihood, and an auxiliary masked-reconstruction or contrastive loss. A Gaussian analysis links the model to partial information decomposition through a conditional-independence adversary. Experiments cover subspace recovery on synthetic data, TriFeatures, and eight real benchmarks (MultiBench and the MCR setups) with two and three modalities.

Critical strengths
  1. The nuisance factor z^c is a reasonable addition. Without it, models in the CCA/JIVE family treat any cross-modal correlation as shared signal, so separating task-irrelevant shared variation from predictive shared variation makes sense for supervised fusion. The model also has a clean interpretation as a generalization of PCCA, JIVE and sJIVE.

  2. The evaluation is broad: synthetic recovery, factor-wise probes on TriFeatures, eight real datasets and three-modality settings. Gains over joint training are clear on CREMA-D and UCF101. This breadth is the main thing in the paper's favour.

Critical weaknesses
  1. The experiments don't support the claim that S2MLVM "consistently outperforms" prior methods. The FactorCL, CoMM and InfMasking numbers are linear probes on frozen self-supervised features, so they aren't comparable to an end-to-end supervised model. Against MCR the gains are small: 0.1 on CREMA-D, 0.5 on MIMIC and 0.8 on MOSI. S2MLVM is lower than MCR on two-modality MOSEI and lower than every baseline on three-modality V&T Contact. The key control is also missing: the same encoders and auxiliary loss without the LVM, which would show whether the gain comes from the structured model at all. MISA and MFM are standard supervised shared/private baselines on these benchmarks, but they are neither compared nor cited. The ablations are hard to trust as well, since the full model scores 73.6 on MOSI validation in Table 5 but 79.0 in Table 9 under the same setting. This is fixable with comparable protocols, the no-LVM control, and multi-seed ablations.

  2. The main story isn't backed by evidence. The paper is motivated by unimodal bias but never measures it. Nothing on real data checks whether the factors are actually disentangled or whether z^c is truly task-irrelevant. The PID section is disconnected from the method: the latent blocks don't map to PID atoms, and no PID quantity is estimated anywhere. Novelty is also limited, since shared/private factorization with a supervised target builds on sJIVE, group factor analysis, MISA and MFM, and Zhao et al. (NeurIPS 2025) already map modalities to latent Gaussians with flows. This is partly fundamental. At minimum, the bias claim needs direct measurement and the decomposition needs to be checked on real data.

Questions for authors
  1. How does S2MLVM compare with the same encoders, auxiliary loss and MLP head but without the LVM and flows? This matters most for judging the contribution.

  2. Were the CoMM and InfMasking rows on CREMA-D, UCF101 and MOSEI (reproduced, no asterisk) evaluated by linear probing or fine-tuning? And was MCR tuned on the MultiBench datasets with the same budget as S2MLVM?

  3. Why does the full model's validation accuracy differ between Table 5 and Table 9 (73.6 vs 79.0 on MOSI) under the same setting?

  4. How is y~ obtained for categorical targets? If it is learned, what prevents it from collapsing under the density loss?

  5. Is there any direct evidence that S2MLVM reduces unimodal bias, e.g. unimodal accuracy of each encoder inside the trained multimodal model, compared with joint training?

1 comment

Q1

We have added this comparison in Appendix A.6 (Table 11). The baseline keeps the same source encoders, auxiliary representation loss, and MLP head, but removes both the flows and the LVM; the encoder representations are concatenated and passed directly to the MLP.

VariantMOSIMUSTARD
S2MLVM-MASK79.8 ± 1.770.3 ± 1.6
w/o LVM & Flow75.7 ± 2.668.8 ± 1.4
S2MLVM-CONTRAST82.7 ± 1.072.2 ± 1.6
w/o LVM & Flow80.2 ± 1.869.1 ± 2.3

Although the LVM and normalizing flows introduce only a small number of additional parameters, they yield a substantial improvement in overall performance.

Q2

CoMM and InfMasking on CREMA-D, UCF101 and MOSEI were reproduced with the official code, evaluated with nonlinear, MLP probing (using the same classification head as our method). For MCR on the MultiBench datasets, we ran the official code with the default hyperparameters on similar datasets. We have now tuned these baselines more exhaustively (We thank the reviewer for pointing this out), with the similar tuning budget as S2MLVM. The updated results are given in Table 3 & 4.

Q3

We have now updated Table 5 (Section 5.3) to use the same hyperparameter settings (batch size, masking ratio, temperature, and loss weights) as Table 9 (Appendix A.4). The relative merits of the methods do not change.

Q4

To handle discrete class labels, we learn continuous label embeddings y~\tilde{y} (Section 2.1). These embeddings avoid collapse due to the joint training objective (Section 2.4). Specifically, the LVM maximizes the likelihood of y~\tilde{y}, while the classification loss ensures that y~\tilde{y} remains highly predictive of the true labels. Any degeneracy in the embeddings would immediately degrade the predictive power of the latent variables, which the classification objective penalizes.

Q5

We have now trained MLP classifiers on the unimodal representations of our trained models and report the test results in Appendix A.5 (Table 10).

DatasetModelVision onlyText onlyFull model
MOSIJoint training50.3 ± 1.967.9 ± 1.472.9 ± 1.6
MOSIS2MLVM-MASK53.1 ± 3.769.2 ± 1.875.8 ± 0.8
MOSIS2MLVM-CONTRAST52.5 ± 2.171.4 ± 1.074.3 ± 1.9
MUSTARDJoint training51.7 ± 3.254.8 ± 2.560.6 ± 2.0
MUSTARDS2MLVM-MASK54.6 ± 2.359.5 ± 1.968.6 ± 5.5
MUSTARDS2MLVM-CONTRAST53.4 ± 3.660.7 ± 1.366.0 ± 2.9

The better representations of our model leads to consistently better accuracy than joint training, especially for weaker modalities.

Reviewer_NyEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Existing methods lack explicit modeling of task-relevant shared information, modality-specific information, and task-irrelevant dependence, which makes it difficult to separate and exploit these distinct components. The authors propose S2MLVM, which maps multimodal representations into a structured Gaussian latent space via normalizing flows and employs a supervised latent variable model to separate shared predictive factors, modality-specific predictive factors, and a shared nuisance factor.

Critical strengths

The structured latent variable model brings shared, unique, and task-irrelevant information together in a single probabilistic framework, making the overall approach clear and consistent. The paper also provides some theoretical insights into how synergistic information arises in this framework.

Critical weaknesses
  1. The structured latent variable model and the training objective derived from it are never clearly explained in terms of why they alleviate modality competition or unimodal bias during training, and this is not connected back to the problem raised in the introduction.

  2. The two auxiliary objectives in Section 2.3 are generic representation learning methods that are not specifically designed to target shared versus unique information, and their motivation is not explained. Masked reconstruction, for instance, may preserve task-relevant and task-irrelevant information alike. What, then, is its relationship to the latent variable modeling in Section 2.1?

  3. There is a gap between the theoretical analysis and the method. The PID analysis in Section 3 mainly discusses which forms of synergistic information the model is able to express, but it does not in turn guide the design of the losses in Section 2.

Questions for authors
  1. On the synthetic TriFeature benchmark, S2MLVM-CONTRAST is clearly the stronger variant (Table 2), whereas on the real-world benchmarks the two variants each win on some datasets and lose on others (Tables 3 and 4). What accounts for this difference?

  2. Can the reported gain in synergy be convincingly attributed to the structured latent variable modeling itself? MCR, an existing synergy-oriented approach, already reaches comparable values (e.g., 90.2% on the TriFeature synergy task, and the best MOSEI result in Table 3).

1 comment

W1

Recent analyses reveal that unimodal bias in joint training stems from inter-modality correlation (Zhang et al., 2024b; Kontras et al., 2025, see introduction). When shared, unique, and nuisance components are forced into a single fused representation, their interference frequently causes the classifier to ignore weaker modalities. Existing balancing techniques like MCR rely on regularizing or re-weighting different information of this entangled representation. S2MLVM instead tries to explicitly disentangle these components, allowing the downstream classifier to optimally combine them without interference.

W2

The auxiliary objectives serve the LVM in Section 2.1; they are not meant to separate shared and unique information. The flow model and the latent model require low-dimensional inputs, so the encoders must reduce the dimensionality of the raw data. At the same time, the reduced representation must keep the information in the data; otherwise the encoders could discard information or collapse to raise the likelihood (Sec. 2.3). Masked reconstruction and contrastive learning are two standard ways to retain information about the input, and we use them for this purpose only.

W3

Because our framework explicitly separates predictive components using individual latent variables, it naturally invites comparisons to Partial Information Decomposition (PID). However, our intention is not to extract latent variables that map one-to-one onto the information-theoretic atoms of BORJA PID. Because most PID research focuses on measuring and understanding information rather than optimizing it, it remains an open question whether strict PID is the ideal decomposition for improving downstream multimodal fusion. Instead, our model's structural design is more directly aligned with the multi-view redundancy assumptions widely adopted in the representation learning literature. This distinction is the primary motivation for Section 3, which clarifies that our LVM components maintain their own distinct functional roles rather than mimicking PID components.

Where our method formally connects to PID is not at the latent variable level, but at the level of the joint distribution induced by S2MLVM. Paralleling the Flow-PID framework, our use of normalizing flows allows S2MLVM to provide a structured Gaussian covariance for the transformed sources and target representation at convergence. Specifically, the nonzero blocks of S2MLVM’s loading matrix AA and the residual covariance matrix—obtained by regressing y~\tilde{y} on (u(1),u(2))(u^{(1)}, u^{(2)})—explicitly determine the Gaussian source-target marginals.

These covariance matrices directly define the reference problem in which Thin-PID minimizes the joint mutual information IQ(u(1),u(2);y~)I_Q(u^{(1)}, u^{(2)}; \tilde{y}) over compatible source couplings. Ultimately, while our latent blocks—which capture shared predictive, source-specific predictive, and shared nuisance variations—establish the rigorous statistical foundation required to define the PID reference problem, they are designed to optimize multimodal fusion rather than to serve as direct analogs to information-theoretic PID atoms.

Q1

On TriFeature, CONTRAST is better on all tasks, and most of the gap comes from the redundancy task R (93.4 / 99.9). Our hypothesis of the performance gap is as follows.

Contrastive learning on images relies on well-established augmentations that preserve the attributes defining the TriFeature labels. On the real-world datasets, many inputs are pre-extracted feature sequences or sensor signals, for which augmentations are less established, and the advantage of CONTRAST disappears. For the masked variant, the input and the reconstruction target come from the same augmented view, so random crops and color perturbations push the model to reconstruct local features rather than the attributes that define the labels. Our framework is quite flexible and we can swap in other representation methods as well.

Q2

Previously, MCR was evaluated only with its own RGB augmentation. We have now also evaluated MCR with the InfMasking augmentations (see Table 2 and Appendix A.2):

MethodRU1U2U (mean)S
MCR (RGB augmentation)99.9 ± 0.192.1 ± 1.493.2 ± 0.992.7 ± 1.290.2 ± 3.9
MCR (InfMasking augmentation)99.9 ± 0.190.9 ± 0.992.3 ± 1.091.6 ± 1.090.0 ± 3.8

Compared with MCR:

  1. Empirically, we use the same augmentation strategy for S2MLVM-CONTRAST and MCR and keep the encoder architecture identical (accuracies updated in Table 2), so the performance difference can be attributed to the LVM.
  2. Conceptually, we explicitly separate different predictive information into individual latent variables, which may better facilitate supervised fusion.
Reviewer_VZ9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes S2MLVM, a supervised structured latent variable framework for multimodal representation learning. It combines modality-specific encoders, invertible normalizing flows, and a low-rank-plus-diagonal Gaussian latent variable model to separate shared, modality-specific predictive, and shared nuisance information, incorporate masked reconstruction or contrastive learning as auxiliary objectives. Experiments on synthetic data and multiple real-world multimodal benchmarks show gains over baselines.

Critical strengths

1.The paper model multimodal learning by distinguishing complementary predictive information from redundant and task-irrelevant cross-modal dependencies. 2.The structured latent variable formulation provides a principled probabilistic perspective on multimodal representation learning. 3.The paper includes synthetic subspace recovery experiments, architectural ablations, and comparisons with supervised and self-supervised baselines.

Critical weaknesses

1.Limited methodological novelty. The framework combines existing components, including structured latent factor models, normalizing flows, and auxiliary representation learning. 2.Insufficient validation of latent disentanglement. The proposed latent factors are assigned semantic roles through the model's loading structure, but the identifiability of these factors is not sufficiently established. 3.Theoretical PID analysis is not sufficiently connected to empirical results. The tightness of the analytical bounds and their relevance to the learned representations on real datasets remain unclear. 4.The paper aims to mitigate modality competition, yet it does not comprehensively compare against established methods such as OGM, PMR, and M3ixup. These comparisons are important for demonstrating the advantage of structured latent modeling over existing optimization-based approaches. [1]Y. Fan, W. Xu, H. Wang, J. Wang and S. Guo, "PMR: Prototypical Modal Rebalance for Multimodal Learning," 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, pp. 20029-20038. [2]R. Lin and H. Hu. 2024. Adapt and explore: Multimodal mixup for representation learning. Information Fusion 105, C (May 2024). [3]Y. Wei, D. Hu, H. Du and J. -R. Wen, "On-the-Fly Modulation for Balanced Multimodal Learning," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 1, pp. 469-485, Jan. 2025. 5.Performance improvements are not consistent across benchmarks while the reasons are not clarified. 6.The paper writing should be further polished to reduce the AI style.

Questions for authors

1.How does S2MLVM differ fundamentally from existing structured multimodal latent variable models, beyond introducing a shared nuisance factor and integrating normalizing flows? 2.What conditions ensure that the shared predictive, modality-specific predictive, and nuisance latent factors are identifiable? In particular, the theoretical analysis assumes orthogonality between nuisance and task-relevant loading matrices. Is this constraint explicitly enforced during model training? 3.Could the authors report separate recovery metrics for the shared predictive, modality-specific predictive, and nuisance subspaces, rather than evaluating only the aggregate predictive subspace? 4.How does the Gaussian latent variable assumption affect the model's ability to capture nonlinear cross-modal dependencies? Would a more flexible joint density model provide additional benefits? 5.Could the authors evaluate the tightness of the proposed PID bounds and provide empirical evidence connecting the estimated information components to the learned latent representations on real-world datasets? 6.Could the authors compare S2MLVM with established modality-balancing methods under consistent experimental settings? 7.How does S2MLVM perform when individual modalities are progressively corrupted or become less predictive? Such experiments would help verify whether the method genuinely reduces modality competition rather than simply improving average predictive performance. 8.Given that improvements over MCR are inconsistent on some benchmarks and relatively small on others, could the authors provide statistical significance tests and additional analyses of when each S2MLVM variant is most effective?

2 comments

Q1: Novelty

S2MLVM builds on the shared-and-individual structure underlying CCA, JIVE, and sJIVE. Its distinction from CCA and unsupervised JIVE is the explicit use of the target to organize predictive source variation. S2MLVM is similar to sJIVE in that sJIVE also learns target-specific joint and individual components. However, S2MLVM explicitly parameterizes a shared target-irrelevant latent variable (zcz^c), alongside the shared and source-specific predictive latent variables. While sJIVE can accommodate shared components with zero target coefficients, it does not explicitly designate a separate latent block for this role.

Our contribution is developing a probabilistic model with loading matrix whose structural zeros distinguish shared predictive, source-specific predictive, and shared target-irrelevant variation. We integrate this structure with source-specific encoders, invertible flows, and a predictor, training all components jointly by stochastic gradient descent under density and task objectives. The standard CCA, JIVE, and sJIVE analyze fixed input features, but our framework jointly learns the representations and their probabilistic structure.

Our model’s novelty also lies in synergistically integrating three components: encoder auxiliary loss (maintaining information and reducing dimensionality), flow likelihood modeling (transforming features to Gaussian space for LVM), and structured LVM (for fine-grained multi-modal interaction). While the first two components have been developed before (to serve different purposes), the third component is entirely new to our knowledge. For example, Gaussian PID does transform features into a Gaussian space for estimating PID components, but it does not perform structured latent variable modeling.

Q2

We have now added proposition 1 on identifiability: individual latent coordinates are identifiable up to rotations within each block. This is why Section 5.1 evaluates recovery with a rotation-invariant subspace metric.

Note, this identifiability is due to the block-sparsity structure of the LVM (see proof of Proposition 1 claim 1), and it does not require the orthogonality mentioned in C.1. The assumption of task-relevant and nuisance spaces being orthogonal is used by the analysis in Appendix C and it is a modeling choice. Our implementation does perform re-orthogonalization after each gradient step, ensuring loading matrices of nuisance (zcz^c) and predictive components ([z12,z1][z^{12}, z^1] or [z12,z2][z^{12}, z^2]) are orthogonal for each input modality.

Q3

We added the recovery metrics for each subspace in Table 1, and individual subspace alignment are strong as for the aggregated subspace.

Q4

While flows in principle can make each modality's marginal Gaussian at its outputs (which are inputs to S2MLVM), the encoder outputs are non-Gaussian. Thus, our model can handle non-Gaussian data, with flexibility coming from the flows. Similarly, linear interaction in the S2MLVM is nonlinear in the input space.

In practice, each modality’s flow output is not exactly Gaussian due to finite capacity, and we make the approximation that flow outputs from all modalities are jointly Gaussian (it is future work to improve on this approximation). Nonetheless, empirical results and ablations show that flow is critical to performance (Table 5), and the fact that we outperform strong baselines indicates that important nonlinear interactions have been captured.

Q5(W3): Connection between the PID analysis and the learned representations.

DatasetVariantIPI_PΔ\DeltaSCES_{CE}SMIS_{MI}RCER_{CE}RMIR_{MI}
MOSIS2MLVM-MASK0.2850.0590.0080.0670.0290.088
MOSIS2MLVM-CONTRAST0.3100.0530.0070.0600.0320.085
MUSTARDS2MLVM-MASK0.4700.1210.1110.2320.0610.183
MUSTARDS2MLVM-CONTRAST0.4500.1190.1110.2290.0570.176

Section 3 explains how the latent structure of S2MLVM relates to PID. Every latent block can contribute to synergy: the shared nuisance factor through cancellation of shared interference, the shared signal through averaging, and the predictive factors through explaining-away. The analysis also gives closed-form bounds through the conditional-entropy adversary qCE∗q_{CE}^{\ast}. The latent blocks do not map one to one onto PID atoms, as stated in Section 3. This gap remains open, and we see it as a direction for future work.

We have added comparison of the estimated synergy and redundancy using qCEq_{CE} and qMIq_{MI}, based on the estimated Gaussian covariances for trained models in C.5 (Table 12).

W3

Given that our framework isolates distinct predictive components via individual latent variables, it is natural to ask how our method relates to Partial Information Decomposition (PID). This motivates our analysis in Section 3, which demonstrates that our LVM components do not map one-to-one onto the information-theoretic atoms of PID. Importantly, it is not our objective to extract latent variables that perfectly match a PID formulation. In fact, it remains an open question whether PID—a framework primarily utilized for quantifying and understanding information—is the optimal decomposition for improving downstream multimodal fusion. Instead, as discussed in the text, our model's structural design is more directly aligned with the multi-view redundancy assumptions widely adopted in the representation learning literature.

Q6

On CREMA-D, UCF101, and MOSEI, we follow the protocol of MCR (Kontras et al., 2025), the most recent balancing method. We have added the OGM results to Tables 3 and 4; OGM underperforms both MCR and S2MLVM.

Q7

Our focus is on how the model uses the information present in real multimodal data, so we measured multimodal fusion/modality competition alleviation directly instead of adding synthetic corruption.

Q8

We performed two-sided Welch's tt-tests over five seeds per method. The table reports pp-values; denotes p<0.05p < 0.05.

DatasetS2MLVM-MASK vs. MCRS2MLVM-CONTRAST vs. MCR
V&T EE0.004 (better)0.001 (worse)
MIMIC0.2590.820
CREMA-D0.2170.933
UCF1010.1530.136
MOSEI0.4820.165

On V&T EE, S2MLVM-MASK is significantly better than MCR, while S2MLVM-CONTRAST is significantly worse; both results remain significant after Holm–Bonferroni correction across the five benchmarks. On the other four benchmarks, both variants are on par with MCR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.