- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce DeepIVA, a latent variable model with the purpose of finding independent vectors that remain identifiable and are linked across multiple modalities. To do so, a two-step architecture is introduced. Step 1 trains individual iVAEs per modality given data and auxiliary variable . Step 2 freezes the part of the encoder that takes in , and uses the MISA loss to align the independent vectors from each modality. This leads to a blind source separation model that is identifiable, multimodal, nonlinear, and links the multiple modalities. Synthetic results show a better mean correlation coefficient (MCC) and minimum distance (MD) than available baselines. Results on neuroimaging find sources that identify auxiliary variables.
Critical strengths
-
Focus on Identifiability: The paper does a good job at focusing on maintaining identifiability of the independent vectors found while introducing non-linearities. Connecting the iVAE to MISA via Sklar’s Copula Theorem is clever conceptual framing and aims to tackle a gap in the BSS literature by aligning the representations across modalities.
-
Promising Results: Positive results on synthetic data show successful recovery of unimodal sources with proper cross-modal linkage. The neuroimaging experiments show an interesting application of recovering SCVs using DeepIVA.
Critical weaknesses
-
Unclear Novelty and Practical Implementation: The methods section is unclear and fails to separate previously-existing methods and architecture from new insights and novelty. At its core, DeepIVA is combining iVAE with a MISA objective via an alternating optimization approach. While this is interesting in concept, the main text focuses on the math and conceptual nature of iVAEs and the MISA loss, and only introduces a sketch of Theorem 2.5 in the main text (which would be considered the main contribution). I would instead like to see more details regarding the alternating optimization, as some things remain unclear. I do not understand how model parameters with respect to remain unchanged in hidden layers given shared downstream weights, or how setting to 0 does not distort encoder features through an out-of-distribution auxiliary vector. The authors should clarify how the layer-freezing scheme preserves the iVAE identifiability.
-
Incomplete Identifiability Claims: There is a critical gap in the identifiability proof of DeepIVA (Theorem 2.5). While the authors show that an identifiable cross-modal solution exists, there is no guarantee that this solution is found via backpropagation. Because the MISA loss changes iVAE encoder parameters in Step 2 without using as a structural anchor, the network can minimize this loss in ways that completely destroy the independent, identifiable structure in Step 1. To complete the identifiability claims, authors need to show that the backpropagation step preserves the iVAE guarantees, not just that the optimal aligned version is identifiable.
Questions for authors
-
Given that the weights of the encoder are changed during the cross-modal alignment step, can you provide evidence that this doesn’t break the identifiability claims presented in Theorem 2.5?
-
Can you provide additional details on the freezing of the encoder for in the MISA training step? It is unclear if only the input layer remains frozen, and how parameters are updated during the MISA step. Besides, setting to 0 during the MISA training step does not guarantee no influence unless strict assumptions are made.
2 comments
Follow-up on our response
Dear Reviewer Pd7D,
Thank you again for your careful reading of our method. As the discussion period is ending soon, we wanted to kindly follow up on our response. In the revised manuscript, we rewrote the description of the alternating optimization to clarify that the auxiliary variable enters the prior network and that the MISA step updates the encoder (Section 2.1.5). We also added a remark after Theorem 2.5 on the relationship between identifiability and optimization (Section 2.1.4), and training-dynamics results showing that unimodal MCC remains at 0.97 while cross-modal MCC increases during joint training (Appendix B.2). In addition, we now separate the existing methods from our contributions (Section 4).
We would greatly appreciate hearing whether these revisions address your concerns, and we are happy to clarify anything further. If they do, we would be grateful if you would consider updating your score.
Response to Reviewer Pd7D
Thank you for your valuable feedback. We respond to each point below (W: Weakness; Q: Question). W1 is addressed together with Q2, and W2 together with Q1. All section, figure, and appendix numbers below refer to the revised manuscript, where changes are highlighted in blue.
W1 novelty.
We agree that the original Methods section did not clearly separate existing methods from our contributions. DeepIVA builds on two existing methods: iVAE [Khemakhem et al., 2020] for unimodal source identification and MISA [Silva et al., 2020] for cross-modal alignment. Our contributions are (i) a framework that combines them through alternating optimization with segment-wise MISA updates, (ii) an identifiability result for linked nonlinear sources across modalities (Theorem 2.5), and (iii) an evaluation on synthetic data and a large multimodal neuroimaging dataset. We have added this statement to the Discussion (Section 4) and expanded the description of the alternating optimization in Section 2.1.5, as detailed below.
W1/Q2 implementation of the MISA step.
Thank you for this careful question. We apologize that our original description of the MISA step was misleading; we have rewritten it in Section 2.1.5 of the revised manuscript. For each modality, the iVAE consists of four separate networks: the encoder mean network g(x), the encoder variance network, the decoder f(z), and the prior network λ(u), which parameterizes the conditional prior p(z∣u). In our implementation, the auxiliary variable u enters only the prior network; the encoder takes only x as input. Consequently, no u-specific weights are shared with the encoder, u is not set to zero, and the encoder never receives an out-of-distribution auxiliary input. The alternating optimization proceeds as follows. (1) iVAE step: all four networks are updated by maximizing the ELBO on (x,u). (2) MISA step: the MISA loss depends only on the encoder output g(x), so only the encoder (all of its layers) is updated, while the decoder, prior network, and variance network remain unchanged. The MISA loss is computed separately for each segment, and each epoch cycles over all segments before the next iVAE step.
W2/Q1 identifiability after the MISA step.
Thank you for this important question. We agree that Theorem 2.5 characterizes the optimum and does not by itself guarantee that alternating gradient-based optimization reaches it. This limitation is shared by identifiability results for deep latent variable models in general, including iVAE [Khemakhem et al., 2020], whose guarantee also holds at the optimum of the ELBO rather than for a particular optimization trajectory. We now state this explicitly in a remark following Theorem 2.5 (Section 2.1.4). We address the concern in three ways.
(i) The auxiliary variable is used in the MISA step. The MISA loss is computed separately for each segment, i.e., for each value of u, after removing the segment mean. The segment structure defined by u therefore anchors both the iVAE and the MISA steps.
(ii) The MISA step is compatible with iVAE identifiability. The iVAE identifies the sources of each modality only up to permutation and component-wise invertible transformations, so the marginal distribution of each recovered source is not determined by the iVAE. Imposing the Kotz distribution of the MISA objective on the source component vectors therefore does not conflict with the iVAE solution; the MISA step uses this remaining freedom to align sources across modalities. In addition, the MISA learning rate is one to two orders of magnitude smaller than the iVAE learning rate (3.2×10−5 to 8×10−5 vs. 10−3), and every round of MISA updates is followed by iVAE updates that restore the ELBO.
(iii) Empirical evidence. If the MISA updates destroyed the identifiable iVAE structure, the unimodal recovery would degrade during joint training. To test this directly, we tracked the MCC on the training set every 5 epochs throughout joint training on synthetic data with known ground truth (8 seeds, dynamic MISA learning-rate schedule; Figure A1). The unimodal MCC of both modalities increases from about 0.69 at initialization to above 0.95 by epoch 300 and then remains at 0.97 (s.d. ≤0.003) until epoch 1000, while the cross-modal MCC between the recovered sources of the two modalities increases from 0.56 to 0.73. The unimodal MCC never drops by more than 0.014 between consecutive evaluations, and at the epoch with the highest cross-modal MCC it ranges from 0.96 to 0.97 across seeds. Results are consistent with a fixed MISA learning rate (unimodal MCC 0.94–0.97, cross-modal MCC 0.71–0.74; Figure A6). The MISA updates therefore align sources across modalities without degrading unimodal identifiability. We have added this analysis to Appendix B.2 (Figure A1) of the revised manuscript.