- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces the Inverse Autoencoder (Inv-AE), which is a framework for multimodal representation learning that first jointly optimizes per-instance latent representations and modality-specific decoders, then trains modality-specific encoders to predict the learned representations. The authors further introduce a multiscale Latent Representation Sequence (LRS), where different modalities use different portions of a hierarchical latent sequence, and an autoregressive Transformer for conditional completion of latent representations.
The CIFAR-10 experiments demonstrate reconstruction, classification, and class-conditional generation using the proposed framework, with encouraging reported results. Additional COCO2017 and ImageNet-1K experiments demonstrate compatibility with pretrained specialist architectures for several downstream prediction tasks, providing evidence for the feasibility of the proposed pipeline. However, the experiments provide limited evidence for the benefits of the training order, hierarchical latent structure, and multimodal information sharing that constitute the central methodological claims.
Critical strengths
- The authors introduce an appealing modular framework for multimodal representation learning. The separation of latent/decoder optimization from encoder training is a potentially useful architectural design. Combined with multiscale latent representations and autoregressive completion, the framework offers a natural approach to accommodating modalities with different information requirements. Its modularity may also facilitate integration with pretrained specialist architectures.
- The experimental results are promising feasibility results across multiple tasks against some prior works. On CIFAR-10, the method demonstrates reconstruction, classification, and class-conditional generation, with encouraging reported performance. The COCO and ImageNet experiments further suggest that pretrained specialist features can be mapped into the proposed latent representation while retaining useful task performance.
Critical weaknesses
-
Generally, th methodological contribution seems insufficiently differentiated from latent-optimization and structured multimodal generative approaches. The central idea of optimizing latent representations and decoders before fitting encoders has been considered before, see [1] which jointly optimizes per-instance latent codes and a generator, and in particular [2] that likewise employs a closely related two-stage procedure for multimodal shared/private representations. Likewise, hierarchical and structured representations are well established in multimodal generative modeling. Nexus [3] learns hierarchical joint and modality-specific representations, while MHVAE [4] develops multiscale hierarchical latent-variable models for cross-modal synthesis. Other multimodal approaches [5-8] explicitly address shared/private information and its implications for conditional generation and representation quality. While these do not contain precisely the same ordered-prefix LRS combined with staged latent optimization, the paper does not clearly identify the methodological advantage of this particular combination relative to existing alternatives, nor are they part of the baselines in the experiments. The experiments do not separately establish the effects of the inverse training procedure, the multiscale latent structure, or the autoregressive model.
-
The experiments establish feasibility but do not convincingly explain the reported gains or validate the central multimodal claims. The CIFAR-10 results are promising, but Table 3 compares the proposed method against separate reconstruction, classification, and generation specialists rather than against existing multimodal generative models evaluated jointly across these tasks. This makes it difficult to assess whether Inv-AE offers a favorable overall trade-off between reconstruction quality, semantic representation, and conditional generation. More relevant comparisons would include representative multimodal models such as multimodal VAEs adapted to a common dataset and supervision setting. These approaches explicitly address generative quality, semantic coherence, shared/private structure, and downstream representation quality. While their published benchmarks differ, a controlled comparison with a few such models would be substantially more informative than assembling independent task-specific scores.
A further limitation is the lack of comparisons with more recent visual tokenization approaches that explicitly address the joint requirements of reconstruction, semantic understanding, and generation such as TokenFlow [9] , QLIP [10], FlexTok [11]. The ladder for instance learns ordered, variable-length latent representations with progressively increasing detail. Although these methods differ in their objectives and training scales, they provide important alternative designs for addressing the same representation trade-offs. The paper does not sufficiently explain what distinguishes the proposed hierarchy from these approaches, or whether its empirical advantages persist when controlling for latent capacity, quantization, decoder architecture, and training procedure. This is particularly relevant given the substantial improvements over the older VQ-based reconstruction baselines in Table 3.
References: [1] Bojanowski, Piotr, et al. "Optimizing the latent space of generative networks." arXiv preprint arXiv:1707.05776 (2017). [2] Zhang, Xinyi, G. V. Shivashankar, and Caroline Uhler. "Partially shared multi-modal embedding learns holistic representation of cell state." Nature Computational Science 6.3 (2026): 285-300. [3] Vasco, Miguel, et al. "Leveraging hierarchy in multimodal generative models for effective cross-modality inference." Neural Networks 146 (2022): 238-255. [4] Dorent, Reuben, et al. "Unified brain MR-ultrasound synthesis using multi-modal hierarchical representations." International conference on medical image computing and computer-assisted intervention. Cham: Springer Nature Switzerland, 2023. [5] Palumbo, Emanuele, Imant Daunhawer, and Julia E. Vogt. "MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises." Fifth Symposium on Advances in Approximate Bayesian Inference-Fast Track. 2023. [6] Märtens, Kaspar, and Christopher Yau. "Disentangling shared and private latent factors in multimodal Variational Autoencoders." Machine Learning in Computational Biology. PMLR, 2024. [7] Hirt, Marcel, et al. "Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives." arXiv preprint arXiv:2309.00380 (2023). [8] Vo, Huyen, María Martínez-García, and Isabel Valera. "Holder++: Improving the Quality-Coherence Trade-off in Multimodal VAEs." arXiv preprint arXiv:2606.13381 (2026). [9] Qu, Liao, et al. "Tokenflow: Unified image tokenizer for multimodal understanding and generation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [10] Zhao, Yue, et al. "Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation." arXiv preprint arXiv:2502.05178 (2025). [11] Bachmann, Roman, et al. "Flextok: Resampling images into 1d token sequences of flexible length." Forty-second International Conference on Machine Learning. 2025.
Questions for authors
-
What is the specific methodological advance relative to previous latent-optimization and hierarchical multimodal generative approaches? How does the proposed two-stage procedure differ from LORD and APOLLO, and what advantage is expected from the LRS compared with hierarchical or shared/private latent representations such as those used by MMVAE+? More recent tokenization approaches, including FlexTok, also learn ordered representations with progressively increasing detail. Which properties of the proposed training order and hierarchy are essential to the claimed benefits? The CIFAR-10 results are promising, but Table 3 compares against separate reconstruction, classification, and generation specialists rather than existing multimodal generative models evaluated across all three tasks. A controlled comparison of conventional joint encoder–decoder training versus inverse training, and of flat, shared/private, and hierarchical representations under matched capacity, would help isolate the contributions.
-
How is consistency of the shared latent representation ensured and demonstrated across modalities? Could the authors report modality-specific code prediction and consistency measures? Relatedly, how do the non-prefix and partly disjoint latent assignments in Table 2 support the proposed shared representation and cross-modal inference?
-
The reason for using the given construction of using mutually exclusive random variables is unclear, why this choice?
3 comments
Ethics flag
Clarification Regarding the Ethics Review Flag
We noticed that our paper was marked with "Flag For Ethics Review: Yes." However, we could not identify any specific ethical concerns in the reviewer's comments, which primarily focus on methodological novelty and experimental comparisons.
We fully respect the review process, but an ethics flag is a serious matter and should be supported by specific ethical concerns. We would therefore kindly ask the reviewer to clarify the reason for this flag and reconsider it if no specific ethical concern exists.
Thank you for your time and consideration.
I thank the authors for their response and updated paper which has now clarified the relation and positioning to prior related works. The ethic flag was an error on my side.
Response to Reviewer 3Tf5
We thank the reviewer for the thorough review.
W1/Q1 (methodological advance). GLO, LORD and APOLLO [2] also involve latent optimization, and LORD and APOLLO additionally fit encoders afterwards; these works are now cited. Nevertheless, the overall framework of Inv-AE is different (C1–C3 in the General Response). Compared with APOLLO, the most related work:
LORD addresses a single image modality. Multimodal VAEs [3–8] train encoders jointly via an ELBO with continuous latents, and their hierarchies are levels of latent variables, whereas the LRS is one ordering by information density shared by all modalities.
W2 (TokenFlow, QLIP, FlexTok). QLIP is cited in Sec. 2, and all three are now discussed. They are image tokenizers; e.g., FlexTok orders tokens within the image modality, whereas the LRS assigns nodes across modalities.
Q2 (consistency). During Step 1, consistency holds by construction, as each instance has one LRS decoded into all modalities. At test time, decoding the first-node tokens predicted by the image encoder with the label decoder yields the correct label for 96.63% of the CIFAR-10 test images.
Q3 (mutually exclusive random variables). This construction makes the latents discrete, which is essential to Step 2: since each latent value is driven to one of two values (Eq. (2)), the encoders are trained with a cross-entropy loss (Eq. (4)) instead of regressing continuous values, reaching a token accuracy of nearly 99%, and a correctly predicted token reproduces the optimized value exactly. Moreover, compared with VQ, no codebook is required, so the instance-specific parameters are optimized directly by gradient descent. We now also cite arXiv:2506.21731 by the same authors.