acceptodds
Under review as a conference paper at ICLR 2027

Inversely Autoencoding Multimodal Data into a Shared Latent Space

Abstract

Learning multimodal representations in a shared latent space is a fundamental challenge towards unified models. In this work, we introduce the Inverse Autoencoder (Inv-AE), a novel framework based on the insight that decoding a shared latent representation into disparate modalities is far more tractable than encoding multimodal data into an aligned space. Inv-AE first optimizes shared latent representations alongside multiple modality-specific decoders, followed by training encoders to map heterogeneous inputs into this established feature space. To accommodate the disparate information densities across modalities, we propose the Latent Representation Sequence (LRS), a multi-scale hierarchical sequence of tensors with progressively increasing sizes, allowing each decoder to selectively leverage an adaptive prefix length. Consequently, low-information-density to high-information-density autoencoding (e.g., images to labels) naturally maps as a classification task, whereas high information density to low-density autoencoding utilizes a Transformer to compensate for finer details via conditional autoregression. Experiments on standard benchmarks demonstrate that our approach delivers competitive performance across multiple tasks while remaining highly extensible to expert architectures with minimal modification.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3Tf5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Inverse Autoencoder (Inv-AE), which is a framework for multimodal representation learning that first jointly optimizes per-instance latent representations and modality-specific decoders, then trains modality-specific encoders to predict the learned representations. The authors further introduce a multiscale Latent Representation Sequence (LRS), where different modalities use different portions of a hierarchical latent sequence, and an autoregressive Transformer for conditional completion of latent representations.

The CIFAR-10 experiments demonstrate reconstruction, classification, and class-conditional generation using the proposed framework, with encouraging reported results. Additional COCO2017 and ImageNet-1K experiments demonstrate compatibility with pretrained specialist architectures for several downstream prediction tasks, providing evidence for the feasibility of the proposed pipeline. However, the experiments provide limited evidence for the benefits of the training order, hierarchical latent structure, and multimodal information sharing that constitute the central methodological claims.

Critical strengths
  1. The authors introduce an appealing modular framework for multimodal representation learning. The separation of latent/decoder optimization from encoder training is a potentially useful architectural design. Combined with multiscale latent representations and autoregressive completion, the framework offers a natural approach to accommodating modalities with different information requirements. Its modularity may also facilitate integration with pretrained specialist architectures.
  2. The experimental results are promising feasibility results across multiple tasks against some prior works. On CIFAR-10, the method demonstrates reconstruction, classification, and class-conditional generation, with encouraging reported performance. The COCO and ImageNet experiments further suggest that pretrained specialist features can be mapped into the proposed latent representation while retaining useful task performance.
Critical weaknesses
  1. Generally, th methodological contribution seems insufficiently differentiated from latent-optimization and structured multimodal generative approaches. The central idea of optimizing latent representations and decoders before fitting encoders has been considered before, see [1] which jointly optimizes per-instance latent codes and a generator, and in particular [2] that likewise employs a closely related two-stage procedure for multimodal shared/private representations. Likewise, hierarchical and structured representations are well established in multimodal generative modeling. Nexus [3] learns hierarchical joint and modality-specific representations, while MHVAE [4] develops multiscale hierarchical latent-variable models for cross-modal synthesis. Other multimodal approaches [5-8] explicitly address shared/private information and its implications for conditional generation and representation quality. While these do not contain precisely the same ordered-prefix LRS combined with staged latent optimization, the paper does not clearly identify the methodological advantage of this particular combination relative to existing alternatives, nor are they part of the baselines in the experiments. The experiments do not separately establish the effects of the inverse training procedure, the multiscale latent structure, or the autoregressive model.

  2. The experiments establish feasibility but do not convincingly explain the reported gains or validate the central multimodal claims. The CIFAR-10 results are promising, but Table 3 compares the proposed method against separate reconstruction, classification, and generation specialists rather than against existing multimodal generative models evaluated jointly across these tasks. This makes it difficult to assess whether Inv-AE offers a favorable overall trade-off between reconstruction quality, semantic representation, and conditional generation. More relevant comparisons would include representative multimodal models such as multimodal VAEs adapted to a common dataset and supervision setting. These approaches explicitly address generative quality, semantic coherence, shared/private structure, and downstream representation quality. While their published benchmarks differ, a controlled comparison with a few such models would be substantially more informative than assembling independent task-specific scores.

A further limitation is the lack of comparisons with more recent visual tokenization approaches that explicitly address the joint requirements of reconstruction, semantic understanding, and generation such as TokenFlow [9] , QLIP [10], FlexTok [11]. The ladder for instance learns ordered, variable-length latent representations with progressively increasing detail. Although these methods differ in their objectives and training scales, they provide important alternative designs for addressing the same representation trade-offs. The paper does not sufficiently explain what distinguishes the proposed hierarchy from these approaches, or whether its empirical advantages persist when controlling for latent capacity, quantization, decoder architecture, and training procedure. This is particularly relevant given the substantial improvements over the older VQ-based reconstruction baselines in Table 3.

References: [1] Bojanowski, Piotr, et al. "Optimizing the latent space of generative networks." arXiv preprint arXiv:1707.05776 (2017). [2] Zhang, Xinyi, G. V. Shivashankar, and Caroline Uhler. "Partially shared multi-modal embedding learns holistic representation of cell state." Nature Computational Science 6.3 (2026): 285-300. [3] Vasco, Miguel, et al. "Leveraging hierarchy in multimodal generative models for effective cross-modality inference." Neural Networks 146 (2022): 238-255. [4] Dorent, Reuben, et al. "Unified brain MR-ultrasound synthesis using multi-modal hierarchical representations." International conference on medical image computing and computer-assisted intervention. Cham: Springer Nature Switzerland, 2023. [5] Palumbo, Emanuele, Imant Daunhawer, and Julia E. Vogt. "MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises." Fifth Symposium on Advances in Approximate Bayesian Inference-Fast Track. 2023. [6] Märtens, Kaspar, and Christopher Yau. "Disentangling shared and private latent factors in multimodal Variational Autoencoders." Machine Learning in Computational Biology. PMLR, 2024. [7] Hirt, Marcel, et al. "Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives." arXiv preprint arXiv:2309.00380 (2023). [8] Vo, Huyen, María Martínez-García, and Isabel Valera. "Holder++: Improving the Quality-Coherence Trade-off in Multimodal VAEs." arXiv preprint arXiv:2606.13381 (2026). [9] Qu, Liao, et al. "Tokenflow: Unified image tokenizer for multimodal understanding and generation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [10] Zhao, Yue, et al. "Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation." arXiv preprint arXiv:2502.05178 (2025). [11] Bachmann, Roman, et al. "Flextok: Resampling images into 1d token sequences of flexible length." Forty-second International Conference on Machine Learning. 2025.

Questions for authors
  1. What is the specific methodological advance relative to previous latent-optimization and hierarchical multimodal generative approaches? How does the proposed two-stage procedure differ from LORD and APOLLO, and what advantage is expected from the LRS compared with hierarchical or shared/private latent representations such as those used by MMVAE+? More recent tokenization approaches, including FlexTok, also learn ordered representations with progressively increasing detail. Which properties of the proposed training order and hierarchy are essential to the claimed benefits? The CIFAR-10 results are promising, but Table 3 compares against separate reconstruction, classification, and generation specialists rather than existing multimodal generative models evaluated across all three tasks. A controlled comparison of conventional joint encoder–decoder training versus inverse training, and of flat, shared/private, and hierarchical representations under matched capacity, would help isolate the contributions.

  2. How is consistency of the shared latent representation ensured and demonstrated across modalities? Could the authors report modality-specific code prediction and consistency measures? Relatedly, how do the non-prefix and partly disjoint latent assignments in Table 2 support the proposed shared representation and cross-modal inference?

  3. The reason for using the given construction of using mutually exclusive random variables is unclear, why this choice?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Tf5

We thank the reviewer for the thorough review.

W1/Q1 (methodological advance). GLO, LORD and APOLLO [2] also involve latent optimization, and LORD and APOLLO additionally fit encoders afterwards; these works are now cited. Nevertheless, the overall framework of Inv-AE is different (C1–C3 in the General Response). Compared with APOLLO, the most related work:

  • Cross-modal autoencoding: APOLLO predicts deterministically from the shared dimensions only, so modality-specific content cannot be produced; ours completes the remaining nodes autoregressively (Eq. (6)), generating content absent from the source (label-to-image, FID 2.72).
  • Discreteness: APOLLO uses real-valued latents; ours drives the latents to binary values (Eq. (2)), making encoder training classification and the LRS a token sequence.
  • Specialists: ours attaches pretrained specialists to the LRS (Sec. 3.5).

LORD addresses a single image modality. Multimodal VAEs [3–8] train encoders jointly via an ELBO with continuous latents, and their hierarchies are levels of latent variables, whereas the LRS is one ordering by information density shared by all modalities.

W2 (TokenFlow, QLIP, FlexTok). QLIP is cited in Sec. 2, and all three are now discussed. They are image tokenizers; e.g., FlexTok orders tokens within the image modality, whereas the LRS assigns nodes across modalities.

Q2 (consistency). During Step 1, consistency holds by construction, as each instance has one LRS decoded into all modalities. At test time, decoding the first-node tokens predicted by the image encoder with the label decoder yields the correct label for 96.63% of the CIFAR-10 test images.

Q3 (mutually exclusive random variables). This construction makes the latents discrete, which is essential to Step 2: since each latent value is driven to one of two values (Eq. (2)), the encoders are trained with a cross-entropy loss (Eq. (4)) instead of regressing continuous values, reaching a token accuracy of nearly 99%, and a correctly predicted token reproduces the optimized value exactly. Moreover, compared with VQ, no codebook is required, so the instance-specific parameters are optimized directly by gradient descent. We now also cite arXiv:2506.21731 by the same authors.

AuthorsOpenReview ↗

Ethics flag

Clarification Regarding the Ethics Review Flag

We noticed that our paper was marked with "Flag For Ethics Review: Yes." However, we could not identify any specific ethical concerns in the reviewer's comments, which primarily focus on methodological novelty and experimental comparisons.

We fully respect the review process, but an ethics flag is a serious matter and should be supported by specific ethical concerns. We would therefore kindly ask the reviewer to clarify the reason for this flag and reconsider it if no specific ethical concern exists.

Thank you for your time and consideration.

I thank the authors for their response and updated paper which has now clarified the relation and positioning to prior related works. The ethic flag was an error on my side.

Reviewer_bfHLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work propose a new method called Inv-AE for multi-modal representation learning. Generally, Inv-AE optimizes a per-sample shared latent jointly with modality-specific decoders, then freezes both and trains encoders to map inputs onto these latents. Experiments cover CIFAR-10 trained from scratch, and COCO and ImageNet using pretrained specialist features.

Critical strengths

The idea of this work is quite clear and interesting. The authors propose that all modalities of the same sample share a same latent variable, so no additional alignment loss is required. The prefix-based design for modalities with different information content is reasonable. The CIFAR-10 results show feasibility of the proposed method, and I appreciate the authors' frank introduction of this work's limitations.

Critical weaknesses

I have few concerns about this work:

First of all, the main claim of this work is not convincing. The paper never compares end-to-end training with the two-stage scheme under the same architecture, and it has no LRS ablation.

Second, the large-scale results rely on specialists. ImageNet (85.80%) falls slightly below SwinV2-B alone (85.99%), and all COCO metrics are below the specialists. Comparing against generalists trained from scratch is unfair. Please name the specialists and report results with and without Inv-AE.

Third, the CIFAR-10 comparisons lack controls. There is no matched latent capacity for reconstruction and no matched training recipe for classification. The generation baselines are few-step models, so compute is unequal, and the EDM number should be checked.

Questions for authors

In Table 4, the detection result (46.32) is bolded although UnifiedIO2-B (47.20) and 4M7-B (49.7) are higher.

Which setting does “COCO Seg. mIoU” refer to? COCO segmentation is usually reported as mask AP or PQ.

Several citations lack years (“DCQ (CHENYUE et al.)”, “CD (Song et al.)”), and “EDM (Zheng et al., 2023)” should be checked.

Missing related work: auto-decoders (GLO, DeepSDF), nested and multi-scale representations (Matryoshka, nested dropout) and coarse-to-fine generation (VAR). The label pathway amounts to learning target codes and then regressing to them.

Scalability: the latent table grows linearly with dataset size, and Step 1 requires every sample to have all modalities. Please discuss unpaired data and adding new modalities.

Please quantify the amortization gap between optimized training latents and encoder-predicted test latents.

Reviewer_oU8oOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Inverse Autoencoder (Inv-AE) for a shared multimodal latent space. It first optimizes a latent per training instance together with modality-specific decoders, then trains encoders to predict these latents. The latent is a sequence of binarized tensors of growing size, so that labels are decoded from a short prefix and images from the full sequence. A causal Transformer handles label-to-image generation, and pretrained backbones can serve as encoders through a small MLP. Experiments cover CIFAR-10 from scratch, and COCO and ImageNet-1K with pretrained feature extractors.

Critical strengths
  1. The decode-first formulation with a prefix-structured latent sequence is simple and gives one interface for attaching decoders and reusing pretrained encoders (Sec. 3.5). If validated, it would be a cheap way to give specialist models a common latent interface.
  2. On CIFAR-10, one model trained from scratch does reconstruction, classification and class-conditional generation reasonably well (Tab. 3). This is the only setting that uses the full pipeline, and it works end to end at small scale.
Critical weaknesses
  1. The central hypothesis is never tested, and the closest prior work is missing. The paper rests on the claim that decoding a shared latent is "far more tractable" than encoding into one, but there are no ablations. There is no standard encoder-decoder baseline with the same latent sequence and decoders, and no ablation of node sizes, prefix assignment or discretization. Step 1 is essentially the GLO and auto-decoder approach [1, 2], and fitting an encoder afterwards is a natural extension. Neither work is cited or compared. As it stands, this is an engineering pipeline with unjustified design choices.
  2. The experiments do not show a shared latent space or any-to-any autoencoding at scale. In Tab. 2 the COCO modalities mostly use different node subsets rather than prefixes, against the prefix design. Sec. 5 admits the label and segmentation latents are "not fully shared", and the COCO setting does not support fully token-based autoregressive generation. The only cross-modal result is label-to-image on CIFAR-10, and the text modality in Fig. 3 is never evaluated. What is shown is separate task heads on one tensor container.
  3. Tab. 3 and Tab. 4 do not isolate what Inv-AE contributes. VQ-VAE latent budgets are not reported, so the large reconstruction gap may just be capacity. Classification uses the TADA+SGD network yet beats TADA+SGD itself (96.63% vs. 95.60%) with no recipe given. The generation baselines are few-step consistency or distillation models with unstated conditioning and step counts, and the "EDM" row cites DPM-Solver-v3. On COCO, the backbone and head per task are not named, and the comparison is against jointly trained multimodal models. The ImageNet result (85.80% vs. 85.99% for SwinV2-B) only shows that SwinV2 features survive re-projection into the first node.
  4. Scalability and reproducibility details are missing. The Tab. 2 configuration stores about 81k latent values per image, growing linearly with the dataset. It is not explained how these were optimized on a single RTX 4080, or whether the full sequence was learned for ImageNet. The discretization, decoders and Transformer are not specified in the main text. The token accuracy of nearly 99% is not tied to a split, which matters since test performance depends on encoders generalizing to latents that exist only for training instances.

References

[1] Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. "Optimizing the Latent Space of Generative Networks." International Conference on Machine Learning (ICML), 2018.

[2] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. "DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

Questions for authors
  1. Can you compare against a jointly trained encoder-decoder with the same latents and against GLO or auto-decoder training [1, 2] plus encoder fitting? What is new relative to [1, 2]?
  2. Why do the COCO modalities use non-prefix node subsets, and can you measure cross-modal transfer between two non-label modalities?
  3. What are the VQ-VAE latent budgets and the classification recipe, and how does each COCO task compare with its specialist backbone alone?
  4. How were the per-instance latents optimized on one GPU, and is the token accuracy measured on training or test images?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.