- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present MIRAGE, a framework for multi-modal generative models that allows for efficient generation of diverse output modalities including images, videos, depth maps, point clouds, and audio. By compressing a pre-trained transformer latent space into a well-behaved product of spherical riemannian manifolds, they obtain a single representation upon which a DiT denoiser can be trained. By exploiting the already-existing task heads of the pretrained model, the latent samples generated by the DiT can then be decoded across task head modalities.
Through MIRAGE, the authors are able to achieve significant gains in performance (both latency and peak memory) and generation quality, while also enabling size transferability of the generative model past its initial training window.
Critical strengths
S1: The product sphere prior which separates similarity (direction on ) and magnitude/concentration () is a contribution to generative modeling in and of itself, as it provides a prior that surpasses the well-known limitations encountered by Gaussian priors at high dimensionalities. The multi-head formulation achieved through the manifold product also further helps expressivity by (1) allowing for independent semantical concepts to be embedded in each spheres, given that the norm constraints between spheres are independend from eachother (2) reduce the curse of dimensionality by reducing the dimensionality of each factor by the amount of spheres. Lastly, the authors have also provided the required scaffolding to perform efficient generation on such a latent space, and have provided a helpful appendix section detailing the training details of such an architecture, helping reproducibility and further adoption.
S2: The method seems to be very generalizable and modality-agnostic, indeed the framework provided by the authors just specifies how to produce a diffusion-ready latent space from a multi-head model, it is therefore reasonable to assume that such a method can be expected to work across a wider range of modalities than the (already wide) set of modalities presented in the manuscript. As such, the paper is likely to be impactful in the field of multi-modal generation as a whole. Additionally, the assumption on the existence of existing task-heads is used in the manuscript to enable zero-shot generation once the informative latent space is obtained, however, it is likely that parts of the methodology presented in the manuscript can be adapted to more common learned-decoder diffusion models, suggesting a potential impact of contributions that goes beyond pretrained multi-modal encoder.
Critical weaknesses
None
Questions for authors
Q1 (Clarifying): The paper does not define the origin of the framework's name "MIRAGE", and could benefit from an acronym expansion upon first occurrence in the manuscript (see the glossaries latex package). Is "MIRAGE" an acronym or is it just the framework's name?
Q2: Did you consider the application of your proposed latent prior to encoder-decoder diffusion models? Did you benchmark its performance on typical VAE reconstruction tasks to isolate it's impact? A small section of the appendix pitting Gaussian VAE vs product VAE could help readers to identify the impact of that individual contribution w.r.t the rest of the method.
2 comments
Thank you for having addressed my remaining questions and concerns appropriately. I will maintain my score.
We thank the reviewer for the careful reading and for recognizing the potential of the product-of-spheres prior beyond our specific application. The revised paper includes the suggested controlled VAE comparison, together with a generative evaluation that isolates the contributions of the latent and flow geometries, and will be uploaded soon.
Q1. Meaning of "MIRAGE"
MIRAGE stands for Multi-modal Implicit Representation Aligned GEneration. We have added the expansion at its first occurrence in the manuscript.
Q2. Applicability to encoder–decoder diffusion and controlled VAE comparison
The hybrid prior is independent of the frozen task heads. Its directional and radial components have closed-form KL divergences, and the same bottleneck can be incorporated into a conventional latent diffusion autoencoder, with the corresponding manifold-aware generative flow. The frozen heads determine how the generated features are interpreted, not the geometry of the generative latent.
Our new DINOv2-B experiment provides evidence for this broader applicability: MIRAGE generates semantic visual features that are decoded into RGB using the learned RAE decoder (Zheng et al., 2026), achieving FID 35.8, FVD 10.8, and CLIP score 0.295 on 500 test clips.
Controlled comparison. Bond et al. (2026) previously compared a direction-only product-of-spheres VAE with a Gaussian VAE for transformer-feature reconstruction. To isolate the contribution of our hybrid extension, we have added a matched comparison of three VGGT VAEs (Gaussian, direction-only spherical, and hybrid), keeping the encoder, decoder, latent width (128), training recipe, and optimization budget fixed. We then train five DiTs with the same architecture, conditioning, data, and optimization budget.
Table 1. Results on the same 500 test clips. Confidence intervals are obtained by paired bootstrap over test clips.
Three findings emerge:
The hybrid latent improves generation at comparable reconstruction quality. The three VAEs differ by less than 0.3% in feature reconstruction cosine similarity, yet both hybrid variants achieve lower Depth-FVD than every alternative latent. With the manifold flow held fixed, the hybrid latent improves Depth-FVD from 5.05 to 4.13 over the Gaussian latent. The depth-value histograms follow the flow rather than the latent: straight interpolants match them more closely on both latents (6.55 vs. 8.29 on the hybrid latent, 4.26 vs. 15.41 on the Gaussian latent). Thus, the advantage in the Gaussian methods comes from using straight-line flows, rather than the bottleneck being Gaussian.
Retaining magnitude information matters. Removing the radii worsens Depth-FVD from 4.13 to 5.52 and increases depth-gradient KL by 5.3×, supporting the hybrid directional–radial design.
The representation and flow geometry have distinct effects. Straight-line flow on the hybrid latent achieves the lowest Depth-FVD but produces substantially more near-static clips (20.2% vs. 2.2% for manifold flow and 10.0% for real clips). Manifold flow improves both depth-gradient metrics, providing a different trade-off between distributional quality, local structure, and temporal evolution.
The key finding is that comparable reconstruction quality does not imply equally effective generative latents. The hybrid representation improves feature-space generation under matched model and training budgets. We believe this motivates its investigation in conventional encoder–decoder diffusion models as well, although a controlled evaluation on pixel-space autoencoders remains future work.