acceptodds
Under review as a conference paper at ICLR 2027

Generative Modeling Beyond Pixels with Any Pretrained Vision Transformers

Abstract

Many pretrained transformers include task heads that decode their features into structured outputs. VGGT and Depth Anything V2 (DAv2), for example, produce depth maps, while VGGT also produces camera poses and point clouds. We show that the feature space of such an encoder can be modeled directly. This turns the encoder into a text-conditioned generative model whose samples are decoded by the heads it already has. Our framework, MIRAGE, compresses encoder tokens with a hybrid VAE onto a product of spheres with log-normal radii, and trains a diffusion transformer on these latents using rectified flow that follows geodesics on each sphere and a straight path in log-radius. Because generation targets the encoder's own feature space, REPA alignment can use the encoder itself as its target, and no external representation model is needed. For long videos, a temporal compressor and per-frame flow times let a single fixed-window model either generate a clip in parallel or roll forward on clean past context, producing videos of arbitrary length. We evaluate four geometric encoders: VGGT, PAGE-4D, Depth Anything V2, DUSt3R, and provide initial audio experiments. A single model generates all outputs supported by each encoder's heads. Against raw-feature diffusion baselines at two scales, MIRAGE's latents are 16-64 smaller, remains within 1-6% in depth-FVD, lowers depth-gradient divergences across all tested encoders, and samples 2 faster with 4-13 less sampling memory beyond model weights. Its compact latent also makes rolling generation of long videos practical.

Primary area:
generative models
Keywords:
diffusion models, generative models, 3d, audio, video

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HKW9OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors present MIRAGE, a framework for multi-modal generative models that allows for efficient generation of diverse output modalities including images, videos, depth maps, point clouds, and audio. By compressing a pre-trained transformer latent space into a well-behaved product of spherical riemannian manifolds, they obtain a single representation upon which a DiT denoiser can be trained. By exploiting the already-existing task heads of the pretrained model, the latent samples generated by the DiT can then be decoded across task head modalities.

Through MIRAGE, the authors are able to achieve significant gains in performance (both latency and peak memory) and generation quality, while also enabling size transferability of the generative model past its initial training window.

Critical strengths

S1: The product sphere prior which separates similarity (direction on Sd−1\mathbb{S}^{d-1}) and magnitude/concentration (logr∈R+\textrm{log} r \in \mathbb{R}^+) is a contribution to generative modeling in and of itself, as it provides a prior that surpasses the well-known limitations encountered by Gaussian N(0,I)\mathcal{N}(0, I) priors at high dimensionalities. The multi-head formulation achieved through the manifold product also further helps expressivity by (1) allowing for independent semantical concepts to be embedded in each spheres, given that the norm constraints between spheres are independend from eachother (2) reduce the curse of dimensionality by reducing the dimensionality of each factor by the amount of spheres. Lastly, the authors have also provided the required scaffolding to perform efficient generation on such a latent space, and have provided a helpful appendix section detailing the training details of such an architecture, helping reproducibility and further adoption.

S2: The method seems to be very generalizable and modality-agnostic, indeed the framework provided by the authors just specifies how to produce a diffusion-ready latent space from a multi-head model, it is therefore reasonable to assume that such a method can be expected to work across a wider range of modalities than the (already wide) set of modalities presented in the manuscript. As such, the paper is likely to be impactful in the field of multi-modal generation as a whole. Additionally, the assumption on the existence of existing task-heads is used in the manuscript to enable zero-shot generation once the informative latent space is obtained, however, it is likely that parts of the methodology presented in the manuscript can be adapted to more common learned-decoder diffusion models, suggesting a potential impact of contributions that goes beyond pretrained multi-modal encoder.

Critical weaknesses

None

Questions for authors

Q1 (Clarifying): The paper does not define the origin of the framework's name "MIRAGE", and could benefit from an acronym expansion upon first occurrence in the manuscript (see the glossaries latex package). Is "MIRAGE" an acronym or is it just the framework's name?

Q2: Did you consider the application of your proposed latent prior to encoder-decoder diffusion models? Did you benchmark its performance on typical VAE reconstruction tasks to isolate it's impact? A small section of the appendix pitting Gaussian VAE vs product VAE could help readers to identify the impact of that individual contribution w.r.t the rest of the method.

2 comments

We thank the reviewer for the careful reading and for recognizing the potential of the product-of-spheres prior beyond our specific application. The revised paper includes the suggested controlled VAE comparison, together with a generative evaluation that isolates the contributions of the latent and flow geometries, and will be uploaded soon.

Q1. Meaning of "MIRAGE"

MIRAGE stands for Multi-modal Implicit Representation Aligned GEneration. We have added the expansion at its first occurrence in the manuscript.

Q2. Applicability to encoder–decoder diffusion and controlled VAE comparison

The hybrid prior is independent of the frozen task heads. Its directional and radial components have closed-form KL divergences, and the same bottleneck can be incorporated into a conventional latent diffusion autoencoder, with the corresponding manifold-aware generative flow. The frozen heads determine how the generated features are interpreted, not the geometry of the generative latent.

Our new DINOv2-B experiment provides evidence for this broader applicability: MIRAGE generates semantic visual features that are decoded into RGB using the learned RAE decoder (Zheng et al., 2026), achieving FID 35.8, FVD 10.8, and CLIP score 0.295 on 500 test clips.

Controlled comparison. Bond et al. (2026) previously compared a direction-only product-of-spheres VAE with a Gaussian VAE for transformer-feature reconstruction. To isolate the contribution of our hybrid extension, we have added a matched comparison of three VGGT VAEs (Gaussian, direction-only spherical, and hybrid), keeping the encoder, decoder, latent width (128), training recipe, and optimization budget fixed. We then train five DiTs with the same architecture, conditioning, data, and optimization budget.

Table 1. Results on the same 500 test clips. Confidence intervals are obtained by paired bootstrap over test clips.

VAE, flowDepth-FVD ↓FVD ratio to ours [95% CI]Hist KL ↓ (×10²)Hist W ↓ (×10²)Grad KL ↓ (×10³)Grad W ↓ (×10³)Near-static clips
Hybrid, manifold (ours)4.1318.2916.0210.390.702.2%
Hybrid, straight-line3.780.92 [0.88, 0.95]6.5511.4210.870.9620.2%
Gaussian, flat Euclidean4.491.09 [1.05, 1.13]4.269.2612.670.670.0%
Gaussian, polar manifold5.051.22 [1.17, 1.27]15.4119.2317.320.851.2%
Direction-only, manifold5.521.34 [1.28, 1.40]18.8020.2055.302.020.0%

Three findings emerge:

  • The hybrid latent improves generation at comparable reconstruction quality. The three VAEs differ by less than 0.3% in feature reconstruction cosine similarity, yet both hybrid variants achieve lower Depth-FVD than every alternative latent. With the manifold flow held fixed, the hybrid latent improves Depth-FVD from 5.05 to 4.13 over the Gaussian latent. The depth-value histograms follow the flow rather than the latent: straight interpolants match them more closely on both latents (6.55 vs. 8.29 on the hybrid latent, 4.26 vs. 15.41 on the Gaussian latent). Thus, the advantage in the Gaussian methods comes from using straight-line flows, rather than the bottleneck being Gaussian.

  • Retaining magnitude information matters. Removing the radii worsens Depth-FVD from 4.13 to 5.52 and increases depth-gradient KL by 5.3×, supporting the hybrid directional–radial design.

  • The representation and flow geometry have distinct effects. Straight-line flow on the hybrid latent achieves the lowest Depth-FVD but produces substantially more near-static clips (20.2% vs. 2.2% for manifold flow and 10.0% for real clips). Manifold flow improves both depth-gradient metrics, providing a different trade-off between distributional quality, local structure, and temporal evolution.

The key finding is that comparable reconstruction quality does not imply equally effective generative latents. The hybrid representation improves feature-space generation under matched model and training budgets. We believe this motivates its investigation in conventional encoder–decoder diffusion models as well, although a controlled evaluation on pixel-space autoencoders remains future work.

Thank you for having addressed my remaining questions and concerns appropriately. I will maintain my score.

Reviewer_we72OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MIRAGE, which uses generative modeling for pretrained transformer features and decodes the generated features through existing frozen task heads. A hybrid VAE compresses features into spherical directions and log-normal radii, followed by geometry-aware flow matching and self-representation alignment. Temporal compression and per-frame flow times extend the framework to rolling video generation.

Critical strengths
  • Reusing existing heads allows one generated representation to support several structured outputs, which is interesting. Experiments across four geometric encoders demonstrate the feasibility of this interface.

  • On PAGE-4D, sampling time decreases from 12.16s to 5.89s relative to RAE-matched, while peak sampling memory decreases from 15.88 GB to 1.28 GB.

Critical weaknesses
  • The central geometry-aware contribution is not adequately isolated. Comparisons against raw-feature RAE baselines change compression, latent geometry, model architecture, and parameter numbers simultaneously. A compressed Euclidean/Gaussian baseline with the same latent width and DiT is needed to distinguish the benefits of the proposed geometry from ordinary compression.

  • The evaluation provides limited direct evidence of geometric consistency. Per-clip normalization can make Depth-FVD insensitive to global depth-scale changes. Although the histogram metrics capture depth and local-gradient statistics, and the paper explicitly analyzes scale bias, these measurements do not directly assess multi-view reprojection consistency, adherence to camera conditioning, or agreement between jointly generated depth, camera poses, and point maps.

  • The main comparison contains a conditioning confound. Section 5.1 states that the Depth Anything V2 MIRAGE model uses short captions, whereas its baselines use long captions.

Questions for authors

I recommend the authors to provide a matched compressed Euclidean baseline, correct the caption mismatch, and report a geometry-consistency or longer-horizon evaluation.

3 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the constructive review. We have added a matched compressed-latent baseline, direct geometric-consistency measurements, and longer-horizon comparisons, and corrected the caption mismatch. We will upload a revised paper soon.

W1. Matched compressed baseline

The RAE comparison establishes whether a compact latent can replace generation in raw encoder features. To isolate the contribution of the latent geometry, we have added the requested Gaussian VAE baseline on VGGT, together with controls that separate the effects of the latent representation and flow geometry.

All three VAEs share the encoder, decoder, latent width (128), training recipe, and optimization budget. All five DiTs share the architecture, conditioning, training data, and optimization budget.

Table 1. Results on the same 500 test clips. Confidence intervals are obtained by paired bootstrap over test clips.

VAE, flowDepth-FVD ↓FVD ratio to ours [95% CI]Hist KL ↓ (×10²)Hist W ↓ (×10²)Grad KL ↓ (×10³)Grad W ↓ (×10³)Near-static clips
Hybrid, manifold (ours)4.1318.2916.0210.390.702.2%
Hybrid, straight-line3.780.92 [0.88, 0.95]6.5511.4210.870.9620.2%
Gaussian, flat Euclidean4.491.09 [1.05, 1.13]4.269.2612.670.670.0%
Gaussian, polar manifold5.051.22 [1.17, 1.27]15.4119.2317.320.851.2%
Direction-only, manifold5.521.34 [1.28, 1.40]18.8020.2055.302.020.0%

Three findings emerge:

  • The hybrid latent improves generation beyond ordinary compression. Both hybrid variants achieve lower Depth-FVD than the matched Gaussian/Euclidean baseline (4.13 and 3.78 vs. 4.49), although the Gaussian baseline achieves better depth-value histogram metrics. With the manifold flow held fixed, replacing the Gaussian latent with the hybrid latent improves Depth-FVD from 5.05 to 4.13 and reduces gradient KL by 1.7×. The depth-value histograms follow the flow rather than the latent: straight interpolants match them more closely on both latents (6.55 vs. 8.29 on the hybrid latent, 4.26 vs. 15.41 on the Gaussian latent). Thus, the advantage in the Gaussian methods comes from using straight-line flows, rather than the bottleneck being Gaussian.

  • The radii contribute to generation quality. Removing the radial component worsens Depth-FVD to 5.52 and increases gradient KL by 5.3×. The three VAEs differ by less than 0.3% in feature reconstruction cosine similarity, suggesting that these differences arise from the suitability of the latent for generation rather than reconstruction quality alone.

  • Manifold flow improves local structure and temporal dynamics. Straight-line flow achieves better Depth-FVD on the hybrid latent but produces substantially more near-static clips (20.2% vs. 2.2% for manifold flow and 10.0% for real clips). Manifold flow improves both depth-gradient metrics. We retain it as a trade-off between distributional quality, local structure, and temporal evolution.

These controls establish the benefit of the hybrid latent independently of compression and DiT capacity, while clarifying the distinct effects of the representation and flow geometry.

W2. Geometric consistency and per-clip normalization

We agree that per-clip normalization makes Depth-FVD insensitive to global depth-scale errors. We therefore complement it with histograms computed on absolute log-depth, using bin edges shared by the generated and real sets. Global scale errors shift mass between bins and increase both KL and Wasserstein distances. This explains the large histogram divergences observed in the original long-rollout experiments and their substantial reduction after median normalization (Sec. 5.2).

AuthorsOpenReview ↗

Response Part 2

To directly assess geometric consistency, we have added the requested measurements:

  • Cross-head consistency: agreement between depth unprojected using the camera head's predictions and the world points predicted by the point-map head.

  • Multi-view reprojection: depth disagreement after reprojecting points from frame tt into frames t+1t+1 and t+4t+4.

  • Camera trajectory agreement: rotation error and Sim(3)-aligned ATE against the camera head's predictions on the paired real clip.

Table 2. Geometric evaluation on the same 500 test clips. Cross-head and reprojection errors are normalized by depth. Camera ATE is normalized by path length. Motion denotes the log ratio of generated to real frame-to-frame depth change (0 indicates equal motion).

EncoderClipsCross-head error ↓Reprojection, k=1 ↓Reprojection, k=4 ↓MotionCamera rot. error (°)Camera ATE
VGGTGenerated0.280 [0.274, 0.288]0.0210.0680.583.630.134
VGGTReal0.276 [0.269, 0.283]0.0140.035
PAGE-4DGenerated0.230 [0.226, 0.235]0.0360.0871.016.370.126
PAGE-4DReal0.215 [0.210, 0.220]0.0120.029

The frozen heads retain much of their mutual consistency on generated features. Cross-head error is 0.280 for generated versus 0.276 for real VGGT features, and 0.230 versus 0.215 for PAGE-4D. These results directly support our central premise that a single generated representation can be decoded into mutually consistent geometric outputs through existing task heads. Reprojection errors remain higher than on real clips, particularly over longer temporal offsets, indicating room for improving multi-view consistency. The revised manuscript reports these metrics alongside the distributional depth metrics.

W3. Caption confound

We thank the reviewer for identifying this mismatch. The DAv2 MIRAGE model was trained on long captions but sampled using short captions. The original manuscript incorrectly described this as a training-time difference.

We have repeated inference using the same long captions as the baselines and updated the corresponding results in the revised manuscript. The DAv2 comparison now uses matched captions during both training and evaluation.

Q. Additional evaluation: longer-horizon generation

Following the reviewer's recommendation, we also compare rolling generation against independent 8-frame windows at 64 and 128 frames. Both sampling strategies use the same fixed-window VGGT MIRAGE model, without camera conditioning, isolating the effect of history conditioning.

Table 3. Results on 128 test clips at 64 and 128 frames.

FramesSamplerDepth-FVD ↓Boundary / within-window changeNear-static clips
64Rolling (ours)9.740.99 [0.95, 1.03]0%
64Independent windows12.1010.07 [9.18, 10.97]0%
128Rolling (ours)9.501.00 [0.97, 1.04]0%
128Independent windows13.3610.13 [9.28, 11.00]0%

Independent generation produces approximately 10× larger depth changes across window boundaries than within windows. Rolling generation eliminates this excess discontinuity and achieves lower full-rollout Depth-FVD at both horizons, without access to the reference camera trajectory. These results establish the benefit of history conditioning, although long-rollout depth-scale stability remains an open limitation.

In summary, the revision includes all three requested controls: matched compressed-latent comparisons, corrected caption conditioning, and direct geometric-consistency and longer-horizon evaluations.

Thank you for the response.

The additional matched compressed baseline experiment in the rebuttal addressed my first concern. I also appreciate the additional metrics on measuring the geometric consistency, which make the evaluation more comprehensive. For the two reasons, I would like to raise my rating to (3) weak accept.

Moreover, from Table 1, one can see that the manifold version improves Grad KL and Near-static clips metric, but falls behind on other metrics. As the rebuttal states, this is a trade-off. I wonder if the trade-off maintains also in geometry consistency in Table 2? Additional evaluation would strengthen the paper.

Reviewer_jPD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MIRAGE, a framework that directly models the feature distribution of a frozen pretrained transformer. Generated features are decoded using the transformer’s existing task heads, allowing one model to produce different types of outputs. They also compress encoder tokens using a hybrid VAE whose latent consists of spherical directions and log-normal radii.

Critical strengths
  1. The main idea of the paper is kind of novel. Modeling a task-aware representation allows existing heads to serve as decoders.
  2. The efficiency gains are substantial.
  3. The method is evaluated across four independently pretrained geometric encoders with substantially different feature spaces and heads.
Critical weaknesses
  1. The paper does not directly ablate its main geometric contribution. The reported ablations cover conditioning, self-REPA, flow-time sharing, and temporal compression, but do not compare the hybrid product-of-spheres VAE against a Gaussian VAE, a direction-only spherical VAE, or Euclidean flow on the same compressed latent. It is therefore difficult to determine how much of the result specifically follows from the proposed manifold geometry, given that it is one of the main claims of the paper.
  2. The quantitative evaluation is dominated by distributional depth metrics. There is no direct measurement of text adherence, first-frame consistency, camera-pose accuracy, or point-track quality. The authors did not use metrics specific to each task, which is a big shortcoming.
  3. The long-form evidence is limited. Only MIRAGE is evaluated in the 64-frame setting; the strongest variant receives the test clip’s camera trajectory, and generated depth becomes approximately twice the reference depth during rollout. The failure analysis also shows cases with very little temporal evolution. These results do not yet establish robust generation at arbitrary lengths.
  4. The authors have not provided video results in the supplementary, so the video claim can not be trusted without looking at the video results.
  5. The generality claim is broader than the evidence. All four main encoders are geometric models and are evaluated on closely related depth and 3D outputs. The paper acknowledges that semantic and other non-visual heads are not tested, while the audio extension consists of a single qualitative spectrogram comparison.
Questions for authors
  1. Can the authors provide a controlled ablation comparing the proposed hybrid product-of-spheres latent and manifold rectified flow against a Gaussian VAE, a direction-only spherical latent, and Euclidean flow with matched latent size and compute?
  2. Can the authors quantitatively evaluate the other outputs supported by the frozen task heads, particularly camera poses, point clouds, and consistency between depth, camera, and 3D geometry? As such, more task-specific metrics, not just distribution metrics.
  3. For long-form generation, can the authors compare MIRAGE against independent chunk generation or another history-conditioned baseline, and report performance as the rollout length increases? There are many recent works tackling this problem, like self-forcing, resampling-forcing.
3 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the detailed and constructive review. We have added controlled latent-geometry ablations, task-specific geometric evaluations, longer-rollout comparisons, and we will provide supplementary videos and a revised paper soon to address the concerns raised.

W1. Ablation of the latent geometry

We agree that isolating the contribution of the latent geometry is important. We have added a matched ablation on VGGT. We train three VAEs with different bottleneck (Gaussian, direction-only spherical, and our hybrid product-of-spheres latent), keeping the encoder, decoder, latent width (128), training recipe, and optimization budget fixed. We then train five DiTs with the same architecture, conditioning, data, and optimization budget:

VAEFlowWhat it isolates
Hybrid (ours)ManifoldReference
HybridStraight-line on directionsThe flow, with the latent held fixed
GaussianFlat EuclideanOrdinary compression at the same width and DiT
GaussianOur polar manifold flowThe latent, with the flow held fixed
Direction-onlyManifold, no radiusContribution of log-normal radii

Table 1. Results on the same 500 test clips. Confidence intervals are obtained by paired bootstrap over test clips.

VAE, flowDepth-FVD ↓FVD ratio to ours [95% CI]Hist KL ↓ (×10²)Hist W ↓ (×10²)Grad KL ↓ (×10³)Grad W ↓ (×10³)Near-static clips
Hybrid, manifold (ours)4.1318.2916.0210.390.702.2%
Hybrid, straight-line3.780.92 [0.88, 0.95]6.5511.4210.870.9620.2%
Gaussian, flat Euclidean4.491.09 [1.05, 1.13]4.269.2612.670.670.0%
Gaussian, polar manifold5.051.22 [1.17, 1.27]15.4119.2317.320.851.2%
Direction-only, manifold5.521.34 [1.28, 1.40]18.8020.2055.302.020.0%

The real test contains 10.0% near-static clips. Three findings emerge:.

  • The hybrid latent improves generation beyond ordinary compression. Both hybrid variants outperform every alternative latent on Depth-FVD. Under the same manifold flow, the hybrid latent improves Depth-FVD from 5.05 to 4.13 over the Gaussian latent and reduces depth-gradient KL by 1.7×. Removing the radii worsens Depth-FVD to 5.52 and increases gradient KL by 5.3×, demonstrating the importance of retaining magnitude information..
  • Reconstruction quality alone does not explain these gains. The three VAEs differ by less than 0.3% in feature reconstruction cosine similarity, yet their generative performance differs substantially. This suggests that the hybrid bottleneck provides a more effective space for generative modeling, beyond its ability to reconstruct features..
  • Manifold flow improves local structure and temporal dynamics. On the hybrid latent, straight-line flow achieves better Depth-FVD (3.78 vs. 4.13) but produces near-static clips twice as frequently as the real data (20.2% vs. 10.0%), exhibiting the failure mode in Fig. A7(b). Manifold flow reduces this rate to 2.2% and improves both depth-gradient metrics. We therefore retain manifold flow as a trade-off between distributional quality, local depth structure, and temporal evolution. The depth-value histograms follow the flow rather than the latent: straight interpolants match them more closely on both latents (6.55 vs. 8.29 on the hybrid latent, 4.26 vs. 15.41 on the Gaussian latent). Thus, the advantage in the Gaussian methods comes from using straight-line flows, rather than the bottleneck being Gaussian.

These results isolate the benefits of the hybrid representation from compression and model capacity, while clarifying the distinct effects of the latent and flow geometries.

AuthorsOpenReview ↗

Response Part 2

W2, Q2. Task-specific metrics and geometric consistency

Our primary metrics are distributional because caption- and first-frame-conditioned generation admits multiple valid future trajectories. Nevertheless, we agree that direct geometric evaluation is essential for assessing the outputs of the frozen heads.

We have added the following task-specific evaluations:

  • First-frame consistency: depth AbsRel and δ1\delta_1 against the real first frame, measuring adherence to the image condition.
  • Camera trajectory agreement: rotation error and Sim(3)-aligned ATE against the camera head’s predictions on the paired real clip.
  • Cross-head consistency: agreement between the 3D points obtained by backprojecting depth with the camera head's predictions and the world points predicted by the point-map head.
  • Multi-view reprojection: depth disagreement after reprojecting points from frame t into frames t+1 and t+4, measuring geometric consistency across views.
  • Text adherence: CLIP for RGB outputs and CLAP for audio outputs (response to W5)..

Table 2. Geometric evaluation on the same 500 test clips. Cross-head and reprojection errors are normalized by depth. Camera ATE is normalized by path length. Motion denotes the log ratio of generated to real frame-to-frame depth change (0 indicates equal motion).

EncoderClipsCross-head error ↓Reprojection, k=1 ↓Reprojection, k=4 ↓MotionCamera rot. error (°)Camera ATE
VGGTGenerated0.280 [0.274, 0.288]0.0210.0680.583.630.134
VGGTReal0.276 [0.269, 0.283]0.0140.035
PAGE-4DGenerated0.230 [0.226, 0.235]0.0360.0871.016.370.126
PAGE-4DReal0.215 [0.210, 0.220]0.0120.029

The frozen heads retain much of their mutual consistency on generated features. Cross-head error is 0.280 on generated versus 0.276 on real VGGT features, and 0.230 versus 0.215 for PAGE-4D. These results directly support the central premise of MIRAGE that a single generated representation can be decoded into mutually consistent geometric outputs through existing task heads.

Reprojection errors remain higher than on real clips, particularly over longer temporal offsets, indicating room for improving multi-view consistency. The camera metrics additionally quantify agreement with the paired reference trajectory, although alternative trajectories can also be valid under our conditioning.

W3,Q3. Long-form generation and history conditioning

We agree that a longer-horizon comparison is needed. The RAE stage-2 DiT has no mechanism to condition on previously generated frames, so its fixed 8-frame window can only be sampled independently. To isolate the effect of history conditioning without confounding it with differences in model quality, we compare two sampling strategies using the same fixed-window VGGT MIRAGE model, without camera conditioning: rolling generation with per-frame flow times and independently generated 8-frame windows from the same caption and first frame.

Table 3. Results on 128 test clips at 64 and 128 frames.

FramesSamplerDepth-FVD ↓Boundary / within-window changeNear-static clips
64Rolling (ours)9.740.99 [0.95, 1.03]0%
64Independent windows12.1010.07 [9.18, 10.97]0%
128Rolling (ours)9.501.00 [0.97, 1.04]0%
128Independent windows13.3610.13 [9.28, 11.00]0%
  • History conditioning eliminates excess discontinuity at window boundaries. Independent generation produces approximately 10× larger depth changes across boundaries than within windows. With rolling generation, the ratio is approximately 1.0 at both horizons.
  • The benefit persists at longer horizons. Rolling generation improves full-rollout Depth-FVD from 12.10 to 9.74 at 64 frames and from 13.36 to 9.50 at 128 frames, without access to the reference camera trajectory.
  • Rolling generation maintains temporal evolution. None of the rollouts is near-static at either length, and the generated clips exhibit substantial temporal depth changes (log motion ratio 0.42 [0.24, 0.60] relative to real clips).

These results establish the benefit of history conditioning beyond the training window. We acknowledge that they do not establish robust generation at arbitrary lengths or resolve the depth-scale bias reported in the original submission.

AuthorsOpenReview ↗

Response Part 3

Regarding Self Forcing (Huang et al., 2025) and resampling forcing, our rolling sampler follows Diffusion Forcing and DFoT: the DiT is trained with independent per-frame flow times and conditions on previously generated latents held near the clean end of the flow during sampling. Self Forcing and resampling forcing instead address exposure bias by incorporating model-generated history during training. These approaches are complementary to MIRAGE's feature-space generation framework and could be incorporated into its DiT training. We have clarified this connection in the revised related work.

W4. Supplementary Videos

We agree that playable videos are important for assessing temporal quality. We will soon update the supplementary material to include video samples.

W5. Encoders beyond geometry

We agree that the original quantitative evaluation focuses on geometric encoders. To test the generality of our framework beyond geometry, we have added quantitative results for two additional encoders, covering RGB video and audio generation.

Semantic visual features. We train MIRAGE on DINOv2-B features and decode the generated features to RGB using the RAE decoder for the same encoder (Zheng et al., 2026). On 500 test clips, MIRAGE achieves FID 35.8, FVD 10.8, and a CLIP text–image score of 0.295. This demonstrates that the same feature-space generation framework also applies to semantic representations with a learned RGB decoder, rather than only geometric encoders with native task heads.

Audio. We replace the original qualitative spectrogram comparison with a quantitative evaluation on 883 held-out AudioCaps clips. We evaluate successive stages of the pipeline to distinguish the effects of the pretrained audio representation, latent compression, and generative modeling.

Table 4. Audio generation and reconstruction quality.

AudioCLAP text-audio score ↑FAD to real recordings (CLAP embeddings) ↓
Real recordings0.583
Dasheng encoder and decoder only0.571
Round trip through our VAE0.5272.16
MIRAGE generated0.4556.56

The generated audio achieves a CLAP score of 0.455, compared with 0.527 for the VAE reconstruction and 0.583 for real recordings. These results demonstrate that the hybrid latent and flow formulation support audio generation, while the reconstruction comparison identifies the remaining quality gap introduced by generative modeling.

These experiments extend MIRAGE beyond geometric representations to semantic visual features and audio, providing quantitative evidence for the generality of the framework. We have also revised the manuscript to distinguish the demonstrated applicability of MIRAGE from the broader range of encoders that its interface could potentially support.

References

Dinkel, Heinrich, et al. “DashengTokenizer: One layer is enough for unified audio understanding and generation”, arXiv preprint arXiv:2602.23765, 2026.

Zheng, Boyang, et al. "Diffusion transformers with representation autoencoders." International Conference on Learning Representations. Vol. 2026. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.