acceptodds
Under review as a conference paper at ICLR 2027

State Before Pixels: Large-scale Text-to-Video Generation in Representation Space

Abstract

Video representations learned by vision foundation models encode scene semantics and dynamics as spatiotemporal world states beyond pixel appearance. While representation autoencoders have shown promise for image generation, representation-space video generation remains largely underexplored, with their higher dimensionality posing additional challenges. In this paper, we demonstrate that native video representations can serve as effective modeling targets for large-scale, open-domain text-to-video generation. Our approach, **WorldFlow**, separates world-state modeling from visual rendering: a text-conditioned flow model first generates latent world states directly in native V-JEPA space, and a second-stage flow model then renders these states into high-quality videos in VAE space. This design retains the learned semantic structure of pretrained video representations as the basis for generation, while allowing fine visual details to be synthesized in a space designed for reconstruction. To enable effective diffusion-transformer denoising in this high-dimensional representation space, we systematically study its geometry and prediction parameterization. We find that by conducting Riemannian Flow Matching on the native representation manifold with *v*-prediction, high-dimensional video representations can be effectively modeled without introducing additional compression. Extensive experiments demonstrate that WorldFlow achieves high-quality text-to-video generation. Beyond RGB, the same generated world states can be readily rendered into well-aligned multimodal outputs such as depth, segmentation, human pose, *etc*. These results demonstrate the feasibility of generating unified world states containing appearance, semantics, geometry, and dynamics within a shared representation.

Primary area:
generative models
Keywords:
Text-to-Video Generation, Video Representations, Riemannian Flow Matching, Multimodal Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TAVfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WorldFlow, a two-stage text-to-video framework that first generates native V-JEPA 2.1 features and then renders videos from these features in LTX VAE space. The state generator uses hyperspherical Riemannian flow matching to respect the centered, fixed-norm feature geometry, while a separately trained renderer uses text and predicted states as conditioning. Both stages are initialized from LTX-2.3. Experiments compare with the released and continued-training LTX baselines, ablate flow geometry and prediction parameterization, and demonstrate appearance editing and adaptation to depth, flow, segmentation, DensePose, and pose. The main contribution is evidence that large, uncompressed video representation grids can be generated effectively and used as an intermediate interface for video synthesis.

Critical strengths
  1. The paper demonstrates a useful, technically coherent route to generating native video representations at substantial scale. The centered hypersphere, source distribution, geodesic interpolation, tangent projection, and integration procedure are explicitly specified (Sec. 3; Appendix A). Importantly, Fig. 3 compares all four combinations of Euclidean/Riemannian geometry and v/x prediction rather than only a weak Euclidean v-prediction baseline. Riemannian v-prediction gives the lowest reported FVD at each checkpoint, while the other successful settings provide context for the size of the advantage.

  2. The evaluation includes a same-data continued-training baseline, which addresses the obvious confound of additional in-domain training. Table 1 reports FVD of 83.57 versus 99.37 for released LTX-2.3 and 103.25 for its continued-training variant; subject consistency and spatial relationship also improve over both. Table 3 separately tests the encoded-to-generated conditioning gap and finds a modest benefit from state noise augmentation. The appearance-editing and multimodal examples further illustrate practical uses of the intermediate representation, although their validation is qualitative.

Critical weaknesses
  1. The evidence establishes the feasibility of native-feature generation more convincingly than the stronger attribution that avoiding compression is responsible for better structure. The statement that VFM representations should not be compressed (Sec. 3.1) rests on selected affinity visualizations (Figs. 5–6), without a native-versus-compressed generation comparison or a fully specified compression control. Moreover, matching the baseline's 100k training steps to stage one does not establish a match for the complete two-stage budget: Sec. 4.1 adds 10k plus 60k renderer-training steps and uses two initialized backbones. This does not invalidate Table 1, but Appendix G's exclusion of compute as an explanation is not established. A bounded compression control and transparent total-cost accounting, or narrower causal claims, would substantially strengthen the interpretation.

  2. The claim that the generated states support spatiotemporally aligned multimodal structure is supported only by qualitative examples (Sec. 4.3; Appendix F; Figs. 7–10 and 12). Empty text prompts correctly remove an additional scene-specific text condition, but do not remove the adapted renderers' learned priors or quantify agreement among their stochastic outputs. The small adaptation subset is also not identified sufficiently to assess the training and evaluation protocol. Thus the results show that several modalities can be rendered from the same conditioning features, but do not yet establish reliable joint geometry or motion. A consistency assessment on existing outputs, together with the adaptation-data and target-generation details, would make this claim much more convincing.

Questions for authors
  1. [Attribution and scope] What is the exact compression configuration used in Figs. 5–6, and is there a controlled native-versus-compressed generation result using the same backbone, data, and comparable optimization budget? If such evidence is unavailable, would you limit the conclusion to the demonstrated feasibility of native-feature generation and the observed affinity differences?

  2. [Comparison clarification] Please report total training and sampling cost for the complete pipeline and continued-training baseline. For Fig. 3, are the renderer, time distribution, effective loss weighting, and guidance settings shared, and how exactly is x-prediction converted to velocity in each geometry? These details would clarify which part of the improvement can be attributed specifically to geometry rather than to the combined training recipe.

  3. [Shared-state validation] Which videos, modality targets, and train/test split were used for multimodal adaptation? Can you provide a bounded quantitative check of agreement among modalities rendered from the same generated state, such as pose/segmentation alignment with RGB or flow consistency across frames, with the reliability of any evaluator stated? This would determine how strongly the shared geometric-state interpretation is supported.

5 comments

We thank the reviewer for the constructive feedback and for recognizing the value of generating native video representations at scale. Below, we clarify the experimental controls and computational costs, and provide new quantitative evidence of cross-modal consistency. More supporting visual evidence can be found on the webpage: https://researchyfish.github.io/submission5463-rebuttal.

W1 & Q1: Compression config and controlled comparison

We address each sub-question below:

  • Compression config: The compressed variant in Figs. 6–7 (original Figs. 5–6) is the V-RAE [1] (V-JEPA 2.1-L) latent, obtained by training a temporal compression module on top of the frozen V-JEPA 2.1 encoder together with a decoder under a reconstruction objective. Following V-RAE [1], the configuration is:
ComponentConfiguration
Frozen encoderV-JEPA 2.1-L (ViT-L/16; 24 transformer blocks)
Temporal poolingAttention pooling with group size 2 (~3M parameters)
Temporal downsampling4× overall: 2× encoder tubelet embedding and 2× temporal pooling
Spatial downsampling16× along each spatial axis; no additional spatial compression
Latent representation1024 channels with non-affine LayerNorm
Reconstruction decoderViT-XL (28 blocks; hidden dimension 1152; full spatiotemporal attention)
Training objectiveL1, LPIPS, adversarial, and Gram losses; latent noise σ∼U(0,0.8)\sigma\sim\mathcal{U}(0,0.8)
Compressor and decoder trainingUCF101 + K600; 50 epochs; global batch size 128; AdamW with learning rate 2×10−42\times10^{-4}
  • Controlled comparison: We provide a controlled native vs. compressed generation comparison in aLdK-W1Q1. Both variants are trained from scratch in a shared framework with matched budgets; native reaches lower FVD at matched steps and converges faster. Nonetheless, we want to clarify that the paper does not claim native representations enable better generation results than compressed ones; it claims that high-dimensional native representations are generatable without compression, removing the need for training an additional compressor that can harm the original semantic organization. We choose this simple design to enable its potential for future research that may require semantic understanding during the generation process. As the reviewer suggests, we limit the conclusion in the revised paper: we tone down the Sec. 3.1 paragraph heading from "VFM representations should not be compressed" to "Compression disrupts structure in representations."

[1] M. Guo et al., "V-RAE: Rethinking Video Latent Spaces for Generation," arXiv 2026.

W1 & Q2: Compute cost

We agree that matching Stage 1 at 100k steps does not account for the full two-stage training budget. Due to organizational policy, we cannot disclose the complete GPU specifications and detailed compute costs. However, we provide approximate GPU-hours based on our training records below; all numbers are measured on the same GPU type. Stage 1 requires substantially fewer GPU-hours than CT, primarily because its target is encoded from a downsampled video (6 fps, 384×224), whereas CT encodes the full-resolution 24 fps video with the LTX VAE.

Trainable ParamsGlobal BatchStepsGPU-h
Stage 113.137B32100k1,118
Stage 23.236B870k441
Ours Total16.373B—170k1,559
LTX-2.3 + CT13.124B32100k2,409

The sampling cost per video (768×448, 5 s, 30 Euler steps per stage, same GPU type) is listed below:

Inference Time / videoPeak Memory
Stage 116.39 s41.30 GiB
Stage 235.47 s42.92 GiB
Ours Total51.86 s42.92 GiB
LTX-2.3 + CT16.89 s42.14 GiB

The complete pipeline requires 51.86 s per video, approximately 3.07× the CT baseline, reflecting the additional cost of state generation and state-conditioned rendering. We acknowledge this inference overhead as a limitation; reducing it through fewer sampling steps or distillation is a direction for future work.

Although our pipeline involves more total training steps, it requires fewer GPU-hours since Stage 1 encodes the downsampled 6 fps, 384×224 video. To ensure a more rigorous statement, we have removed the sentence excluding "any difference in data or compute" (Appendix G in the original submission), since the qualitative comparison it accompanied is now in the main text. We further provide a training-step-matched comparison between WorldFlow (Stage 1 100k + Stage 2 10k) and CT (110k) below. WorldFlow's advantage over the CT baseline remains clear under the same total number of training steps.

MethodSubject Consistency ↑Background Consistency ↑Spatial Relationship ↑Aesthetic Quality ↑Imaging Quality ↑Color ↑
CT (110k)89.8094.2379.9963.0666.3385.61
WorldFlow (100k + 10k)92.1294.9184.6864.1866.7685.03

All four variants in Fig. 4 (original Fig. 3) use the same Stage-2 renderer checkpoint, logit-normal schedule with μ=12ln⁡(NC/dref)≈3.84\mu=\tfrac12\ln(NC/d_{\mathrm{ref}})\approx3.84, guidance settings, and 30 Euler steps.

The difference in effective loss weighting arises from converting xx-predictions to velocity before applying the vv-loss, and is therefore inherent to the prediction parameterization, as discussed in JiT [1].

[1] T. Li and K. He, "Back to Basics: Let Denoising Generative Models Denoise," CVPR 2026.

The xx-prediction velocity is computed as follows:

Euclidean:

v^=(xt−x^0)/t.\hat{\mathbf v}=(\mathbf x_t-\hat{\mathbf x}_0)/t.

Hyperspherical:

v^=−log⁡st(s^0)/t.\hat{\mathbf v}=-\log_{\mathbf s_t}(\hat{\mathbf s}_0)/t.

The conversion is direct in both geometries. In the hyperspherical branch the predicted endpoint is centered and rescaled onto MR\mathcal{M}_R before taking the logarithm, after which the sampler's tangent projection applies as usual; the Euclidean branch needs neither step. The xx-prediction variants floor the denominator at tϵ=0.05t_\epsilon=0.05. Details are in Appendix A (added in the revision).

W2 & Q3: Multimodal validation

The adaptations are conducted on a random subset of OpenUni, a large multimodal video dataset proposed in UnityVideo [1], which we cite in the revised paper. For each modality, we optimize the renderer for 1k iterations at batch size 16. The data details are summarized below:

ModalityAnnotation source in OpenUni#Train videos#Test videos
DepthDepth Anything 2$\sim$10k100
SegmentationSAM$\sim$10k100
Optical flowRAFT$\sim$10k100
DensePoseMeta DensePose$\sim$10k100

The 100 test videos are held out and disjoint from the adaptation videos.

We report quantitative results of depth, segmentation, optical flow and DensePose estimation on these held-out test videos, rendering from states encoded from each video and comparing against its OpenUni annotations. The "unpaired" row computes metrics on unrelated {prediction, annotation} pairs to establish a reference value on structurally non-correlated videos. Our metrics are significantly better, indicating promising alignment across different modalities.

Based on the visual comparisons covering samples with scores from best to worst at https://researchyfish.github.io/submission5463-rebuttal/#encoded, in several cases our outputs appear more temporally stable and spatially accurate than the provided annotations.

[1] J. Huang et al., "UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation," CVPR 2026.

Depth (AbsRel ↓)Segmentation (FG IoU% ↑)Optical flow (Normalized EPE ↓)DensePose (mIoU% ↑)
Unpaired2.55613.830.2938.09
Ours0.52582.540.10863.89

Note that the multimodal renderers are adapted independently and receive empty prompts, so their alignment cannot come from a shared input (e.g., text prompt) other than the state. The significant gap between paired/unpaired results further indicates that the geometry and dynamics originate in the shared state instead of leaking through renderer priors.

We further compare our multimodal rendering results from generated states with the outputs of pretrained estimators (Depth Anything 3, SAM 3, RAFT, Meta DensePose) applied to the generated videos of 100 randomly selected VBench prompts. The following scores use the estimators' outputs as pseudo-GT. The numbers are lower than in the first setting, partly because the estimators are less robust on generated frames, as shown in the visual samples at https://researchyfish.github.io/submission5463-rebuttal/#generated. The gap to the unpaired control further demonstrates the rich geometry and dynamics information in our generated states.

Depth (AbsRel ↓)Segmentation (FG IoU% ↑)Optical flow (Normalized EPE ↓)DensePose (mIoU% ↑)
Unpaired2.86313.590.3121.19
Ours0.90367.230.15738.41

To check agreement between independently adapted renderers rendering from the same state, we compute the foreground IoU between the DensePose output and the segmentation output, with an unpaired control that pairs outputs from unrelated states:

DensePose FG vs. Seg. FG (IoU% ↑)
Unpaired11.59
Ours59.61

The two renderers are adapted separately with different supervision and share no conditioning input other than the state. Their outputs achieve a foreground overlap of 59.61% IoU, compared with 11.59% for the unpaired control. This higher overlap supports a consistent spatial layout across independently adapted renderers conditioned on the same state.

Reviewer_cxKsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorldFlow, a two-stage text-to-video system. Stage 1 generates native V-JEPA 2.1 representation tokens as latent “world states” using hyperspherical Riemannian flow matching with vv-prediction; Stage 2 renders these states and text into VAE video latents/RGB. The authors adapt LTX Video transformers, report improved video metrics over pretrained and continued-training LTX baselines, and show state-conditioned RGB re-rendering and lightweight renderers for depth, flow, segmentation, and pose.

Critical strengths

S1: Separating semantic/dynamic state generation from appearance rendering is a clear and timely design hypothesis for text-to-video generation. The paper makes the hypothesis operational at a meaningful scale rather than only demonstrating representation reconstruction.

S2: The geometry study is well motivated. Normalized V-JEPA tokens make a hyperspherical model plausible, and the Euclidean/Riemannian and xx/vv parameterization comparisons help justify the chosen Stage-1 recipe.

Critical weaknesses

W1: The experimental controls do not isolate the paper's central causal claim. WorldFlow differs from its continued-training baseline in representation target, two-stage factorization, conditioning mechanism, geometric flow formulation, and possibly capacity/optimization details. A same-compute two-stage VAE-latent control, an encoded-state oracle for Stage 2, and an equal-capacity compressed-representation control are needed to establish that native V-JEPA state generation is responsible for the reported gains.

W2: The paper has not yet demonstrated that Stage-1 tokens are unified world states rather than useful features for a powerful conditional renderer. Each non-RGB modality uses a learned renderer, and Stage 2 still receives the text prompt. The results therefore do not establish cross-modal calibration, identifiability, or that geometry/dynamics originate in the generated state instead of leaking through text or renderer priors.

W3: The novelty claim needs a sharper positioning relative to the broader semantic-representation-to-generative-renderer paradigm. For example, BLIP3-o studies generating CLIP features through a diffusion stage followed by a diffusion-based visual decoder, establishing the high-level idea that semantically meaningful visual representations can be an intermediate generative target rather than using VAE features alone. WorldFlow differs materially in its video setting, native dense V-JEPA states, Riemannian vv-flow, and shared multimodal state interface. However, the current experiments do not isolate whether its gains come from these specific ingredients or from the more general two-stage semantic-state-plus-renderer factorization. A matched end-to-end compressed-V-JEPA or alternative pretrained video-representation control would clarify the distinctive contribution.

Questions for authors

Q1: Can the authors provide matched-compute controls for a two-stage VAE-latent model, a compressed V-JEPA target, and Stage 2 conditioned on encoded ground-truth states?

Q2: How much does Stage 2 depend on text versus the state? Please report state/text dropout or swap experiments, and quantify structural preservation under appearance-prompt edits.

Q3: Can the authors position WorldFlow explicitly relative to semantic-representation-to-diffusion-renderer approaches such as BLIP3-o, and provide a matched end-to-end control using compressed V-JEPA or another pretrained video representation? This would distinguish the value of native V-JEPA states and Riemannian flow from the general two-stage factorization.

Q4: Can the multimodal claim be supported with calibrated cross-modal metrics (depth/flow/segmentation/pose) and consistency between independently trained renderers, rather than selected visual examples?

1 comment

We thank the reviewer for the constructive comments and for acknowledging that making native-representation generation operational at scale is meaningful and technically coherent. Our replies follow.

W1 & Q1: Controls

The three sub-questions are addressed below:

  • Same-compute two-stage VAE-latent control: We acknowledge the intent of this control. However, Stage 2 renders by conditioning on Stage 1's output, so a pipeline whose two stages share the same VAE latent space degenerates to a one-stage VAE generator plus a copy-and-paste Stage 2: the second stage receives what the first stage already produced, and only re-renders within the same space. It therefore does not open a distinct modeling space to compare against — it reduces to the one-stage VAE baseline, for which we report a step-matched comparison (TAVf-W1Q2). The factorization itself is a natural design to adapt the V-JEPA+VAE paradigm, since generating in a representation space requires a renderer back to pixels; it is not a mechanism we claim improves generation quality by itself.

  • Compressed V-JEPA target: To exclude the influence of the two-stage pipeline and only compare the generation quality of native vs. compressed V-JEPA, we follow the V-RAE setting and train two lightweight DiT models from scratch on UCF101 under identical backbone, data, optimizer, and training budget, differing only in the modeling target: native V-JEPA 2.1 tokens versus the V-RAE-compressed version. As shown in the table in aLdK-W1Q1, the uncompressed variant converges faster and reaches lower FVD at matched steps.

  • Encoded-state oracle for Stage 2: The results of Stage-2 conditioned on encoded ground-truth states are already provided in the encoded column of manuscript Table 3 (14.59/15.12), extended by the new state-perturbation sensitivity analysis in aLdK-W3Q4.

Q2: Important role of stage-1 tokens in stage-2 rendering

We select 100 random prompts from VBench, fix the seed, generate a reference video using the original two-stage pipeline, then vary Stage 2's conditions, and compare the results with the reference using DINO similarity. As shown in the table below, removing the state has a much larger impact on the generation results than removing the prompt.

ConditionStatePromptDINO SIM ↑
w/o prompt✓∅0.98
w/o state∅✓0.36
w/o both∅∅0.07
edited prompt (change color)✓edited0.92
edited state (change seed)edited✓0.76

To quantify structural preservation under appearance-prompt edits, we further alter each of the 100 prompts for Stage 2 and compute the DINO similarity with the original output. The score (0.92) is lower than removing the prompt (0.98) and higher than using a different seed for Stage-1 state generation (0.76), suggesting successful appearance editing while preserving the overall structure.

Visual results at https://researchyfish.github.io/submission5463-rebuttal/#conditioning illustrate these conclusions more intuitively: removing the prompt only affects fine details, while removing the state corrupts the structure, leaving only prompt-related textures. Editing the prompt does not change the structure.

W3 & Q3: Positioning relative to other semantic-to-render approaches

We want to emphasize that our core contribution is enabling text-to-video generation in native video representation space at scale. As stated in W1Q1, we do not claim the two-stage factorization itself as a novel idea relative to BLIP3-o and other semantic-space generation methods, but as a useful design for our specific task.

For matched control, we isolate two ingredients: the native target (aLdK-W1Q1) and the RFM geometry (Fig. 4, original Fig. 3). Regarding other pretrained video representations, our Stage-1 recipe only relies on the normalized (hyperspherical) token geometry shared by most LayerNorm-based VFM encoders, so it is not specific to V-JEPA; we chose V-JEPA 2.1 for its dense, temporally predictive features, and a systematic comparison across encoders at our 22B scale is beyond the rebuttal budget. BLIP3-o-style pipelines have been added to related work in the revised paper.

W2 & Q4: Cross-modal metrics

Please refer to TAVf-W2Q3 for the full quantitative results, including the agreement check between independently adapted renderers. The multimodal renderers are adapted independently and receive empty prompts, so their agreement cannot come from a shared input (e.g., text prompt) other than the state. The significant gap between the unpaired and paired results indicates that the geometry and dynamics originate in the generated state instead of leaking through renderer priors.

Reviewer_aLdKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WorldFlow, a two-stage text-to-video generation framework that separates world-state generation from visual rendering. The first stage generates states directly in native V-JEPA 2.1 representation space using hyperspherical Riemannian Flow Matching with v-prediction, while the second stage renders them into RGB videos in VAE latent space. The framework also enables appearance editing and multimodal rendering. Experiments demonstrate improved video generation quality and spatiotemporal consistency over LTX-2.3 baselines.

Critical strengths
  1. The paper explores generating videos directly from native video foundation model representations, offering an interesting alternative to conventional VAE-based video generation.
  2. The authors analyze the hyperspherical structure of V-JEPA representations and systematically compare different flow geometries and prediction parameterizations. The results support the effectiveness of Riemannian Flow Matching with v-prediction.
  3. WorldFlow achieves better FVD and improvements in several structural and temporal consistency metrics compared with the LTX-2.3 baselines.
  4. The separation between state generation and visual rendering enables appearance editing and multimodal generation from shared representations, demonstrating the potential versatility of the framework.
  5. The authors include an LTX-2.3 continued-training baseline using the same training data, helping assess the benefits of the proposed framework beyond additional data exposure.
Critical weaknesses
  1. The claimed advantage of native V-JEPA representations is mainly supported by feature similarity visualizations, without a direct end-to-end comparison against compressed representations under the same generation framework.
  2. Although WorldFlow aims to improve structural and temporal consistency, Table 1 omits important VBench metrics such as motion smoothness, dynamic degree, and temporal flickering. These results would provide stronger evidence for the claimed benefits of world-state modeling.
  3. Table 3 shows a substantial performance gap when the renderer uses generated states instead of encoded states. Although noise augmentation provides some improvement, the paper does not sufficiently investigate error propagation between the two stages or the renderer's sensitivity to imperfect state representations.
  4. The multimodal results are mainly demonstrated through qualitative examples. More quantitative evaluations are needed to establish whether the generated states accurately preserve geometric and motion information across different modalities.
Questions for authors
  1. Could the authors compare native and compressed V-JEPA representations within the same two-stage framework to directly demonstrate the advantages of avoiding representation compression?
  2. Could the authors include comparisons with recent representation-space video generation methods, or clarify the challenges that prevent such comparisons?
  3. Could the authors provide more details about the internal 3M-video training dataset, including its sources, filtering and deduplication procedures, and potential overlap with evaluation benchmarks?
  4. Table 3 shows a large FVD gap between rendering encoded and generated states. Could the authors further analyze how Stage 1 generation errors affect the renderer and the final video quality?
  5. Could the authors provide quantitative results for multimodal rendering, such as depth estimation, optical flow, and segmentation quality, and compare them with outputs obtained by applying pretrained estimators to generated RGB videos?
  6. Could the authors provide a more comprehensive VBench evaluation, particularly on motion-related metrics such as motion smoothness, dynamic degree, and temporal flickering, to better support the claimed improvements in temporal consistency and dynamics?
4 comments

We thank the reviewer for the positive assessment and for acknowledging that exploring generation in native video foundation model representation space is interesting, that the geometry and parameterization analysis is well motivated, and that the continued-training baseline addresses the data confound. We respond to each point below.

W1 & Q1: Native vs. compressed

The paper does not claim native representations achieve better generation quality than compressed ones; it claims that high-dimensional native representations are generatable without compression, removing the need for training an additional compressor that can harm the original semantic organization. We choose this simple design to enable its potential for future research that requires semantic understanding during the generation process. Accordingly, we revise the Sec. 3.1 paragraph heading from "VFM representations should not be compressed" to "Compression disrupts structure in representations."

To compare the generation quality of native vs. compressed representations under a restricted compute budget, we run a small-scale controlled experiment rather than retraining the full 22B pipeline twice: following the V-RAE setting [1], we train two lightweight DiT models from scratch on UCF101 under identical backbone, data, optimizer, and training budget, differing only in the modeling target: native V-JEPA 2.1 tokens versus the V-RAE-compressed version. As shown in the table below, the uncompressed variant converges faster and reaches lower FVD at matched steps.

StepsCompressed gFVD ↓Native gFVD ↓
10k606.46337.27
20k224.09203.45
30k191.29187.64

[1] M. Guo et al., "V-RAE: Rethinking Video Latent Spaces for Generation," arXiv 2026.

Q2: Comparison with representation-space methods

V-RAE [1] and VideoRAE [2] are the two closest concurrent works on representation-space video generation. V-RAE only trains small class-conditional models (~1.36B) on small datasets (UCF101, K600, Cityscapes) without open-domain text-to-video training, so a fair comparison is intractable. VideoRAE reports a subset of VBench metrics for the text-to-video task; we list the comparison below.

MethodTotal Score ↑Quality Score ↑Semantic Score ↑Subject Consistency ↑Background Consistency ↑Imaging Quality ↑Object Class ↑Human Action ↑Scene ↑
VideoRAE71.2676.0951.9291.2797.0246.0259.0760.8027.57
Ours83.5784.0681.6394.5694.9466.9593.5099.0054.46

WorldFlow achieves significantly better scores on all dimensions except background consistency. Note that while preparing this comparison we found an error in the previously reported subject consistency of WorldFlow; the correct value is 94.56 (previously 92.10), which is now fixed in Table 1 of the revised paper.

[1] M. Guo et al., "V-RAE: Rethinking Video Latent Spaces for Generation," arXiv 2026.

[2] Z. Xie et al., "VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders," arXiv 2026.

Q3: Dataset

The original internal corpus comprises high-quality video-caption pairs covering a variety of sources like movies, news, sports, animations, gameplay, etc. The parent long videos are cut into short shots using pretrained scene-cut detectors. At the curation stage, we retain landscape clips with durations in [5, 6) seconds, at least 130 frames, an effective frame rate of at least 23.9 fps, and source dimensions of at least 1344 by 768 pixels. Clips are then center-cropped and resized to 768 by 448 for training at 24 fps. The upstream parent video index is already deduplicated by exact video object key, retaining one occurrence for repeated keys. The train/validation split is made at the parent-video level, so clips from the same source video never appear in both sets. Existing structured captions were rewritten into LTX-style visual descriptions using Qwen3.6-35B-A3B-FP8 at temperature 0. The 2,048 validation videos are disjoint from training, and VBench uses only its prompt suite and contains no videos, so video-level overlap is impossible.

W3 & Q4: Error propagation

The "encoded" column of Table 3 refers to the FVD between real videos and renderings of the representations encoded from themselves, thus 14.59/15.12 is close to conventional rFVD (reconstruction FVD). In contrast, the "generated" column refers to the FVD between real videos and renderings of the representations generated with their captions, so the gap from 15.12 to 83.57 mostly reflects the generation uncertainty of stage 1 rather than a rendering performance gap, analogous to the usual gap between rFVD and gFVD.

To measure sensitivity directly, we corrupt encoded states with noise s~=ΠM(s+σk)\tilde{\mathbf{s}}=\Pi_{\mathcal{M}}(\mathbf{s}+\sigma\mathbf{k}) and render:

σ\sigma00.10.20.40.60.81.2
w/o aug. FVD ↓14.5915.7925.66197.97975.292189.934052.22
Ours FVD ↓15.1215.1515.4516.3517.8419.9028.84

Without augmentation, FVD grows significantly once σ>0.2\sigma>0.2; with augmentation it stays nearly flat. This shows that noise augmentation substantially improves the renderer's robustness to imperfect state representations.

W4 & Q5: Multimodal quantitative

We report quantitative results of depth, segmentation, optical flow and DensePose estimation on a held-out 100-video OpenUni [1] test set with annotations, and further compare our multimodal rendering results with the outputs of pretrained estimators applied to generated videos of 100 randomly selected VBench prompts. Please refer to TAVf-W2Q3 for details. Visual results are shown at https://researchyfish.github.io/submission5463-rebuttal/#rebuttal.

[1] J. Huang et al., "UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation," CVPR 2026.

W2 & Q6: VBench motion

We compare our results with the baseline on more motion-related metrics below. The slightly lower motion smoothness score comes with a higher dynamic degree; temporal flickering is slightly better than the baseline.

MethodMotion Smoothness ↑Dynamic Degree ↑Temporal Flickering ↑
LTX-2.398.7171.3998.92
Ours97.7975.5699.06

I am very grateful to the authors for providing additional results and clarifications, which have addressed my main concerns. I will therefore maintain my original rating.

Thank you for your positive feedback and for taking the time to review our rebuttal. We appreciate your consideration.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. All reviewers agree that exploring video generation in native video foundation model representation space is interesting (aLdK), and making it operational at scale is meaningful, useful and technically coherent (cxKs, TAVf). The geometry & parameterization analysis is found to be well motivated (cxKs) and systematic (aLdK, TAVf). Reviewers also agree that the continued-training baseline addresses the data confound (aLdK, TAVf), and the potential practical uses of the framework are demonstrated by the appearance-editing and multimodal examples (aLdK, TAVf).

To further address the concerns, we conduct new experiments below and make minor paper revisions. Supporting visual evidence can be found on the webpage: https://researchyfish.github.io/submission5463-rebuttal.

New experiments:

  • E1. Native vs. compressed V-JEPA. We follow V-RAE to train two models from scratch on UCF101 with identical backbone, data, and compute budget, differing only in the stage-1 target. Native converges faster and reaches lower FVD at matched steps (aLdK-W1Q1, cxKs-W1Q1, TAVf-W1Q1).

  • E2. Training steps matched comparison. We do use more total training steps than the continued-training (CT) baseline due to the two-stage design. However, at equal total training steps our VBench advantage over the CT baseline still holds (TAVf-W1Q2, cxKs-W1Q1).

  • E3. Error propagation check. We run a sensitivity analysis over different corruption levels on the stage-1 state and check its influence on stage-2 rendering. With our noise augmentation, the FVD stays robust, which largely prevents error propagation caused by imperfect stage-1 outputs. In contrast, the FVD of the no-augmentation version grows significantly once σ>0.2\sigma>0.2 (aLdK-W3Q4).

  • E4. Validation of the importance of stage-1 tokens. We compare ablations of removing or changing the prompt and/or the state tokens in the stage-2 renderer conditions. Removing the states collapses structure; removing or editing the prompt does not (cxKs-Q2, TAVf-W2Q3).

  • E5. Beyond qualitative results on multimodal rendering. We compute quantitative metrics on depth, segmentation, optical flow and DensePose outputs, and compare them with pretrained estimators' outputs on generated RGB videos (aLdK-W4Q5, cxKs-W2Q4, TAVf-W2Q3).

  • E6. Additional VBench motion dimensions (aLdK Q6). We provide more VBench metrics related to temporal consistency and motion quality.

Paper revisions:

  1. Sec. 2 adds a comparison with BLIP3-o-style semantic-space generation methods.

  2. The Sec. 3.1 paragraph "VFM representations should not be compressed" is retitled "Compression disrupts structure in representations", and the Appendix sentence excluding compute as an explanation is removed.

  3. Sec. 4.1 adds the multimodal data source and the parent-video-level train/validation split.

  4. Sec. 4.2 moves the qualitative comparison with LTX-2.3 from the Appendix to the main text, and Table 1 corrects the subject consistency of WorldFlow (92.10 → 94.56).

  5. Sec. 4.4 adds the stage-2 condition ablation (E4).

  6. The Appendix adds the x-prediction-to-velocity conversion (Appendix A), the cross-frame affinity analysis (Appendix C), the state-perturbation sensitivity analysis (E3), and the quantitative multimodal results on encoded states (E5).

All edited parts are marked in blue in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.