- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes WorldFlow, a two-stage text-to-video framework that first generates native V-JEPA 2.1 features and then renders videos from these features in LTX VAE space. The state generator uses hyperspherical Riemannian flow matching to respect the centered, fixed-norm feature geometry, while a separately trained renderer uses text and predicted states as conditioning. Both stages are initialized from LTX-2.3. Experiments compare with the released and continued-training LTX baselines, ablate flow geometry and prediction parameterization, and demonstrate appearance editing and adaptation to depth, flow, segmentation, DensePose, and pose. The main contribution is evidence that large, uncompressed video representation grids can be generated effectively and used as an intermediate interface for video synthesis.
Critical strengths
-
The paper demonstrates a useful, technically coherent route to generating native video representations at substantial scale. The centered hypersphere, source distribution, geodesic interpolation, tangent projection, and integration procedure are explicitly specified (Sec. 3; Appendix A). Importantly, Fig. 3 compares all four combinations of Euclidean/Riemannian geometry and v/x prediction rather than only a weak Euclidean v-prediction baseline. Riemannian v-prediction gives the lowest reported FVD at each checkpoint, while the other successful settings provide context for the size of the advantage.
-
The evaluation includes a same-data continued-training baseline, which addresses the obvious confound of additional in-domain training. Table 1 reports FVD of 83.57 versus 99.37 for released LTX-2.3 and 103.25 for its continued-training variant; subject consistency and spatial relationship also improve over both. Table 3 separately tests the encoded-to-generated conditioning gap and finds a modest benefit from state noise augmentation. The appearance-editing and multimodal examples further illustrate practical uses of the intermediate representation, although their validation is qualitative.
Critical weaknesses
-
The evidence establishes the feasibility of native-feature generation more convincingly than the stronger attribution that avoiding compression is responsible for better structure. The statement that VFM representations should not be compressed (Sec. 3.1) rests on selected affinity visualizations (Figs. 5–6), without a native-versus-compressed generation comparison or a fully specified compression control. Moreover, matching the baseline's 100k training steps to stage one does not establish a match for the complete two-stage budget: Sec. 4.1 adds 10k plus 60k renderer-training steps and uses two initialized backbones. This does not invalidate Table 1, but Appendix G's exclusion of compute as an explanation is not established. A bounded compression control and transparent total-cost accounting, or narrower causal claims, would substantially strengthen the interpretation.
-
The claim that the generated states support spatiotemporally aligned multimodal structure is supported only by qualitative examples (Sec. 4.3; Appendix F; Figs. 7–10 and 12). Empty text prompts correctly remove an additional scene-specific text condition, but do not remove the adapted renderers' learned priors or quantify agreement among their stochastic outputs. The small adaptation subset is also not identified sufficiently to assess the training and evaluation protocol. Thus the results show that several modalities can be rendered from the same conditioning features, but do not yet establish reliable joint geometry or motion. A consistency assessment on existing outputs, together with the adaptation-data and target-generation details, would make this claim much more convincing.
Questions for authors
-
[Attribution and scope] What is the exact compression configuration used in Figs. 5–6, and is there a controlled native-versus-compressed generation result using the same backbone, data, and comparable optimization budget? If such evidence is unavailable, would you limit the conclusion to the demonstrated feasibility of native-feature generation and the observed affinity differences?
-
[Comparison clarification] Please report total training and sampling cost for the complete pipeline and continued-training baseline. For Fig. 3, are the renderer, time distribution, effective loss weighting, and guidance settings shared, and how exactly is x-prediction converted to velocity in each geometry? These details would clarify which part of the improvement can be attributed specifically to geometry rather than to the combined training recipe.
-
[Shared-state validation] Which videos, modality targets, and train/test split were used for multimodal adaptation? Can you provide a bounded quantitative check of agreement among modalities rendered from the same generated state, such as pose/segmentation alignment with RGB or flow consistency across frames, with the reliability of any evaluator stated? This would determine how strongly the shared geometric-state interpretation is supported.
5 comments
W1 & Q2: Compute cost
We agree that matching Stage 1 at 100k steps does not account for the full two-stage training budget. Due to organizational policy, we cannot disclose the complete GPU specifications and detailed compute costs. However, we provide approximate GPU-hours based on our training records below; all numbers are measured on the same GPU type. Stage 1 requires substantially fewer GPU-hours than CT, primarily because its target is encoded from a downsampled video (6 fps, 384×224), whereas CT encodes the full-resolution 24 fps video with the LTX VAE.
| Trainable Params | Global Batch | Steps | GPU-h | |
|---|---|---|---|---|
| Stage 1 | 13.137B | 32 | 100k | 1,118 |
| Stage 2 | 3.236B | 8 | 70k | 441 |
| Ours Total | 16.373B | — | 170k | 1,559 |
| LTX-2.3 + CT | 13.124B | 32 | 100k | 2,409 |
The sampling cost per video (768×448, 5 s, 30 Euler steps per stage, same GPU type) is listed below:
| Inference Time / video | Peak Memory | |
|---|---|---|
| Stage 1 | 16.39 s | 41.30 GiB |
| Stage 2 | 35.47 s | 42.92 GiB |
| Ours Total | 51.86 s | 42.92 GiB |
| LTX-2.3 + CT | 16.89 s | 42.14 GiB |
The complete pipeline requires 51.86 s per video, approximately 3.07× the CT baseline, reflecting the additional cost of state generation and state-conditioned rendering. We acknowledge this inference overhead as a limitation; reducing it through fewer sampling steps or distillation is a direction for future work.
Although our pipeline involves more total training steps, it requires fewer GPU-hours since Stage 1 encodes the downsampled 6 fps, 384×224 video. To ensure a more rigorous statement, we have removed the sentence excluding "any difference in data or compute" (Appendix G in the original submission), since the qualitative comparison it accompanied is now in the main text. We further provide a training-step-matched comparison between WorldFlow (Stage 1 100k + Stage 2 10k) and CT (110k) below. WorldFlow's advantage over the CT baseline remains clear under the same total number of training steps.
| Method | Subject Consistency ↑ | Background Consistency ↑ | Spatial Relationship ↑ | Aesthetic Quality ↑ | Imaging Quality ↑ | Color ↑ |
|---|---|---|---|---|---|---|
| CT (110k) | 89.80 | 94.23 | 79.99 | 63.06 | 66.33 | 85.61 |
| WorldFlow (100k + 10k) | 92.12 | 94.91 | 84.68 | 64.18 | 66.76 | 85.03 |
All four variants in Fig. 4 (original Fig. 3) use the same Stage-2 renderer checkpoint, logit-normal schedule with , guidance settings, and 30 Euler steps.
The difference in effective loss weighting arises from converting -predictions to velocity before applying the -loss, and is therefore inherent to the prediction parameterization, as discussed in JiT [1].
[1] T. Li and K. He, "Back to Basics: Let Denoising Generative Models Denoise," CVPR 2026.
The -prediction velocity is computed as follows:
Euclidean:
Hyperspherical:
The conversion is direct in both geometries. In the hyperspherical branch the predicted endpoint is centered and rescaled onto before taking the logarithm, after which the sampler's tangent projection applies as usual; the Euclidean branch needs neither step. The -prediction variants floor the denominator at . Details are in Appendix A (added in the revision).
W2 & Q3: Multimodal validation
The adaptations are conducted on a random subset of OpenUni, a large multimodal video dataset proposed in UnityVideo [1], which we cite in the revised paper. For each modality, we optimize the renderer for 1k iterations at batch size 16. The data details are summarized below:
| Modality | Annotation source in OpenUni | #Train videos | #Test videos |
|---|---|---|---|
| Depth | Depth Anything 2 | $\sim$10k | 100 |
| Segmentation | SAM | $\sim$10k | 100 |
| Optical flow | RAFT | $\sim$10k | 100 |
| DensePose | Meta DensePose | $\sim$10k | 100 |
The 100 test videos are held out and disjoint from the adaptation videos.
We report quantitative results of depth, segmentation, optical flow and DensePose estimation on these held-out test videos, rendering from states encoded from each video and comparing against its OpenUni annotations. The "unpaired" row computes metrics on unrelated {prediction, annotation} pairs to establish a reference value on structurally non-correlated videos. Our metrics are significantly better, indicating promising alignment across different modalities.
Based on the visual comparisons covering samples with scores from best to worst at https://researchyfish.github.io/submission5463-rebuttal/#encoded, in several cases our outputs appear more temporally stable and spatially accurate than the provided annotations.
[1] J. Huang et al., "UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation," CVPR 2026.
| Depth (AbsRel ↓) | Segmentation (FG IoU% ↑) | Optical flow (Normalized EPE ↓) | DensePose (mIoU% ↑) | |
|---|---|---|---|---|
| Unpaired | 2.556 | 13.83 | 0.293 | 8.09 |
| Ours | 0.525 | 82.54 | 0.108 | 63.89 |
Note that the multimodal renderers are adapted independently and receive empty prompts, so their alignment cannot come from a shared input (e.g., text prompt) other than the state. The significant gap between paired/unpaired results further indicates that the geometry and dynamics originate in the shared state instead of leaking through renderer priors.
We further compare our multimodal rendering results from generated states with the outputs of pretrained estimators (Depth Anything 3, SAM 3, RAFT, Meta DensePose) applied to the generated videos of 100 randomly selected VBench prompts. The following scores use the estimators' outputs as pseudo-GT. The numbers are lower than in the first setting, partly because the estimators are less robust on generated frames, as shown in the visual samples at https://researchyfish.github.io/submission5463-rebuttal/#generated. The gap to the unpaired control further demonstrates the rich geometry and dynamics information in our generated states.
| Depth (AbsRel ↓) | Segmentation (FG IoU% ↑) | Optical flow (Normalized EPE ↓) | DensePose (mIoU% ↑) | |
|---|---|---|---|---|
| Unpaired | 2.863 | 13.59 | 0.312 | 1.19 |
| Ours | 0.903 | 67.23 | 0.157 | 38.41 |
To check agreement between independently adapted renderers rendering from the same state, we compute the foreground IoU between the DensePose output and the segmentation output, with an unpaired control that pairs outputs from unrelated states:
| DensePose FG vs. Seg. FG (IoU% ↑) | |
|---|---|
| Unpaired | 11.59 |
| Ours | 59.61 |
The two renderers are adapted separately with different supervision and share no conditioning input other than the state. Their outputs achieve a foreground overlap of 59.61% IoU, compared with 11.59% for the unpaired control. This higher overlap supports a consistent spatial layout across independently adapted renderers conditioned on the same state.
We thank the reviewer for the constructive feedback and for recognizing the value of generating native video representations at scale. Below, we clarify the experimental controls and computational costs, and provide new quantitative evidence of cross-modal consistency. More supporting visual evidence can be found on the webpage: https://researchyfish.github.io/submission5463-rebuttal.
W1 & Q1: Compression config and controlled comparison
We address each sub-question below:
[1] M. Guo et al., "V-RAE: Rethinking Video Latent Spaces for Generation," arXiv 2026.