acceptodds
Under review as a conference paper at ICLR 2027

JVT: Just Video Transformer makes pixel-space video diffusion simple

15 views

Abstract

Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.

Primary area:
generative models
Keywords:
video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V56nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents JVT, a pixel-space video diffusion model. It generates videos without a pretrained VAE, cascaded stages, or an extra detail network. The paper studies video patch embeddings, register tokens, and adaptive layer normalization. It also extends the model to jointly generate RGB and disparity under camera control. Experiments cover short-video generation and camera-conditioned generation.

Critical strengths
  1. The model has a simple architecture. It achieves competitive results on the tested short-video benchmarks without a pretrained VAE, cascaded generation, or an extra detail network.

  2. The authors provide ablation experiments to support their final design choices for patch embeddings, register tokens, and adaptive layer normalization.

Critical weaknesses
  1. The novelty beyond prior work is limited. The method mainly extends JiT to video generation. Register tokens, register guidance, and joint RGB–geometry modeling have already been explored. The gain from the spatial-first embedding over the bottleneck tubelet baseline is also small: FVD improves from 135.6 to 135.0, without a reported range of variation.

  2. The evaluation is limited to short videos at 256×256 resolution. There are no experiments on text-conditioned generation, longer videos, or higher resolutions. It remains unclear whether the proposed design works well in these settings.

  3. The geometry extension lacks direct quantitative evaluation of its geometry outputs. The paper shows disparity maps and point clouds and reports camera-following accuracy. However, it does not directly measure depth or disparity accuracy, or cross-frame geometric consistency. Such results are needed to assess the quality of the generated geometry.

  4. Important training details are missing. It is unclear whether the geometry model is trained from scratch or fine-tuned from an RGB model. The paper also does not specify how the base register parameters R₀ are initialized. Are they initialized to zero or sampled from a distribution? If sampled, what distribution and parameters are used? Please also provide the total training steps and the sampling settings used for each main result.

Questions for authors

Please address the questions and concerns raised in Critical Weaknesses.

5 comments

We thank the reviewer for recognizing our simple architecture and ablation experiments. We will try to clarify our contributions and add additional experimental details.

1. Limited novelty

We would like to clarify that our contributions go beyond a direct extension of JiT to video, with some notable findings below:

  • Patch-first embedding for efficiency-quality trade-off. We propose a patch-first embedding design, which achieves a better efficiency-quality trade-off compared to the bottlenecked tubelet embedding. In Table 2, we report only the minimum FVD, as most prior works[1-3] did not report statistical FVD results. Below are the statistics from 10 runs of FVD evaluation with different random seeds. Although FVD min improves little (135.6 to 135.0), the FVD mean greatly benefits from patch-first embedding (147.3 to 142.6). Moreover, efficiency metrics also yield gains in FLOPs, Latency and Peak memory.
Embedding│Tubelet│Patch-first│Frame-first
CbC_b│-3264128│163264│81632
FVD min│145.00147.42135.58142.28│143.55135.02143.48│151.20149.71146.62
FVD max│174.42167.85157.93170.62│155.46148.34166.90│164.09176.46160.77
FVD mean│156.69156.10147.30150.48│150.90142.59152.74│157.65158.00153.44
FVD std│9.667.146.638.27│3.883.848.08│4.078.034.74
  • The real role register tokens play in pixel-space video diffusion. Previous works[8,10] propose using register tokens with class embedding added, which works on class-conditioned image generation on ImageNet. Rather than simply adopt this conclusion, we provide an empirical analysis that delineates when they are useful. Specifically, we show that if register guidance is not intended to be applied at inference time, incorporating registers into the JVT model design brings no significant benefit. Only on complex datasets, and when one is willing to apply register guidance, does the register module yield gains that surpass simply increasing the number of diffusion solver sampling steps. We believe this finding offers practical reference for future pixel-space diffusion designs.

  • A paradigm for joint modeling direct in pixel space. Previous works[11-12] try to force other modalities to be projected to RGB-specific VAE latent space, which causes information loss. What's more, some pixel-aligned modalities such as raymaps have long been difficult to align into VAE latent space. Although our experiments use RGB–disparity as an instantiation, the paradigm we propose is general: JVT is suited for jointly modeling RGB with other pixel-aligned modalities that are inherently ill-suited for RGB-specific VAE compression. To the best of our knowledge, our work is the first to align RGB and such pixel-aligned modalities directly in pixel space, which is fundamentally different from prior approaches that approximately map auxiliary modalities into the VAE's RGB-specific latent space.

In summary, we believe JVT's contributions lie in its simple yet effective embedding design, a clarified analysis of register usage, and a general pixel-space direct joint modeling paradigm, etc. We agree that our work is an initial exploration in this direction: extending it to more complex tasks, larger model sizes, higher resolutions, longer videos, and richer multi-modal joint generation requires further investigation in future work.

2. t2v, longer videos and higher resolutions

Regarding text-to-video generation, we are actively working on it. However, the progress has been slower than we hoped due to the difficulty of the task and our limited computation resources. We include preliminary results from a small-scale training in the rebuttal version of supplementary material, and we plan to release the model when the generation quality is acceptable.

To generate longer video at inference time, using the predict-and-interpolate method similar to DFoT, our camera-conditioned video generation is inherently supports generating longer videos at inference time.

As for pretraining directly on longer videos and at higher resolutions, this is difficult for us to pursue in the short term given our limited computation resources. We will release our training and inference code as soon as possible, and we eagerly anticipate the community can scale this simple pixel-space video generation paradigm to longer videos, higher resolutions, and more complex tasks.

3. Quantitative evaluation for depth and pointmap

As the generated videos are usually different from ground truth videos, and the ground truth dense depth annotation is also unavailabel for RealEstate10K, we didn't evaluate accuracy for depth and pointmap in our initial submission. Here we use commonly used geometry estimators (Pi3X[6], VGGT-Omega[4] and Depth Anything 3[5]) to estimate pseudo-depth and pseudo-pointmap of the generated video, and use them as ground truth to evaluate accuracy. We select Aether[11], a general latent-space world model jointly modeling rgb, depth and raymap, as our baseline. We use point cloud accuracy evaluation to quantify cross-frame consistency. Additional qualitative results compared with Aether are included in the rebuttal version of supplementary material.

For depth accuracy, we report the Absolute Relative Error (AbsRel) and the prediction accuracy at a threshold of δ<1.25\delta<1.25. For pointmap accuracy, we report Accuracy (Acc.), Completion (Comp.), and Normal Consistency (N.C.). Predicted point maps are aligned to the ground truth using the Umeyama algorithm for a coarse Sim(3) alignment, followed by refinement with the Iterative Closest Point (ICP) algorithm. (s-inv) and (a-inv) denote scale-invariant alignment and affine-invariant alignment, respectively.

  • Pi3X

    MethodAbsRel (s-inv) ↓\downarrow│δ<1.25\delta<1.25 (s-inv) ↑\uparrow│AbsRel (a-inv) ↓\downarrow│δ<1.25\delta<1.25 (a-inv) ↑\uparrow│Acc. ↓\downarrow│Comp. ↓\downarrow│N.C. ↑\uparrow
    Aether[11]121.8795│0.557│0.2321│0.745│0.0446│0.2095│0.619
    Aether[11] w/ disp4.0301│0.687│0.1729│0.818│0.0503│0.1588│0.654
    JVT w/ disp0.0853│0.935│0.0489│0.968│0.0325│0.0331│0.740
  • VGGT-Omega

    MethodAbsRel (s-inv) ↓\downarrow│δ<1.25\delta<1.25 (s-inv) ↑\uparrow│AbsRel (a-inv) ↓\downarrow│δ<1.25\delta<1.25 (a-inv) ↑\uparrow│Acc. ↓\downarrow│Comp. ↓\downarrow│N.C. ↑\uparrow
    Aether[11]658.7059│0.578│0.2252│0.743│0.0415│0.2059│0.618
    Aether[11] w/ disp328.1501│0.706│0.1631│0.819│0.0476│0.1443│0.650
    JVT w/ disp from Pi3X0.1118│0.868│0.0764│0.926│0.0342│0.0465│0.723
    JVT w/ disp from VGGT-Omega2.0320│0.922│0.0588│0.955│0.0330│0.0369│0.724
  • Depth Anything 3 (DA3)

    MethodAbsRel (s-inv) ↓\downarrow│δ<1.25\delta<1.25 (s-inv) ↑\uparrow│AbsRel (a-inv) ↓\downarrow│δ<1.25\delta<1.25 (a-inv) ↑\uparrow│Acc. ↓\downarrow│Comp. ↓\downarrow│N.C. ↑\uparrow
    Aether[11]418.2016│0.563│0.2320│0.751│0.0410│0.2173│0.627
    Aether[11] w/ disp353.3358│0.693│0.1694│0.831│0.0441│0.1577│0.665
    JVT w/ disp from Pi3X0.1070│0.869│0.0648│0.942│0.0328│0.0407│0.732
    JVT w/ disp from DA30.1468│0.921│0.0437│0.975│0.0263│0.0277│0.738

4. Implementation details

All our models, including the class-conditioned, unconditional and RGB / RGB-disparity camera-conditioned models are trained from scratch.The base registers R0R_0 are learnable tokens. When the training (from scratch) starts, they are initialized i.i.d. from a zero-mean Gaussian with standard deviation 0.020.02. We follow the training recipe of Latte[1], using training iterations of ~1M, constant learning rate with warm-up, max learning rate of 0.0001, and 8 NVIDIA 80GB GPUs for full-scale training. For camera-conditioned experiments, we use training iterations of ~300K.

For sampling, we use the Euler solver for class-conditioned and camera-conditioned generation, and the Heun solver for unconditional generation. All samplers run for 50 denoising steps, with the timestep schedule shifted by a noise shift. Depending on the task, we apply classifier-free guidance (CFG)[7], register guidance (RG)[8], or history guidance (HG)[9]. The CFG scale is selected by a linear search with a step size of 1, and the RG scale with a step size of 0.1. For FVD evaluation, we generate 2,048 clips per method and use the same number of real clips as the reference.

We also add additional training and sampling details to the appendix A.1 of the updated rebuttal version of our paper.

[1] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025.

[2] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3626–3636, June 2022.

[3] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18456–18466, June 2023.

[4] Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, Christian Rupprecht. VGGT-Ω\Omega. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21486-21499, June 2026.

[5] Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, Bingyi Kang. Depth anything 3: recovering the visual space from any views. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=yirunib8l8.

[6] Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π3\pi^3: Permutation-equivariant visual geometry learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=DTQIjngDta.

[7] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview.net/forum?id=qw8AKxfYbI.

[8] Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers. arXiv preprint arXiv: 2605.16147, 2026.

[9] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=j8Vr3E3vhy.

[10] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 36115–36125, June 2026.

[11] Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, and Tong He. Aether: Geometric-aware unified world modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8535– 8546, October 2025.

[12] Jiehui Huang, Yuechen Zhang, Xu He, Yuan Gao, Zhi Cen, Bin Xia, Yan Zhou, Xin Tao, Pengfei Wan, and Jiaya Jia. Unityvideo: Unified multi-modal multi-task learning for enhancing world-aware video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4471–4481, June 2026a.

Reviewer_WvE7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends the clean-data prediction approach of JiT to pixel-space video generation using a simple Transformer without an autoencoder or specialized refinement modules. It studies spatiotemporal patch embeddings, register guidance, and conditioning mechanisms, and demonstrates competitive FVD across four video datasets. An additional camera-conditioned RGB–disparity model improves camera following and produces point clouds through geometric unprojection.

Critical strengths
  1. A useful demonstration of architectural simplicity. Competitive generation across four datasets shows that pixel-space video modeling does not necessarily require specialized multiscale architectures. The embedding and conditioning ablations make this more informative than a single successful configuration.

  2. A practical extension to joint appearance and geometry generation. In Table 5, adding disparity improves RTA@5 from 72.4 to 79.5 while changing FVD only from 27.0 to 27.5. This supports the usefulness of jointly modeling aligned modalities.

Critical weaknesses

1.The comparative evidence is not sufficiently controlled. Table 1 largely combines published results, while total training updates and compute for JVT are not specified in Appendix A.1. The efficiency analysis primarily compares JVT variants rather than matched latent baselines. Consequently, the results establish feasibility more convincingly than a favorable quality–compute trade-off against latent diffusion.

2.The geometric contribution is only indirectly validated. Camera-following scores evaluate poses estimated from generated RGB, rather than the accuracy or cross-view consistency of the generated disparity. Pi3X also supplies both training depth targets and evaluation poses. The point-cloud visualizations are encouraging, but independent geometric evaluation is needed to substantiate the broader scene-modeling claims.

Questions for authors

1.Experimental clarification: What are the total training budgets and exact sampling configurations for the main results? How does JVT compare with a representative latent baseline under matched training and inference budgets?

2.Geometric validation: Does the camera-following improvement persist with an independent pose estimator, and can you quantify the generated disparity’s cross-view consistency or geometric accuracy?

3 comments

We thank the reviewer for recognizing our demonstration. We try to explore the potential of this simple paradigm for video generation.

1. Table 1 results

Most prior works[1-3] just compare their own results with the reported results of previous works, and it's hard to reproduce every baseline, because many of them didn't release code/checkpoint/evaluation protocal. Therefore, following the convention of prior works, we compare against the FVD values reported by each work.

2. Training budgets and sampling configurations

For training, we follow the training recipe of Latte[1], using training iterations of ~1M, constant learning rate with warm-up, max learning rate of 0.0001, and 8 NVIDIA 80GB GPUs for full-scale training. For camera-conditioned experiments, we use training iterations of ~300K.

For sampling, we use the Euler solver for class-conditioned and camera-conditioned generation, and the Heun solver for unconditional generation. All samplers run for 50 denoising steps, with the timestep schedule shifted by a noise shift. Depending on the task, we apply classifier-free guidance (CFG)[4], register guidance (RG)[5], or history guidance (HG)[6]. The CFG scale is selected by a linear search with a step size of 1, and the RG scale with a step size of 0.1. For FVD evaluation, we generate 2,048 clips per method and use the same number of real clips as the reference.

We also add additional training and sampling details to the appendix A.1 of the updated rebuttal version of our paper.

3. Comparison with latent baselines under matched budget

Here we compare JVT with other 3 latent baselines, in which Latte is our main baseline. These methods train on the same resolution of 16×256×25616\times 256\times 256, and share similar model sizes and training recipes. The default settings of the 3 baselines all have large NFE and large latency, so we try to compare them under the same inference NFE. The UCF-101 checkpoint of LVDM is not released, so the data is missing.

MethodSamplingConditionNFEFVD (UCF-101, c2v)FVD (UCF-101, uncond)FVD (FFS, uncond)FVD (Sky, uncond)FVD (Taichi, uncond)
PVDM-L[3]DDIM 100Uncond.100455.1231.3
LVDM[2]DDIM 100Uncond.100553.6247.6
Latte-XL/2[1]DDIM 50, CFG 7Cond.100165.2
DDIM 100Uncond.10047.0791.66367.2
JVTEuler 50, CFGCond.100136.5
Heun 50Uncond.100413.532.8646.58119.1
JVT+RGEuler 50, CFG+RGCond.150129.2
Heun 50, RGUncond.200215.931.8047.80124.3

We also report the inference efficiency details of JVT and 3 latent baselines. We find that even under the sample NFE (e.g. Latte DDIM 50 and JVT Euler 50), VAE-free JVT still has clear gain on TFLOPs, latency and peak memory.

MethodDefaultsettingSamplingConditionNFEDenoiser (M)VAE params(M)Total (M)TFLOPs / videoVAE decode (TFLOPs)Latency (s / video)Peak mem (GiB)
PVDM-L[3]✓DDIM 400Uncond.400528.326.9555.198.41.1611.203.59
DDIM 100Uncond.100528.326.9555.125.51.162.963.59
LVDM[2]✓DDPM 1000Uncond.1000548.912.0560.9364.62.8519.843.31
DDIM 100Uncond.100548.912.0560.939.02.852.043.31
Latte-XL/2[1]✓DDPM 250, CFG 7Cond.500674.149.5723.61902.99.9519.086.36
✓DDPM 250Uncond.250674.049.5723.5956.49.9510.366.36
DDIM 50, CFG 7Cond.100674.149.5723.6388.59.953.986.36
DDIM 100Uncond.100674.049.5723.5388.59.954.286.36
JVT✓Euler 50, CFGCond.100678.80678.8105.701.152.73
✓Heun 50Uncond.100678.60678.6105.701.472.70
JVT+RG✓Euler 50, CFG+RGCond.150678.80678.8161.001.952.75
✓Heun 50, RGUncond.200678.70678.7213.903.082.72

4. Independent pose estimator for camera following evaluation

We add two commonly used independent pose estimators, VGGT-Omega[7] and Depth Anything 3[8], to evaluate camera following. The resulting metrics are numerically close to those estimated by Pi3X[9], and RGB-disparity joint generation still shows a significant improvement over RGB-only generation in terms of translation.

  • VGGT-Omega
MethodRRA@5RRA@30RTA@5RTA@30AUC@5AUC@30
DFoT[6]99.910082.398.462.689.9
Geometry Forcing[10]99.810077.397.754.887.3
LIVE[11]98.710048.891.827.472.6
JVT RGB-disp99.910081.099.059.589.7
JVT RGB99.910075.297.154.186.0
  • Depth Anything 3
MethodRRA@5RRA@30RTA@5RTA@30AUC@5AUC@30
DFoT[6]99.910080.898.561.489.3
Geometry Forcing[10]99.810075.997.853.686.7
LIVE[11]98.910042.091.722.569.9
JVT RGB-disp10010082.099.162.190.3
JVT RGB99.910077.197.757.787.5

5. Quantitative evaluation for depth and pointmap

As the generated videos are usually different from ground truth videos, and the ground truth dense depth annotation is also unavailabel for RealEstate10K, we didn't evaluate accuracy for depth and pointmap in our initial submission. Here we use commonly used geometry estimators (Pi3X[9], VGGT-Omega[7] and DA3[8]) to estimate pseudo-depth and pseudo-pointmap of the generated video, and use them as ground truth to evaluate accuracy. We select Aether[12], a general latent-space world model jointly modeling rgb, depth and raymap, as our baseline. We use point cloud accuracy evaluation to quantify cross-frame consistency. Additional qualitative results compared with Aether are included in the rebuttal version of supplementary material.

For depth accuracy, we report the Absolute Relative Error (AbsRel) and the prediction accuracy at a threshold of δ<1.25\delta<1.25. For pointmap accuracy, we report Accuracy (Acc.), Completion (Comp.), and Normal Consistency (N.C.). Predicted point maps are aligned to the ground truth using the Umeyama algorithm for a coarse Sim(3) alignment, followed by refinement with the Iterative Closest Point (ICP) algorithm. (s-inv) and (a-inv) denote scale-invariant alignment and affine-invariant alignment, respectively.

  • Pi3X

    MethodAbsRel (s-inv) ↓\downarrow│δ<1.25\delta<1.25 (s-inv) ↑\uparrow│AbsRel (a-inv) ↓\downarrow│δ<1.25\delta<1.25 (a-inv) ↑\uparrow│Acc. ↓\downarrow│Comp. ↓\downarrow│N.C. ↑\uparrow
    Aether[12]121.8795│0.557│0.2321│0.745│0.0446│0.2095│0.619
    Aether[12] w/ disp4.0301│0.687│0.1729│0.818│0.0503│0.1588│0.654
    JVT w/ disp0.0853│0.935│0.0489│0.968│0.0325│0.0331│0.740
  • VGGT-Omega

    MethodAbsRel (s-inv) ↓\downarrow│δ<1.25\delta<1.25 (s-inv) ↑\uparrow│AbsRel (a-inv) ↓\downarrow│δ<1.25\delta<1.25 (a-inv) ↑\uparrow│Acc. ↓\downarrow│Comp. ↓\downarrow│N.C. ↑\uparrow
    Aether[12]658.7059│0.578│0.2252│0.743│0.0415│0.2059│0.618
    Aether[12] w/ disp328.1501│0.706│0.1631│0.819│0.0476│0.1443│0.650
    JVT w/ disp from Pi3X0.1118│0.868│0.0764│0.926│0.0342│0.0465│0.723
    JVT w/ disp from VGGT-Omega2.0320│0.922│0.0588│0.955│0.0330│0.0369│0.724
  • Depth Anything 3 (DA3)

    MethodAbsRel (s-inv) ↓\downarrow│δ<1.25\delta<1.25 (s-inv) ↑\uparrow│AbsRel (a-inv) ↓\downarrow│δ<1.25\delta<1.25 (a-inv) ↑\uparrow│Acc. ↓\downarrow│Comp. ↓\downarrow│N.C. ↑\uparrow
    Aether[12]418.2016│0.563│0.2320│0.751│0.0410│0.2173│0.627
    Aether[12] w/ disp353.3358│0.693│0.1694│0.831│0.0441│0.1577│0.665
    JVT w/ disp from Pi3X0.1070│0.869│0.0648│0.942│0.0328│0.0407│0.732
    JVT w/ disp from DA30.1468│0.921│0.0437│0.975│0.0263│0.0277│0.738

[1] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025.

[2] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Czsdv-S4-w9.

[3] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18456–18466, June 2023.

[4] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview.net/forum?id=qw8AKxfYbI.

[5] Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers. arXiv preprint arXiv: 2605.16147, 2026.

[6] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=j8Vr3E3vhy.

[7] Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, Christian Rupprecht. VGGT-Ω\Omega. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21486-21499, June 2026.

[8] Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, Bingyi Kang. Depth anything 3: recovering the visual space from any views. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=yirunib8l8.

[9] Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π3\pi^3: Permutation-equivariant visual geometry learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=DTQIjngDta.

[10] Haoyu Wu, Diankun Wu, Tianyu He, Junliang Guo, Yang Ye, Yueqi Duan, and Jiang Bian. Geometry forcing: Marrying video diffusion and 3d representation for consistent world modeling. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ULXYZCms41.

[11] Junchao Huang, Ziyang Ye, Xinting Hu, Tianyu He, Guiyu Zhang, Shaoshuai Shi, Jiang Bian, and Li Jiang. LIVE: Long-horizon interactive video world modeling. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=iKKZmgxkbG.

[12] Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, and Tong He. Aether: Geometric-aware unified world modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8535– 8546, October 2025.

Reviewer_HsWsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends JiT to pixel-space video generation with a plain DiT, ablating patch embedding, registers with register guidance, and AdaLN. It reports strong FVD on UCF-101 and three other datasets, and extends to camera-conditioned RGB+disparity joint generation.

Critical strengths

The design is simple and the results are strong (conditional UCF-101 FVD 136.5 vs. 478 for Latte), showing that video diffusion can work without a VAE. Adding a disparity channel improves camera following and yields geometry directly, which is a natural advantage of pixel space.

Critical weaknesses

Novelty is limited: it is essentially JiT for video, and most ablations are null or within FVD noise, with single runs. The comparison with latent models is not controlled, the training budget is not reported, and the appendix shows Latte+IMG, FAR and others are better on several benchmarks.

Questions for authors

What is the total training budget, and can you add a latent DiT trained with the same recipe? Pi3X provides both the depth labels and the pose evaluation; can you evaluate with an independent pose estimator?

3 comments

We thank the reviewer for identifying our simple design and extension to RGB+disparity joint generation. Below we provide additional details that were not covered in our initial submission.

1. Most ablations are null or within FVD noise with single runs

In Table 2, we report only the minimum FVD, following most prior works[1-3] which did not report statistical FVD results. Here we report below the statistics from 10 runs of FVD evaluation with different random seeds:

Embedding│Tubelet│Patch-first│Frame-first
CbC_b│-3264128│163264│81632
FVD min│145.00147.42135.58142.28│143.55135.02143.48│151.20149.71146.62
FVD max│174.42167.85157.93170.62│155.46148.34166.90│164.09176.46160.77
FVD mean│156.69156.10147.30150.48│150.90142.59152.74│157.65158.00153.44
FVD std│9.667.146.638.27│3.883.848.08│4.078.034.74

We also conduct 10 runs of FVD evaluation with different random seeds for the results in Table 3 and Table 4.

  • Table 3
Guidance│Sampling│FVD min/max
CFGRG│SolverStepsNFE│No reg.er=0e_r=0er=eye_r=e_yer=ete_r=e_ter=ey+ete_r=e_y+e_t
✓✗│Euler50100│136.32/147.08136.10/159.73138.63/161.95137.13/150.48137.56/160.13
✓✗│Euler100200│134.12/146.59131.84/145.17129.39/144.13124.94/142.58137.91/156.50
✓✗│Heun50200│157.43/168.46143.28/158.83143.36/161.34147.87/160.98151.20/170.80
✓✓│Euler50150│–129.19/144.81130.42/144.15133.32/144.89131.98/155.74
Guidance│Sampling│FVD mean$\pm$std
CFGRG│SolverStepsNFE│No reg.er=0e_r=0er=eye_r=e_yer=ete_r=e_ter=ey+ete_r=e_y+e_t
✓✗│Euler50100│141.39$\pm$3.89145.89$\pm$8.21149.88$\pm$7.91142.67$\pm$4.69146.50$\pm$6.87
✓✗│Euler100200│141.16$\pm$3.95137.18$\pm$4.07138.02$\pm$4.82136.88$\pm$4.86144.85$\pm$6.57
✓✗│Heun50200│163.15$\pm$3.56152.43$\pm$5.10151.81$\pm$4.81156.19$\pm$3.91162.16$\pm$6.07
✓✓│Euler50150│–138.82$\pm$4.47137.47$\pm$3.77138.39$\pm$3.78143.26$\pm$6.40
  • Table 4
VariantAdaLN conditionere_rFVD minFVD maxFVD meanFVD std
baselineet+eye_t+e_y–136.32147.08141.393.89
w/ er=eye_r=e_yet+eye_t+e_yeye_y138.63161.95149.887.91
w/o noise leveleye_y–140.45158.95148.905.92
w/o AdaLN–eye_y156.26170.91161.794.96

2. Limited novelty

We would like to clarify that our contributions go beyond a direct extension of JiT to video, with some notable findings below:

  • Patch-first embedding for efficiency-quality trade-off. We propose a patch-first embedding design, which achieves a better efficiency-quality trade-off compared to the bottlenecked tubelet embedding. From bottlenecked tubelet embedding to patch-first embedding, FVD mean improves from 147.3 to 142.6, while the efficiency metrics also yield gains in FLOPs, Latency and Peak memory.

  • The real role register tokens play in pixel-space video diffusion. Previous works[8,14] propose using register tokens with class embedding added, which works on class-conditioned image generation on ImageNet. Rather than simply adopt this conclusion, we provide an empirical analysis that delineates when they are useful. Specifically, we show that if register guidance is not intended to be applied at inference time, incorporating registers into the JVT model design brings no significant benefit. Only on complex datasets, and when one is willing to apply register guidance, does the register module yield gains that surpass simply increasing the number of diffusion solver sampling steps. We believe this finding offers practical reference for future pixel-space diffusion designs.

  • A paradigm for joint modeling direct in pixel space. Previous works[10-11] try to force other modalities to be projected to RGB-specific VAE latent space, which causes information loss. What's more, some pixel-aligned modalities such as raymaps have long been difficult to align into VAE latent space. Although our experiments use RGB–disparity as an instantiation, the paradigm we propose is general: JVT is suited for jointly modeling RGB with other pixel-aligned modalities that are inherently ill-suited for RGB-specific VAE compression. To the best of our knowledge, our work is the first to align RGB and such pixel-aligned modalities directly in pixel space, which is fundamentally different from prior approaches that approximately map auxiliary modalities into the VAE's RGB-specific latent space.

In summary, we believe JVT's contributions lie in its simple yet effective embedding design, a clarified analysis of register usage, and a general pixel-space direct joint modeling paradigm, etc. We agree that our work is an initial exploration in this direction: extending it to more complex tasks, larger model sizes, higher resolutions, longer videos, and richer multi-modal joint generation requires further investigation in future work.

3. Training budget and controlled comparison with latent DiT

We follow the training recipe of Latte[1], using training iterations of ~1M, constant learning rate with warm-up, max learning rate of 0.0001, and 8 NVIDIA 80GB GPUs for full-scale training. Therefore, we regard our comparison with Latte as a fair comparison against a standard latent DiT, and adopt Latte as our main baseline. For camera-conditioned experiments, we use training iterations of ~300K. We also add additional training details to the appendix A.1 of the updated rebuttal version of our paper.

4. Appendix shows Latte+IMG, FAR and others are better on several benchmarks

When designing the JVT architecture, we aim to keep it as simple and fundamental as possible, and therefore did not incorporate more sophisticated training or modeling techniques, such as the video-image joint training used in Latte+IMG[1], or the hybrid AR+diffusion modeling used in FAR[4]. That said, we believe these techniques are complementary to JVT and hold strong potential for further improving its performance when combined with our framework. We look forward to subsequent studies that explore practical training and modeling techniques and fill this gap.

5. Independent pose estimator for camera following evaluation

We add two commonly used independent pose estimators, VGGT-Omega[5] and Depth Anything 3[6], to evaluate camera following. The resulting metrics are numerically close to those estimated by Pi3X[7], and RGB-disparity joint generation still shows a significant improvement over RGB-only generation in terms of translation.

  • VGGT-Omega
MethodRRA@5RRA@30RTA@5RTA@30AUC@5AUC@30
DFoT[9]99.910082.398.462.689.9
Geometry Forcing[12]99.810077.397.754.887.3
LIVE[13]98.710048.891.827.472.6
JVT RGB-disp99.910081.099.059.589.7
JVT RGB99.910075.297.154.186.0
  • Depth Anything 3
MethodRRA@5RRA@30RTA@5RTA@30AUC@5AUC@30
DFoT[9]99.910080.898.561.489.3
Geometry Forcing[12]99.810075.997.853.686.7
LIVE[13]98.910042.091.722.569.9
JVT RGB-disp10010082.099.162.190.3
JVT RGB99.910077.197.757.787.5

[1] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025.

[2] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3626–3636, June 2022.

[3] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18456–18466, June 2023.

[4] Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv: 2503.19325, 2025.

[5] Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, Christian Rupprecht. VGGT-Ω\Omega. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21486-21499, June 2026.

[6] Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, Bingyi Kang. Depth anything 3: recovering the visual space from any views. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=yirunib8l8.

[7] Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π3\pi^3: Permutation-equivariant visual geometry learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=DTQIjngDta.

[8] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 36115–36125, June 2026.

[9] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=j8Vr3E3vhy.

[10] Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, and Tong He. Aether: Geometric-aware unified world modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8535– 8546, October 2025.

[11] Jiehui Huang, Yuechen Zhang, Xu He, Yuan Gao, Zhi Cen, Bin Xia, Yan Zhou, Xin Tao, Pengfei Wan, and Jiaya Jia. Unityvideo: Unified multi-modal multi-task learning for enhancing world-aware video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4471–4481, June 2026a.

[12] Haoyu Wu, Diankun Wu, Tianyu He, Junliang Guo, Yang Ye, Yueqi Duan, and Jiang Bian. Geometry forcing: Marrying video diffusion and 3d representation for consistent world modeling. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ULXYZCms41.

[13] Junchao Huang, Ziyang Ye, Xinting Hu, Tianyu He, Guiyu Zhang, Shaoshuai Shi, Jiang Bian, and Li Jiang. LIVE: Long-horizon interactive video world modeling. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=iKKZmgxkbG.

[14] Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers. arXiv preprint arXiv: 2605.16147, 2026.

Comments on the paper

2 comments
Area_Chair_751BOpenReview ↗

Title: Gentle Reminder: Discussion Closing Soon

Dear Reviewer,

As the discussion period closes soon, we kindly encourage you to engage in the discussion and share any remaining thoughts or feedback.

Thank you for your time and valuable contributions!

AuthorsOpenReview ↗

Summary of our recognized contribution and how we address the reviewers' concerns

Dear Area Chair,

We thank area chair and all reviewers for taking time to review our work and give constructive feedback. We would like to summarize our recognized strengths and how we address the reviewers' concerns.

  1. Our architecture is simple and effective, achieving competitive generation quality against latent baselines. (Reviewer HsWs, WvE7, V56n) Pretrained VAE, cascaded generation, multiscale designs and detailers are not mandatory for high-quality pixel-space video generation.
  2. We prove a natural advantage of pixel-space modeling: pixel-aligned joint modeling. (Reviewer HsWs, WvE7) As a practical example, we extend JVT to joint RGB-disparity generation, which generates geometry directly without loss from VAE compression and postprocessing of another 3D reconstruction model.
  3. We do ablations and give design choices for patch embeddings, register tokens, and adaptive layer normalization. (Reviewer V56n) We ablate only the necessary modules, keeping JVT's simplicity.

During the discussion period, we address all reviewers' concerns.

  1. Limited novelty. (Reviewer HsWs, V56n) We clarify our key findings: (1) patch-first embedding for efficiency-quality trade-off (2) real role of register tokens in pixel-space video diffusion (3) a paradigm for joint modeling direct in pixel space.
  2. FVD noise in ablation. (Reviewer HsWs, V56n) For all ablations, we report statistical FVDs from 10 runs of FVD evaluation with different random seeds. Following most prior works, we did not report statistical FVD results in the submission version.
  3. Implementation details. (Reviewer HsWs, WvE7, V56n) We answer the reviewers the details about designing, training and sampling. We also update the details to the appendix A.1 of the rebuttal version of our paper.
  4. Controlled comparison with latent DiT baselines. (Reviewer HsWs, WvE7) Latte serves as our main baseline, as we align our training budget with Latte. We further give efficiency comparison with latent baselines under matched budget.
  5. Independent pose estimator for camera following evaluation. (Reviewer HsWs, WvE7) We add two commonly used independent pose estimators to evaluate camera following. The resulting metrics are numerically close to those estimated by Pi3X, and RGB-disparity joint generation still shows a significant improvement on translation over RGB-only generation.
  6. Quantitative evaluation for depth and pointmap. (Reviewer WvE7, V56n) We use three commonly used geometry estimators to estimate pseudo-depth and pseudo-pointmap of the generated video, and use them as ground truth to evaluate accuracy. We select Aether, a general latent-space world model jointly modeling rgb, depth and raymap, as our baseline. We also provide additional qualitative results in the rebuttal version of supplementary material.
  7. Other concerns. We explain our FVD results in Appendix. (Reviewer HsWs) We explain we follow the convention of prior works to gather the reported FVDs in Table 1. (Reviewer WvE7) We update our preliminary results on text-to-video generation in the rebuttal version of supplementary material, and explain the longer avideo and higher resolution settings. (Reviewer V56n)

Thank you again for spending time reviewing our work. We hope our rebuttal resolve the reviewers' concerns and look forward to your decision.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.