- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents JVT, a pixel-space video diffusion model. It generates videos without a pretrained VAE, cascaded stages, or an extra detail network. The paper studies video patch embeddings, register tokens, and adaptive layer normalization. It also extends the model to jointly generate RGB and disparity under camera control. Experiments cover short-video generation and camera-conditioned generation.
Critical strengths
-
The model has a simple architecture. It achieves competitive results on the tested short-video benchmarks without a pretrained VAE, cascaded generation, or an extra detail network.
-
The authors provide ablation experiments to support their final design choices for patch embeddings, register tokens, and adaptive layer normalization.
Critical weaknesses
-
The novelty beyond prior work is limited. The method mainly extends JiT to video generation. Register tokens, register guidance, and joint RGB–geometry modeling have already been explored. The gain from the spatial-first embedding over the bottleneck tubelet baseline is also small: FVD improves from 135.6 to 135.0, without a reported range of variation.
-
The evaluation is limited to short videos at 256×256 resolution. There are no experiments on text-conditioned generation, longer videos, or higher resolutions. It remains unclear whether the proposed design works well in these settings.
-
The geometry extension lacks direct quantitative evaluation of its geometry outputs. The paper shows disparity maps and point clouds and reports camera-following accuracy. However, it does not directly measure depth or disparity accuracy, or cross-frame geometric consistency. Such results are needed to assess the quality of the generated geometry.
-
Important training details are missing. It is unclear whether the geometry model is trained from scratch or fine-tuned from an RGB model. The paper also does not specify how the base register parameters R₀ are initialized. Are they initialized to zero or sampled from a distribution? If sampled, what distribution and parameters are used? Please also provide the total training steps and the sampling settings used for each main result.
Questions for authors
Please address the questions and concerns raised in Critical Weaknesses.
5 comments
3. Quantitative evaluation for depth and pointmap
As the generated videos are usually different from ground truth videos, and the ground truth dense depth annotation is also unavailabel for RealEstate10K, we didn't evaluate accuracy for depth and pointmap in our initial submission. Here we use commonly used geometry estimators (Pi3X[6], VGGT-Omega[4] and Depth Anything 3[5]) to estimate pseudo-depth and pseudo-pointmap of the generated video, and use them as ground truth to evaluate accuracy. We select Aether[11], a general latent-space world model jointly modeling rgb, depth and raymap, as our baseline. We use point cloud accuracy evaluation to quantify cross-frame consistency. Additional qualitative results compared with Aether are included in the rebuttal version of supplementary material.
For depth accuracy, we report the Absolute Relative Error (AbsRel) and the prediction accuracy at a threshold of . For pointmap accuracy, we report Accuracy (Acc.), Completion (Comp.), and Normal Consistency (N.C.). Predicted point maps are aligned to the ground truth using the Umeyama algorithm for a coarse Sim(3) alignment, followed by refinement with the Iterative Closest Point (ICP) algorithm. (s-inv) and (a-inv) denote scale-invariant alignment and affine-invariant alignment, respectively.
-
Pi3X
Method AbsRel (s-inv) │ (s-inv) │ AbsRel (a-inv) │ (a-inv) │ Acc. │ Comp. │ N.C. Aether[11] 121.8795 │ 0.557 │ 0.2321 │ 0.745 │ 0.0446 │ 0.2095 │ 0.619 Aether[11] w/ disp 4.0301 │ 0.687 │ 0.1729 │ 0.818 │ 0.0503 │ 0.1588 │ 0.654 JVT w/ disp 0.0853 │ 0.935 │ 0.0489 │ 0.968 │ 0.0325 │ 0.0331 │ 0.740 -
VGGT-Omega
Method AbsRel (s-inv) │ (s-inv) │ AbsRel (a-inv) │ (a-inv) │ Acc. │ Comp. │ N.C. Aether[11] 658.7059 │ 0.578 │ 0.2252 │ 0.743 │ 0.0415 │ 0.2059 │ 0.618 Aether[11] w/ disp 328.1501 │ 0.706 │ 0.1631 │ 0.819 │ 0.0476 │ 0.1443 │ 0.650 JVT w/ disp from Pi3X 0.1118 │ 0.868 │ 0.0764 │ 0.926 │ 0.0342 │ 0.0465 │ 0.723 JVT w/ disp from VGGT-Omega 2.0320 │ 0.922 │ 0.0588 │ 0.955 │ 0.0330 │ 0.0369 │ 0.724 -
Depth Anything 3 (DA3)
Method AbsRel (s-inv) │ (s-inv) │ AbsRel (a-inv) │ (a-inv) │ Acc. │ Comp. │ N.C. Aether[11] 418.2016 │ 0.563 │ 0.2320 │ 0.751 │ 0.0410 │ 0.2173 │ 0.627 Aether[11] w/ disp 353.3358 │ 0.693 │ 0.1694 │ 0.831 │ 0.0441 │ 0.1577 │ 0.665 JVT w/ disp from Pi3X 0.1070 │ 0.869 │ 0.0648 │ 0.942 │ 0.0328 │ 0.0407 │ 0.732 JVT w/ disp from DA3 0.1468 │ 0.921 │ 0.0437 │ 0.975 │ 0.0263 │ 0.0277 │ 0.738
4. Implementation details
All our models, including the class-conditioned, unconditional and RGB / RGB-disparity camera-conditioned models are trained from scratch.The base registers are learnable tokens. When the training (from scratch) starts, they are initialized i.i.d. from a zero-mean Gaussian with standard deviation . We follow the training recipe of Latte[1], using training iterations of ~1M, constant learning rate with warm-up, max learning rate of 0.0001, and 8 NVIDIA 80GB GPUs for full-scale training. For camera-conditioned experiments, we use training iterations of ~300K.
For sampling, we use the Euler solver for class-conditioned and camera-conditioned generation, and the Heun solver for unconditional generation. All samplers run for 50 denoising steps, with the timestep schedule shifted by a noise shift. Depending on the task, we apply classifier-free guidance (CFG)[7], register guidance (RG)[8], or history guidance (HG)[9]. The CFG scale is selected by a linear search with a step size of 1, and the RG scale with a step size of 0.1. For FVD evaluation, we generate 2,048 clips per method and use the same number of real clips as the reference.
We also add additional training and sampling details to the appendix A.1 of the updated rebuttal version of our paper.
[1] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025.
[2] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3626–3636, June 2022.
[3] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18456–18466, June 2023.
[4] Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, Christian Rupprecht. VGGT-. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21486-21499, June 2026.
[5] Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, Bingyi Kang. Depth anything 3: recovering the visual space from any views. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=yirunib8l8.
[6] Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. : Permutation-equivariant visual geometry learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=DTQIjngDta.
[7] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview.net/forum?id=qw8AKxfYbI.
[8] Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers. arXiv preprint arXiv: 2605.16147, 2026.
[9] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=j8Vr3E3vhy.
[10] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 36115–36125, June 2026.
[11] Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, and Tong He. Aether: Geometric-aware unified world modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8535– 8546, October 2025.
[12] Jiehui Huang, Yuechen Zhang, Xu He, Yuan Gao, Zhi Cen, Bin Xia, Yan Zhou, Xin Tao, Pengfei Wan, and Jiaya Jia. Unityvideo: Unified multi-modal multi-task learning for enhancing world-aware video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4471–4481, June 2026a.
We thank the reviewer for recognizing our simple architecture and ablation experiments. We will try to clarify our contributions and add additional experimental details.
1. Limited novelty
We would like to clarify that our contributions go beyond a direct extension of JiT to video, with some notable findings below:
The real role register tokens play in pixel-space video diffusion. Previous works[8,10] propose using register tokens with class embedding added, which works on class-conditioned image generation on ImageNet. Rather than simply adopt this conclusion, we provide an empirical analysis that delineates when they are useful. Specifically, we show that if register guidance is not intended to be applied at inference time, incorporating registers into the JVT model design brings no significant benefit. Only on complex datasets, and when one is willing to apply register guidance, does the register module yield gains that surpass simply increasing the number of diffusion solver sampling steps. We believe this finding offers practical reference for future pixel-space diffusion designs.
A paradigm for joint modeling direct in pixel space. Previous works[11-12] try to force other modalities to be projected to RGB-specific VAE latent space, which causes information loss. What's more, some pixel-aligned modalities such as raymaps have long been difficult to align into VAE latent space. Although our experiments use RGB–disparity as an instantiation, the paradigm we propose is general: JVT is suited for jointly modeling RGB with other pixel-aligned modalities that are inherently ill-suited for RGB-specific VAE compression. To the best of our knowledge, our work is the first to align RGB and such pixel-aligned modalities directly in pixel space, which is fundamentally different from prior approaches that approximately map auxiliary modalities into the VAE's RGB-specific latent space.
In summary, we believe JVT's contributions lie in its simple yet effective embedding design, a clarified analysis of register usage, and a general pixel-space direct joint modeling paradigm, etc. We agree that our work is an initial exploration in this direction: extending it to more complex tasks, larger model sizes, higher resolutions, longer videos, and richer multi-modal joint generation requires further investigation in future work.
2. t2v, longer videos and higher resolutions
Regarding text-to-video generation, we are actively working on it. However, the progress has been slower than we hoped due to the difficulty of the task and our limited computation resources. We include preliminary results from a small-scale training in the rebuttal version of supplementary material, and we plan to release the model when the generation quality is acceptable.
To generate longer video at inference time, using the predict-and-interpolate method similar to DFoT, our camera-conditioned video generation is inherently supports generating longer videos at inference time.
As for pretraining directly on longer videos and at higher resolutions, this is difficult for us to pursue in the short term given our limited computation resources. We will release our training and inference code as soon as possible, and we eagerly anticipate the community can scale this simple pixel-space video generation paradigm to longer videos, higher resolutions, and more complex tasks.