- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that a flat, large-patch transformer is a poor fit for pixel-space video, and proposes U-JIT, a U-shaped transformer whose patch size grows with depth. The key evidence is a PCA spectrum (Figure 2): JiT's signal dimension is about 596, versus 1308 for real clips and 1284 for U-JIT. The authors also borrow three things from latent decoders, namely time-shifting, CFG, and an LPIPS loss, though LPIPS is only applied to U-JIT. All models are trained from scratch on Kinetics-600. U-JIT matches the best retrained latent baseline on FVD and beats it on FID, from f8t4 up to f32t8, on frames-to-video, image-to-video, and class-to-video.
Critical strengths
-
Figure 2 shows JiT's spectrum collapsing to a noise floor, missing high frequencies. Table S4 confirms that removing the U shape at matched FLOPs drastically worsens both FVD and FID. Progressive patchification effectively solves the high dimensional data bottleneck through learned nonlinear compression.
-
The latent baselines are rigorously retrained under matching objectives, datasets, and FLOP budgets. U-JIT uses fewer parameters yet ties the best latent FVD and shows strong robustness to extreme spatiotemporal compression rates like f64t8.
-
Section 3.3 effectively justifies the borrowed sampling tricks. By jittering latents and isolating frequency bands, the authors prove how the decoder maps isotropic noise to perceptually aligned directions. This perfectly explains why time shifting, CFG, and LPIPS are necessary here.
Critical weaknesses
-
The FID gap over latent models is mostly the LPIPS loss, not the architecture. LPIPS is only on U-JIT; removing it (Table S4, line 2) drops FID from 19.20 to 35.47. FID is also only on the last frame. On FVD, with matched sampling, the lead is tiny: 0.61 vs. 0.62, and 2.55 vs. 2.66.
-
The reported headline numbers need to be clarified. In the introduction (lines 084 to 086), the authors claim a 29% FVD improvement and a 90% FID improvement over the JIT pixel space baseline. However, this compares against the base sampling JIT (0.86 FVD and 32.43 FID), not the tuned JIT (0.71 FVD and 5.29 FID). The comparison against the tuned JIT is the 14% FVD improvement actually mentioned in the abstract (line 025). Furthermore, the 64% FID drop claimed in the abstract (line 028) is against Cosmos, not Wan2.1, which has the best FVD.
-
Scope is thin for the conclusion. The experiments only cover Kinetics-600, resolutions of 128 or 256, and up to 65 frames. Class conditional FVD and FID are computed on the training set. CFG is picked by grid search, and the paper does not specify whether the grid uses the same videos as the reported scores.
-
Figure 2 is missing the details needed to be credible. There is no sample count, no patches per video, and no patch dimension explicitly linked to the math. You cannot estimate a roughly 1300 dimension signal from 1000 samples if each video gives one 3072 dimension patch.
Questions for authors
-
Please train the JiT and l-JIT baselines with the same LPIPS weight applied to U-JIT (at least for Table 3) and report the FID across all generated frames. If U-JIT fails to outperform Wan2.2 on FVD under these matched conditions, my score will remain a weak reject.
-
Please clarify the exact sampling methodology for Figure 2. Specifically, state the total number of videos, the number of patches extracted per video, and the exact patch dimensions. Reaching an estimated 1300 dimension subspace from 1000 video samples mathematically requires extracting multiple patches per video, which must be detailed.
-
Were the CFG scales in Tables 2 through 4 tuned directly on the reported evaluation sets? Please provide class to video generation metrics computed on a strictly held out validation set.
-
How exactly are the conditioning frames fed into the model? Please specify whether they are concatenated, used as clean token replacements, or provided via a separate embedding. The experiments in Tables 2 and 3 cannot be reproduced without this information.
We would like to thank the reviewer for their time and for highlighting that “U-JIT [...] ties the best latent FVD and shows strong robustness to extreme spatiotemporal compression” and that our paper “effectively justifies the borrowed sampling tricks”.
We addressed the reviewer first and main weakness/question in the general answer, with new additional experimental results. Additionally, we would like to address individual points below:
Weakness 1 / Question 1: we addressed the main concern on LPIPS-controlled experiments in the general answer to all reviewers, with new experimental results. Additionally, we would like to explain the choice of reference frame for FID: on frames-to-video experiments, the first few generated frames are strongly correlated with the input conditioning, while the last frame is more representative of the model generative capabilities. We added in Appendix C.1 an analysis reporting the per-frame FID, as well as the overall FID on 12 generated frames. We see that for all models, the per-frame FID degrades with frame position, and that the overall FID on all frames gives same model ordering as the last-frame FID. We report below an extract of those results for comparison:
Weakness 2: We agree with the reviewer that the reported percentage improvements need clarification. As we do not want to overstate the improvements brought by our method, we use in the abstract relative improvements compared to the best FID and best FVD independently, and using our improved sampling for JiT. We updated the paper to clarify those numbers in the introduction, in which we explicitly name the setting (latent encoder) for both latent FID and latent FVD. We also now report relative improvements to JiT both with and without improved sampling.
Weakness 3 / Question 3: to evaluate class-conditional metrics, we followed the widely accepted methodology of evaluating over the training set, as used in diffusion literature (Rombach et al., 2022, Esser et al., 2021, Li and He, 2026, etc.). This practice is usually followed as weakly-conditioned generation (class) learns to generate the training set distribution, while strongly-conditioned generators (frames) can adapt to the target set statistics. To improve the soundness of our results, we added in the revision of the paper the evaluation of FID and FVD on the held-out Kinetics-600 validation set in Table 4. We use the same CFG that was picked for the training set, and do not tune in on the validation set, to ensure strict held-out evaluation. We observe only slight variance for each metrics, keeping the same relative ordering of the methods: this confirms that our evaluations were not overfitting on the training set.
Table below: class-to-video, train vs test metrics
Weakness 4 / Question 2: We apologise to the reviewer if the methodology for the PCA decomposition was unclear. We corrected the caption of Figure 2, and added detailed explanations and throughout analysis of the method used for Figure 2 in appendix B.4. Specifically, we detail how we generate 100 videos per model, extract 12 frames from each, and split them in a total of 76,800 4x16x16 patches, which is enough to compute the 3000 PCA components. We also justify using this patch size, as we aim to study the impact of patchification on frequencies. We additionally add explicit frequency decomposition through DFT and DCT to show the same conclusion.
Question 4: we thank the reviewer for pointing out this oversight, to which we remedy in two ways. First, we added a “Frames conditioning” paragraph in the Implementation details (Appendix A). Secondly, we commit to release the full codebase for training and inference upon paper acceptance, to ensure full reproducibility of our results on all Kinetics-600 tasks.