acceptodds
Under review as a conference paper at ICLR 2027

Pixel-space video generation with U-JiT

11 views

Abstract

Latent diffusion and flow matching models are ubiquitous in state-of-the-art generative video models. While highly effective, relying on pretrained autoencoders to compress the data into a lower-dimensional latent space also comes with drawbacks. First, training proceeds in two stages: autoencoder pretraining, with compression and reconstruction objectives, followed by latent generative model training, precluding fully end-to-end training. Second, autoencoders also lead to an inherent loss of signal, and the trade-off between reconstruction quality and generative performance in the latent space is still an active area of research. In this paper we explore the viability of generative video models directly in pixel space. We analyze recent approaches in pixel-space generative modeling of images that rely on basic transformer designs, and identify several shortcomings that limit their effectiveness for the high-dimensional pixel-space video generation. To overcome these limitations, we propose a UNet-like transformer architecture along with improvements to the training recipe and sampling strategy. We conduct controlled experiments, training latent and pixel-space models from scratch on Kinetics-600. For the standard frames-to-video task at 128 resolution, our U-JiT model improves generative FVD by 27% over prior state-of-the-art results, and by 14% against the JiT pixel-space baseline. Scaling to higher 256 resolution, we find the improvement over pixel-space and latent-based models grows as we increase the spatiotemporal compression rate up to . Finally, we find that U-JiT improves per-frame FID by 27% over latent models and 61% over pixel-space models on class-to-video generation, with respectively 9% and 56% reduction in FVD.

Primary area:
generative models
Keywords:
video, flow matching, generative

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mjT9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a flat, large-patch transformer is a poor fit for pixel-space video, and proposes U-JIT, a U-shaped transformer whose patch size grows with depth. The key evidence is a PCA spectrum (Figure 2): JiT's signal dimension is about 596, versus 1308 for real clips and 1284 for U-JIT. The authors also borrow three things from latent decoders, namely time-shifting, CFG, and an LPIPS loss, though LPIPS is only applied to U-JIT. All models are trained from scratch on Kinetics-600. U-JIT matches the best retrained latent baseline on FVD and beats it on FID, from f8t4 up to f32t8, on frames-to-video, image-to-video, and class-to-video.

Critical strengths
  1. Figure 2 shows JiT's spectrum collapsing to a noise floor, missing high frequencies. Table S4 confirms that removing the U shape at matched FLOPs drastically worsens both FVD and FID. Progressive patchification effectively solves the high dimensional data bottleneck through learned nonlinear compression.

  2. The latent baselines are rigorously retrained under matching objectives, datasets, and FLOP budgets. U-JIT uses fewer parameters yet ties the best latent FVD and shows strong robustness to extreme spatiotemporal compression rates like f64t8.

  3. Section 3.3 effectively justifies the borrowed sampling tricks. By jittering latents and isolating frequency bands, the authors prove how the decoder maps isotropic noise to perceptually aligned directions. This perfectly explains why time shifting, CFG, and LPIPS are necessary here.

Critical weaknesses
  1. The FID gap over latent models is mostly the LPIPS loss, not the architecture. LPIPS is only on U-JIT; removing it (Table S4, line 2) drops FID from 19.20 to 35.47. FID is also only on the last frame. On FVD, with matched sampling, the lead is tiny: 0.61 vs. 0.62, and 2.55 vs. 2.66.

  2. The reported headline numbers need to be clarified. In the introduction (lines 084 to 086), the authors claim a 29% FVD improvement and a 90% FID improvement over the JIT pixel space baseline. However, this compares against the base sampling JIT (0.86 FVD and 32.43 FID), not the tuned JIT (0.71 FVD and 5.29 FID). The comparison against the tuned JIT is the 14% FVD improvement actually mentioned in the abstract (line 025). Furthermore, the 64% FID drop claimed in the abstract (line 028) is against Cosmos, not Wan2.1, which has the best FVD.

  3. Scope is thin for the conclusion. The experiments only cover Kinetics-600, resolutions of 128 or 256, and up to 65 frames. Class conditional FVD and FID are computed on the training set. CFG is picked by grid search, and the paper does not specify whether the grid uses the same videos as the reported scores.

  4. Figure 2 is missing the details needed to be credible. There is no sample count, no patches per video, and no patch dimension explicitly linked to the math. You cannot estimate a roughly 1300 dimension signal from 1000 samples if each video gives one 3072 dimension patch.

Questions for authors
  1. Please train the JiT and l-JIT baselines with the same LPIPS weight applied to U-JIT (at least for Table 3) and report the FID across all generated frames. If U-JIT fails to outperform Wan2.2 on FVD under these matched conditions, my score will remain a weak reject.

  2. Please clarify the exact sampling methodology for Figure 2. Specifically, state the total number of videos, the number of patches extracted per video, and the exact patch dimensions. Reaching an estimated 1300 dimension subspace from 1000 video samples mathematically requires extracting multiple patches per video, which must be detailed.

  3. Were the CFG scales in Tables 2 through 4 tuned directly on the reported evaluation sets? Please provide class to video generation metrics computed on a strictly held out validation set.

  4. How exactly are the conditioning frames fed into the model? Please specify whether they are concatenated, used as clean token replacements, or provided via a separate embedding. The experiments in Tables 2 and 3 cannot be reproduced without this information.

1 comment

We would like to thank the reviewer for their time and for highlighting that “U-JIT [...] ties the best latent FVD and shows strong robustness to extreme spatiotemporal compression” and that our paper “effectively justifies the borrowed sampling tricks”.

We addressed the reviewer first and main weakness/question in the general answer, with new additional experimental results. Additionally, we would like to address individual points below:

Weakness 1 / Question 1: we addressed the main concern on LPIPS-controlled experiments in the general answer to all reviewers, with new experimental results. Additionally, we would like to explain the choice of reference frame for FID: on frames-to-video experiments, the first few generated frames are strongly correlated with the input conditioning, while the last frame is more representative of the model generative capabilities. We added in Appendix C.1 an analysis reporting the per-frame FID, as well as the overall FID on 12 generated frames. We see that for all models, the per-frame FID degrades with frame position, and that the overall FID on all frames gives same model ordering as the last-frame FID. We report below an extract of those results for comparison:

FID over:First generated frameLast generated frameAll 12 frames
l-JiT-B, Cosmos 4x8x87.928.356.48
l-JiT-B, Wan2.1 4x8x86.8110.047.46
JiT-B3.687.283.78
U-JiT-B1.393.081.58

Weakness 2: We agree with the reviewer that the reported percentage improvements need clarification. As we do not want to overstate the improvements brought by our method, we use in the abstract relative improvements compared to the best FID and best FVD independently, and using our improved sampling for JiT. We updated the paper to clarify those numbers in the introduction, in which we explicitly name the setting (latent encoder) for both latent FID and latent FVD. We also now report relative improvements to JiT both with and without improved sampling.

Weakness 3 / Question 3: to evaluate class-conditional metrics, we followed the widely accepted methodology of evaluating over the training set, as used in diffusion literature (Rombach et al., 2022, Esser et al., 2021, Li and He, 2026, etc.). This practice is usually followed as weakly-conditioned generation (class) learns to generate the training set distribution, while strongly-conditioned generators (frames) can adapt to the target set statistics. To improve the soundness of our results, we added in the revision of the paper the evaluation of FID and FVD on the held-out Kinetics-600 validation set in Table 4. We use the same CFG that was picked for the training set, and do not tune in on the validation set, to ensure strict held-out evaluation. We observe only slight variance for each metrics, keeping the same relative ordering of the methods: this confirms that our evaluations were not overfitting on the training set.

Table below: class-to-video, train vs test metrics

Model (f32t8)Train FVD ↓Train FID ↓Val FVD ↓Val FID ↓
l-JiT-XL (Cosmos Tokenizer)4.78<u>17.44</u>4.68<u>18.45</u>
l-JiT-XL (Wan2.1 Tokenizer)<u>3.31</u>21.52<u>3.36</u>22.49
l-JiT-XL (Wan2.2 Tokenizer)3.5518.083.5218.92
l-JiT-XL (LTX Tokenizer)3.9622.243.8223.29
JiT-XL (base sampling)9.4565.499.2166.36
JiT-XL6.8832.606.7233.36
U-JiT-XL (ours)3.0012.653.0713.72

Weakness 4 / Question 2: We apologise to the reviewer if the methodology for the PCA decomposition was unclear. We corrected the caption of Figure 2, and added detailed explanations and throughout analysis of the method used for Figure 2 in appendix B.4. Specifically, we detail how we generate 100 videos per model, extract 12 frames from each, and split them in a total of 76,800 4x16x164x16x16 patches, which is enough to compute the 3000 PCA components. We also justify using this patch size, as we aim to study the impact of patchification on frequencies. We additionally add explicit frequency decomposition through DFT and DCT to show the same conclusion.

Question 4: we thank the reviewer for pointing out this oversight, to which we remedy in two ways. First, we added a “Frames conditioning” paragraph in the Implementation details (Appendix A). Secondly, we commit to release the full codebase for training and inference upon paper acceptance, to ensure full reproducibility of our results on all Kinetics-600 tasks.

Reviewer_d3zSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes U-JiT, a U-shaped Transformer for pixel-space video generation. By incorporating progressive spatiotemporal patch compression, multi-scale attention, and skip connections, U-JiT aims to alleviate the representation bottleneck of conventional JiT under high compression ratios. The method further integrates time-shifting, classifier-free guidance (CFG), and perceptual loss to improve generation quality. Experiments on Kinetics-600 demonstrate that U-JiT outperforms several pixel-space and latent-space baselines in video generation quality, particularly under high compression ratios.

Critical strengths

1.The research problem is meaningful. The paper addresses the representation bottleneck in high-dimensional pixel-space video generation and proposes a multi-scale architecture, offering a promising alternative to relying on pretrained video tokenizers. 2.The experimental performance is competitive. On the 128×128 video prediction task, U-JiT reduces FVD by approximately 27% compared to the previous best result. Under the high-compression setting at 256×256 resolution, it also outperforms the compared JiT and latent-space models. 3.The ablation and scaling experiments are relatively comprehensive. The experiments cover different model sizes, spatiotemporal compression ratios, and generation tasks. Ablation studies further demonstrate the contributions of progressive patch compression, skip connections, and sampling strategies.

Critical weaknesses

1.The comparisons are confounded by inconsistent training objectives. Table S1 shows that only U-JiT uses LPIPS loss, whereas JiT and the latent-space models do not. Table S4 further shows that removing LPIPS increases FID from 19.20 to 35.47. Therefore, the current comparisons do not adequately disentangle the contribution of the proposed architectural improvements from that of the additional perceptual supervision. 2.The explanation of the core representation bottleneck lacks direct validation. The paper uses PCA eigenvalue spectra to argue that JiT loses high-frequency information. However, PCA directions do not directly correspond to spatiotemporal frequencies. Explicit frequency-domain error analysis is needed to establish whether the performance degradation is indeed caused by the loss of high-frequency information. 3.The long-video evaluation is insufficient to demonstrate long-term consistency. Section 4.4 evaluates generated 65-frame videos by splitting them into four 16-frame segments and computing FVD. This evaluation cannot capture motion continuity across segments or long-term temporal consistency, and therefore does not provide sufficient evidence to support the claimed long-video generation capability.

Questions for authors

See Critical Weaknesses.

1 comment

We would like to thank the reviewer for their time and for highlighting our “competitive” experimental performance with “relatively comprehensive” ablation and scaling experiments on this “meaningful” research problem.

Weakness 1: we addressed the main concern on LPIPS-controlled experiments in the general answer to all reviewers, with new experimental results, and believe they now properly disentangle the contribution of the architecture (which enabled modeling high-dimensional space) and of the additional perceptual supervision (which guides toward the correct signal in this high dimensional space).

Weakness 2: we agree with the reviewer that, while in practice PCA decomposition aligns with frequency decomposition (Hancock et al., 1992; Simoncelli and Olshausen, 2001), we would greatly strengthen the analysis by including explicit frequency-domain analysis. We expand our analysis in appendix B.4, in which we perform DFT and DCT to compute variance along each frequency. We observe that these results correlate with our PCA decomposition, where low and mid frequencies are correlated across all models with the real data, and JiT high frequencies diverge and keep explicitly variance above a certain threshold. We note that the DFT and DCT are noisy on small patches, producing spiky results: as such, we keep in Figure 2 the PCA decomposition, and explicitly reference the appendix to justify the soundness of our analysis.

Weakness 3: First, we would like to highlight that our contributions focus on the pixel-to-token patchification, in which we learn progressive mapping from high to low / low to high dimensional representations in the early and late layers of the model. We do not affect the middle layers' attention, responsible for long-video coherence. As such, we postulate similar long-term consistency to latent and JiT models. We conduct additional analysis by adding the VBench consistency metric, evaluated over all 65 generated frames generated by class-to-video models, and report them in Table 4. We observe that all models generate videos with close-to-ground truth consistency (1% variance around ground-truth for latent models, 1.5% for our U-JiT, and up to 5.2% for JiT). Additional quantitative examples in Appendix D show that U-JiT is competitive with latent models at preserving subject consistency.

Table below: class-to-video, consistency evaluated over 65 frames

Model (f32t8)FVD ↓ (train)Consistency ↑
l-JiT-XL (Wan2.1 Tokenizer)3.3184.3
l-JiT-XL (Wan2.2 Tokenizer)3.5583.8
JiT-XL6.8881.9
U-JiT-XL (ours)3.0082.9
Reviewer_YnwoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether competitive video generation can be performed directly in pixel space. It introduces U-JiT, a U-shaped transformer that progressively changes the spatial and temporal patch size across network depth. Fine-resolution blocks, skip connections, attention tiling, and attention patchification are used to preserve high-frequency information.

Critical strengths
  1. The paper addresses an important question of translating the success of image generation to video generation.
  2. The proposed architecture is conceptually simple and technically well motivated. Progressive patchification gives the model fine-resolution access near the input and output while retaining a computationally efficient transformer at the central level.
  3. The empirical results are strong and consistent across model sizes, generation settings, and compression ratios.
  4. The ablation study is extensive.
Critical weaknesses
  1. The main architectural comparison is not fully controlled because the LPIPS loss is applied to U-JiT but not to JiT or the latent JiT baselines. This loss has a very large effect.
  2. The quantitative evaluation relies only on FVD and FID. The authors do not share any visual results in the supplementary material; it is hard to judge video generation improvement without looking at videos.
  3. All main experiments use Kinetics-600 at 128 or 256 resolution and 8 FPS, not even Taichi, UCF101, etc among other small-scale datasets. The results therefore do not yet establish whether the method scales to higher-resolution, open-domain, or text-conditioned video generation.
  4. Not having large-scale text-to-video is a limitation; it is unclear if these improvements work at scale. These days, it's important to validate the claims on text-to-video generation.
Questions for authors
  1. Could the authors report matched-loss comparisons where JiT and, where feasible, latent JiT receive the same LPIPS objective as U-JiT? Alternatively, could the main tables include U-JiT without LPIPS?
  2. Can the authors run text-to-video generation with their approach and compare with baselines? (Similar to what they did for Kinetics)? Currently, it's hard to accept the paper without seeing visual results and without larger scale text-to-video validation.
1 comment

We would like to thank the reviewer for their time and for highlighting our “strong and consistent” empirical results with “extensive” ablation study on the “important question of translating the success of image generation to video generation”.

Weakness 1: we addressed the main concern on LPIPS-controlled experiments in the general answer to all reviewers, with new experimental results, which clarify how both the architecture and perceptual loss complement each other, by respectively enabling modeling of high-dimensional space, and guiding toward the correct signal in this space.

Weakness 2: We agree with the reviewers that the single-frame examples of Figure 1 are not enough to compare motion and visual quality across all models. We added in Appendix D visual results to showcase the improvements brought by our model on visual quality. These new qualitative results match the metrics of Table 2, as our method obtains results visually similar to the best latent model (Wan2.1) on this task, and clear improvements over JiT and latent Cosmos.

Weakness 3 (“small-scale datasets”): First, we would like to point out that “small-scale datasets” such as Taichi and UCF101 do not share the same scale as Kinetics-600, with 1 to two orders of magnitude in training set size. While Kinetics-600 contains 427k videos, Taichi and UCF101 are limited to 12k and 2.8k videos, which would be insufficient to train pixel-space models from scratch. Nevertheless, to evaluate our method on a lower-scale dataset, we add experiments on Tai-Chi-HD in Appendix B.5, and report the table below. We observe that latent models trained from scratch outperform pixel-space at small scale, as they benefit from the large-scale pre-training of their auto-encoders. On the other hand, by initializing both latent and pixel-space models from pre-trained Kinetics-600 models to reduce discrepancy in pre-training scale, we obtain similar results for U-JiT and the best latent space model. As such, we can conclude that U-JiT matches and outperforms latent models with scaling.

Model (f8t4)From scratch FVD ↓From scratch FID ↓Fine-tuned FVD ↓Fine-tuned FID ↓
l-JiT-B (Cosmos Tokenizer)2.2524.132.0421.55
l-JiT-B (Wan2.1 Tokenizer)1.5020.031.2816.42
JiT-B (base sampling)3.7593.451.3846.13
U-JiT-B (ours)1.7515.631.2812.73

Weakness 3 and 4 (“higher-scale” and “text-conditioning”): we started training text-to-video models at higher resolution (768×512768\times512). Due to the limited rebuttal time, we were not able to complete the training in time, but we will update the paper with proper results for the final version. Early results show the same clear improvements compared to JiT for U-JiT models, with performance close to latent models. Additionally, we would like to point out that the scaling of pixel-space pre-training of text-to-images models is in itself an active research direction, which requires methods beyond the architecture and training objective (see “An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models”, Jiang et al.). As such, we believe that U-JiT provides the architectural and trainings components enabling further research in this direction. Additionally, for the scaling, Kinetics-600 is not usable for higher resolution than 256, which is why we compared deeper compression settings.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global answer to all reviewers (1/2)

We want to thank the reviewers for their time, and for pointing out that the “research problem is meaningful”, our “proposed architecture is conceptually simple and technically well motivated”, “empirical results are strong and consistent” with “latent baselines [...] rigorously retrained under matching objectives”.

We would like to first address the main point raised by all three reviewers: the controlled comparison under LPIPS loss. We first justify the choice of our comparison in our paper, and add experiments to justify our comparisons and conclusions.

Peceptual loss on latent models: We first would like to address the misunderstanding that “only U-JiT uses LPIPS loss, whereas [...] latent-space models do not”. Latent-space models are effectively two-stage generative models, with auto-encoder and latent models trained successively. All decoders used in our work are trained with perceptual losses, of which LPIPS is the most common one. Our models train end-to-end, integrating this perceptual loss directly on the generator as we drop the decoder: as such, we use similar objectives over the whole model training. Additionally, we would like to clarify why the perceptual loss is necessary for controlled comparison. On the high-dimensional space of videos, L1 / L2 are not well-aligned to measure perceptual distance between frames, and uniform variations along dimensions result in blurry frames. Variational auto-encoders solve this by mapping this high-dimensional space into a small latent space, in which uniform variance is mapped to perceptually-aligned variance: we show this in paragraph 3 of section 3.3 in the paper. Adding LPIPS on pixel-space generators brings the same effect of aligning the generations variance with perceptually-aligned directions (see third part of section 3.4).

JiT with LPIPS loss: As for JiT, we first report the base JiT with base samplings in all tables, as the baseline model. Our work analyses the limitations of JiT, and bridges the gap between latent and pixel-space models through architecture (U-JiT), sampling, and losses. Reviewers are right to point out that we didn’t fully control for each of those components in each table. As such: we are adding JiT + LPIPS (with our sampling) in the 3 main result tables (Tables 2, 3 and 4), using the same training and sampling as for U-JiT. We observe that LPIPS consistently improves FID, and improves FVD in most models. Despite that, the gap between JiT+LPIPS and U-JiT remains significant, highlighting the necessity of our architecture. Additionally, we observe that the relative improvements brought by U-JiT architectural changes increase with higher spatiotemporal compression (Table 3 and 4), highlighting the gains brought by the progressive encoding against the linear patchification of JiT.

Table below: extracts from Table 2, 3 and 4 with the impact of LPIPS on JiT, compared to pure JiT and U-JiT, using the same training and sampling.

Compression / TableModelFVDFID
f8t4, Table 2JiT-B1.227.28
f8t4, Table 2JiT-B + LPIPS1.183.43
f8t4, Table 2U-JiT-B1.063.08
f8t4, Table 2JiT-XL0.715.29
f8t4, Table 2JiT-XL + LPIPS0.733.29
f8t4, Table 2U-JiT-XL0.613.15
f32t8, Table 3JiT-XL5.8630.27
f32t8, Table 3JiT-XL + LPIPS5.0813.18
f32t8, Table 3U-JiT-XL2.557.20
f32t8, Table 4JiT-XL6.8832.60
f32t8, Table 4JiT-XL + LPIPS4.3318.12
f32t8, Table 4U-JiT-XL3.0012.65
AuthorsOpenReview ↗

Global answer to all reviewers (2/2)

To properly justify our previous statements, we added new controlled comparisons in the appendix B.3 of all models trained with and without LPIPS. We keep a setting similar to our main result table (Table 2) while increasing frame resolution to 256x256 to ensure meaningful results on perceptual quality, and increasing spatial compression to 16x16 to preserve the same computational cost. We compare one latent model (Wan2.2) with JiT and U-JiT, training each of those with and without LPIPS loss. Below is the table included in B.3 in the paper.

Model (f16t4)FVD ↓FID ↓
l-JiT-B (Wan2.2 Tokenizer)1.274.23
+ LPIPS1.31 (+0.04)5.23 (+1.00)
JiT-B1.419.43
+ LPIPS1.37 (−0.04)4.16 (−5.27)
U-JiT-B1.418.13
+ LPIPS1.28 (−0.13)3.57 (−4.56)

We believe those results disentangle architecture contribution from the loss, in the following ways:

  • LPIPS doesn’t improve the latent-space model and even hurt its performance, which is coherent with our previous statements on the inclusion of perceptual losses in decoder training. Perceptual loss guides models toward perceptually-aligned signal in the high-dimensional space, and as such, improves all pixel-space models
  • LPIPS alone is not enough: JiT inherent bottleneck (section 3.1) limits its modeling capabilities, deteriorating with compression strength (Table 3, Figure 8)
  • U-JiT architecture itself enables modeling higher-dimensional space, but doesn’t help guiding toward the correct low-dimensional manifold embedded in pixel-space.
  • U-JiT and perceptual loss completement each other, reaching similar results to latent space for B-sized models, and improving over them at larger scale (XL models in Tables 3 and 4)

We provide additional per-reviewer answers addressing each point raised with additional experiments and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.