- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors show that JiT pixel-space image generative models benefit from adding a convolutional pixel-level head to improve image quality. They also show that this pathway can be enabled only during part of the sampling process, by interpolating between the linear projection and convolutional head output depending on the timestep. They show visible improvements both on metrics and quantitative results. While adding a pixel-head to JiT is not novel (PixelDiT, PiD, etc.), the specific design (ConvNeXt blocks) and adding the head as a post-training component are novel.
Critical strengths
The specific design of the pixel head combined with the post-training method are new and could be applied to post-train existing models to improve their generative quality if proven to be effective.
Critical weaknesses
The main weakness is that the experiments very weakly support each design point of the model:
- Architecture: while the paper introduce a specific architecture, this specific design choice is orthogonal to the main component (post-training with a head). No result justifies this specific design and its effectiveness over existing ones.
- Post-training method: the cost to run the pixel-head compared to the backbone vanishes as the model scale, which means the head could as well be applied during the full training and simplify the two-stage training into a single stage.
- Effectiveness compared to JiT: the post-trained JiT models models seem to use a t-clipping of 0, rendering the comparison unfair: the improvement of the head over a model trained for less epochs or sampled with the time-clipping dropped do not help justify the method effectiveness.
- Time-clipping: the use of time-clipping seems to be necessary in JiT to produce good results, and the removal of it during sampling for this method is not well justified here, even less so for the post-training of JiT.
- Effectiveness compared to existing methods: the results are clearly inferior to some methods reported, even when such methods use smaller models, such as PixelDiT-XL. While the use of REPA by such methods is mentioned by the authors, a controlled comparison between the different design points would clarify if the proposed method is effective compared to existing literature.
Questions for authors
Could the authors provide clear experiments supporting:
- The use of the head as post-training
- The specific design of the head
- The effectiveness compared to a post-trained JiT model
Response to reviewer Pivs
Thank you for taking the time to review our paper! We would greatly appreciate your feedback on the below response.
We agree that our work could benefit from an additional ablation to understand the effect of different pixel heads. However, it was not our aim to find the best possible pixel head architecture. Our aim was to show that (1) with x-prediction, the pixel head is not necessary throughout the entire denoising process and (2) to provide a correction for the original JiT sampling.
That being said, we already compare two convolutional implementations of the head in appendix (ConvNeXT-like vs ResNet). We also ran additional ablations with JiT-B/16 with PixelDiT-style head, Swin transformer head on pixel level and a wide transformer head (similar to RAE's latent refinement layers) with identical training and sweep settings. PixelDiT-style head FID is 3.68, wide transformer head FID is 3.89, Swin head FID is 3.60. The best of these results are slightly worse than our best ConvNeXt head equivalent (3.42), however, we can't rule out that these other heads need more hyperparameter tuning.
This might be true for our simple convolutional head at 256x256 but the story changes entirely when using more expensive heads at higher resolution. For example, when using the original PixelDiT with H/32 backbone at 512x512 (~1B parameters), the PixelDiT head occupies about 50% of total forward compute (this number is ~10% for our convolutional head). This just illustrates that with the late head activation, both during training and sampling, we can dramatically save on compute, especially when more expensive heads are used. Nevertheless, the pixel head can also be trained from the very beginning as well, this is only expected to improve results once the correct hyperparameters are found.
As we show in Section 3, the original clipping during sampling is not correct and does not sample from the learned flow.
Please consider the following scenario to intuitively illustrate why: consider a pixel-level model trained entirely on icons/logos/text (smooth noise-free shapes). Ideally, we want the final generation results to be completely noise-free as well, which we can accomplish if we properly learn a model on this data and then sample from it. However, JiT’s timestep clamp during sampling artificially retains a small amount of the initial random noise in the generator’s final output (as we show in and after equation 5). This artificially retained noise will thus still be present in our generated icons/logos/text, even though it was never present in the training images and thus shouldn’t be present in the generator’s distribution.
In fact, we just found concurrent work, PixText (on arxiv since mid-October), which also drops clamping to accomplish noise-free text-generation. If we test the vanilla models before or after matched fine-tuning, with the sampling correction (no clipping), our results indicate substantial improvement.
PixText: https://arxiv.org/pdf/2610.18284
We justify the removal of clipping during sampling for all methods in Sec. 3, arguing with theory and experiments that it is incorrect and only fools the FID metric to give better scores because of the reference dataset used (real photographs with photo grain which is “faked” by the noise retained through clipped sampling). See also previous point. During training, we always keep timestep clipping.
Our main goal was to understand what caused JiT's image quality issues, with the results of the investigation being a correction of the sampler (some recent works still use the incorrect sampling), and the finding that a dedicated pixel head can be activated late in the denoising process.
Regarding REPA: applying it on JiT is not trivial, as PixelREPA reports. Also, it is not clear how to apply it in a post-training phase. We believe it is an orthogonal technique that boosts training through knowledge transfer from a pretrained model.
Tthere are no inherent advantages of training the head only during post-training but activating it throughout the entire training might require additional hyperparameter tuning. It was just a convenience on our part to reuse original pretrained models in order to use our limited compute to run more ablations and explore training schedules etc.
We already have a comparison with a post-trained vanilla JiT model for B/16 and L/16 in Table 1: "Finetuned (100 epochs)", indicating there is improvement compared to unclamped original. Not reported is the clamped performance of the finetuned model, which is worse than the clamped performance of the original model (FID of ~4.0)