acceptodds
Withdrawn from ICLR 2027

Just Image Transformers with Late-Stage Pixel Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent work on latent-free diffusion transformers has demonstrated the feasibility of training image generators without a latent VAE. This avoids compression artifacts from the VAE and simplifies the training pipeline. In particular, JiT showed that even a standard transformer can be trained to generate images when the model is trained to predict the final image (-prediction). Even though JiT unlocks training efficiency comparable to latent models, it still suffers from reduced image quality likely arising from modeling the image as a collection of large patches. In this work, we revisit JiT and show that adding simple convolutional pixel-pathway layers in a post-training phase significantly improves generation quality. Moreover, we show that the added pixel-pathway does not need to be active in the early denoising stages, and can be activated as late as in the last 20% of the denoising process. Our experiments on ImageNet@256 show improvements compared to the original JiT results, achieving an FID of , without using additional losses.

Primary area:
generative models
Keywords:
pixel-level generative models, latent-free generative modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PivsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors show that JiT pixel-space image generative models benefit from adding a convolutional pixel-level head to improve image quality. They also show that this pathway can be enabled only during part of the sampling process, by interpolating between the linear projection and convolutional head output depending on the timestep. They show visible improvements both on metrics and quantitative results. While adding a pixel-head to JiT is not novel (PixelDiT, PiD, etc.), the specific design (ConvNeXt blocks) and adding the head as a post-training component are novel.

Critical strengths

The specific design of the pixel head combined with the post-training method are new and could be applied to post-train existing models to improve their generative quality if proven to be effective.

Critical weaknesses

The main weakness is that the experiments very weakly support each design point of the model:

  • Architecture: while the paper introduce a specific architecture, this specific design choice is orthogonal to the main component (post-training with a head). No result justifies this specific design and its effectiveness over existing ones.
  • Post-training method: the cost to run the pixel-head compared to the backbone vanishes as the model scale, which means the head could as well be applied during the full training and simplify the two-stage training into a single stage.
  • Effectiveness compared to JiT: the post-trained JiT models models seem to use a t-clipping of 0, rendering the comparison unfair: the improvement of the head over a model trained for less epochs or sampled with the time-clipping dropped do not help justify the method effectiveness.
  • Time-clipping: the use of time-clipping seems to be necessary in JiT to produce good results, and the removal of it during sampling for this method is not well justified here, even less so for the post-training of JiT.
  • Effectiveness compared to existing methods: the results are clearly inferior to some methods reported, even when such methods use smaller models, such as PixelDiT-XL. While the use of REPA by such methods is mentioned by the authors, a controlled comparison between the different design points would clarify if the proposed method is effective compared to existing literature.
Questions for authors

Could the authors provide clear experiments supporting:

  • The use of the head as post-training
  • The specific design of the head
  • The effectiveness compared to a post-trained JiT model
1 comment
AuthorsOpenReview ↗

Response to reviewer Pivs

Thank you for taking the time to review our paper! We would greatly appreciate your feedback on the below response.

Architecture

We agree that our work could benefit from an additional ablation to understand the effect of different pixel heads. However, it was not our aim to find the best possible pixel head architecture. Our aim was to show that (1) with x-prediction, the pixel head is not necessary throughout the entire denoising process and (2) to provide a correction for the original JiT sampling.

That being said, we already compare two convolutional implementations of the head in appendix (ConvNeXT-like vs ResNet). We also ran additional ablations with JiT-B/16 with PixelDiT-style head, Swin transformer head on pixel level and a wide transformer head (similar to RAE's latent refinement layers) with identical training and sweep settings. PixelDiT-style head FID is 3.68, wide transformer head FID is 3.89, Swin head FID is 3.60. The best of these results are slightly worse than our best ConvNeXt head equivalent (3.42), however, we can't rule out that these other heads need more hyperparameter tuning.

Post-training method

This might be true for our simple convolutional head at 256x256 but the story changes entirely when using more expensive heads at higher resolution. For example, when using the original PixelDiT with H/32 backbone at 512x512 (~1B parameters), the PixelDiT head occupies about 50% of total forward compute (this number is ~10% for our convolutional head). This just illustrates that with the late head activation, both during training and sampling, we can dramatically save on compute, especially when more expensive heads are used. Nevertheless, the pixel head can also be trained from the very beginning as well, this is only expected to improve results once the correct hyperparameters are found.

Effectiveness compared to JiT

As we show in Section 3, the original clipping during sampling is not correct and does not sample from the learned flow.

Please consider the following scenario to intuitively illustrate why: consider a pixel-level model trained entirely on icons/logos/text (smooth noise-free shapes). Ideally, we want the final generation results to be completely noise-free as well, which we can accomplish if we properly learn a model on this data and then sample from it. However, JiT’s timestep clamp during sampling artificially retains a small amount of the initial random noise in the generator’s final output (as we show in and after equation 5). This artificially retained noise will thus still be present in our generated icons/logos/text, even though it was never present in the training images and thus shouldn’t be present in the generator’s distribution.

In fact, we just found concurrent work, PixText (on arxiv since mid-October), which also drops clamping to accomplish noise-free text-generation. If we test the vanilla models before or after matched fine-tuning, with the sampling correction (no clipping), our results indicate substantial improvement.

PixText: https://arxiv.org/pdf/2610.18284

Time-clipping

We justify the removal of clipping during sampling for all methods in Sec. 3, arguing with theory and experiments that it is incorrect and only fools the FID metric to give better scores because of the reference dataset used (real photographs with photo grain which is “faked” by the noise retained through clipped sampling). See also previous point. During training, we always keep timestep clipping.

Effectiveness compared to existing methods:

Our main goal was to understand what caused JiT's image quality issues, with the results of the investigation being a correction of the sampler (some recent works still use the incorrect sampling), and the finding that a dedicated pixel head can be activated late in the denoising process.

Regarding REPA: applying it on JiT is not trivial, as PixelREPA reports. Also, it is not clear how to apply it in a post-training phase. We believe it is an orthogonal technique that boosts training through knowledge transfer from a pretrained model.

The use of the head as post-training

Tthere are no inherent advantages of training the head only during post-training but activating it throughout the entire training might require additional hyperparameter tuning. It was just a convenience on our part to reuse original pretrained models in order to use our limited compute to run more ablations and explore training schedules etc.

The effectiveness compared to a post-trained JiT model

We already have a comparison with a post-trained vanilla JiT model for B/16 and L/16 in Table 1: "Finetuned (100 epochs)", indicating there is improvement compared to unclamped original. Not reported is the clamped performance of the finetuned model, which is worse than the clamped performance of the original model (FID of ~4.0)

Reviewer_d3CtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper's primary observation is that the late denoising predictions in JiT-style x-prediction image diffusion lacks sharp detail. The authors propose a minimal, investigatory direction to address this, proposing that a lightweight convolutional output head can help with blurriness. This head can be lightly finetuned into the model (100 epochs) and only needs to operate at late timesteps. Using this approach, the FID of JiT is improved and sharp details are increased.

Critical strengths

The authors identify a testable, specific problem in the extremely widely used JiT architecture, specifically a lack of sharp detail at late denoising steps. Although not a fundamental problem in deep learning, this is clearly an interesting enough problem to deserve specific analysis.

The proposed solution from the authors is simple, well motivated, and most importantly does not require external models or perceptual losses, which a large amount of related work requires. The solution is effective and noticably improves FID.

In general, this is a well-scoped useful paper for the community.

Critical weaknesses

The primary concern I see in this work is a lack of ablation of the bottleneck layer in JiT. The default bottleneck applied to JiT-B and JiT-L is a 128 dimension pca compression of the space, which necessarily does not provide fine grained details. Therefore, it is unclear whether the improvements in this paper are specifically from the CNN architecture proposed, rather than a more fundamental investigation into the required bottleneck behavior of JiT architectures at a small scale, with the CNN implicitly removing the bottleneck. As a result, I am borderline on this paper, however acknowledge that the work is self-contained, easy to read, and generally high-quality. Specifically, the CNN layer does not contain a bottleneck in the input. Is it the case that the JiT architecture could use a varying granularity bottleneck throughout inference"

The other location of worthwhile investigation is additional architectures. RAE [1] addressed the dimensionality issue similarly to this work, with a specific output head dedicated for noise prediciton, that also conditions on x_t. Therefore, the use of a final stage output head to output fine-grained detail is not novel, although doubtlessly the proposed CNN is a more FLOP-efficient strategy.

Finally, in addition to the RAE architecture, Asymmetric Flow Models [2] are a cited but uncompared baseline with substantial overlap to the focus of this work. Asymflow outperforms this approach with a 1.76 FID vs 1.78 in this work (Asymflow Table 1, without REPA), without adding additional training epochs and only a linear layer of additional parameters. Asymflow additionally investigates the clamp used during inference, significantly overlap this work. Asymflow instead makes a bottleneck rank argument for JiT. Although the proposed approach may be complementary to Asymflow, the work deserves more discussion than a baseline comparison in the appendix.

[1] RAE https://arxiv.org/abs/2510.11690 [2] Asymmetric Flow Models https://arxiv.org/pdf/2605.12964

Questions for authors

Are the gains in the CNN from locality (no patch boundaries) or from removing the bottleneck?

How does a CNN compare versus a high-rank output head as in RAE?

How does this approach relate to relevant prior work, such as Asymflow?

2 comments
AuthorsOpenReview ↗

Response to reviewer d3Ct

Thank you for taking the time to review our paper! We would greatly appreciate your feedback on the below response.

The primary concern I see in this work is a lack of ablation of the bottleneck layer in JiT.

We were also wondering the same thing. The reason we didn't look more into this is that the original JiT[1] paper already has an ablation on this (see Fig. 4 in the JiT paper), showing that no bottleneck actually gives worse FIDs. Nevertheless, we will run experiments on this without clamping (training from scratch takes time) and include them in our paper.

[1] https://arxiv.org/pdf/2511.13720

RAE, final stage output head to output fine-grained detail is not novel

We don’t claim the skip connection as novelty since skipping the backbone to feed the input directly to a head has also been investigated in PixelDiT, DiP and other recent papers on pDiT’s. Note that RAE’s DiT with its wide-transformer head does not operate directly in pixel-space but in the representation space of a frozen pretrained encoder. Please not that it was not our goal to find the best possible head, but to (1) point out the clamping issue during sampling, and (2) show that the pixel head is only needed in later denoising steps, justifying using JiT's original model and x-prediction formulation in early denoising. Nevertheless, we ran an ablation on JiT-B/16 with a RAE-like wide transformer head and it gave an FID of 3.89. This is lower than ConvNeXT's FID of 3.42, but we can't rule out that the results can be improved by hyperparameter tuning.

AsymFlow

Asymflow is orthogonal to our paper since it introduces a modified optimization objective (and a different initialization for the bottleneck although this was never ablated).

Asymflow does not provide an investigation of the reason timestep clamping during sampling helps, which we provide in Section 3. They only compare clamped and unclamped results. In our work, we argue why clamping during denoising is incorrect and show that similar improvements can be obtained simply by adding a random noise after unclamped generation.

Moreover, looking at Tab. 1 for Asymflow actually reveals that our approach beats theirs when clamping is disabled. As we have shown in Section 3, clamping the timestep during denoising is incorrect since it takes the trajectory outside the learned flow by retaining some initial random noise in the final image (FID likely registers this as higher image quality due to its similarity to photo grain in the reference photo data). If you compare the correct \sigma_min = 0 settings to our reported settings, we beat their 2.28 FID with our 1.78. We also evaluated AsymFlow's H/16 model trained with REPA, without clamping, and it gave an FID of 1.83. This indicates that even with transfer learning from DINO, AsymFlow wasn't able to beat our numbers.

Note that AsymFlow trained from scratch for 600 epochs with their modified objective and initialization whereas we start from the pre-trained JiT model provided by the authors.

Less relevant: We also found another issue with AsymFlow’s numbers: they report 3.27 FID for unclipped JiT-H/16. In our work, we performed a hyperparameter search for the unclamped pretrained JiT-H/16 and it gave 2.34 FID for the best setting (which was different from the best setting for clamped H/16). This is reported in Tab 2. A possible explanation for this discrepancy is if Asymflow performed the hyperparameter search on the clamped generation and then applied those setting on clamped generation.

Extra: an intuitive explanation why timestep clamping during denoising is incorrect: Consider a pixel-level model trained entirely on icons/logos/text (smooth noise-free shapes). Ideally, we want the final generation results to be completely noise-free as well, which we can accomplish if we properly learn a model on this data and then sample from it. However, JiT’s timestep clamp during sampling artificially retains a small amount of the initial random noise in the generator’s final output (as we show in and after equation 5). This artificially retained noise will thus still be present in our generated icons/logos/text, even though it was never present in the training images and thus shouldn’t be present in the generator’s distribution. In fact, we just found concurrent work, PixText (on arxiv since mid-October), which also drops clamping to accomplish noise-free text-generation. If we test the vanilla models before or after matched fine-tuning, with the sampling correction (no clipping), our results indicate substantial improvement.

JiT Bottleneck.

The point I was making was not the inclusion of the JiT Bottleneck at the beginning of the model, but rather the architecture's late introduction of a bottleneck free output head for refinement. Although the authors are correct that JiT views worse performance from the bottleneck, the fairer comparison here would be removing the bottleneck only at late layers, as is currently done in the Authors' proposed method. A similar late-bottleneck introduction is performed by RAE. If you add the bottleneck back to the final CNN layer, does performance again start to fail?

Timestep Clamping

The reason I bring up Asymflow is because of the known behavior of JiT losses: A v loss does not have clamping issues, however struggles to model the high-dimensional noise manifold. An x loss helps model high-dimensional data, however has clamping. Therefore, it's extremely natural to look at the interpolation between a v-loss and an x-loss, investigated by Asymflow, as the method to avoid timestep clamping related issues.

My current understanding is that the contribution of this paper is demonstration of the capability to remove the timestep clamping (or at least push the required clamping down to the last integration step) with a tuned architecture. I am convinced that the authors' proposed approach is state-of-the-art for this setting. However, because this gap does not provide a gain in performance, the analysis does not answer reasonably natural followup questions, and the proposed gain seems likely to go away with scale once the JiT model has enough capacity to handle the full v-loss, I maintain my rating. However, I am borderline and would not mind if this paper were accepted.

Reviewer_iYj7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Paper identifies 2 key issues of existing Just Image Transformer, a type of model learns to directly predict final image (x-prediciton), directly in pixel space, which are patching artifact and sharpness. This allows authors to revisit the architecture design and propose to a post training scheme by introducing a few convolution layers stacking on top of transformer backbone for refining output pixels and notably such layers is only kicked in at later denoising stage. Experiments on ImageNet shows that the method achieves better results compared to the vanilla JiT.

Critical strengths
  • Observation of patching artifact and sharpness is justifiable
Critical weaknesses
  • Patch artifact is very common in diffusion transformer, not just only for pixel-based methods but also latent one since it is the standard way to patchify the input image into tokens for processing by transformer. Some methods introduce multi-scale feature architecture that use only transformer at lowest resolution and use convolution blocks for higher resolution [1] or window attention [2] to mitigate the discontinutity artifact of patchifying. Though these methods are for latent diffusion, It is still very relevant and is important to know the position of the proposed method compared to existing ones. At least, comparison is needed.
  • The motivation of two types of pixel head is not clear. What is the role of each and Why we need them. Why just not simply use a single head?
  • Table 1: the improvement in metrics is very marginal and hard to justify the advantage of the method. I would suggest back this up with more concrete evidence for example visual comparison or some settings like high-resolution generation, where the method clearly wins over the vanilla model.

[1] Liu et al. "Alleviating distortion in image generation via multi-resolution diffusion models and time-dependent layer normalization.", in NeurIPS 2024. [2] Wu et al. "Swin dit: Diffusion transformer using pseudo shifted windows." arXiv e-prints (2025): arXiv-2505. [3] Yuyao Zhang, and Yu-Wing Tai. "UltraGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention." arXiv preprint arXiv:2510.16325 (2025).

Questions for authors
  • What if the pixel head is runned every denoising step?
  • Why 2x2 patch grid is used? Any rationale.
1 comment
AuthorsOpenReview ↗

Respone to reviewer iYj722

Thank you for taking the time to review our paper! We would appreciate your feedback on the below response.

It is still very relevant and is important to know the position of the proposed method compared to existing ones. At least, comparison is needed.

With the used convolutional head, our goal was to show that even a simple cheap head improves the result. It was not our goal to find the best possible head, but to (1) point out the clamping issue during sampling, and (2) show that the pixel head is only needed in later denoising steps, justifying using JiT's original model and x-prediction formulation in early denoising.

That being said, we already compare two convolutional implementations of the head in appendix (ConvNeXT-like vs ResNet). We also ran additional ablations with JiT-B/16 with PixelDiT-style head, Swin transformer head on pixel level and a wide transformer head (similar to RAE's latent refinement layers) with identical training and sweep settings. PixelDiT-style head FID is 3.68, wide transformer head FID is 3.89, Swin head FID is 3.60. The best of these results are slightly worse than our best ConvNeXt head equivalent (3.42), however, we can't rule out that these other heads need more hyperparameter tuning.

The motivation of two types of pixel head is not clear. What is the role of each and Why we need them. Why just not simply use a single head?

In the dual-head version, there is (1) a very cheap linear head (equivalent to the one in the original JiT) and (2) a more expensive multi-layer convolutional head. The dual-head version means that we switch from a simple and cheap linear head to the more expensive convolutional head in the middle of denoising. The advantage of this approach is that we can use more expensive heads for final pixel refinement while saving on computation in earlier denoising steps, where the expensive head is not necessary and the cheap linear head is used instead.

the improvement in metrics is very marginal and hard to justify the advantage of the method.

We would disagree that the improvement is marginal. Firstly, the original JiT numbers used incorrect timestep clipping during denoising, which introduced artificial noise to the final images, which led the reported FID metric to be too optimistic (see Section 3 in the paper). If you compare with corrected sampling, we improve FID from 2.72 to 2.20 (L model) and 2.34 -> 1.78 (H model). However, there is still improvement even when comparing the original (incorrect) sampling on the original JiT model with our corrected sampling on the convolutional head model: 2.39 -> 2.20 (L model) and 1.86 -> 1.78 (H model). We additionally ran 512x512 experiments with similar findings: 2.53->2.20 for L-model, with incorrect sampling for the vanilla model. H/32 experiments haven't finished yet.

Extra: an intuitive explanation why timestep clamping during denoising is incorrect: Consider a pixel-level model trained entirely on icons/logos/text (smooth noise-free shapes). Ideally, we want the final generation results to be completely noise-free as well, which we can accomplish if we properly learn a model on this data and then sample from it. However, JiT’s timestep clamp during sampling artificially retains a small amount of the initial random noise in the generator’s final output (as we show in and after equation 5). This artificially retained noise will thus still be present in our generated icons/logos/text, even though it was never present in the training images and thus shouldn’t be present in the generator’s distribution.

What if the pixel head is runned every denoising step?

That’s what we did with the Conv-only experiments. The first point is that the convolutional head improves the generation of details in the image, but uniquely to our work, we also show that actually, it’s only necessary in the later stages and quantify this thorough experiments.

Why 2x2 patch grid is used? Any rationale.

For the pixel-head, 2x2 patches are used in the pixel head to reduce the required computations and give the pixel head’s outputs slightly larger receptive windows given the same number of layers. In principle, 1x1 should also work, it was just much more expensive in our experiments.

Comments on the paper

2 comments
~Shanchuan_Lin1OpenReview ↗

Public question

Could the authors also report the guidance-free FID? I am curious whether the architectural change yields an even larger improvement without the guidance. This is also useful for future work.

Area_Chair_YHnDOpenReview ↗

Reviewers, please help to join the discussion.

Dear reviewers,

The authors have uploaded the rebuttal. Please help to join the discussion and the further process.

Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.