- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces PiT, a pixel diffusion transformer that replaces symmetric U-Net-style skip connections with asymmetric skips and introduces adaGLU for adaptive feature fusion. The method achieves competitive results on ImageNet-256/512 and demonstrates good scalability and text-to-image generation performance.
Critical strengths
- Strong experimental results. PiT-G/16 achieves competitive FID scores of 1.62 and 1.70 on ImageNet-256 and ImageNet-512, respectively, with good scalability across model sizes.
- Simple and effective design. The proposed asymmetric skips and adaGLU improve generation quality without requiring additional pretraining or complex training procedures.
Critical weaknesses
- Limited novelty. The main architectural components are adaptations of existing skip connections and SwiGLU, making the methodological novelty somewhat limited.
- Insufficient experimental analysis. The paper lacks straight evidence of noise-adaptive gating behavior and parameter-matched comparisons to isolate the benefits of adaGLU from increased model capacity.
Questions for authors
- Could the authors analyze how adaGLU's gating behavior changes across different noise levels?
- How does adaGLU compare with standard MLP-based fusion under matched parameter counts or FLOPs?
- Could the authors clarify the performance comparison with JiT under equal training budgets, particularly on ImageNet-512?
2 comments
Response to Authors' Rebuttal
Thanks for your detailed responses. I have carefully checked the rebuttal, and am particularly impressed by the significant improvements brought by adaGLU. All my concerns are addressed completely, and I will maintain my positive assessment.
Thank you for your valuable feedback and recognition of our work. We provide our responses below and hope they could properly address your concerns.
W.1 Limited Novelty
We agree that long skip connections and SwiGLU are established primitives. Our contribution lies not in claiming either primitive as new, but in offering a novel perspective on their roles and organization in pixel diffusion.
First, we empirically separate the information-routing and optimization roles of long skips. Detaching skip gradients does not degrade performance (52.1 → 50.1 FID), and replacing symmetric encoder–decoder connections with asymmetric input-to-decoder pathways slightly improves FID to 47.8.
Second, although adaGLU is implemented using a standard SwiGLU MLP, its key distinction is that it introduces multiplicative interaction between two distinct representations (the backbone and skip features) rather than applying SwiGLU-style self-modulation to a single input.
Finally, these findings yield a simple architecture that scales to 1.6B parameters without AdaLN, in-context embeddings, pretraining, auxiliary losses, or scale-specific hyperparameters.
We hope these clarifications help you reassess the contributions of our work.
W.2
(Also Q.1) Direct evidence of noise-adaptive gating behavior
We visualize the relative contribution of adaGLU across noise levels and decoder layers in Appendix Figure 6 and discuss the results in Section D. For decoder layer ℓ, let zℓ denote the backbone feature before fusion and Δzℓ=adaGLU(hℓ,zℓ) denote the adaGLU update. We measure the relative update magnitude $r_{\ell}(\sigma)=\frac{\lVert \Delta z_{\ell}\rVert_2 }{ \lVert z_{\ell} \rVert_2} $, averaged over 1000 samples at each noise level σ. The results show that rℓ(σ) decreases as the noise level increases across the examined decoder layers.
Since the features are fused by addition: zℓout=zℓ+Δzℓ, this ratio measures the relative change that the adaGLU update induces in the backbone representation. Its decrease at higher noise levels therefore shows that the relative contribution of the adaGLU update diminishes. This observation is consistent with the network relying less on skip-path information when the input is heavily corrupted.
Conversely, the larger relative updates at low noise levels support our hypothesis that adaGLU helps propagate fine-grained input information to deeper layers to facilitate pixel prediction (Section 4.2, lines 249–254).
(Also Q.2) Parameter-matched comparison with the baseline without adaGLU
To isolate the benefit of adaGLU from increased model capacity, we compare it with the standard MLP fusion baseline without adaGLU under the same training configuration. We increase the baseline’s hidden dimension to 1344 so that it has slightly more parameters than PiT-XL (709M v.s. 640M).
Despite having fewer parameters, PiT-XL substantially outperforms the baseline, achieving 30.8 versus 62.0 FID-10K. This improvement cannot be explained by an increase in model capacity alone and instead supports the benefit of backbone-conditioned adaptive fusion.
Q.3 Performance comparison with JiT at equal training epochs on ImageNet-512
We provide an epoch-matched comparison with JiT on ImageNet-512 in Appendix Table 10 and attach the results below.
In the 600-epoch comparison on ImageNet-512, JiT-L and JiT-H outperform PiT-L and PiT-XL, respectively, while PiT-G nearly matches JiT-G (1.83 v.s. 1.78 FID) with approximately 20% fewer parameters. The performance gap narrows as model scale increases. With longer training and unchanged hyperparameters, all PiT models improve; PiT-G reaches 1.72 FID at 800 epochs. On ImageNet-256, PiT-G outperforms JiT-G at 600 epochs (1.61 v.s. 1.82).
These results support our claim that PiT continues to improve with longer training without scale-specific hyperparameter tuning. They do not imply uniform superiority over JiT.
Thank you for your constructive advice to help us strengthen our work. We hope these clarifications and additional experiments address your concerns, and we would be grateful if you would reconsider your assessment in light of them.