- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes JAx, a prediction-level alignment method for accelerating pixel-space generative training with Just image Transformers (JiT). Motivated by the limitations of hidden representation alignment in clean-image prediction models, the authors propose aligning clean-image predictions across different noise levels instead of imposing additional constraints on intermediate representations.
The method constructs Markov-coupled noisy observations that preserve the original JiT input distribution. The authors theoretically show that, under an oracle teacher, the proposed supervision preserves the conditional mean of the original prediction target while reducing its conditional covariance. In practice, JAx employs an EMA teacher and a reliability-gated coupling strategy to select appropriate teacher states.
Experiments on ImageNet 256×256 demonstrate consistent FID improvements across JiT-B/16, L/16, and H/16. Additional ablations and gradient variance analyses support the effectiveness of the proposed design.
Critical strengths
Strength 1: Well-motivated method with a clear theoretical foundation.
The paper identifies a meaningful limitation of hidden representation alignment in pixel-space generative models and proposes directly supervising clean-image predictions instead. The Markov coupling formulation provides a principled way to preserve the original regression target while reducing conditional target variance under an oracle teacher. This connection between the theoretical analysis and method design is a major strength.
Strength 2: Consistent experimental improvements and informative ablations.
JAx improves FID across multiple JiT model scales without requiring external pretrained encoders or changing the inference architecture. The comparisons with Self-Flow, SRA, and PixelREPA are useful. In particular, the ablations on Markov coupling, reliability-gated pair selection, and time reweighting help isolate the contributions of the main components. The gradient variance measurements further support the proposed optimization mechanism.
Critical weaknesses
Weakness 1: Relatively modest improvements at larger model scales and additional training overhead.
Although JAx consistently improves FID, the absolute gains become relatively small for larger models. For example, the FID of JiT-H/16 decreases from 1.86 to 1.82 after 600 epochs. Meanwhile, the Inception Score decreases across all three evaluated scales, particularly for JiT-L/16.
Furthermore, JAx introduces approximately 21% additional training computation. While the reported improvements in epoch-based convergence are encouraging, a more comprehensive compute-matched evaluation would help establish the practical efficiency advantages. Reporting variability across random seeds would also strengthen the significance of the relatively small improvements.
Weakness 2: Limited robustness analysis of the practical reliability-gating strategy.
The theoretical analysis establishes useful properties for oracle predictions, but the practical method relies on an imperfect EMA teacher and empirically selected reliability thresholds. Although the paper provides theoretical discussion of teacher approximation errors and ablations on coupling intervals, the sensitivity to the prediction-risk threshold and calibration procedure remains insufficiently explored.
In addition, the experiments focus exclusively on ImageNet class-conditional generation with JiT. It remains unclear whether similar improvements would be observed for other prediction architectures or more general conditional generation tasks.
Questions for authors
Q1. How significant are the improvements under matched training computation?
JAx introduces additional EMA teacher evaluations, resulting in approximately 21% higher training computation. Could the authors provide FID comparisons under matched training FLOPs or GPU-hours rather than only matched training epochs? Additionally, given the relatively small FID improvement for JiT-H/16, are the results consistent across multiple random seeds?
Q2. Why does the Inception Score decrease despite improvements in FID?
Table 5 shows that JAx consistently improves FID but reduces Inception Score across the evaluated model scales, particularly for JiT-L/16. Could the authors provide additional analysis, such as Precision and Recall, to investigate whether the proposed supervision introduces a quality–diversity trade-off?
Q3. How robust is reliability-gated pair selection to its calibration parameters?
The practical method relies on an empirical prediction-risk threshold and a reliability frontier. Could the authors provide sensitivity analyses for the risk threshold and safety margin, or clarify how reliably the selected teacher states satisfy the intended prediction-risk criterion throughout training? This would help assess whether the method requires careful calibration when transferred to other datasets or architectures.
2 comments
Thans for the rebuttal. Most of my doubts have been resolved through experiments. However, there are still some questions regarding IS reduction, strict computational matching, and cross-architecture generalization. Given my initial optimistic viewpoint, I have decided to maintain the original score.
With the assistance and understanding provided by AI, I first completed the initial manual review and scoring. Based on this, I allowed AI to refine and supplement the content, without altering my own views or scoring. I have uploaded my initial manual review to AI Documentation For Reviewing