acceptodds
Under review as a conference paper at ICLR 2027

Just Align : Aligning Predictions, Not Representations

Abstract

Representation alignment has become an effective way to accelerate diffusion training, but its benefits do not transfer reliably to pixel-space clean-image prediction. In JiT, we find that auxiliary feature alignment can improve access to semantic features while reducing access to image variation needed for clean-image prediction, creating a mismatch between the auxiliary objective and the denoising task. This suggests a different principle: *auxiliary supervision should improve the prediction target itself rather than impose a separate representation target*. We introduce JAx (Just Align **x**), a prediction-supervision method that aligns clean-image predictions across noise levels. JAx couples a noisier student observation with a cleaner observation through a Markov degradation that preserves the original JiT input distribution. Under this coupling, the oracle prediction from the cleaner state has the same conditional mean as the optimal JiT target, while its conditional target covariance is no greater. Thus, oracle prediction alignment preserves the population JiT objective up to a constant while providing a lower-variance training target. To make this construction practical with an imperfect EMA teacher, JAx combines ground-truth supervision with a reliability-gated coupling band that selects nearby teacher states based on prediction risk. On ImageNet 256×256, JAx consistently improves FID and accelerates convergence across JiT-B/16, L/16, and H/16, without an external encoder or changes to the architecture or sampling procedure. Gradient diagnostics further show reduced minibatch gradient variance, while ablations demonstrate that the gains cannot be explained by time reweighting alone. These results show that prediction-space supervision provides a simple and principled alternative to representation alignment for pixel-space generative models.

Primary area:
generative models
Keywords:
Diffusion, Image Generation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QbycOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JAx, a prediction-level alignment method for accelerating pixel-space generative training with Just image Transformers (JiT). Motivated by the limitations of hidden representation alignment in clean-image prediction models, the authors propose aligning clean-image predictions across different noise levels instead of imposing additional constraints on intermediate representations.

The method constructs Markov-coupled noisy observations that preserve the original JiT input distribution. The authors theoretically show that, under an oracle teacher, the proposed supervision preserves the conditional mean of the original prediction target while reducing its conditional covariance. In practice, JAx employs an EMA teacher and a reliability-gated coupling strategy to select appropriate teacher states.

Experiments on ImageNet 256×256 demonstrate consistent FID improvements across JiT-B/16, L/16, and H/16. Additional ablations and gradient variance analyses support the effectiveness of the proposed design.

Critical strengths

Strength 1: Well-motivated method with a clear theoretical foundation.

The paper identifies a meaningful limitation of hidden representation alignment in pixel-space generative models and proposes directly supervising clean-image predictions instead. The Markov coupling formulation provides a principled way to preserve the original regression target while reducing conditional target variance under an oracle teacher. This connection between the theoretical analysis and method design is a major strength.

Strength 2: Consistent experimental improvements and informative ablations.

JAx improves FID across multiple JiT model scales without requiring external pretrained encoders or changing the inference architecture. The comparisons with Self-Flow, SRA, and PixelREPA are useful. In particular, the ablations on Markov coupling, reliability-gated pair selection, and time reweighting help isolate the contributions of the main components. The gradient variance measurements further support the proposed optimization mechanism.

Critical weaknesses

Weakness 1: Relatively modest improvements at larger model scales and additional training overhead.

Although JAx consistently improves FID, the absolute gains become relatively small for larger models. For example, the FID of JiT-H/16 decreases from 1.86 to 1.82 after 600 epochs. Meanwhile, the Inception Score decreases across all three evaluated scales, particularly for JiT-L/16.

Furthermore, JAx introduces approximately 21% additional training computation. While the reported improvements in epoch-based convergence are encouraging, a more comprehensive compute-matched evaluation would help establish the practical efficiency advantages. Reporting variability across random seeds would also strengthen the significance of the relatively small improvements.

Weakness 2: Limited robustness analysis of the practical reliability-gating strategy.

The theoretical analysis establishes useful properties for oracle predictions, but the practical method relies on an imperfect EMA teacher and empirically selected reliability thresholds. Although the paper provides theoretical discussion of teacher approximation errors and ablations on coupling intervals, the sensitivity to the prediction-risk threshold and calibration procedure remains insufficiently explored.

In addition, the experiments focus exclusively on ImageNet class-conditional generation with JiT. It remains unclear whether similar improvements would be observed for other prediction architectures or more general conditional generation tasks.

Questions for authors

Q1. How significant are the improvements under matched training computation?

JAx introduces additional EMA teacher evaluations, resulting in approximately 21% higher training computation. Could the authors provide FID comparisons under matched training FLOPs or GPU-hours rather than only matched training epochs? Additionally, given the relatively small FID improvement for JiT-H/16, are the results consistent across multiple random seeds?

Q2. Why does the Inception Score decrease despite improvements in FID?

Table 5 shows that JAx consistently improves FID but reduces Inception Score across the evaluated model scales, particularly for JiT-L/16. Could the authors provide additional analysis, such as Precision and Recall, to investigate whether the proposed supervision introduces a quality–diversity trade-off?

Q3. How robust is reliability-gated pair selection to its calibration parameters?

The practical method relies on an empirical prediction-risk threshold and a reliability frontier. Could the authors provide sensitivity analyses for the risk threshold and safety margin, or clarify how reliably the selected teacher states satisfy the intended prediction-risk criterion throughout training? This would help assess whether the method requires careful calibration when transferred to other datasets or architectures.

2 comments

With the assistance and understanding provided by AI, I first completed the initial manual review and scoring. Based on this, I allowed AI to refine and supplement the content, without altering my own views or scoring. I have uploaded my initial manual review to AI Documentation For Reviewing

Thans for the rebuttal. Most of my doubts have been resolved through experiments. However, there are still some questions regarding IS reduction, strict computational matching, and cross-architecture generalization. Given my initial optimistic viewpoint, I have decided to maintain the original score.

Reviewer_5r7rOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a self-alignment training method for pixel diffusion models. The training composes of the flow matching loss and clean-image prediction alignment loss. The alignment loss is computed between the predictions of the student model and the teacher model, whose weights are moving average of those from student model. The loss aligns their predictions given input pairs of various noise levels. The inputs are crafted using Markov coupling, which helps reduce target variance.

Critical strengths
  1. The motivation is clear and easy to understand.
  2. The method operates without external models.
Critical weaknesses

The major limitation is that the experiments are insufficient to support the main claim of the paper (aligning predictions rather than representations).

  1. The authors claim that the hidden representation alignment is not always effective when transferred from latent diffusion to pixel-space. However they only validate this on small scale model (Table 1) under constrained training steps.
  2. Compared with the JiT baseline, the actual performance gains quickly diminish on larger models. For example, In line 373-375, JAx reduces FID from 3.66 to 3.41 on B/16, from 2.36 to 2.26 on L/16, and from 1.86 to 1.82 on H/16. Moreover, in figure 4, the authors only show JiT-L and -XL performance at 200 and 600 train epochs respectively, making it difficult to evaluate the actual benefits regarding training convergence speed.

Overall, these marginal performance gains undermine the central claims and the proposed framework components (such as the admissible prediction risk R∗R^*, safety margin σsafe\sigma_{safe}, risk-estimation grid spacing, and calibration update interval), making the overall method and arguments less persuasive.

Minor:

  1. To improve clarity, I suggest providing an intuitive explanation before presenting the theoretical results.
Questions for authors

Please see weakness

1 comment

Thank you for the clarifications and additional results. However, the evidence against representation alignment remains limited to JiT-B/16 in a specific setting and does not support the broader “align predictions rather than representations” argument. Addressing this would require a major revision of the paper’s framing, positioning JAx as an effective auxiliary objective in the evaluated settings rather than evidence of a general advantage over representation alignment.

My concerns about overclaiming and limited insight therefore remain, and I decide to maintain my assessment.

Reviewer_FzRkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JAx improves JiT training with a teacher built by EMA. The teacher sees an image with less noise. The student learns from both the teacher’s prediction and the original image. The method uses teacher predictions only when the teacher’s image reconstruction error is low enough. It improves ImageNet generation measured by FID.

Critical strengths
  • The theory explains how to construct the teacher and student inputs. With an ideal teacher, the method keeps JiT’s learning goal and makes training updates less noisy. Appendix A also explains how errors from a real teacher can change what the student learns.

  • The experiments support a useful training method. Improvements appear at three model sizes. Simply changing the weights of the original training loss does not give the same benefit. Figure 5 also shows that the new training updates are closer to the expected JiT updates on average. Combining JAx with PixelREPA further improves the result.

Critical weaknesses
  • My main concern is how the method decides which teacher predictions are useful. A teacher can have low reconstruction error because its input contains little noise. This does not necessarily mean that its prediction helps the student more. Appendix A explains this trade-off, and Figure 5 supports the predictions that the method selected. However, it remains unclear how well the selection rule distinguishes helpful predictions from less helpful ones. This is central to the method: using a fixed noise difference between teacher and student makes FID worse in Table 2. I would like a clearer explanation of why the proposed rule selects better training targets.

  • The broader message about prediction versus feature alignment is less convincing. Figure 2 does not show that feature alignment removes information needed for generation, as the authors acknowledge. Table 1 reports 4.00 FID for PixelREPA, 4.17 for JAx, and 3.91 for their combination. Appendix B also recognizes that the methods can work together. These results support adding JAx to existing methods. They do not clearly show that learning from features is the underlying problem. The paper should explain whether the problem is feature alignment itself or the particular way REPA is applied to JiT.

Questions for authors
  • At the same student noise level, what evidence shows that the selection rule finds more helpful teacher predictions, rather than simply finding images that are easier for the teacher to reconstruct?

  • Given the PixelREPA and combined results, is the main contribution a better way to train JiT alongside feature alignment? What evidence supports a broader preference for prediction alignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.