- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether simple representation alignment can improve pixel-space JiT generation. The authors apply an MLP projection head to align intermediate JiT features with frozen DINOv2 features during training. They show that early-layer alignment improves FID on ImageNet-256, while alignment at the final transformer block can hurt performance. The paper further investigates low-rank teacher representations using PCA and random projections. The experiments suggest that much of the improvement from representation alignment can be retained using only a small number of teacher feature directions. The authors also provide an analysis of the cosine alignment objective to discuss why retained feature variance alone may not explain generation performance.
Critical strengths
-
The paper addresses a relevant question about representation alignment in pixel-space diffusion models.
-
The main FID improvement is reasonably consistent across three training seeds. The authors also evaluate different CFG scales and provide an additional experiment using the released PixelREPA implementation.
Critical weaknesses
-
The main approach applies existing REPA-style alignment to pixel-space JiT using a simple MLP projection head. Both representation alignment and its sensitivity to attachment depth have been investigated in previous work. The main contribution therefore appears to be an empirical observation under a particular training configuration.
-
The paper mainly compares simple REPA against vanilla JiT. However, PixelREPA and iREPA are highly relevant to this problem, and the authors do not provide matched comparisons with their alignment approaches. Without these comparisons, it is difficult to assess the practical value of the proposed baseline.
-
The authors report that rank-32 PCA targets retain most of the FID improvement, and that a random rank-32 projection performs similarly in one experiment. However, the paper does not establish what information in these restricted representations is responsible for the improvement. Important controls, including mean-only targets, mean removal, and patch shuffling, are discussed but not evaluated.
-
The writing and organization need substantial improvement. The abstract is particularly difficult to read, as it presents many technical details and experimental results without clearly introducing the problem and main contribution. Similar issues appear throughout the manuscript, where extensive discussion of evaluation settings, caveats, and analytical observations makes the main message difficult to follow.
Questions for authors
-
In Table 4, random rank-32 projection achieves almost the same FID as PCA rank-32, despite retaining much less teacher variance. Does this suggest that the semantic information of the teacher features is not essential for the improvement? Have the authors considered mean-only targets, mean removal, or patch shuffling to investigate what information is actually being transferred?
-
The results show that early alignment improves FID while final-block alignment significantly worsens it. What causes this difference? Could the authors provide direct evidence, such as gradient similarity between the alignment and denoising objectives, rather than relying mainly on the discussion in Section 6?