acceptodds
Withdrawn from ICLR 2027

Simple Alignment for Pixel-Space JiT: Matched Controls and Low-Rank Targets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Simple early-layer representation alignment is a strong baseline for pixel-space JiT on ImageNet-256. With a training-only MLP head, searched FID-50K improves from 4.80 ± 0.16 to 3.69 ± 0.06 across three matched training seeds, a 23.1% reduction. The benefit persists at every common guided CFG scale tested in two complete sweeps and is corroborated by one fixed-CFG pair in the released PixelREPA implementation. Attachment matters: the final-block endpoint is harmful in two seeds. Mean-preserving rank-32 PCA targets retain approximately 87% of the searched-FID gain over two seeds, despite retaining only 26.5% of centered teacher variance. A single random rank-32 basis reaches a similar searched endpoint with 4.2% variance, without establishing equivalence between subspaces. Analysis of the cosine objective separates variance retention, target direction, and conditional predictability; it supplies testable hypotheses rather than a causal account of generation quality. These results support including simple early alignment in matched JiT evaluations and show that much of its measured benefit survives restricted teacher targets.

Primary area:
generative models
Keywords:
pixel-space diffusion, representation alignment, controlled experiments, low-rank teacher targets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HB7gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether simple representation alignment can improve pixel-space JiT generation. The authors apply an MLP projection head to align intermediate JiT features with frozen DINOv2 features during training. They show that early-layer alignment improves FID on ImageNet-256, while alignment at the final transformer block can hurt performance. The paper further investigates low-rank teacher representations using PCA and random projections. The experiments suggest that much of the improvement from representation alignment can be retained using only a small number of teacher feature directions. The authors also provide an analysis of the cosine alignment objective to discuss why retained feature variance alone may not explain generation performance.

Critical strengths
  • The paper addresses a relevant question about representation alignment in pixel-space diffusion models.

  • The main FID improvement is reasonably consistent across three training seeds. The authors also evaluate different CFG scales and provide an additional experiment using the released PixelREPA implementation.

Critical weaknesses
  • The main approach applies existing REPA-style alignment to pixel-space JiT using a simple MLP projection head. Both representation alignment and its sensitivity to attachment depth have been investigated in previous work. The main contribution therefore appears to be an empirical observation under a particular training configuration.

  • The paper mainly compares simple REPA against vanilla JiT. However, PixelREPA and iREPA are highly relevant to this problem, and the authors do not provide matched comparisons with their alignment approaches. Without these comparisons, it is difficult to assess the practical value of the proposed baseline.

  • The authors report that rank-32 PCA targets retain most of the FID improvement, and that a random rank-32 projection performs similarly in one experiment. However, the paper does not establish what information in these restricted representations is responsible for the improvement. Important controls, including mean-only targets, mean removal, and patch shuffling, are discussed but not evaluated.

  • The writing and organization need substantial improvement. The abstract is particularly difficult to read, as it presents many technical details and experimental results without clearly introducing the problem and main contribution. Similar issues appear throughout the manuscript, where extensive discussion of evaluation settings, caveats, and analytical observations makes the main message difficult to follow.

Questions for authors
  • In Table 4, random rank-32 projection achieves almost the same FID as PCA rank-32, despite retaining much less teacher variance. Does this suggest that the semantic information of the teacher features is not essential for the improvement? Have the authors considered mean-only targets, mean removal, or patch shuffling to investigate what information is actually being transferred?

  • The results show that early alignment improves FID while final-block alignment significantly worsens it. What causes this difference? Could the authors provide direct evidence, such as gradient similarity between the alignment and denoising objectives, rather than relying mainly on the discussion in Section 6?

Reviewer_K2yUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates representation alignment (REPA) for pixel-space Just image Transformers (JiT-B/16) on ImageNet-256. Motivated by conflicting literature where PixelREPA reported that plain MLP-based REPA degrades FID (4.37→5.144.37 \to 5.14) and requires a specialized masked transformer adapter (MTA), the authors conduct a controlled audit. They show that attaching a simple training-only 3-layer MLP head with DINOv2-B cosine supervision at an early block (L3, after the 4th block, right before class tokens are injected at index 4) consistently improves searched FID-50K from 4.80±0.164.80 \pm 0.16 to 3.69±0.063.69 \pm 0.06 across three paired training seeds in their JAX stack, and from 4.704.70 to 4.124.12 (at fixed CFG 3.1) in a single-seed control patched into the released PixelREPA codebase. Conversely, attaching the alignment head at the final block (L11) degrades FID to 7.70±0.337.70 \pm 0.33 across two seeds. Next, the authors intervene on the DINOv2-B targets via affine low-rank projection z(r)=μ+VrVr⊤(z−μ)z^{(r)} = \mu + V_r V_r^\top (z - \mu), finding that a rank-32 top-PCA target retains ~87% of the FID improvement (3.87±0.043.87 \pm 0.04 over two seeds) despite capturing only 26.5% of centered teacher variance. Strikingly, a single-seed control using a random rank-32 orthonormal basis (4.2%4.2\% centered variance) achieves the exact same searched FID (3.843.84) as top-PCA rank-32 on seed 42. Section 6 provides elementary analytical propositions regarding the cosine loss, pointing out that preserving the global mean μ\mu makes z(r)z^{(r)} span an (r+1)(r+1)-dimensional subspace and that conditional predictability E∥E[Q∣H]∥\mathbb{E}\Vert\mathbb{E}[Q \mid H]\Vert and mean-dominance (tr⁡(PC)/∥μ∥2\operatorname{tr}(PC)/\Vert\mu\Vert^2) govern cosine risk rather than unconditional centered variance tr⁡(PC)/tr⁡(C)\operatorname{tr}(PC)/\operatorname{tr}(C).

Critical strengths
  • Valuable empirical baseline correction for pixel-space diffusion: When prior work (PixelREPA) claims a standard technique (plain MLP REPA) fails and proposes a complex architectural fix (masked transformer adapters), a careful reproduction showing that plain MLP REPA at L3 yields a 23.1% FID reduction (4.80→3.694.80 \to 3.69) across seeds and across all guided CFG scales (Table 2) is a genuinely useful scientific service to the community.

  • Intriguing low-rank and random-projection empirical finding: Demonstrating that projecting 768-dim DINOv2 targets onto 32 PCA dimensions, or even 32 random orthogonal directions plus the global patch mean μ\mu, preserves ~87% of the FID gain (3.843.84 vs. 3.713.71 full teacher on seed 42) challenges the widespread assumption that REPA works by transferring fine-grained, high-variance semantic features from the teacher.

Critical weaknesses
  • Unresolved confounds in the paper's central findings (Mean-only/Centered controls and MTA comparison):

    • Missing Rank-0 (μ\mu-only) and Centered (μ=0\mu=0) ablations: The most surprising result in the paper is that Random rank 32 (4.2%4.2\% variance, FID 3.843.84) matches Top-PCA rank 32 (26.5%26.5\% variance, FID 3.843.84). In Section 6.1 and Result 4 (Eq. 10), the authors themselves show that because Eq. (2) is an affine projection z(r)=μ+P(z−μ)z^{(r)} = \mu + P(z-\mu), the normalized target direction depends heavily on ∥μ∥\Vert\mu\Vert vs. tr⁡(PC)\operatorname{tr}(PC). Yet the paper never runs the two trivial controls that their own Section 6 explicitly calls for (lines 334, 375, 411): (i) Centered Rank-32 (z(r)=P(z−μ)z^{(r)} = P(z-\mu), removing μ\mu) and (ii) Mean-only / Rank-0 (P=0P=0) or spatial patch-shuffling. Nor do they even report the numerical value of ∥μ∥2/tr⁡(C)\Vert\mu\Vert^2 / \operatorname{tr}(C) for the cached DINOv2 features! If Random rank 32 only works because ∥μ∥\Vert\mu\Vert dominates or because random linear projections preserve pairwise spatial token distances (Johnson-Lindenstrauss effect), the entire interpretation of Section 5 changes.

    • No head-to-head comparison against PixelREPA's MTA or iREPA: While the authors patch a plain MLP into the released PixelREPA codebase for 1 seed (getting 4.70→4.124.70 \to 4.12 at F3.1), they never run the actual PixelREPA MTA branch under the exact same matched seed/setup to show how much gap (if any) remains between a plain MLP and MTA.

    • Depth vs. Class-Token boundary confound: In Section 4, L3 (beneficial) is immediately before the 32 in-context class tokens are inserted at block index 4, whereas L5-L11 are after class-token insertion. Without an ablation shifting in_context_start, the claim that alignment depth is the driver is confounded with attention redistribution from class-token injection.

  • Trivial analytical section that is disconnected from the empirical features:

    • Section 6 ("An Analytical View of Target Restriction and Alignment") consists of four elementary textbook identities: the Jacobian of cosine normalization (Result 1), the fact that arg⁡max⁡∥u∥=1u⊤E[Q∣H]=E[Q∣H]/∥E[Q∣H]∥\arg\max_{\Vert u\Vert=1} u^\top \mathbb{E}[Q\mid H] = \mathbb{E}[Q\mid H]/\Vert\mathbb{E}[Q\mid H]\Vert by Cauchy-Schwarz (Result 2), the triangle inequality on conditional expectations (Result 3), and a 2-line bound for a constant predictor u=μ^u=\hat{\mu} (Result 4).

    • None of the quantities in Results 1-4 (∥μ∥2\Vert\mu\Vert^2, tr⁡(PC)/∥μ∥2\operatorname{tr}(PC)/\Vert\mu\Vert^2, δH(P)\delta_H(P), or the gradient inner product ⟨gD,gA⟩\langle g_D, g_A \rangle in Eq. 11) are actually measured on the DINOv2 targets or trained JiT checkpoints. Presenting basic Cauchy-Schwarz identities and then stating in the text that "none is measured here" leaves a large gap between the theoretical framing and the empirical investigation.

Questions for authors
  • Mean/centering diagnostics on the DINOv2-B features: What is the empirical ratio tr⁡(C)/∥μ∥2\operatorname{tr}(C) / \Vert\mu\Vert^2 for your DINOv2-B patch features, and what happens when μ\mu is removed? Since you already have the PCA basis and mean μ\mu computed over 25.6M tokens (Section 5), you can immediately compute ∥μ∥2\Vert\mu\Vert^2, tr⁡(C)\operatorname{tr}(C), and tr⁡(PC)/∥μ∥2\operatorname{tr}(PC)/\Vert\mu\Vert^2 (from Result 4) without any new training. Furthermore, what happens to Top-PCA rank 32 and Random rank 32 when you center the targets (z(r)=VrVr⊤(z−μ)z^{(r)} = V_r V_r^\top (z - \mu), without adding μ\mu back)?

  • Replication of Random rank 32 and Johnson-Lindenstrauss spatial structure: Random rank 32 is currently a single training seed (seed 42) and a single basis seed (seed 0) run on a different accelerator (v5e vs. v5p). Can you replicate Random rank 32 on seed 43 (and/or a second basis seed)? Do you hypothesize that Random rank 32 works because random projections preserve pairwise spatial token cosine similarities across the 16×1616 \times 16 image grid (similar to iREPA's spatial structure hypothesis)?

  • Comparison with PixelREPA's MTA in the released stack: In Table 1 (bottom), you ran Vanilla (4.704.70) and REPA, fourth-block plain MLP (4.124.12) in the pinned PixelREPA repository. What FID does the default PixelREPA Masked Transformer Adapter (MTA) achieve under that exact same seed and 200-epoch protocol?

  • Measuring gradient compatibility ⟨gD,gA⟩\langle g_D, g_A \rangle across depth: In Eq. (11), you hypothesize that L11 fails while L3 succeeds due to gradient interference ⟨gD,gA⟩\langle g_D, g_A \rangle. Since you have the saved checkpoints for seed 42 at L3 and L11, can you compute the cosine similarity between ∇θLJiT\nabla_\theta \mathcal{L}_{\text{JiT}} and ∇θLREPA\nabla_\theta \mathcal{L}_{\text{REPA}} on a few minibatches to verify whether ⟨gD,gA⟩\langle g_D, g_A \rangle actually turns negative at L11?

Reviewer_9euEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effectiveness of plain representation alignment (REPA) in pixel-space JiT. The experiments show that early-layer alignment improves generation quality, while final-block alignment is harmful. The paper also finds that low-rank teacher targets retain most of the benefit.

Critical strengths

The authors report results from three training seeds and compare performance at common CFG scales. These experiments provide solid support for the benefits of early-layer alignment.

Critical weaknesses
  1. The contribution is limited and incremental. Prior work has studied REPA on JiT and alignment-depth sensitivity. This paper strengthens the empirical evidence, but it does not identify the factors behind beneficial or harmful alignment. While the matched experiments are useful, I find the additional insight insufficient for acceptance.

  2. The low-rank result remains insufficiently explained. Although the finding that low-rank targets retain most of the gain is an important result of this paper, the paper does not further identify which retained information actually matters.The random-subspace control also uses only one training seed and one basis seed. These limitations reduce the depth and generality of the low-rank finding.

Questions for authors

Section 6 acknowledges that mean-only, mean-removal, and patch-shuffling interventions have not been measured. Do the authors have additional results that could help distinguish the roles of mean retention, spatial correspondence, and subspace selection in the observed gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.