- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies representation alignment in pixel-space Just Image Transformers. Motivated by a possible conflict between semantic alignment and preserving image details, RouteREPA computes an input-dependent channel gate and uses complementary masks for generation and alignment. It also aggregates alignment features from multiple depths. ImageNet-256 experiments compare several model sizes, training budgets, and routing alternatives, and report improved generation with modest overhead. Feature-similarity and reconstruction probes are used to motivate the proposed division of representational roles.
Critical strengths
- The empirical gain is substantial across scales under the same training duration. In Table 1, JiT-B at 200 epochs improves from 32.2 FID without alignment and 27.5 with REPA to 22.2 with RouteREPA; JiT-L improves from 16.1 and 12.8 to 10.6. RouteREPA-H reaches 4.96 versus 6.68 for REPA at 200 epochs. These differences are more compelling than the smaller headline guided-FID differences. Appendix A also re-evaluates long-training checkpoints through a common pipeline for the unguided comparisons.
- The ablations usefully separate routing from multi-depth supervision. In Table 3, single-depth routing at layer 4 improves 27.5 to 24.2, while multi-depth routing reaches 22.2; multi-depth alignment alone is insufficient. Generation-only, alignment-only, independent-mask, and orthogonal alternatives are included. Measured overhead is small at the largest scale: about 0.3% additional parameters, 1.6% training-step time, and 1.7% inference-forward time.
Critical weaknesses
- The claimed mechanism is not isolated from generic channel reweighting or optimization effects. Lower reconstruction accuracy from aligned features under a particular decoder does not establish loss of information, and reduced feature similarity does not directly measure gradient conflict. Complementary gates can change feature magnitudes and conditioning even without a meaningful separation of semantic and detail channels. The existing routing ablations are valuable, but they do not include an equally parameterized learned static gate or an input-dependent channel gate without complementary task separation. Consequently, the method is better supported than the explanation that motivates it.
- The strongest comparative and mechanistic claims have limited scope. The principal study uses ImageNet and a DINOv2 teacher. Table 2 compares guided FID 1.72 for RouteREPA at 400 epochs against 1.81 for PixelREPA and 1.86 for JiT at 600 epochs, using published results rather than a fully matched training and evaluation setup. In Table 5, latent SiT-B/2 with REPA gives 24.40 FID and with RouteREPA gives 21.85 at 400K iterations and CFG 1. This is useful evidence of transfer, but Figure 2(c) reports no comparable reconstruction degradation for a different latent model, Latent JiT. The paper should clarify whether the latent-space benefit follows the same conflict-resolution mechanism or a more general optimization effect. A matched alignment-strength sweep would also clarify the advantage over a tuned REPA baseline.
Questions for authors
- Mechanism test: How does RouteREPA compare with a learned static complementary gate and a similarly parameterized input-dependent channel gate that does not separate the two tasks? A matched alignment-strength sweep would also help determine whether the benefit is specific to complementary routing rather than rescaling the auxiliary loss.
- Mechanism scope: Figure 2(c) probes Latent JiT, whereas Table 5 evaluates latent SiT-B/2. Can the same reconstruction probe be applied to the SiT models to determine whether routing helps when alignment does not reduce reconstructability? This would clarify the scope of the proposed conflict-resolution explanation.
- Can you extend RouteREPA to text-to-image on CC12M, LAION, or COCO with a text encoder, and compare against DiT/SiT-based T2I methods under matched data (like FID, CLIP score, GenEval, DPG-Bench)?
- Can you evaluate on other baseline methods at 512×512, with multiple teachers such as DINOv2-L, CLIP-L, and I-JEPA?
5 comments
Thanks the authors for adding some experiments to solve my problems, I raise my score to 3. However, the text-to-image experiment could not be completed due to paper contribution. I understand the computational costs, but the current FID score for the c2i model is already very low. If the authors have the opportunity to supplement this experiment, or if they could provide a discussion on new and comprehensive metrics, like FDrk (https://arxiv.org/abs/2604.28190), I would consider further increasing the score.
Follow-up response
Thank you very much for taking the time to review our additional experiments and for raising your score. We especially appreciate your understanding of the computational demands of text-to-image training and evaluation. Following your suggestion, we evaluated class-conditional generation on ImageNet at 256 × 256 using FDr⁶.
| Method | FDr⁶ ↓ |
|---|---|
| JiT-H/16 | 7.68 |
| PixelREPA-H/16 | 5.94 |
| RouteREPA-H/16 | 5.22 |
RouteREPA achieves the lowest FDr⁶, reducing it by 32.1% relative to JiT and 12.2% relative to PixelREPA.
We are making effort to complete the text-to-image experiments. However, with the deadline fast approaching and limited computation resources, we may need more time to finish training and evaluation.
Response to Reviewer TUEa
Thank you for your feedback. We address each concern below.
W1 & Q1. Reconstruction error, gradient conflict, alternative gating, and REPA tuning
1. What the reconstruction probe measures
We agree that higher reconstruction error alone provides insufficient evidence of information loss. The probe measures how difficult it is to recover clean pixels from intermediate features with a given decoder and training budget. This matters for JiT because its remaining layers must likewise predict clean pixels from those features with finite capacity.
2. Gradient evidence for conflict mitigation
We measured the angle between the generation and alignment gradients on shared parameters. RouteREPA reduces both the mean angle and the fraction above 90°, providing direct evidence of reduced gradient conflict.
3. Static complementary gating and input-dependent gating without task separation
We compare RouteREPA with a learned static complementary gate and an input-dependent gate without task separation. Both methods uses the same parameter count as RouteREPA. RouteREPA achieves the lowest FID after 200 epochs at CFG 1.
4. Alignment coefficient sweep of REPA
With standard REPA, JiT's generation loss is roughly an order of magnitude smaller than SiT's (Appendix A.1). We reduced λ from 0.5 in SiT to 0.05 in JiT to bring the generation and weighted alignment losses to comparable scales. Both methods use 0.05 in the paper.
We swept REPA's λ and compared it with RouteREPA at the paper setting. After 200 epochs with JiT-B/16 on ImageNet-256, RouteREPA's CFG 1 FID is 5.3 points lower than the best REPA result.
W2a. Matched main comparisons
We trained JiT-H/16 + RouteREPA on ImageNet-256 for 600 epochs using the same training, sampling, and evaluation settings as the published JiT and PixelREPA results. RouteREPA achieves a guided FID of 1.67, outperforming both published baselines.
W2b & Q2. Conflict mitigation in latent space
We applied the Latent JiT reconstruction probe from Figure 2(c) to SiT-B/2 with and without REPA. The table reports RGB reconstruction MSE. In SiT, t=0 is a clean image and t=1 is pure noise.
On SiT-B/2, REPA increases reconstruction MSE at three of four timesteps, and RouteREPA lowers FID from 24.40 to 21.85. On Latent JiT, where REPA does not consistently increase reconstruction error, RouteREPA lowers FID only slightly, from 27.0 to 26.9. These results suggest that RouteREPA is more effective when REPA makes reconstruction more difficult.
Q3. Concerns about text-to-image comparison with DiT- and SiT-based methods
A comparison with DiT- and SiT-based text-to-image methods would be informative at the system level. However, the differences in backbone architecture and representation space would make it difficult to isolate RouteREPA’s contribution. To assess that contribution, we would compare pixel-space JiT with and without RouteREPA. This paper focuses on pixel-space class-conditional generation, and we leave the controlled text-to-image comparison for future work.
Q4. Additional baselines and teachers at 512 × 512
We compare REPA and RouteREPA with DINOv2 and CLIP on ImageNet-512 using JiT-H/32. RouteREPA achieves lower CFG 1 FID than REPA with both teachers after 100 epochs.