- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates which properties of diffusion-transformer representations are associated with generation quality. It compares eight properties along a training trajectory, across 16 REPA/iREPA models, and across eight models trained with different auxiliary objectives. The main finding is that strong correlations during training often weaken across independently trained models, whereas linear-probe accuracy remains comparatively predictive.
Critical strengths
The paper addresses an important methodological question for representation-guided generative modeling. The distinction between training-time correlation and cross-model evidence is useful and clearly motivated. Recomputing FID under a shared sampling protocol, controlling for alignment-method families, and reporting bootstrap intervals and layer/time sweeps make the empirical study substantially more informative than a single-trajectory analysis. I also appreciate the explicit discussion of the assumptions and limitations of the effective-rank result.
Critical weaknesses
-
The across-objective comparison remains heterogeneous despite the shared sampler. Models differ in training budget, VAE, and conditioning; REG additionally receives image-derived encoder information during feature extraction. With only eight models, these differences limit how confidently the correlations can be attributed to representation properties.
-
The intervention experiments provide narrower evidence than a clean causal test of the properties themselves. Each treatment is run once, starts with a reset optimizer, and targets block 8, while the main observational comparison uses block 12. The augmentation-invariance treatment also substantially changes feature scale and the input-independent component. These results support conclusions about the tested objectives and settings, but do not establish that the corresponding properties are generally ineffective targets.
-
Uniformity illustrates substantial estimator dependence. Appendix E.1 shows that the sign reversal disappears when random image pairs replace the class-biased within-batch pairs. This qualification deserves greater prominence in the main discussion of reversed correlations.
Questions for authors
-
Do the principal cross-objective findings remain similar after excluding REG and REPA-E, which introduce additional conditioning or representation differences?
-
Is there a matched optimizer-restart control for the intervention experiments, or other evidence separating the auxiliary-loss effect from restarting optimization?
-
Could the main uniformity results emphasize the random-pair estimator and distinguish estimator-sensitive findings from the more robust conclusions?
3 comments
Thank you for the additional analyses. Excluding REG and REPA-E, adding an optimizer restart matched control with interventions at block 12, and using random pair uniformity as the primary estimator directly address my three questions.
These additions strengthen my assessment, and I maintain my weak accept recommendation.
Thank you for your reply. We are glad our additional analyses address your questions and strengthen your assessment.
Thank you very much for the detailed and constructive review, for agreeing with us that our central research question is important and that our empirical study is substantially more informative than a single-trajectory analysis.
W1, Q1: Heterogeneous models across objectives
We removed REG and REPA-E from the cross-objective test (third row). Spearman correlations with FID:
Our findings here are similar: Linear probe accuracy and ICR are the strongest predictors, and the correlations of LDS and uniformity weaken relative to the training run. Across objectives, training budget and FID are essentially uncorrelated (ρ = −0.07), and controlling for log training steps leaves the correlations unchanged or stronger (second row).
Also within the second setting, the budget is fixed (all models exactly 400K steps, identical recipe), and the weakening appears there as well.
W2, Q2: Interventions (optimizer reset, block 8 vs. 12)
While we agree that there could potentially be settings in which targeting the property helps, we think our chosen setting covers the most relevant one: We designed it to be the simplest intervention that results in the desired change of the property while keeping training healthy, which is arguably what any practitioner would do. For a fairer comparison, we reran the interventions on block 12 and added a matched control that branches at 25K steps with a reset optimizer and no auxiliary loss. Our previous findings hold in the new setting: there are no or only marginal gains relative to the control, even though the property is effectively optimized. From 50K steps on, FID stays within 1% (LDS) and 5.5% (augmentation invariance) of the control. The final FID is 17.85 and 18.49 vs. 17.98 (block 12; Section 4, Figure 4). The block-8 runs give the same result (18.13 and 17.91 vs. 17.98; Appendix D.5).
W3, Q3: Uniformity
Agreed. We now use the random-pair estimator as the headline number. We also comment on the sensitivity to the choice of estimator in the main body of the revised manuscript (Section 3, after Table 1) and in Appendix E ("Uniformity from pairs within batches").
We hope this addresses your concerns. If you have any remaining concerns, please let us know.