acceptodds
Under review as a conference paper at ICLR 2027

Representation Geometry in Diffusion Transformers: What Matters for Generation Quality?

10 views

Abstract

Diffusion and flow-matching models are the dominant paradigm for image generation, but are notoriously expensive to train. A recent line of work accelerates training and improves generative performance by shaping the internal representations of these models through auxiliary objectives. However, it is unclear which properties actually matter and how to reliably identify them. To address this, we measure eight properties of diffusion transformer (SiT-XL/2) representations and test whether they track generative performance in three complementary settings: a single training trajectory, 16 REPA and iREPA models that differ only in the vision encoder they are aligned to, and models trained with different representation guidance objectives. Several properties correlate strongly with generative performance, measured by Fréchet Inception Distance (FID), along the training trajectory, yet several of these properties do not separate better from worse models once training has converged. This shows that correlation along a trajectory alone is insufficient evidence that a property matters for generation. We confirm this with interventional experiments, in which turning such properties into auxiliary training objectives yields no or only marginal gains. We find that properties that remain predictive throughout our experiments correspond largely to those that are explicitly or implicitly optimized by recent successful methods. These results offer principled guidance for the design of new representation-shaping objectives.

Primary area:
generative models
Keywords:
diffusion model, flow matching, representation learning, diffusion transformer

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_o3esOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates which properties of diffusion-transformer representations are associated with generation quality. It compares eight properties along a training trajectory, across 16 REPA/iREPA models, and across eight models trained with different auxiliary objectives. The main finding is that strong correlations during training often weaken across independently trained models, whereas linear-probe accuracy remains comparatively predictive.

Critical strengths

The paper addresses an important methodological question for representation-guided generative modeling. The distinction between training-time correlation and cross-model evidence is useful and clearly motivated. Recomputing FID under a shared sampling protocol, controlling for alignment-method families, and reporting bootstrap intervals and layer/time sweeps make the empirical study substantially more informative than a single-trajectory analysis. I also appreciate the explicit discussion of the assumptions and limitations of the effective-rank result.

Critical weaknesses
  • The across-objective comparison remains heterogeneous despite the shared sampler. Models differ in training budget, VAE, and conditioning; REG additionally receives image-derived encoder information during feature extraction. With only eight models, these differences limit how confidently the correlations can be attributed to representation properties.

  • The intervention experiments provide narrower evidence than a clean causal test of the properties themselves. Each treatment is run once, starts with a reset optimizer, and targets block 8, while the main observational comparison uses block 12. The augmentation-invariance treatment also substantially changes feature scale and the input-independent component. These results support conclusions about the tested objectives and settings, but do not establish that the corresponding properties are generally ineffective targets.

  • Uniformity illustrates substantial estimator dependence. Appendix E.1 shows that the sign reversal disappears when random image pairs replace the class-biased within-batch pairs. This qualification deserves greater prominence in the main discussion of reversed correlations.

Questions for authors
  1. Do the principal cross-objective findings remain similar after excluding REG and REPA-E, which introduce additional conditioning or representation differences?

  2. Is there a matched optimizer-restart control for the intervention experiments, or other evidence separating the auxiliary-loss effect from restarting optimization?

  3. Could the main uniformity results emphasize the random-pair estimator and distinguish estimator-sensitive findings from the more robust conclusions?

3 comments

Thank you very much for the detailed and constructive review, for agreeing with us that our central research question is important and that our empirical study is substantially more informative than a single-trajectory analysis.

W1, Q1: Heterogeneous models across objectives

The across-objective comparison remains heterogeneous despite the shared sampler. Models differ in training budget, VAE, and conditioning; REG additionally receives image-derived encoder information during feature extraction. [...] Q1: Do the principal cross-objective findings remain similar after excluding REG and REPA-E [...]?

We removed REG and REPA-E from the cross-objective test (third row). Spearman correlations with FID:

Linear probeICREff. rankCKNNAUniformityLDSAug. inv.Class scatter
All models (n = 8)−0.79+0.83−0.50−0.38+0.36+0.19−0.02−0.31
Controlled for log steps−0.80+0.86−0.71−0.50+0.50+0.240.00−0.31
Without REG and REPA-E (n = 6)−0.60+0.60−0.14−0.26−0.09+0.20−0.03−0.03

Our findings here are similar: Linear probe accuracy and ICR are the strongest predictors, and the correlations of LDS and uniformity weaken relative to the training run. Across objectives, training budget and FID are essentially uncorrelated (ρ = −0.07), and controlling for log training steps leaves the correlations unchanged or stronger (second row).

Also within the second setting, the budget is fixed (all models exactly 400K steps, identical recipe), and the weakening appears there as well.

W2, Q2: Interventions (optimizer reset, block 8 vs. 12)

The intervention experiments provide narrower evidence than a clean causal test of the properties themselves. Each treatment is run once, starts with a reset optimizer, and targets block 8, while the main observational comparison uses block 12. [...] Q2: Is there a matched optimizer-restart control for the intervention experiments [...]?

While we agree that there could potentially be settings in which targeting the property helps, we think our chosen setting covers the most relevant one: We designed it to be the simplest intervention that results in the desired change of the property while keeping training healthy, which is arguably what any practitioner would do. For a fairer comparison, we reran the interventions on block 12 and added a matched control that branches at 25K steps with a reset optimizer and no auxiliary loss. Our previous findings hold in the new setting: there are no or only marginal gains relative to the control, even though the property is effectively optimized. From 50K steps on, FID stays within 1% (LDS) and 5.5% (augmentation invariance) of the control. The final FID is 17.85 and 18.49 vs. 17.98 (block 12; Section 4, Figure 4). The block-8 runs give the same result (18.13 and 17.91 vs. 17.98; Appendix D.5).

W3, Q3: Uniformity

Uniformity illustrates substantial estimator dependence. [...] Q3: Could the main uniformity results emphasize the random-pair estimator [...]?

Agreed. We now use the random-pair estimator as the headline number. We also comment on the sensitivity to the choice of estimator in the main body of the revised manuscript (Section 3, after Table 1) and in Appendix E ("Uniformity from pairs within batches").

We hope this addresses your concerns. If you have any remaining concerns, please let us know.

Thank you for the additional analyses. Excluding REG and REPA-E, adding an optimizer restart matched control with interventions at block 12, and using random pair uniformity as the primary estimator directly address my three questions.

These additions strengthen my assessment, and I maintain my weak accept recommendation.

Thank you for your reply. We are glad our additional analyses address your questions and strengthen your assessment.

Reviewer_ZdbkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper evaluates 8 internal feature properties of diffusion models to figure out what actually drives generation quality (FID).

Critical strengths

1 The problem to solve in this paper is important and well-motivated

2 The appendix is very detailed, including 27 pages.

Critical weaknesses

1 The claimed "cross-model reversal" for Uniformity relies on biased in-batch sampling (33.5% same-class pairs). Random sampling results in the appendix contradict this. Authors should use random sampling as the primary metric.

  1. The cross-model comparisons are confounded by drastically different training budgets (400K-7M steps) and components (VAEs, decoders)). Additionally, confidence intervals for many claimed "reversals" still cross zero, lacking statistical significance.

  2. The "Effective rank" theorem relies on a highly unrealistic "shared affine map" assumption, completely ignoring the non-linearities and attention mechanisms of actual transformers.

  3. The core conclusions are strictly restricted to the SiT-XL/2 architecture, ImageNet-256 dataset, and unguided FID evaluation. Validation on other architectures (like DiT) and guided generation is mising.

Questions for authors

See weaknesses

3 comments

Thank you for your careful review. To address your concerns, we changed the manuscript in four places: random pairs are now the headline uniformity estimator (Table 1; Section 3, l.304–305), and we added analyses of training budget (App. E.1, Table 7), VAE and conditioning (Section 3, l.307–309; App. E.1) and guided FID (App. D.3, Table 6).

W1: Uniformity

The claimed "cross-model reversal" for Uniformity relies on biased in-batch sampling [...]. Authors should use random sampling as the primary metric.

You're right. With random pairs as the headline estimator, uniformity no longer reverses. Its correlation weakens from +0.82 along the training run to +0.50 within families and +0.36 across objectives.

W2: Training budgets, VAEs, and intervals

The cross-model comparisons are confounded by drastically different training budgets [...] and components (VAEs, decoders). Additionally, confidence intervals for many claimed "reversals" still cross zero [...].

The variation in training budget across objectives does not explain the correlations: training budget and FID are essentially uncorrelated (ρ = −0.07), and controlling for log training steps leaves the correlations unchanged or stronger (second row). Within families, all models use the same budget (exactly 400K steps, identical recipe), and the weakening appears there as well.

All models except REPA-E share the SD-VAE encoder. Their decoders (ft-EMA, ft-MSE) are fine-tuned from the same autoencoder, so the representations we measure are unaffected. To rule out remaining differences, we removed REPA-E (own VAE) and REG (image information through its class token) from the cross-objective test (third row):

Linear probeICREff. rankCKNNAUniformityLDSAug. inv.Class scatter
All models (n = 8)−0.79+0.83−0.50−0.38+0.36+0.19−0.02−0.31
Controlled for log steps−0.80+0.86−0.71−0.50+0.50+0.240.00−0.31
Without REG and REPA-E (n = 6)−0.60+0.60−0.14−0.26−0.09+0.20−0.03−0.03

Our findings here are similar: linear-probe accuracy and ICR are the strongest predictors, and the correlations of LDS and uniformity weaken relative to the training run.

Regarding significance: You're right that the sign change itself is not significant. Our central claim is that the training-run correlations overstate how much these properties matter. This is significant: within families, the confidence intervals of LDS and augmentation invariance ([−0.67, +0.32], [−0.46, +0.68]) exclude their training-run correlations (+0.97, −0.95).

W3: Affine-map assumption in Prop. 5.1

The "Effective rank" theorem relies on a highly unrealistic "shared affine map" assumption [...].

Our goal with Proposition 5.1 was to provide intuition on why a link between FID and effective rank exists, not to come up with theory that fully predicts our empirical findings. No empirical conclusion in the paper rests on it. We agree that the assumptions are not realistic and state this as a limitation (Section 7, l.495–496). Such simplifications are standard in related work: Yang & Wang [1] use a linear autoencoder to model a nonlinear U-Net, and Li et al. [2] assume a linear map from hidden features to the output of a simplified U-Net. We revised Section 5 to make this role clearer (l.397–400).

W4: Architecture, dataset, guidance

The core conclusions are strictly restricted to the SiT-XL/2 architecture, ImageNet-256 dataset, and unguided FID evaluation. [...]

Our central finding does not need a test on every architecture: a strong correlation along a training run is not sufficient evidence that a property matters (Section 7, l.480–481). One counterexample is enough to show this, and LDS is the clearest one: along the training run, lower LDS goes with lower FID, but a loss that lowers LDS does not improve FID (Section 4). Hence we propose comparing independently trained checkpoints as an additional, cheap test.

We agree that results for individual properties may differ on other architectures and datasets (Section 7, l.490–492). We chose SiT-XL/2 because it is the standard model in this line of work, so most converged public checkpoints are SiT-XL/2. Further note that SiT uses the DiT backbone. In the training-run correlations where we don't rely on external checkpoints, we also used SiT-B/2 and SiT-L/2 (App. E.3).

We chose unguided FID for an unbiased comparison. Guidance affects the models very differently: at guidance strength 1.5, FID falls by 75% for SiT-XL/2, 64% for REPA and 49% for Self-Flow, and Self-Flow goes from the best of the three models to the worst (App. D.3, Table 6).

We hope this addresses your concerns. If you have any remaining concerns, please let us know.

References

[1] X. Yang and X. Wang. Diffusion Model as Representation Learner. ICCV 2023.

[2] X. Li et al. Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling. NeurIPS 2025.

Thank you again for your careful review. As the discussion period ends soon, we wanted to check whether our response and the revised manuscript address your concerns. If so, we would be grateful if you would consider updating your score. We are happy to clarify anything else before the deadline.

Thank you for the response. However, my concerns remain unresolved, and I maintain my original score.

Reviewer_8SbcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper conducts a systematic evaluation on what properties should have the internal representations of flow models. They studies eight different properties and compute the correlations of these properties with the generative performance measured by unguided FID, across three settings: along a single training pipeline, across multiple model ckpts independently trained, and across models trained with different auxiliary losses that enforce difference properties explicitly. It's not clear what properties are causal for the generative performance.

Critical strengths
  1. The paper runs evaluation on multiple model checkpoints and provides many results.
Critical weaknesses
  1. The presentation of the paper is more like a report that collects all results, rather than an academic paper. The results are hard to parse, and it is hard to understand what is the take away from this analysis paper.

  2. The writing is hard to follow. The presentation is not structured, and not logical. See questions for detailed examples.

  3. The novelty of the paper is not clear. It seems that almost all properties have been studies in previous literature, and have been shown useful for diffusion generative models. Running experiments on a collection of previous literature does not form a contribution by itself.

  4. The criteria to choose the eight properties is not clear, or misjustified. The authors say in line 149 that the properties are those "that representation learning values". There is no citation of representation learning literature supporting these eight properties are indeed valued by representation learning. Furthermore, "Similarity to a pretrained encoder" and "Local spatial structure" are chosen, according to lines 194-201, because they have been shown useful for image generation, not because of representation learning.

  5. Inconsistency in results. Section 3 says uniformity, LDS and augmentation invariance do not correlate as strong as other properties with FID. However, in Section 4, adding these properties as additional auxiliary losses improves FID as shown in Figure 4, 400K steps. It is not explain why and how this is the case.

  6. It is not clear what is the point of the proposition 5.1. See questions below.

  7. The paper only evaluates on one model SiT-XL/2, which is a specific model architecture (flow-based training), and a specific model size. Although authors acknowledged this as a limitation, it is not sufficient for an analysis paper to draw its conclusion only from a single model architecture and a single model size.

Questions for authors
  1. Can the authors explain what is the take away of the paper? How should we design auxiliary objectives for diffusion / flow models for image gneration?

  2. I am confused by the presentation in many places, here is a non-exhaustive list

a) The three settings in abstract is presented in a confusing way.

b) In line 60, The answer to the question "Is emergence during training then a reliable indicator of what matters?" appears before this question.

c) line 76, the properties appear firstly in abbreviations without any explanations.

d) line 79, what is "face value"?

e) line 81, it is not defined what are "auxiliary losses"

f) line 84-85, it is hard to understand the statement in the block without reading the following sections in details.

g) line 101, what are these properties?

  1. Can the authors position their paper better with previous analysis, including analysis done in methodology papers? There is no related work section in the paper, and I strongly suggest the authors to add one.

  2. Can the authors justify the choice of the properties evaluted in the paper?

  3. Can the authors clarify the weakness 5?

  4. It seems that proposition 5.1 does not contribute to the paper's empirical results. What's the purpose of the proposition? How tight is the bound? Does it match the empirical results? Does it establish the theory between FID and erank?

  5. It is strongly recommended to add more model architectures and model sizes.

  6. Why the correlations are computed in absolute value?

3 comments

Thanks a lot for the detailed review and the concrete list of presentation problems.

To speed up training and improve generation quality, several recent works add losses on the hidden representations of the model, in addition to the usual flow-matching loss. These losses are usually motivated by showing that some property of the representations (like linear separability) correlates with FID along a training run. We show that this is an unreliable test. Several of these properties correlate strongly with FID along a single training run, but the correlation disappears or even reverses when we compare independently trained models. We therefore propose as a better test to check how a property behaves across converged checkpoints. Some properties that were thought to be promising, such as LDS and augmentation invariance, do not pass it, and in our runs, auxiliary losses on them give no clear gain in FID.

Scope of the paper (weaknesses 1, 2 and 3, questions 1 and 2)

We hope that this summary clarifies the scope of the paper and its main takeaway (weakness 1 and question 1).

Regarding novelty (weakness 3), it is true that the properties we study have been studied or used by prior works, but this is exactly the point of the paper. We want to study representation-shaping objectives in diffusion models, so we take the properties that people use there. The analysis itself, comparing these properties across independently trained models, to our knowledge, has not been done systematically before. This is also why we chose exactly these eight properties (weakness 4 and question 4).

Clarity (weakness 2, questions 2 and 3) Because this is an analysis paper, we have to cover several properties and methods, each of which was a separate contribution of its own. We see that this can be overwhelming. But given that several reviewers point out clarity, we acknowledge this as a weakness of the submitted version. In the revision, we have made several changes to improve clarity. We now explicitly number the three settings in the abstract, and we name all eight properties in words in the introduction. We also state the scope and main takeaway of the paper more explicitly. We think with these changes, the paper is clearer and easier to read.

Inconsistency in results (weakness 5, question 5) You noted that in Figure 4, the auxiliary losses on LDS and augmentation invariance end with a lower FID than the control at 400K, although Section 3 finds that these properties do not predict FID well across models. This is only because the FID of the control spikes at the end of training. For most of training, up to 300K steps, both auxiliary-loss runs have a worse FID than the control.

The control in the submission was our vanilla training run, while the auxiliary-loss runs branch off it at 25K steps with a restarted optimizer. For a fairer comparison, we trained a new control that branches off in the same way, without an auxiliary loss. It has no spike and ends at nearly the same FID as both auxiliary-loss runs, so neither loss gives a clear gain. The revised Figure 4 uses this control. The same holds with the losses at block 12, where Section 3 measures the properties.

Purpose of Proposition 5.1 (weakness 6, question 6) The proposition gives intuition for why effective rank stays positively associated with generation quality in all three settings. A flow-matching loss close to its minimum bounds the Wasserstein distance between generated and data distribution, which keeps FID small (Section 5, l.397–400). Under simplifying assumptions, the same small loss requires an effective rank above a threshold. This necessary condition does not imply that raising effective rank improves FID, and we do not claim that the bound is tight.

Architectures and sizes (weakness 7, question 7)

  1. To reject a hypothesis (here, that correlation along a training run is sufficient evidence for a property being necessary for good generation) a single counterexample suffices. As the scope of the paper is to provide a test to show that a property is not necessary (rather than showing that it is necessary) focusing on one architecture is sufficient.
  2. More architectures and sizes would be desirable, but they are hardly feasible and would make the analysis complex. Our comparisons rely on released, converged checkpoints, which do not exist for other architectures to the same extent as for SiT. Finally, we study the properties that prior work on representation alignment uses, and almost all of this work uses SiT.

Absolute correlations (question 8) All correlations in the paper are signed, and sign reversals are defined on the signed values (Appendix C.6). We use absolute values only where we compare the strength of correlations with different signs (line 211).

We hope this addresses your concerns and are happy to answer remaining questions.

Thank you for the rebuttal!

The empirical analysis is limited and restricted to a very specific setup: previously trained checkpoints of a specific model (SiT with SD-VAE) with certain amount of differences that the authors cannot control (like decoder, optimizer, etc.), and the selected representation layers. The authors argued that focusing on this certain type is sufficient to prove that the properties are not universally necessary for image generation training. This is logically valid for refuting a universal necessity claim. However, I do not think it addresses the concerns of the limited generality of the empirical findings. For example, they offer limited insight into when or why these properties are unnecessary, or whether the findings extend to other architectures, encoders, and generative modeling approaches.

Overall, I acknowledge the value of these observations. Nevertheless, I remain unconvinced that the experimental evidence supports a sufficiently broad or substantial contribution. I remain leaning towards reject. Thank you again for your rebuttal.

Thank you very much for your response!

We do agree that with our analysis, we cannot conclude that a property of a model's internal representation is (un)necessary for good generative performance in general (i.e. for different architectures, generative methods, etc.). The scope of the paper was to study this specifically for diffusion transformers.

Our argument to why this restriction is reasonable is that i) diffusion models with transformers are currently SOTA in generative modeling, and basically all relevant representation alignment methods for diffusion models use the transformer architecture, and ii) without such a restriction, it would be very hard to perform a similar analysis. It would not even be clear how to obtain comparable representations across different architectures (e.g. in transformers, we pool over the sequence dimension, while in convolutional models, we'd pool over spatial dimensions).

The methodological finding is general. A strong correlation along a training run is used as evidence that a property matters, and we show that this evidence can mislead in the standard setup where it is used. That makes it unreliable as a selection criterion in general. As a better criterion, we propose to check whether a property still predicts generation quality across independently trained models, which only requires forward passes on existing checkpoints.

The second experiment with iREPA/REPA models is a controlled experiment. These models share the decoder, the optimizer, the training budget and the layer at which the loss is applied, and only the alignment encoder varies. Some difference in how the models are trained is unavoidable, since otherwise they would not differ in generation quality. Our central findings appear in this controlled setting.

The comparison across objectives then shows that these findings are robust to larger differences between models. In the revision, we explicitly check the differences you mention. Controlling for training budget and excluding REG and REPA-E leave the main findings unchanged (App. E.1, Table 7).

The findings also do not depend on the chosen layer. The weakening for LDS and augmentation invariance appears at most layers and noise levels we test (App. E.1, Figures 5 and 6).

We hope this clarifies the scope of our contribution, and we thank you again for the discussion.

Reviewer_Se6cOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates which properties of diffusion transformer representations are genuinely informative for generation quality. The authors measure eight representation properties and study their relationship with generative performance in three settings. They find that many properties correlate strongly with FID during training, but several lose predictive value or even reverse sign across independently trained models.

Critical strengths
  1. The paper provides a relatively systematic comparison of representation properties, rather than building its case around a single favored metric. The authors evaluate eight representation properties in a unified framework and compare them across three complementary settings: a single training trajectory, independently trained models within the same method family, and models trained with different auxiliary objectives. This breadth is consequential because it reveals that many properties look similarly compelling along training, yet separate sharply once independently trained models are compared.

  2. The paper delivers a useful methodological correction to a common line of argument in the recent literature. Another one contribution is the empirical demonstration that correlation with FID along a single training run is not, by itself, reliable evidence that a representation property is worth targeting with an auxiliary objective. Given how often such arguments are used to motivate representation-shaping methods, this is a meaningful takeaway.

Critical weaknesses
  1. The paper remains fundamentally an observational screening study, rather than an identification of the intrinsic factors that drive generation quality. Its main evidence comes from whether a property co-evolves with FID during training and whether it remains predictive across trained models. This is useful for screening which metrics are more robust indicators, but it does not establish that the retained properties are causal drivers or intrinsic determinants of good generation. In that sense, the paper is better understood as identifying which representation measures track strong models more reliably, rather than what truly matters for generation in a mechanistic sense.

  2. The framing is broader than what the empirical study directly supports. The title and motivation suggest an answer to what matters for generation quality and, by implication, for improving diffusion training. In practice, however, the paper mainly performs post hoc representational diagnostics on existing checkpoints, rather than systematically studying how these properties affect optimization, convergence, or sample efficiency. As a result, the work is more convincing as an analysis paper about evidence quality than as a direct account of training improvement mechanisms.

  3. The constructive payoff is limited: the paper is stronger at ruling out weak evidence than at establishing what should be optimized instead. The paper makes a persuasive negative case that correlations along a single training run are insufficient, and the interventions on LDS and augmentation invariance reinforce this point. However, it does not provide equally strong positive validation, within a unified experimental framework, for the properties that appear more robust across models. This leaves the paper with stronger negative methodological guidance than positive design guidance.

  4. The treatment of generation quality remains relatively narrow. Generation quality is inherently multi-dimensional, potentially involving fidelity, diversity, coverage, and semantic consistency. Although the appendix includes sFID, IS, precision, and recall, the main analysis and conclusions remain centered on FID and closely related score-based metrics. The paper therefore more clearly establishes which properties correlate with these evaluation scores than which properties matter for generation quality in a broader sense.

Questions for authors
  1. Clarification on the motivation. Because the paper is framed partly around what matters for generation quality and diffusion training, it would help to clarify whether the intended contribution is primarily an analysis of evidential methodology, or whether the authors believe the current experiments already support conclusions about optimization behavior, convergence, or sample efficiency.

  2. Clarification on constructive implications. Do the authors have any additional evidence, beyond the LDS and augmentation-invariance interventions, that the properties with more stable cross-model associations (e.g., linear-probe accuracy, effective rank, ICR, or CKNNA) are better targets for auxiliary objectives in a controlled setting? This would materially strengthen the paper’s practical takeaway for objective design.

  3. Clarification on generation-quality scope. Given that the main narrative is centered on FID, how do the authors view the extent to which their conclusions should generalize to broader notions of generation quality beyond score-based image metrics? A clearer discussion here would help calibrate the scope of the title and conclusions.

2 comments

Thanks a lot for the very detailed review and for agreeing that it is important to systematically study properties that are used to motivate representation-shaping objectives. Our main contribution is an analysis of how properties of hidden representations in diffusion transformers correlate with generative performance across independently trained models. The main conclusion is that the usual heuristic of checking correlation along a single training run is not sufficient. Instead, we propose checking correlation across independently trained models using converged checkpoints as a better test.

Regarding weaknesses 1, 2 and 3 and questions 1 and 2 (Scope of the paper) We fully agree that the paper is more convincing as an analysis paper, and this was actually our intention.

It is also true that our experiments can rule out candidate properties to optimize rather than give evidence that a property is necessary. However:

  1. Showing that a property is necessary for good generation is intrinsically hard.
  2. A framework that is good at ruling out is also valuable. Given a set of candidate properties, knowing which ones are not promising is much more useful than knowing nothing. To our knowledge, this is the first systematic comparison of many such properties across independently trained models. LDS (local spatial structure of the features) and augmentation invariance both correlate strongly with FID along the training run (this was the kind of evidence used to motivate them), but within families their correlations weaken significantly, and auxiliary losses on them do not improve FID (Section 4).

Regarding positive results (weakness 3 and question 2) Properties that survive the test can indeed be good candidates. Linear-probe accuracy, for instance, passes the test most clearly. During the rebuttal, we trained SiT-XL/2 from scratch with the LSEP loss (Yun et al., 2025) that optimizes linear-probe accuracy. This lowers FID-50K at 400K steps by 28%. We did not include such a result in the submission, because (as you noted) the main contribution of the paper is the analysis and a way to refute candidate properties, not a new method to improve generation. However, we agree that it is good to see that properties that survive the test can indeed be promising candidates. We now include this result (Section 4, l.362–364; App. D.5). Thank you for the suggestion!

Regarding weakness 4 and question 3. (Evaluation beyond FID) We fully agree that metrics like FID evaluate only part of generation quality and only partially agree with human judgment. Beyond FID, we also report sFID, Inception Score, precision and recall in the appendix (Appendix E.2). The pattern is the same under each of them. We now state this in Section 3.

Fundamentally, we evaluate the properties that the field uses against the standard evaluation metrics that the field uses. Better evaluation metrics are of course highly desirable, but developing them would be a separate project, beyond the scope of this paper.

We hope this addresses your concerns. If you have any remaining concerns, please let us know.

Thank you again for your detailed review. As the discussion period ends soon, we wanted to check whether our response and the revised manuscript address your concerns. If so, we would be grateful if you would consider updating your score. We are happy to clarify anything else before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.