acceptodds
Under review as a conference paper at ICLR 2027

Purify Before You Align: Improving Representation Alignment for Diffusion Models

Abstract

Representation alignment improves and accelerates diffusion training by encouraging intermediate features to match a pretrained visual encoder. Yet choosing the right encoder remains largely empirical: DINOv2 provides substantial gains, whereas MAE, despite retaining more reconstructive information, offers little benefit. We introduce local appearance purification (LAP), which improves generation quality and makes encoders such as MAE effective alignment targets by removing locally predictable content and aligning to the residual. LAP modifies only the alignment target and requires no changes to diffusion training or inference. Through controlled interventions within a fixed encoder, we show that effective alignment depends on what information the target emphasizes, not simply how much image information it contains. Our theory interprets standard alignment as maximizing a variational lower bound on the information that the denoiser's hidden state carries about the target, with purified alignment yielding a corresponding bound conditioned on clean latent patches. We further derive an exact constant-readout baseline that measures how well the target can be matched without using the hidden state. LAP improves generation across five pretrained representations, with the largest gains for MAE: on ImageNet-256 with SiT-B/2, it reduces FID from to . Its gains persist when combined with stronger alignment recipes and tokenizers. Across 13 encoders, the constant-readout score predicts the benefit of purification before diffusion training (Spearman ).

Primary area:
generative models
Keywords:
Representation Alignment, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_GDr4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies which components of a frozen teacher representation help diffusion-model alignment. Latent-Aware Purification (LAP) subtracts a component predictable from clean latent patches before using the residual as an alignment target. The paper considers constant, linear, and nonlinear predictors, develops a conditional-information interpretation, and proposes a target-concentration statistic, C0, as a diagnostic. Fixed-teacher interventions, experiments with multiple teachers and alignment recipes, tokenizer variations, larger models, and aerial-image datasets support the central finding: making teacher features more directly reconstructive need not improve generation, while removing predictable components can substantially help some teachers.

Critical strengths
  1. The strongest contribution is the fixed-teacher experimental analysis. The predictable-only DINOv2 target yields FID 56.71, worse than the no-alignment baseline at 53.92, while the residual target reaches 41.47, close to raw-teacher alignment at 41.75 (Figure 2 and Table 21). Separately, the auxiliary-prediction experiment reports velocity-probe R-squared increasing from 0.115 to 0.534 (Table 5), while generation FID worsens from 41.75 to 46.79 (Table 21). These interventions meaningfully challenge a simple reconstruction-based explanation. The appendices further examine centering, spatial effects, and fixed versus redrawn predictable targets, substantially reducing easy alternative explanations.
  2. LAP is simple and produces large improvements where the proposed diagnosis predicts a problem. In Table 1, MAE alignment improves from 53.05 to 42.30 FID with LAP-N; the SiT-L/2, batch-256 MAE experiment in Table 17 improves from 15.39 to 12.06. Compositions with several alignment recipes and tokenizer settings broaden the evidence. Reporting weak or negative results for already effective teachers, transfer to RESISC45 and AID in Table 4, and limitations of the information-theoretic interpretation makes the overall argument more credible.
Critical weaknesses
  1. The practical generality is weaker than an unqualified reading of the headline suggests. Large gains are concentrated in high-concentration targets such as MAE, while DINOv2 improvements are small and sometimes negative. For example, the block-4 experiments in Table 18 give 36.28 versus 36.79 for raw versus purified DINOv2 REPA, and 27.05 versus 27.23 for REG. Table 2 also selects the best LAP variant per setting; moreover, its iREPA uses a depth-matched projector and its VA comparison isolates the alignment schedule rather than the full Stable Velocity objective. These are disclosed, but the main presentation should distinguish composability in these controlled implementations from superiority over each complete published method.
  2. C0 and the conditional-information argument have a more limited explanatory scope than the core intervention results. The 13-encoder study is encouraging, but encoders share families and a common training recipe. In the external iREPA analysis for SiT-XL/2 at 100K steps, the partial rank correlation with improvement after controlling for raw FID is -0.13, so C0 is not yet a generally calibrated predictor of benefit. Likewise, the conditional bound may become uninformative when the full clean latent grid is highly informative, and raw targets also admit a conditional bound. The theory motivates purification but does not establish that the removed component is a causally distinct nuisance factor.
Questions for authors
  1. Clarification: Can the main tables explicitly identify the adapted iREPA/VA implementations and distinguish best-per-setting LAP selection from a fixed selection rule? Do the conclusions remain similar when the variant chosen in the default setting is carried unchanged to the block-4 and larger-model experiments? This would determine how broadly the practical recommendation should be stated.
  2. Analysis request: Does the C0-benefit relationship retain its predictive ordering when an entire encoder family is held out, rather than another size of a family already present in the sweep? Given the external partial-correlation result, evidence that the diagnostic transfers to a held-out family or alignment recipe would materially strengthen the claim that it can guide teacher selection.
  3. Can you evaluate LAP on a text-to-image setting such as CC12M, LAION, or COCO with a text encoder (e.g., T5, CLIP, or Gemma) and a DiT/SiT/PixArt-style backbone?
3 comments

Thank you very much to the author for revising some of the expressions and adding some experiments, including the results on the text-to-image public MS-COCO dataset. However, I don't see the revised manuscript at the moment. I will decide whether to raise score based on whether the revised manuscript is consistent with the response.

Dear Reviewer GDr4,

Thank you so much for your prompt reply and positive feedback! We've revised the manuscript following your comments and updated the PDF accordingly. The revision was marked in blue. Please check out and let us know if there's any other comments. We would greatly appreciate your consideration of a higher score.

Thank you again for your time and we are always happy to address any questions you may have!

All my problems are solved. And the response is consistent with the updated manuscript. So I raise my score to 4.

Reviewer_B6uqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what makes a pretrained encoder an effective target for REPA-style recipe, motivated by the observation that MAE features provide little benefit despite predicting clean-image content well. Through fixed-encoder interventions on DINOv2, the authors show that aligning only to the component predictable from clean latent patches hurts generation while the residual retains the benefit. Based on this, authors propose LAP, which subtracts a local prediction of the target from clean latent patches (via a constant, affine, or small CNN predictor) and aligns to the residual. They also provide a variational interpretation of alignment and a constant-readout score C0, which measures how much cosine alignment is attainable without using the hidden state. Experiments across different models, encoders, and recipes shows correlation between the gain and C0 score which particularly improves MAE in various settings.

Critical strengths
  • The paper analyzes for which targets REPA works well and when its benefit becomes marginal, using fixed-encoder interventions which separate locally predictable content from the residual. The effectiveness of alignment depends on what information the target emphasizes rather than how much image information it contains, is valuable for understanding REPA-style methods.

  • The claims are clearly stated and well matched by the experimental results across various setups - including different teachers, alignment recipes, tokenizers, etc. The combination of clear claims and supporting experiments makes the findings feel solid.

Critical weaknesses
  • As the authors’ own pre-specified prediction anticipates, the gains are concentrated on teachers with high C0, while the improvements for commonly used teachers (e.g., DINO) are within seed variation. The practical value of LAP therefore lies mainly in settings where only weaker alignment targets such as MAE are available, for example domain-specific imagery. However, this scenario is motivated but not directly tested: the aerial experiments transfer ImageNet-pretrained teachers rather than using teachers pretrained on in-domain data. Demonstrating a benefit with an in-domain teacher would considerably strengthen the practical case for the method.

  • Most of the MAE gain appears to come from removing the shared mean direction: LAP-C alone accounts for about 73% of the improvement (53.05 → 45.17 for LAP-C vs. 42.30 for LAP-N; Table 1). The authors also note that under iREPA-style operations, LAP-C has no effect because per-image spatial standardization already removes the batch-mean token (Table 21). Since C0 measures this shared direction, its ability to predict the gain may partly follow from its definition. As a result, the practical contribution of LAP seems closer to a form of target normalization, and its distinct benefit beyond existing operations could be clarified further.

Questions for authors
  • Since the main practical motivation is domains without suitable discriminative teachers, could the authors evaluate LAP with a teacher pretrained on in-domain data?

  • The iREPA results in Table 2 are obtained with the authors’ own variant of iREPA, rather than the original one. Since the distinct benefit of LAP is best judged against the published method, could the authors report results with the original iREPA? In addition, given that LAP-C has no effect when per-image spatial normalization is applied, what does LAP-N remove beyond the shared mean direction?

2 comments

Dear Reviewer B6uq,

Thank you so much for your helpful suggestions and constructive feedback! We've revised the manuscript following your comments and updated the PDF accordingly. The revision was marked in blue. Please check out and let us know if there's any other comments. We would greatly appreciate your consideration of a higher score.

Thank you again for your time and we are always happy to address any questions you may have!

Thank you for the detailed response and the additional experiments. The response answers most of the questions but there remain limitations which are clearly acknowledged. Hence, I decided to keep my score.

Reviewer_gcz1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the teacher representation and the image information it contains affect the alignment quality in diffusion training. Through fixed-encoder interventions, the authors show that locally predictable content can hurt generation. Thus, they propose Local Appearance Purification (LAP), where the locally predictable content is subtracted from the teacher feature and only the residual is used in alignment training. On ImageNet-256 with SiT-B/2, LAP improves MAE-based FID from 53.05 to 42.30 and demonstrates compatibility with other alignment recipes and tokenizers.

Critical strengths
  1. Section 3.2 modifies the teacher feature from raw DINOv2. The alignment results from the raw teacher, the predictable component and the residual show substantial differences. This supports a valuable conclusion: teacher content that is easy to predict or match need not provide useful generative supervision.
  2. LAP-N improves MAE-based FID from 53.05 to 42.30 without adding sampling-time computation. Its compatibility with other alignment recipes and tokenizers strengthens its practical relevance.
Critical weaknesses
  1. For MAE, centering already improves FID from 53.05 to 45.17, with LAP-N further improving it to 42.30. If simple geometric transformations can recover such additional benefit, the distinct value of local prediction as the central method would be reduced.
  2. In Table 1, the best mean improvements for DINOv2 and DINOv3 are only approximately 0.28 and 0.66 FID. Some variants of LAP worsen performance. These issues do not undermine the substantial MAE gains, but they affect the significance of LAP as a general alignment improvement.
Questions for authors
  1. In Table 1, LAP-C and LAP-N achieve FIDs of 45.17 and 42.30 with MAE, demonstrating a benefit from nonlinear local prediction beyond centering. How do the authors interpret this additional gain? Which existing results support the explanation that removing locally predictable content is responsible, beyond showing that residualization improves the geometry or optimization properties of the alignment target?
  2. Table 2 explicitly reports the LAP variant with the best FID for each configuration. Please clarify whether each variant was selected using the mean FID across the two seeds, and whether the unselected variants also show consistent improvements. In particular, for the approximately 0.3–1.2 FID gains with DINOv2, do the results primarily establish that at least one helpful variant exists for each recipe, or do they support broader robustness across variants?
3 comments

Dear Reviewer gcz1,

Thank you so much for your helpful suggestions and constructive feedback! We've revised the manuscript following your comments and updated the PDF accordingly. The revision was marked in blue. Please check out and let us know if there's any other comments. We would greatly appreciate your consideration of a higher score.

Thank you again for your time and we are always happy to address any questions you may have!

solid work. And the RBT solved my concerns; I increased my score to 4. good luck

Thank you so much for your positive feedback!! We are pleased that the additional experiments have addressed your concerns. Your suggestions really helped us strengthen the evaluation and improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.