acceptodds
Withdrawn from ICLR 2027

What Visual Generators Need from Teachers: Rethinking Representation Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation alignment accelerates diffusion transformer training by pulling an intermediate block of the model (student) toward features of a frozen pretrained encoder (teacher). However, which teacher layer to align with, and for how long, are still chosen largely by convention, with each alternative requiring a separate training run. We find that alignment helps where the student cannot linearly recover the teacher's features, rather than where the two representations already resemble each other. Since a deep teacher layer is largely predictable from the layer below it, we isolate what each layer adds—its increment—and measure how much of that increment an unaligned student recovers. The student fills the teacher's hierarchy from the bottom up and stalls near the top, a phenomenon we call hierarchy filling: even after 400K training steps, it recovers almost none of the deepest increments. We define the recoverability gap as the unrecovered share of an increment, which can be estimated from a single unaligned checkpoint. Across short runs that each align one teacher layer at one student block, the recoverability gap nearly reproduces the ranking of teacher layers by FID improvement, while CKA, a measure of feature similarity, largely reverses this ranking. Based on these findings, we introduce Representation Alignment and Recoverability Estimation (RARE). Before training, RARE selects the teacher layer with the largest recoverability gap. During training, it tracks each token's remaining distance to that layer as an online counterpart of the gap, weights tokens according to this distance, and phases out the alignment loss once the average distance stops decreasing. With SiT-B/2 on ImageNet 256×256, RARE achieves an FID of 18.02 without classifier-free guidance and 4.46 with guidance, outperforming seven representation-alignment baselines, including REPA, iREPA, and HASTE. RARE also requires 14% fewer GPU-hours than iREPA. Its FID remains lower than iREPA's across model scales, teacher encoders, datasets, and diffusion backbones.

Primary area:
generative models
Keywords:
generative models, diffusion transformers, representation alignment, recoverability gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uDixOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to select appropriate teacher representations and determine the duration of representation alignment when training diffusion transformers. The authors observe a phenomenon termed hierarchy filling, where an unaligned generative model can linearly recover information contributed by shallower teacher layers but struggles to recover the incremental information from deeper layers. Based on this observation, they propose the recoverability gap, which measures the unrecovered portion of a teacher-layer increment and serves as a criterion for selecting alignment targets. The proposed RARE framework further introduces residual-based token weighting and an adaptive schedule that gradually releases alignment supervision when its residual improvement plateaus.

The experiments demonstrate a strong correlation between the recoverability gap and alignment benefit across different diagnostic configurations, outperforming several alternative selection criteria. RARE also achieves competitive improvements over existing representation alignment methods, across different architectures and pretrained teachers. Overall, the paper offers an interesting empirical perspective on representation alignment and an effective approach to improving the efficiency of generative model training.

Critical strengths

The paper goes beyond conventional representation similarity measurements to investigate what information the student can recover from different teacher layers. The observation of hierarchy filling, together with the proposed recoverability gap, provides an interesting perspective on when external representation supervision may be useful.

In particular, the reported Spearman correlation of 0.92 between the recoverability gap and alignment benefit across 14 diagnostic configurations (Table 1 and Figure 3) provides encouraging evidence for the proposed criterion. The comparison against similarity-based measures and alternative selection criteria further strengthens this empirical finding. This is a meaningful contribution because identifying useful alignment targets without extensive training searches remains an important practical problem.

The proposed RARE framework demonstrates consistent improvements over several established representation alignment methods. On ImageNet-256 with SiT-B/2, RARE achieves an FID of 18.02 without classifier-free guidance and 4.46 with guidance, while reducing GPU-hours by approximately 14% compared with iREPA.

The evaluation covers multiple model scales, pretrained teachers, image domains, and generative architectures. The ablation studies also investigate several design components, including target selection, token weighting, and alignment release. These experiments provide useful evidence for the practical effectiveness of the proposed training recipe, independently of whether its broader mechanistic interpretation fully generalizes.

Critical weaknesses

  1. The relationship between linear recoverability and useful missing knowledge remains insufficiently established.

The central motivation assumes that information which cannot be linearly recovered from the student's intermediate representations constitutes a useful knowledge gap that should be addressed through teacher alignment.

However, low linear recoverability does not necessarily indicate that information is absent from the student. The information may be encoded in a nonlinear form or remain accessible to subsequent layers. Furthermore, information that is difficult to recover is not necessarily beneficial for the generative objective. Its usefulness may also depend on the relevance and compatibility of the teacher representation with the student's generative task.

Although the strong empirical correlation supports the proposed metric as a useful predictor within the evaluated configurations, it does not fully establish the broader interpretation that unrecoverable teacher information represents the knowledge most needed for generative training.

This distinction matters because it affects the generality of the paper's central scientific claim. A clearer discussion of linear recoverability as an operational diagnostic, rather than a direct measurement of missing knowledge, would strengthen the contribution. Additional comparisons with nonlinear probing or teacher targets of different generative relevance would further substantiate the interpretation.


  1. The experiments do not fully isolate the contribution of recoverability-guided target selection from the other improvements in RARE.

An important contribution of the paper is the use of the recoverability gap to identify the most beneficial teacher layer. However, on the primary ImageNet benchmark, RARE selects Layer 12, which is also the default alignment target used by REPA and iREPA. Consequently, the improved performance on this benchmark does not directly demonstrate the advantage of recoverability-guided teacher-layer selection.

Moreover, the ablation study in Table 3 indicates that releasing alignment supervision and introducing residual-based token weighting contribute substantially to the performance improvements. The results therefore establish the effectiveness of the combined training strategy more directly than the individual advantage of the proposed selection criterion.

While the diagnostic correlation experiments provide encouraging evidence, a more direct evaluation of gap-guided selection against fixed teacher-layer choices under an otherwise identical RARE training procedure would help isolate and validate this contribution. This would also clarify when and why recoverability-guided selection provides advantages beyond existing alignment strategies.

Questions for authors

  1. Linear recoverability and missing knowledge.

The recoverability gap is defined through linear readouts from the student's hidden representations. To what extent does the observed hierarchy filling reflect information genuinely unavailable to the student, rather than information encoded in a form that is not linearly accessible? Have the authors examined whether the hierarchy and its correlation with alignment benefit remain consistent under more expressive readouts?

Clarifying this point would help establish whether the metric captures a broader knowledge deficiency or a particular form of representational mismatch.


  1. Is missing information necessarily beneficial for generation?

The proposed criterion primarily measures how much of the teacher-layer increment the student cannot recover. However, the potential benefit of alignment may also depend on whether this information is useful for the generative objective.

Could the authors clarify how the recoverability gap distinguishes beneficial missing information from information that is difficult to recover but less relevant to generation? For example, would teacher targets with similar recoverability gaps but different generative relevance be expected to provide comparable improvements?


  1. Isolating the contribution of teacher-layer selection.

Since RARE selects the same teacher layer as the existing baselines on the primary ImageNet benchmark, could the authors provide or clarify comparisons that isolate the contribution of recoverability-guided target selection while keeping the alignment loss, token weighting, student block, and release schedule fixed?

In particular, the Places365 experiment, where RARE selects Layer 9 instead of the default Layer 12, appears to be a useful setting for this comparison.

Reviewer_Am7SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RARE, a representation-alignment method that selects a teacher layer according to how poorly an unaligned diffusion model can linearly recover its incremental features. Training then aligns against the selected whole-layer representation, using token-dependent weights and adaptive early termination. The results show some improvements over previous works.

Critical strengths

1/ The question of which teacher features provide useful supervision is relevant, and the proposed teacher-layer selection criterion offers an interesting perspective.

2/ Experiments cover several architectures, teachers and datasets, with no additional inference cost.

Critical weaknesses

The main results do not establish the value of the central contribution. On ImageNet, RARE selects the same teacher layer and student block as REPA/iREPA, so its improvement does not demonstrate a benefit from layer selection. The criterion largely favors deeper teacher layers and cannot reliably distinguish its strongest candidates or choose the student block. More fundamentally, teacher features that are difficult to predict are not necessarily useful for generation. Some may be irrelevant or counterproductive, and the student could benefit from discarding them. The criterion does not explicitly distinguish such features from useful missing information. Meanwhile, the reported gains come from changes to the training recipe, including early stopping already established by HASTE. Token weighting and the stopping schedule are also changed together, leaving their individual contributions unclear. What the selection mechanism adds beyond a simpler alignment recipe therefore remains unestablished.

The experimental evidence and comparisons are insufficient. Larger-model studies stop at 100K steps, and final comparisons use one training seed. Longer scaled experiments and uncertainty across training seeds are needed to establish reliable gains. Important simple alternatives are also missing: aggregating several deep teacher layers, motivated by RAEv2 and REGLUE, and selecting targets through depth-based or low-similarity rules. These comparisons should use the same target dimension and training recipe to isolate the value of the proposed criterion. Without them, it is unclear whether the method offers a robust advantage over straightforward alternatives.

The writing makes a relatively simple training recipe unnecessarily difficult to understand. The paper lacks clarity and the writing should be improved.

Questions for authors

See Weaknesses for the most part.

  • How exactly are features whitened and reduced to 128 dimensions? Please specify how the projection matrix and retained subspace are computed. Appendix A states the transformation but leaves its actual construction unclear.

  • Can token weighting and early stopping be ablated independently?

Reviewer_5gpLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to select useful teacher alignment targets based on the representations a generative model has already learned. The authors decompose each teacher layer into an inherited component that can be linearly predicted from the preceding layer and an incremental component that cannot, then use linear readouts from a frozen student to measure how much of each layer's increment can be recovered. The experiments show that an unaligned student develops linearly readable representations of shallow teacher increments relatively early, whereas deep increments remain difficult to recover within the tested training budgets. The authors call this phenomenon "hierarchy filling." Based on this observation, the paper introduces a recoverability gap and tests its ranking relationship with alignment benefit using short, controlled training branches. The resulting method, RARE, selects a teacher layer using this gap and combines alignment-residual-based token weighting with an adaptive phase-out of the alignment loss. Evaluations on ImageNet and Places365, across model scales, teachers, and backbones, report improvements in generation quality and training cost.

Critical strengths
  1. Analyzing alignment needs in terms of the student's existing representations offers explanatory value and is supported by targeted experiments. A deep teacher layer can retain substantial shallow information, so directly comparing whole-layer representations can obscure the student's lack of newly introduced deep-layer content. The paper makes this issue measurable through increment decomposition and held-out linear readouts. Section 4.2 further examines the relationship between the gap and realized alignment benefit using 14 controlled training branches, reporting a Spearman correlation as high as 0.92 and including within-layer comparisons, content controls, and gradient-signal controls. These experiments provide meaningful empirical evidence that alignment benefit is related to content the student has not yet acquired.

  2. The method has practical value, with experiments spanning multiple settings and reporting efficiency gains. In the main ImageNet-256 experiment with SiT-B/2, RARE achieves an unguided FID of 18.02, improving over iREPA's 20.40 by a moderate margin. The paper also reports approximately 14% fewer training GPU-hours, without additional modules at inference. The authors evaluate different student scales, teachers, datasets, and backbones, and report multiple metrics beyond FID. These results support the value of further studying and using this training recipe.

Critical weaknesses
  1. The connection between the core gap analysis and the final method's gains has not been sufficiently isolated, limiting the assessment of the added contribution. In the main ImageNet experiment, the gap selects teacher layer 12, which is also the default target of REPA and iREPA, while the student alignment location is inherited from the baselines. Section 4.2 also explicitly states that the gap cannot effectively select the student block. Thus, evidence that the overall training recipe works cannot yet be directly attributed to gap-guided layer selection. In Table 3, the whitened target with always-on alignment obtains an FID of 20.40; stopping at a fixed 250K steps gives 18.83; token weighting with a manually scheduled phase-out gives 17.85; and full RARE gives 18.02. These results indicate important roles for early stopping and the combination of weighting and phase-out, but the adaptive strategy's value over the manual schedule is primarily earlier termination and lower cost; an additional benefit in final quality has not been established. Given that HASTE already includes early termination of alignment and RARE inherits iREPA's convolutional projector and spatial normalization, isolating the independent contribution of gap-based layer selection under an otherwise identical training recipe is particularly important. On Places365, the selected layer changes from the default layer 12 to layer 9, but I did not find a direct comparison that changes only these two teacher layers while holding the other settings fixed. Moreover, the offline gap measures linear recoverability of teacher increments, whereas online weighting and stopping use the cosine residual between convolutionally projected features and a whole-layer target. Their connection as offline and online measurements of the same gap still lacks direct validation.

  2. The gains over key competing methods are small, and the lack of experiments with multiple training seeds leaves insufficient statistical evidence for consistent superiority over the strongest baseline. Section 6.1 states that the main settings use only one training seed per method. In Table 2, RARE's unguided FID is 18.02 versus HASTE's 18.99, a difference of only 0.97. In Table 4, the differences relative to iREPA are 0.45 on SiT-XL/2 and 0.38 in the unguided Places365 setting. The paper itself cautions that FID differences below approximately 1.5 should primarily be interpreted as directional evidence. Although the reported results show a fairly consistent direction of improvement, it remains unclear whether the smaller gains are stable. Some scale and teacher transfer experiments also stop at 100K steps, restricting these conclusions mainly to limited training budgets. Multiple-seed results for the key RARE, HASTE, and iREPA comparisons in the main experiment would directly affect my assessment of whether the method maintains its relative advantage.

Questions for authors
  1. How much does gap-based layer selection contribute independently? On Places365, could the authors compare the gap-selected layer 9 with the default layer 12 while fixing whitening, the student block, the projector, token weights, and the stopping policy? Are there other settings where the gap selects a different layer from the default and the benefit has been verified through such a controlled comparison? This evidence would help distinguish the value of the selection criterion from that of the other training changes.

  2. Is the improvement over the strongest baseline statistically robust? Are multiple-training-seed results available for RARE, HASTE, and iREPA on ImageNet-256 with SiT-B/2 at 400K steps? If so, please report the mean and standard deviation of FID, along with the differences in individual runs. If not, could this key comparison be added? This would help assess whether the 0.97 FID improvement over HASTE is stable and inform my view of the smaller advantages in the transfer experiments.

  3. To what extent does the online cosine residual reflect the offline recoverability gap? The former uses a trainable convolutional projector and a whole-layer target, while the latter uses a linear readout and teacher increments. Could the authors show how the two quantities evolve during training at the same teacher layer and student block, and explain how improved fitting by the projector itself can be distinguished from improved student recoverability of teacher increments? This would affect the assessment of whether the gap analysis directly supports the token-weighting and stopping criteria.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.