- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to select appropriate teacher representations and determine the duration of representation alignment when training diffusion transformers. The authors observe a phenomenon termed hierarchy filling, where an unaligned generative model can linearly recover information contributed by shallower teacher layers but struggles to recover the incremental information from deeper layers. Based on this observation, they propose the recoverability gap, which measures the unrecovered portion of a teacher-layer increment and serves as a criterion for selecting alignment targets. The proposed RARE framework further introduces residual-based token weighting and an adaptive schedule that gradually releases alignment supervision when its residual improvement plateaus.
The experiments demonstrate a strong correlation between the recoverability gap and alignment benefit across different diagnostic configurations, outperforming several alternative selection criteria. RARE also achieves competitive improvements over existing representation alignment methods, across different architectures and pretrained teachers. Overall, the paper offers an interesting empirical perspective on representation alignment and an effective approach to improving the efficiency of generative model training.
Critical strengths
The paper goes beyond conventional representation similarity measurements to investigate what information the student can recover from different teacher layers. The observation of hierarchy filling, together with the proposed recoverability gap, provides an interesting perspective on when external representation supervision may be useful.
In particular, the reported Spearman correlation of 0.92 between the recoverability gap and alignment benefit across 14 diagnostic configurations (Table 1 and Figure 3) provides encouraging evidence for the proposed criterion. The comparison against similarity-based measures and alternative selection criteria further strengthens this empirical finding. This is a meaningful contribution because identifying useful alignment targets without extensive training searches remains an important practical problem.
The proposed RARE framework demonstrates consistent improvements over several established representation alignment methods. On ImageNet-256 with SiT-B/2, RARE achieves an FID of 18.02 without classifier-free guidance and 4.46 with guidance, while reducing GPU-hours by approximately 14% compared with iREPA.
The evaluation covers multiple model scales, pretrained teachers, image domains, and generative architectures. The ablation studies also investigate several design components, including target selection, token weighting, and alignment release. These experiments provide useful evidence for the practical effectiveness of the proposed training recipe, independently of whether its broader mechanistic interpretation fully generalizes.
Critical weaknesses
- The relationship between linear recoverability and useful missing knowledge remains insufficiently established.
The central motivation assumes that information which cannot be linearly recovered from the student's intermediate representations constitutes a useful knowledge gap that should be addressed through teacher alignment.
However, low linear recoverability does not necessarily indicate that information is absent from the student. The information may be encoded in a nonlinear form or remain accessible to subsequent layers. Furthermore, information that is difficult to recover is not necessarily beneficial for the generative objective. Its usefulness may also depend on the relevance and compatibility of the teacher representation with the student's generative task.
Although the strong empirical correlation supports the proposed metric as a useful predictor within the evaluated configurations, it does not fully establish the broader interpretation that unrecoverable teacher information represents the knowledge most needed for generative training.
This distinction matters because it affects the generality of the paper's central scientific claim. A clearer discussion of linear recoverability as an operational diagnostic, rather than a direct measurement of missing knowledge, would strengthen the contribution. Additional comparisons with nonlinear probing or teacher targets of different generative relevance would further substantiate the interpretation.
- The experiments do not fully isolate the contribution of recoverability-guided target selection from the other improvements in RARE.
An important contribution of the paper is the use of the recoverability gap to identify the most beneficial teacher layer. However, on the primary ImageNet benchmark, RARE selects Layer 12, which is also the default alignment target used by REPA and iREPA. Consequently, the improved performance on this benchmark does not directly demonstrate the advantage of recoverability-guided teacher-layer selection.
Moreover, the ablation study in Table 3 indicates that releasing alignment supervision and introducing residual-based token weighting contribute substantially to the performance improvements. The results therefore establish the effectiveness of the combined training strategy more directly than the individual advantage of the proposed selection criterion.
While the diagnostic correlation experiments provide encouraging evidence, a more direct evaluation of gap-guided selection against fixed teacher-layer choices under an otherwise identical RARE training procedure would help isolate and validate this contribution. This would also clarify when and why recoverability-guided selection provides advantages beyond existing alignment strategies.
Questions for authors
- Linear recoverability and missing knowledge.
The recoverability gap is defined through linear readouts from the student's hidden representations. To what extent does the observed hierarchy filling reflect information genuinely unavailable to the student, rather than information encoded in a form that is not linearly accessible? Have the authors examined whether the hierarchy and its correlation with alignment benefit remain consistent under more expressive readouts?
Clarifying this point would help establish whether the metric captures a broader knowledge deficiency or a particular form of representational mismatch.
- Is missing information necessarily beneficial for generation?
The proposed criterion primarily measures how much of the teacher-layer increment the student cannot recover. However, the potential benefit of alignment may also depend on whether this information is useful for the generative objective.
Could the authors clarify how the recoverability gap distinguishes beneficial missing information from information that is difficult to recover but less relevant to generation? For example, would teacher targets with similar recoverability gaps but different generative relevance be expected to provide comparable improvements?
- Isolating the contribution of teacher-layer selection.
Since RARE selects the same teacher layer as the existing baselines on the primary ImageNet benchmark, could the authors provide or clarify comparisons that isolate the contribution of recoverability-guided target selection while keeping the alignment loss, token weighting, student block, and release schedule fixed?
In particular, the Places365 experiment, where RARE selects Layer 9 instead of the default Layer 12, appears to be a useful setting for this comparison.