acceptodds
Under review as a conference paper at ICLR 2027

Distilling Drifting Transformers with Representation Autoencoders

Abstract

Despite the significant training acceleration and promising performance, effectively distilling models in Representation Autoencoder (RAE) latent spaces remains challenging. In this work, we argue that RAE is competent at high-quality one-step generation. We achieve 1.48 FID with only 16-epoch distillation on ImageNet 256 dataset, surpassing various state-of-the-art methods. To achieve this, we quantitatively study the geometrical behavior of different underlying latent spaces. We conclude that conventional trajectory-based distillation methods heavily rely on priors of plain teacher trajectories, while RAE incurs more complex trajectories with poor properties due to the highly anisotropic latent space. We introduce the recently proposed drifting field as the distillation methodology, which makes use of semantically rich RAE latents and provides supervision without relying on teacher trajectories. Bridging our Drift-RAE with previous generative paradigms, we propose several insightful modifications, including the first extrapolation-based guided sampling method for one-step generation with minimal additional overhead.

Primary area:
generative models
Keywords:
Drifting model, representation autoencoder, diffusion distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ubYyOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method to distill diffusion transformers in representation autoencoder (RAE) space from a manystep to a onestep generative model by using the recently proposed drifting field method for distillation. The authors argue that RAE space has higher curvature and is thus difficult to learn with convential distillation trajectories. On the other hand, drifting fields naturally match two distributions using two batches of samples. Experiments are conducted on class-conditional 256x256 ImageNet. The proposed method reaches sota performance in only 16 epochs.

Critical strengths

The method has a sound motivation, interesting analysis and evidence of the problem (difference between standard VAE and RAE spaces), and convincing positive results.

Critical weaknesses

There are a few parts in the paper where more thorough evaluation and ablation would be required. For example:

  • The paper does not report compute & time comparisons
  • The paper does not evaluate the drifting method with an MAE
  • The paper does not explain the 32 flow trajectories to compute curvature (why only 32, classes, stds)
  • The paper reports no stds in all the tables (main and appendix)
Questions for authors
  • How are the positive samples constructed? can the be any or do they have to e.g. belong to the same class?
  • Can the authors explain the choice of an exponential distribution to perturb inputs with noise?
  • Is the FID reference set constructed by using 50 imgs per class as well?
  • Can the authors ablate the register token choice?
  • The authors claim "more stable distillation" in Sec 4.3, how is that measured?
  • Why did the authors choose to flatten each 2x2 latent in SDVAE? how was RAE handled?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ubYy (Part 1/3)

We thank the reviewer for recognizing the soundness and effectiveness of our work. Below, we address the reviewer's valuable suggestions and questions.

1. Comparing computational cost and training time

Thank you for this suggestion. Here we compare the FLOPs and training time of the distillation methods with publicly available implementations under a common GPU resource budget. The benchmarking protocol is detailed below.

Benchmarking protocol. We evaluate STEI, π\pi-Flow, FACM, MF-RAE, and our Drift-RAE on a single NVIDIA A800 GPU with 80 GiB of memory. We assume a 32-GPU configuration for each method and benchmark its corresponding per-GPU workload on a single GPU. We use the official open-source implementations, adjusting only gradient accumulation where necessary to fit within the memory budget. This preserves each implementation's effective batch size, numerical precision, and memory-saving techniques, such as the 8-bit AdamW optimizer and gradient checkpointing used by π\pi-Flow. We report teacher-pretraining epochs separately and exclude teacher pretraining from the distillation-cost comparison.

Training time and FLOPs. Drift-RAE requires substantially less training time and FLOPs than π\pi-Flow, FACM, and MF-RAE. Although STEI has a similar training cost, Drift-RAE achieves a substantially better FID (1.48 vs. 7.12). These results demonstrate Drift-RAE's favorable efficiency.

Peak memory. Under each method's original batch-size and precision settings, Drift-RAE fits within the 80 GiB memory budget without gradient accumulation, whereas MF-RAE and FACM require gradient accumulation to fit on a single GPU. We additionally report peak memory under a matched setting, using FP32, a local batch size of 16, and no gradient accumulation for all methods. This allows every method to fit on a single GPU. Under these matched batch-size and precision settings, Drift-RAE also exhibits relatively low peak memory usage. This is because it does not require additional EMA student or teacher models to reside in GPU memory, nor does it require memory-intensive operations such as Jacobian–vector products (JVPs).

We thank the reviewer again for this valuable suggestion. We have added the comparison table and the accompanying analysis to the appendix of the revised manuscript.

MethodEpochsItersGlobal batchs / iterA800 h totalTotal FLOPs (×1018\times 10^{18})Peak memory: Original (GiB)Peak memory: Matched setting (GiB)
Teacher: SiT-XL/21400
STEI (1-step)20100K2560.29626354.6621.2933.56
Teacher: SiT-XL/2 + REPA800
π\pi-Flow (1-step)448140K40962.0262521109112.3423.01
Teacher: LightningDiT-XL800
FACM (2-step)6075K10241.411941182.559.03 (accum=2)76.57
Teacher: DiTᴰᴴ-XL800
MF-RAE47207K512 / 2560.535985163.954.37 (accum=2)68.94
Drift-RAE1610K20482.85925445.2874.7529.78

2. Drifting with an additional MAE

We note that one of the key findings from our analysis is that the highly concentrated RAE latent space enables effective drifting-based distillation without an additional MAE. This simplifies the distillation pipeline by removing both a separate MAE training stage and the additional forward and backward passes through the MAE during distillation. Therefore, our primary focus is on MAE-free drifting-based distillation in RAE, which is also an important practical advantage of Drift-RAE over the original Drifting Model.

Exploring MAE-based feature extraction for drifting in RAE could be a valuable direction for future work. Such an extension would require designing and training an MAE specifically for the RAE latent space, introducing additional methodological and computational complexity. We thank the reviewer for highlighting this potential extension.

AuthorsOpenReview ↗

Response to Reviewer ubYy (Part 2/3)

3. Details of the curvature computation

Thank you for the question. In our original analysis, we randomly sampled class labels for each trajectory and evaluated 32 trajectories, matching the choice of [Chen et al., 2024].

During the rebuttal period, we further refine the comparison protocol. First, we adopt DDT [Wang et al., 2026] as the flow-based model in the SD-VAE latent space, as its architecture and schedule are more closely aligned with those of DiTDH\text{DiT}^{\text{DH}}, and use identical sampling time grids for both models. Second, we adopt arc-length-normalized curvature, which removes the effect of uniform trajectory scaling and enables a fairer comparison across latent spaces.

Using this revised protocol, we also examine how the number of sampled trajectories affects the mean curvature and its standard deviation. We sample one trajectory per class and evaluate nested subsets of 32, 64, 128, 256, 512, and 1000 distinct ImageNet classes. For each trajectory, we average the normalized curvature over its 49 interior points, then report the mean and sample standard deviation across trajectories. The results are reported in the table below. As the number of trajectories increases, the mean curvature in both latent spaces remains largely stable. The results consistently support our observation that RAE trajectories exhibit higher normalized curvature than SD-VAE trajectories.

We thank the reviewer for this helpful suggestion. We have added these supplementary results, along with visualizations based on 1000 trajectories, to the appendix of the revised manuscript.

Number of trajectoriesRAE (DiTᴰᴴ-XL)SD-VAE (DDT-XL/2)
325.2415 ± 0.65852.2235 ± 0.6910
645.2824 ± 0.70102.1340 ± 0.6073
1285.2550 ± 0.67242.1606 ± 0.6326
2565.2626 ± 0.71022.1430 ± 0.6419
5125.3263 ± 0.76252.2297 ± 0.7045
10005.3474 ± 0.76192.2256 ± 0.6862

4. Standard deviations for our experiments

Thank you for this suggestion. Here we separately report the mean and sample standard deviation of FID across five independent training runs and five sampling runs with different seeds. We have added these results to the appendix of the revised manuscript.

Varied seedRun 1Run 2Run 3Run 4Run 5Mean ± std
Training1.4841.5451.5111.4701.5291.508 ± 0.031
Sampling1.4841.5331.5201.4681.4951.500 ± 0.026

5. How are the positive samples constructed?

Positive samples are drawn from the same class as the negative samples being optimized. Thus, all positive and negative samples used to compute the drifting field belong to the same class, consistent with the original Drifting formulation. To support this, we maintain a per-class FIFO cache which is updated as new samples arrive. Thank you for the question, and this procedure is described in Appendix B.2.

6. The choice of an exponential distribution for noise perturbations

This design emphasizes local supervision near generated samples while covering neighborhoods at varying distances. The exponential distribution assigns greater probability to smaller perturbation magnitudes while still allowing occasional larger perturbations. Our theoretical connection to Diffusion-GAN motivates evaluating supervision at perturbed query points, and the exponential distribution is a practical choice that balances locality with coverage across perturbation scales. Thank you for the question, and the corresponding description is in Appendix A.2.

7. Is the FID reference set also constructed using 50 images per class?

The reference mean and covariance for FID are computed using the full ImageNet training set, consistent with standard practice in prior work. The 50 images per class setting applies only to the generated samples, yielding 50,000 generated images across the 1,000 ImageNet classes. This also follows the standard evaluation protocol. Thank you for the question, and we have clarified this distinction in the Experiments section.

AuthorsOpenReview ↗

Response to Reviewer ubYy (Part 3/3)

8. Ablation on the number of register tokens

Thank you for this suggestion. The table below reports an ablation with 0, 2, 4, 6, and 8 register tokens. Using register tokens generally yields improvement, while performance is relatively insensitive to the exact number. We use 4 register tokens by default, matching the configuration of DINOv2 with registers. We have added these results to the appendix of the revised manuscript.

Number of register tokens024 (default)68
FID (ω = 1.0)1.771.751.741.741.76
FID (ω = 1.4)1.521.481.481.491.51

9. Clarifying “more stable distillation”

In Section 4.3, "more stable distillation" refers specifically to numerical stability during training. Both the official RAE repository and MF-RAE report issues such as NaNs or exploding gradients when training in the RAE latent space. During our development experiments, we also encountered NaNs under some random seeds. With both modifications enabled as described in Section 4.3, we observed fewer such failures. This is a qualitative observation. Thank you for the question, and we have clarified this point in the revised manuscript.

10. Tokenization in the SD-VAE and RAE analyses

We flatten each 2×22\times2 spatial patch in the SD-VAE latent because this matches the input tokenization of the Transformer models considered in our analysis. Each patch is flattened into a 16-dimensional token, yielding a token sequence of shape 256×16256\times16, where the 256256 tokens correspond to a 16×1616\times16 spatial grid. Our analysis therefore uses the model's actual input tokens as its basic units.

For RAE, we likewise conduct the analysis at the token level. The latent representation already has the shape 256×768256\times768, comprising 256256 tokens with 768768 features each. DiTDH\text{DiT}^{\text{DH}} directly takes this latent token sequence as input, so no additional spatial rearrangement is needed for RAE.

Thus, both analyses follow the respective models' input tokenization. We thank the reviewer for this question, and we have added an explicit description of the RAE processing procedure to the appendix.

[Chen et al., 2024] Chen, Defang, et al. "On the trajectory regularity of ODE-based diffusion sampling." ICML, 2024.

[Wang et al., 2026] Wang, Shuai, et al. "DDT: Decoupled diffusion transformer." CVPR. 2026.

Reviewer_QxWhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Drift-RAE, which utilizes a pre-trained flow model to initialize the single-step generator and directly performs token-by-token drifting distribution matching in the RAE latent space, thereby eliminating the need for an additional MAE feature extractor. The authors combined latent space geometric analysis and proposed improvements such as input perturbation, softmax adjustment, and additional gradient-free negative samples. They also utilized Internal Drifting Guidance (IDG) to perform extrapolation during inference using intermediate predictions. On ImageNet-256, the method achieved a single-step FID of 1.48.

Critical strengths
  1. The generation quality is competitive, and the ablation can support the actual value of the main components.
  2. This plan has practical value.
Critical weaknesses
  1. Some of the statements in the paper are problematic. The introduction's generalization of the existing distillation methods based on the teacher's trajectory is too broad. DMD/DMD2 has adopted distribution matching, where DMD2 explicitly removes the regression term based on the teacher-generated paired data. Therefore, "not relying on the trajectory" alone is not sufficient to explain why drifting must be chosen.
  2. Also in the introduction section of this article, it is argued that anisotropy leads to high curvature, thereby making distillation difficult. Firstly, the experimental comparison seems to be rather unfair. SD-VAE and RAE use different models respectively, and the sampling and other settings are not aligned. Secondly, the increase in anisotropy does not seem to be sufficient to prove that it leads to more difficult distillation. For example, anisotropic Gaussian targets can still be generated from standard Gaussian through a simple linear mapping.
Questions for authors

I believe that this paper can be further enhanced in terms of completeness and rigor by improving the accuracy of its writing expression and aligning the experiments. Given that the proposed method is so straightforward and the potential for improved completeness is significant, this paper is ultimately a weak acceptable work.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QxWh (Part 1/2)

We thank the reviewer for recognizing the effectiveness and practical value of our work. Below, we address the reviewer's valuable suggestions.

1. Clarifying the scope of trajectory-based methods and the motivation for drifting

Thank you for this suggestion. We agree that distribution-matching methods, particularly DMD2, also do not rely on teacher trajectories. We therefore position drifting as a competitive alternative for RAE distillation, while acknowledging that other methods can also be suitable for this task.

To make this distinction clear, we have revised the Introduction as follows:

First, we explicitly qualify our discussion of trajectory-based distillation:

Conventional trajectory-based methods distill the pre-trained models using the intrinsic priors of the denoising process.

Second, we clarify that drifting offers a potentially competitive alternative for RAE distillation:

The drifting field enforces the generated distribution to coincide with the real one without relying on teacher trajectories, making it a potentially competitive alternative for RAE distillation.

2. Further discussion on distribution matching

Building on the clarification above, here we further discuss the connection between drifting and distribution matching.

As discussed in Appendix A.2, the drifting field can be interpreted as a Monte Carlo estimate of a score difference. Table 2 shows that the RAE latent distribution is highly concentrated, which can reduce the estimation error when using discrete samples. Our design further improves this estimate by increasing the number of positive samples and introducing detached negative samples. From this perspective, Drift-RAE and DMD are closely related: the former estimates the update field using discrete samples, whereas the latter uses learned score networks. Therefore, a practical advantage of drifting is that it requires neither an online-updated fake-score network nor the additional adversarial training components used in DMD2. This can reduce both computation and memory overhead.

Here we additionally report results for Consistency Distillation (CD) and DMD2 in the RAE latent space, both trained with the same batch size of 2048 and the same number of iterations as our Drift-RAE. The FID results are reported in the table below. CD achieves suboptimal FID, consistent with our motivation that trajectory-based distillation can be challenging in the presence of curved RAE trajectories. DMD2 substantially narrows the FID gap to Drift-RAE compared with CD. This supports that DMD2, which likewise avoids reliance on teacher trajectories, is also a viable method for RAE distillation.

We further compare the per-iteration training time and peak GPU memory of CD, DMD2, and Drift-RAE on a single NVIDIA A800 GPU. We use an effective local batch size of 64, corresponding to a global batch size of 2048 across 32 GPUs. Without gradient checkpointing or other additional memory-saving techniques, CD and DMD2 require 2 and 8 gradient accumulation steps, respectively, to fit their workload on a single GPU. Drift-RAE fits without gradient accumulation. Moreover, Drift-RAE takes approximately half the per-iteration training time of DMD2.

DMD2's additional memory and computation overhead includes maintaining and updating the fake-score network, as well as backpropagating through the fake-score network for adversarial supervision. In contrast, the additional samples used by Drift-RAE are detached and therefore do not require retaining their computation graphs, limiting the associated memory overhead.

We thank the reviewer again for this valuable suggestion. We have incorporated these additional discussions into the revised manuscript.

FID comparison on ImageNet 256×256 in RAE space.

MethodFID ↓
CD4.61
DMD21.93
Drift-RAE (w/o IDG)1.74
Drift-RAE (w/ IDG)1.48

Computational cost on a single NVIDIA A800 80GB at effective local batch 64.

MethodAccums / iterPeak GPU memory (GiB)
CD1-OOM
22.41554.41
DMD24-OOM
85.25156.32
Drift-RAE (w/o IDG)12.78872.54
Drift-RAE (w/ IDG)12.85974.75
AuthorsOpenReview ↗

Response to Reviewer QxWh (Part 2/2)

  1. A better-aligned setting for the curvature comparison

Thank you for this valuable suggestion. We agree that using comparable model architectures and identical sampling procedures makes the comparison fairer. We therefore use DDT [Wang et al., 2026] as the flow-matching model in the SD-VAE latent space. Since the DiTDH\text{DiT}^{\text{DH}} architecture used in RAE draws on DDT's design, DDT provides the closest publicly available architectural counterpart among flow-matching models in the SD-VAE latent space. For sampling, we use the same time grid, including the time shift used in RAE, and the same first-order Euler solver to obtain trajectories for both models. As an additional reference, we also evaluate the curvatures under DDT's officially recommended uniform time grid.

While preparing this rebuttal, we also identify that the metric previously computed following the protocol of [Chen et al., 2024] is standard Euclidean curvature, whose magnitude depends on the absolute scale of the trajectory. This can confound comparisons across latent spaces, particularly when their dimensionalities and feature scales differ. We therefore adopt arc-length-normalized curvature to remove the effect of uniform trajectory scaling.

Specifically, for a sampled trajectory consisting of K+1K+1 states, zkk=0K\\{\mathbf{z}_k\\}_{k=0}^{K}, we flatten each complete latent state into a vector and estimate curvature at each interior point using three consecutive states. Let ak=zk−zk−1,bk=zk+1−zk,\mathbf{a}_k=\mathbf{z}_k-\mathbf{z}_{k-1}, \qquad \mathbf{b}_k=\mathbf{z}_{k+1}-\mathbf{z}_k, then the curvature is computed as

κk=2∣∣ak∣∣22∣∣bk∣∣22−⟨ak,bk⟩2∣∣ak∣∣2∣∣bk∣∣2∣∣ak+bk∣∣2,k=1,…,K−1.\kappa_k= \frac{ 2\sqrt{ ||\mathbf{a}_k||_2^2||\mathbf{b}_k||_2^2 -\langle\mathbf{a}_k,\mathbf{b}_k\rangle^2 } }{ ||\mathbf{a}_k||_2 ||\mathbf{b}_k||_2 ||\mathbf{a}_k+\mathbf{b}_k||_2 }, \qquad k=1,\ldots,K-1.

This expression equals the reciprocal of the circumradius and applies directly to high-dimensional latent states. We then estimate the total arc length of the trajectory as L=∑k=0K−1∣zk+1−zk∣2,L=\sum_{k=0}^{K-1} \\|\mathbf{z}_{k+1}-\mathbf{z}_k\\|_2, and define the dimensionless normalized curvature as κ^k=Lκk.\widehat{\kappa}_k=L\kappa_k. This normalization makes the curvature invariant to uniform scaling of the latent space, enabling a more meaningful comparison between RAE and SD-VAE trajectories.

We recompute the curvature comparison and update the figure using this protocol. Under this better-aligned setting, we again observe that flow trajectories in the RAE latent space exhibit curvature substantially higher than SD-VAE near the pure-noise endpoint. This pronounced bending near the noise endpoint is particularly relevant to one-step distillation, where the student must map directly from the noise endpoint to clean data in a single step.

We thank the reviewer again for this suggestion, which has helped us make the comparison more rigorous and its interpretation clearer. We have updated the figure and the accompanying text in the revised manuscript.

4. The relationship between anisotropy and curvature

Thank you for this helpful suggestion. We agree that anisotropy may contribute to higher trajectory curvature, but it does not necessarily imply it. Our analysis empirically shows that RAE trajectories exhibit substantially higher curvature near the noise endpoint. We have therefore revised the Introduction to avoid implying that higher curvature follows directly from anisotropy and to report our empirical findings more precisely:

We first examine the geometry of RAE latent spaces in Section 3.2 and identify pronounced anisotropy in the token distributions at individual patch positions. Furthermore, under matched flow-matching settings, we observe that RAE trajectories exhibit substantially higher normalized curvature near the noise endpoint than SD-VAE trajectories, which may complicate trajectory-based distillation, particularly when distilling to very few sampling steps.

We thank the reviewer again for these valuable suggestions, which have helped us improve the precision and clarity of our manuscript.

[Wang et al., 2026] Wang, Shuai, et al. "DDT: Decoupled diffusion transformer." CVPR. 2026.

[Chen et al., 2024] Chen, Defang, et al. "On the trajectory regularity of ODE-based diffusion sampling." ICML, 2024.

Thank you for your detailed reply, and my concerns have been addressed.

Thank you again for your review and for taking the time to consider our response. We are glad that our clarifications have addressed your concerns.

Reviewer_fcFMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Drift-RAE converts a pretrained diffusion transformer in representation-autoencoder latent space into a one-step generator using a drifting objective. The method adapts drifting to this setting through tokenwise supervision, modified weighting, input perturbations, additional detached samples, and an intermediate guidance head. It reports strong ImageNet generation after a short distillation stage.

Critical strengths
  1. The paper addresses a meaningful practical problem: efficient one-step generation in high-dimensional RAE spaces without online teacher-trajectory supervision. The reported FID of 1.48 is competitive, with relatively high recall (Table 4).
  2. The ablations are extensive and include the softmax axes, perturbations, sample allocation, CLS tokens, and guidance-head objectives. This gives useful evidence about the recipe beyond the final result.
  3. The theoretical discussion gives a plausible motivation for why high-dimensional latent geometry can affect drifting. The failure of training from scratch is also reported, making the dependence on a pretrained generator visible.
Critical weaknesses
  1. Lack of a direct computational-cost comparison. The reported 16 epochs count gradient-carrying samples but exclude additional detached generated samples. Please compare distillation GPU-hours and peak memory with relevant baselines on comparable hardware, and report pretrained-model preparation costs separately. Epoch counts alone do not establish the claimed efficiency advantage.
Questions for authors
  1. What is the measured distillation cost relative to relevant baselines, including the additional detached samples?
  2. How are cross-token dependencies preserved? Matching each token's marginal distribution does not generally guarantee matching the joint distribution of all tokens. In the original feature-space Drifting formulation, losses are backpropagated through a feature encoder whose outputs can depend on multiple input positions; thus separate feature losses can induce coupled updates to the generated pixels or latents through the encoder Jacobian. Drift-RAE instead applies drifting directly to generated RAE tokens. What mechanism preserves their joint structure: the pretrained initialization, shared generator, CLS-token supervision, or another component? Could the authors clarify this distinction and provide evidence that cross-token dependencies are preserved during distillation? The encoder coupling in the original formulation is not, by itself, a guarantee of joint-distribution matching either.
4 comments
AuthorsOpenReview ↗

Response to Reviewer fcFM (Part 1/2)

We thank the reviewer for recognizing the significance and effectiveness of our work. Below, we address the reviewer's valuable suggestions and questions.

1. A direct computational-cost comparison

Thank you for suggesting this comparison. Here we benchmark the computational efficiency of STEI, π\pi-Flow, FACM, MF-RAE, and our Drift-RAE on a single NVIDIA A800 GPU with 80 GiB of memory.

Benchmarking protocol. To compare the methods under the same GPU resource budget, we assume a 32-GPU configuration for each method and benchmark its corresponding per-GPU workload on a single A800 GPU. We use the official open-source implementations for all methods, adjusting only gradient accumulation where necessary to fit within the 80 GiB memory budget. This preserves each implementation's effective batch size, numerical precision, and memory-saving techniques, such as the 8-bit AdamW optimizer and gradient checkpointing used by π\pi-Flow. The resulting timings therefore compare the official implementations under a common GPU resource budget.

We additionally report peak memory under a matched setting: all methods use FP32, a local batch size of 16, and no gradient accumulation, which allows every method to fit on a single GPU. This comparison helps isolate differences in memory requirements under the same batch size and precision, including those arising from GPU-resident models and operations such as Jacobian–vector products (JVPs). The results are attached in the following Table.

Training time. Drift-RAE requires substantially less training time than π\pi-Flow, FACM, and MF-RAE. Although STEI has a similar training cost, Drift-RAE achieves a substantially better FID (1.48 vs. 7.12). These results demonstrate Drift-RAE's favorable efficiency. We emphasize that the reported time includes all components of Drift-RAE training, i.e., encoding real images into DINOv2 features to construct positive samples, generating all detached negative samples, the forward and backward passes for all negative samples that contribute gradients, and computing the drifting field.

Peak memory. Under each method's original batch-size and precision settings, Drift-RAE fits within the 80 GiB memory budget without gradient accumulation, whereas MF-RAE and FACM require gradient accumulation to fit on a single GPU. Under the matched batch-size and precision settings, Drift-RAE also exhibits relatively low peak memory usage. This is because it does not require additional EMA student or teacher models to reside in GPU memory, nor does it require memory-intensive operations such as JVPs.

Following the reviewer's suggestion, we also group the methods by their teachers and separately report teacher-training epochs. For MF-RAE, we distinguish its two training stages, which use batch sizes of 512 and 256, respectively. Its average time per iteration is computed as the iteration-weighted average across the two stages. We also correct MF-RAE's total training epochs to 47; our previous calculation incorrectly used a batch size of 256 for both stages.

We have added this table and the accompanying analysis to the appendix of the revised manuscript. We thank the reviewer again for this valuable suggestion.

MethodEpochsItersGlobal batchs / iterA800 h totalTotal FLOPs (×1018\times 10^{18})Peak memory: Original (GiB)Peak memory: Matched setting (GiB)
Teacher: SiT-XL/21400
STEI (1-step)20100K2560.29626354.6621.2933.56
Teacher: SiT-XL/2 + REPA800
π\pi-Flow (1-step)448140K40962.0262521109112.3423.01
Teacher: LightningDiT-XL800
FACM (2-step)6075K10241.411941182.559.03 (accum=2)76.57
Teacher: DiTᴰᴴ-XL800
MF-RAE47207K512 / 2560.535985163.954.37 (accum=2)68.94
Drift-RAE1610K20482.85925445.2874.7529.78
AuthorsOpenReview ↗

Response to Reviewer fcFM (Part 2/2)

  1. How are cross-token dependencies preserved?

Thank you for raising this important point. We agree that matching token-wise marginal distributions does not, by itself, guarantee matching the joint distribution across tokens. In practice, we find that token-wise supervision during distillation can already achieve strong one-step generation performance. We posit that cross-token dependencies are largely inherited from the pretrained teacher through weight initialization. Moreover, although the losses are defined token-wise, all tokens are generated jointly by a shared Transformer, whose self-attention and shared parameter updates couple their outputs. These mechanisms provide a potential explanation for retaining cross-token structure without an additional feature-encoder Jacobian.

To assess whether the distilled model retains cross-token correlation structure under the token-wise drifting objective, we compare cross-token correlation matrices computed from generated and real samples. Specifically, we randomly select 32 ImageNet classes. For each class, we construct a reference correlation matrix using 128 real samples and use a separate, non-overlapping set of 128 real samples to estimate the discrepancy attributable to finite sampling. Then we generate 128 samples per class from each of three configurations: the teacher with 50 NFE, the distilled model without IDG, and the distilled model with an IDG scale of 1.4. We compute the RMSE between the off-diagonal entries of each resulting correlation matrix and those of the real-data reference matrix, and average the results over the 32 classes.

We additionally construct a shuffled control setting. Within each class, we independently permute the sample indices at each fixed token position. This preserves the empirical marginal distribution at every position while disrupting the cross-token correspondence.

The results, as reported in the table below, show that Drift-RAE largely retains the teacher's cross-token correlation structure after distillation and remains close to that of real data. In comparison, independently shuffling each token substantially disrupts these correlations and yields a much larger RMSE. These findings provide empirical support that cross-token correlations are preserved despite the use of a token-wise objective.

We thank the reviewer again for this valuable suggestion. We have added these results, the accompanying discussion, and visualizations of the cross-token correlation matrices to the appendix of the revised manuscript.

SourceCorrelation RMSE ↓
Real data, disjoint split0.0295
Teacher (50-step ODE)0.0342
Drift-RAE (IDG scale = 1)0.0301
Drift-RAE (IDG scale = 1.4)0.0406
Drift-RAE (IDG scale = 1), shuffled0.2714
Reviewer_fcFMOpenReview ↗

Response

I thank the author for the new results. It's convincing to me that driftRAE has compute advantages. Also it's interesting to know that using drifting as post-training doesn't corrupt the token dependency learned in pretraining. I'll therefore keep my positive score.

And my review is human written...

Thank you for the thoughtful review and for recognizing our additional results. We appreciate your positive assessment and are glad that our clarifications addressed your concerns.

Comments on the paper

1 comment
Area_Chair_hQoNOpenReview ↗

Reminder: Discussion Deadline — November 18

Dear Reviewers,

Thank you for your valuable time and contributions to ICLR. This is a friendly reminder to review the authors' responses and engage in discussions with them. The discussion deadline is November 18.

Thank you for your continued efforts!

ACs

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.