- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new method to distill diffusion transformers in representation autoencoder (RAE) space from a manystep to a onestep generative model by using the recently proposed drifting field method for distillation. The authors argue that RAE space has higher curvature and is thus difficult to learn with convential distillation trajectories. On the other hand, drifting fields naturally match two distributions using two batches of samples. Experiments are conducted on class-conditional 256x256 ImageNet. The proposed method reaches sota performance in only 16 epochs.
Critical strengths
The method has a sound motivation, interesting analysis and evidence of the problem (difference between standard VAE and RAE spaces), and convincing positive results.
Critical weaknesses
There are a few parts in the paper where more thorough evaluation and ablation would be required. For example:
- The paper does not report compute & time comparisons
- The paper does not evaluate the drifting method with an MAE
- The paper does not explain the 32 flow trajectories to compute curvature (why only 32, classes, stds)
- The paper reports no stds in all the tables (main and appendix)
Questions for authors
- How are the positive samples constructed? can the be any or do they have to e.g. belong to the same class?
- Can the authors explain the choice of an exponential distribution to perturb inputs with noise?
- Is the FID reference set constructed by using 50 imgs per class as well?
- Can the authors ablate the register token choice?
- The authors claim "more stable distillation" in Sec 4.3, how is that measured?
- Why did the authors choose to flatten each 2x2 latent in SDVAE? how was RAE handled?
5 comments
Response to Reviewer ubYy (Part 2/3)
3. Details of the curvature computation
Thank you for the question. In our original analysis, we randomly sampled class labels for each trajectory and evaluated 32 trajectories, matching the choice of [Chen et al., 2024].
During the rebuttal period, we further refine the comparison protocol. First, we adopt DDT [Wang et al., 2026] as the flow-based model in the SD-VAE latent space, as its architecture and schedule are more closely aligned with those of , and use identical sampling time grids for both models. Second, we adopt arc-length-normalized curvature, which removes the effect of uniform trajectory scaling and enables a fairer comparison across latent spaces.
Using this revised protocol, we also examine how the number of sampled trajectories affects the mean curvature and its standard deviation. We sample one trajectory per class and evaluate nested subsets of 32, 64, 128, 256, 512, and 1000 distinct ImageNet classes. For each trajectory, we average the normalized curvature over its 49 interior points, then report the mean and sample standard deviation across trajectories. The results are reported in the table below. As the number of trajectories increases, the mean curvature in both latent spaces remains largely stable. The results consistently support our observation that RAE trajectories exhibit higher normalized curvature than SD-VAE trajectories.
We thank the reviewer for this helpful suggestion. We have added these supplementary results, along with visualizations based on 1000 trajectories, to the appendix of the revised manuscript.
| Number of trajectories | RAE (DiTᴰᴴ-XL) | SD-VAE (DDT-XL/2) |
|---|---|---|
| 32 | 5.2415 ± 0.6585 | 2.2235 ± 0.6910 |
| 64 | 5.2824 ± 0.7010 | 2.1340 ± 0.6073 |
| 128 | 5.2550 ± 0.6724 | 2.1606 ± 0.6326 |
| 256 | 5.2626 ± 0.7102 | 2.1430 ± 0.6419 |
| 512 | 5.3263 ± 0.7625 | 2.2297 ± 0.7045 |
| 1000 | 5.3474 ± 0.7619 | 2.2256 ± 0.6862 |
4. Standard deviations for our experiments
Thank you for this suggestion. Here we separately report the mean and sample standard deviation of FID across five independent training runs and five sampling runs with different seeds. We have added these results to the appendix of the revised manuscript.
| Varied seed | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Mean ± std |
|---|---|---|---|---|---|---|
| Training | 1.484 | 1.545 | 1.511 | 1.470 | 1.529 | 1.508 ± 0.031 |
| Sampling | 1.484 | 1.533 | 1.520 | 1.468 | 1.495 | 1.500 ± 0.026 |
5. How are the positive samples constructed?
Positive samples are drawn from the same class as the negative samples being optimized. Thus, all positive and negative samples used to compute the drifting field belong to the same class, consistent with the original Drifting formulation. To support this, we maintain a per-class FIFO cache which is updated as new samples arrive. Thank you for the question, and this procedure is described in Appendix B.2.
6. The choice of an exponential distribution for noise perturbations
This design emphasizes local supervision near generated samples while covering neighborhoods at varying distances. The exponential distribution assigns greater probability to smaller perturbation magnitudes while still allowing occasional larger perturbations. Our theoretical connection to Diffusion-GAN motivates evaluating supervision at perturbed query points, and the exponential distribution is a practical choice that balances locality with coverage across perturbation scales. Thank you for the question, and the corresponding description is in Appendix A.2.
7. Is the FID reference set also constructed using 50 images per class?
The reference mean and covariance for FID are computed using the full ImageNet training set, consistent with standard practice in prior work. The 50 images per class setting applies only to the generated samples, yielding 50,000 generated images across the 1,000 ImageNet classes. This also follows the standard evaluation protocol. Thank you for the question, and we have clarified this distinction in the Experiments section.
Response to Reviewer ubYy (Part 3/3)
8. Ablation on the number of register tokens
Thank you for this suggestion. The table below reports an ablation with 0, 2, 4, 6, and 8 register tokens. Using register tokens generally yields improvement, while performance is relatively insensitive to the exact number. We use 4 register tokens by default, matching the configuration of DINOv2 with registers. We have added these results to the appendix of the revised manuscript.
| Number of register tokens | 0 | 2 | 4 (default) | 6 | 8 |
|---|---|---|---|---|---|
| FID (ω = 1.0) | 1.77 | 1.75 | 1.74 | 1.74 | 1.76 |
| FID (ω = 1.4) | 1.52 | 1.48 | 1.48 | 1.49 | 1.51 |
9. Clarifying “more stable distillation”
In Section 4.3, "more stable distillation" refers specifically to numerical stability during training. Both the official RAE repository and MF-RAE report issues such as NaNs or exploding gradients when training in the RAE latent space. During our development experiments, we also encountered NaNs under some random seeds. With both modifications enabled as described in Section 4.3, we observed fewer such failures. This is a qualitative observation. Thank you for the question, and we have clarified this point in the revised manuscript.
10. Tokenization in the SD-VAE and RAE analyses
We flatten each spatial patch in the SD-VAE latent because this matches the input tokenization of the Transformer models considered in our analysis. Each patch is flattened into a 16-dimensional token, yielding a token sequence of shape , where the tokens correspond to a spatial grid. Our analysis therefore uses the model's actual input tokens as its basic units.
For RAE, we likewise conduct the analysis at the token level. The latent representation already has the shape , comprising tokens with features each. directly takes this latent token sequence as input, so no additional spatial rearrangement is needed for RAE.
Thus, both analyses follow the respective models' input tokenization. We thank the reviewer for this question, and we have added an explicit description of the RAE processing procedure to the appendix.
[Chen et al., 2024] Chen, Defang, et al. "On the trajectory regularity of ODE-based diffusion sampling." ICML, 2024.
[Wang et al., 2026] Wang, Shuai, et al. "DDT: Decoupled diffusion transformer." CVPR. 2026.
Response to Reviewer ubYy (Part 1/3)
We thank the reviewer for recognizing the soundness and effectiveness of our work. Below, we address the reviewer's valuable suggestions and questions.
1. Comparing computational cost and training time
Thank you for this suggestion. Here we compare the FLOPs and training time of the distillation methods with publicly available implementations under a common GPU resource budget. The benchmarking protocol is detailed below.
Benchmarking protocol. We evaluate STEI, π-Flow, FACM, MF-RAE, and our Drift-RAE on a single NVIDIA A800 GPU with 80 GiB of memory. We assume a 32-GPU configuration for each method and benchmark its corresponding per-GPU workload on a single GPU. We use the official open-source implementations, adjusting only gradient accumulation where necessary to fit within the memory budget. This preserves each implementation's effective batch size, numerical precision, and memory-saving techniques, such as the 8-bit AdamW optimizer and gradient checkpointing used by π-Flow. We report teacher-pretraining epochs separately and exclude teacher pretraining from the distillation-cost comparison.
Training time and FLOPs. Drift-RAE requires substantially less training time and FLOPs than π-Flow, FACM, and MF-RAE. Although STEI has a similar training cost, Drift-RAE achieves a substantially better FID (1.48 vs. 7.12). These results demonstrate Drift-RAE's favorable efficiency.
Peak memory. Under each method's original batch-size and precision settings, Drift-RAE fits within the 80 GiB memory budget without gradient accumulation, whereas MF-RAE and FACM require gradient accumulation to fit on a single GPU. We additionally report peak memory under a matched setting, using FP32, a local batch size of 16, and no gradient accumulation for all methods. This allows every method to fit on a single GPU. Under these matched batch-size and precision settings, Drift-RAE also exhibits relatively low peak memory usage. This is because it does not require additional EMA student or teacher models to reside in GPU memory, nor does it require memory-intensive operations such as Jacobian–vector products (JVPs).
We thank the reviewer again for this valuable suggestion. We have added the comparison table and the accompanying analysis to the appendix of the revised manuscript.
2. Drifting with an additional MAE
We note that one of the key findings from our analysis is that the highly concentrated RAE latent space enables effective drifting-based distillation without an additional MAE. This simplifies the distillation pipeline by removing both a separate MAE training stage and the additional forward and backward passes through the MAE during distillation. Therefore, our primary focus is on MAE-free drifting-based distillation in RAE, which is also an important practical advantage of Drift-RAE over the original Drifting Model.
Exploring MAE-based feature extraction for drifting in RAE could be a valuable direction for future work. Such an extension would require designing and training an MAE specifically for the RAE latent space, introducing additional methodological and computational complexity. We thank the reviewer for highlighting this potential extension.