- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a framework for improving the efficiency of latent diffusion models by jointly addressing spatial resolution, model architecture, and sampling complexity. The authors propose Score-Compatible Tokenizer Distillation, which enables the learning of a more compact latent representation while aiming to maintain compatibility with a pretrained diffusion teacher. A lightweight projection network maps the compressed latents into the teacher's latent space, where score-based distribution matching guides the joint optimization of the encoder and projection network. A reconstruction objective is also incorporated to maintain image reconstruction quality. Using the resulting tokenizer, the authors train a smaller diffusion model in the compressed latent space and further reduce the sampling process to a single step through distribution-matching distillation. Experiments on ImageNet with diffusion transformers show savings in computational cost and memory consumption, accompanied by some loss in generation quality.
Critical strengths
- The paper addresses a limitation of existing diffusion acceleration approaches by combining latent spatial compression with architectural and temporal optimization.
- The use of a pretrained diffusion teacher to guide compressed tokenizer learning through score-based distribution alignment is interesting, extending conventional reconstruction-based compression.
- The training-time bridge enables smaller student models with different architectures and latent resolutions to benefit from teacher score supervision. The bridge and auxiliary score network are removed during inference.
Critical weaknesses
- Although the proposed framework combines spatial, architectural, and temporal acceleration, it remains unclear whether similar benefits could be achieved by combining existing acceleration techniques. Furthermore, the additional training costs of the compressed tokenizer and auxiliary score network are not sufficiently analyzed. A comparison of overall training costs would help clarify whether the inference-efficiency gains justify the additional training overhead.
- The architectural compression strategy relies on training a smaller diffusion backbone from scratch rather than pruning the pretrained teacher. Could the authors discuss the advantages of this choice over pruning-based alternatives, particularly in terms of training cost and generation quality?
- DC-Gen addresses the representation gap through embedding alignment and post-training adaptation. Although the authors compare SCTD with a DC-Gen tokenizer baseline, the comparison does not cover its full adaptation pipeline. Moreover, DC-Gen has demonstrated compatibility with few-step distillation methods such as AnyFlow. Further discussion would help clarify the practical advantages of SCTD over these approaches, particularly for temporal distillation.
- The evaluation primarily relies on FID and IS. Additional metrics, such as class-conditional accuracy and generative precision and recall, would help provide a more comprehensive assessment of generation quality and diversity.
- The empirical validation is limited to class-conditional ImageNet generation. Experiments on text-to-image models would help demonstrate the applicability of SCTD to more practical diffusion systems.
Questions for authors
- How much additional training compute does SCTD require, including tokenizer training and the auxiliary score network? Could similar gains be achieved by combining existing acceleration methods?
- How does training a smaller DiT from scratch compare with pruning-based alternatives in terms of training cost and generation quality?
- Table 1 seems to consider only a DC-Gen tokenizer baseline. Given that DC-Gen has demonstrated compatibility with few-step distillation, could the authors clarify why its combination with temporal distillation is marked as not directly applicable?
- Additional metrics, such as class-conditional accuracy and generative precision and recall, would help assess generation quality and diversity beyond FID and IS.
- Evaluation on text-to-image models would help establish whether SCTD generalizes beyond class-conditional ImageNet generation.