acceptodds
Under review as a conference paper at ICLR 2027

Holistic Latent Diffusion Acceleration: Unifying Spatial, Temporal, and Architectural Efficiency

Abstract

Latent Diffusion Models (LDM) face three compounding efficiency challenges in practical deployment: i) the temporal latency of iterative sampling; ii) the architectural overhead of heavy backbone parameter counts; and iii) the spatial cost of high-dimensional latent grids. While recent acceleration methods have made substantial progress on temporal distillation and architectural compression, the spatial axis is often inherited from the teacher tokenizer and treated as fixed. In this work, we recast latent resolution as an optimizable efficiency axis and introduce a unified framework that optimizes all three dimensions simultaneously. We introduce a novel strategy of Score-Compatible Tokenizer Distillation (SCTD), which leverages score-matching principles to align a spatially compact latent space with the induced distribution of a frozen, powerful teacher model, distilling the teacher's generative prior into a compressed, lower-dimensional compatible manifold. With flexibility provided by SCTD, we can surrogate a computationally heavy teacher backbone with a lightweight student architecture operating strictly within this new compressed space. Finally, we apply temporal distillation to collapse the sampling trajectory, producing a one-step generator that operates at peak efficiency. Our method yields a student generator outperforming existing single-axis acceleration methods in efficiency and throughput, while maintaining competitive generation quality. With reduced peak memory usage and latency, our method enables resource-constrained deployment and high-volume serving of high-fidelity LDM.

Primary area:
generative models
Keywords:
Diffusion Model Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yPwHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework for improving the efficiency of latent diffusion models by jointly addressing spatial resolution, model architecture, and sampling complexity. The authors propose Score-Compatible Tokenizer Distillation, which enables the learning of a more compact latent representation while aiming to maintain compatibility with a pretrained diffusion teacher. A lightweight projection network maps the compressed latents into the teacher's latent space, where score-based distribution matching guides the joint optimization of the encoder and projection network. A reconstruction objective is also incorporated to maintain image reconstruction quality. Using the resulting tokenizer, the authors train a smaller diffusion model in the compressed latent space and further reduce the sampling process to a single step through distribution-matching distillation. Experiments on ImageNet with diffusion transformers show savings in computational cost and memory consumption, accompanied by some loss in generation quality.

Critical strengths
  • The paper addresses a limitation of existing diffusion acceleration approaches by combining latent spatial compression with architectural and temporal optimization.
  • The use of a pretrained diffusion teacher to guide compressed tokenizer learning through score-based distribution alignment is interesting, extending conventional reconstruction-based compression.
  • The training-time bridge enables smaller student models with different architectures and latent resolutions to benefit from teacher score supervision. The bridge and auxiliary score network are removed during inference.
Critical weaknesses
  • Although the proposed framework combines spatial, architectural, and temporal acceleration, it remains unclear whether similar benefits could be achieved by combining existing acceleration techniques. Furthermore, the additional training costs of the compressed tokenizer and auxiliary score network are not sufficiently analyzed. A comparison of overall training costs would help clarify whether the inference-efficiency gains justify the additional training overhead.
  • The architectural compression strategy relies on training a smaller diffusion backbone from scratch rather than pruning the pretrained teacher. Could the authors discuss the advantages of this choice over pruning-based alternatives, particularly in terms of training cost and generation quality?
  • DC-Gen addresses the representation gap through embedding alignment and post-training adaptation. Although the authors compare SCTD with a DC-Gen tokenizer baseline, the comparison does not cover its full adaptation pipeline. Moreover, DC-Gen has demonstrated compatibility with few-step distillation methods such as AnyFlow. Further discussion would help clarify the practical advantages of SCTD over these approaches, particularly for temporal distillation.
  • The evaluation primarily relies on FID and IS. Additional metrics, such as class-conditional accuracy and generative precision and recall, would help provide a more comprehensive assessment of generation quality and diversity.
  • The empirical validation is limited to class-conditional ImageNet generation. Experiments on text-to-image models would help demonstrate the applicability of SCTD to more practical diffusion systems.
Questions for authors
  • How much additional training compute does SCTD require, including tokenizer training and the auxiliary score network? Could similar gains be achieved by combining existing acceleration methods?
  • How does training a smaller DiT from scratch compare with pruning-based alternatives in terms of training cost and generation quality?
  • Table 1 seems to consider only a DC-Gen tokenizer baseline. Given that DC-Gen has demonstrated compatibility with few-step distillation, could the authors clarify why its combination with temporal distillation is marked as not directly applicable?
  • Additional metrics, such as class-conditional accuracy and generative precision and recall, would help assess generation quality and diversity beyond FID and IS.
  • Evaluation on text-to-image models would help establish whether SCTD generalizes beyond class-conditional ImageNet generation.
Reviewer_QtbGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors identify latent resolution as an important factor in accelerating latent diffusion models (LDMs) and propose optimizing it jointly with temporal and architectural factors. To this end, they introduce Score-Compatible Tokenizer Distillation (SCTD), which uses score-matching principles to align a spatially compressed latent space with the distribution induced by a frozen teacher diffusion model. This approach aims to distill the teacher's generative prior into a lower-dimensional latent space. The encoder is further integrated into temporal distillation to reduce the sampling process to a single step. Experimental results demonstrate improvements in generation efficiency and quality.

Critical strengths
  1. Score-based encoder training. The authors propose using the teacher's diffusion model to guide the training of the encoder and projection map. Two divergence objectives, including VSD-style KL divergence and SiD-style Fisher divergence, are considered to align the compressed latent representation with the teacher's generative distribution.
Critical weaknesses
  1. Insufficient comparison with straightforward latent compression approaches. The paper does not sufficiently discuss the differences between the proposed SCTD and DC-Gen approaches. Since reducing the spatial resolution of the latent representation is a main contribution, a straightforward baseline would be to retrain a VAE with a smaller latent spatial size. It is unclear how much improvement comes from the proposed score-based distillation rather than simply reducing the latent resolution. A comparison with this baseline would help clarify the contribution of SCTD.
  2. High training cost and unclear advantages over simpler approaches. The proposed method requires an auxiliary score network, which is trained alternately with SCTD. This introduces additional computational costs associated with score function learning. However, the paper does not provide a sufficient comparison of training costs with baseline methods. It is therefore unclear whether the improvements in generation efficiency justify the additional training complexity and computational cost.

Minor Comments

  1. Inconsistent citation style. The in-text citation style is inconsistent. For example, the citations in lines 342–351 use a different style from those in lines 353–364. The authors should use a consistent citation format throughout the paper.
  2. Inconsistent notation. In lines 207–209, the parameter is denoted by θ\theta, whereas φ\varphi is used elsewhere. The authors should check and unify the notation.
Questions for authors
  1. Comparison with a straightforward baseline. The authors should compare the proposed SCTD framework with a straightforward baseline, such as retraining a VAE with a reduced latent spatial resolution, to better demonstrate the benefits of the proposed method.
  2. Training details of SCTD. The authors should provide more details on how the encoder and projection map are trained in the SCTD framework, especially the training procedures.
Reviewer_oPjNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies joint acceleration of latent diffusion models across latent spatial resolution, network size, and sampling steps. It proposes SCTD, which maps compressed encodings into the teacher’s latent space through a bridge and trains the tokenizer using distribution-matching signals from teacher and auxiliary score networks alongside reconstruction objectives. A lightweight diffusion model is then trained in the compressed space and distilled into a one-step generator. Experiments on ImageNet demonstrate a tradeoff between generation quality and inference compute and memory.

Critical strengths
  • Uses teacher score guidance to train compressed tokenizers, providing a practical mechanism for cross-space distillation.
  • Reports compute, measured latency, memory, and generation quality, presenting multiple aspects of the efficiency–quality tradeoff.
Critical weaknesses
  • Missing ablations for the core design choices. The experiments do not establish whether adapting the tokenizer is necessary compared with freezing a reconstruction-trained tokenizer and training only the bridge. A comparison against sample-wise MSE alignment between student and teacher encodings is also missing. These controls would clarify the contribution of tokenizer adaptation and the advantage of score-based distribution matching over simpler alternatives.
  • Insufficient evidence of semantic consistency across latent spaces. Matching conditional distributions after the bridge does not explicitly ensure that semantic content is preserved across the mapping. A projected latent may match the requested class in the teacher’s space while the original student latent decodes to different content. The reported FID and IS cannot rule out this potential mismatch, and conditional-consistency evaluation is needed.
Questions for authors
  • What are the latent channel counts for the teacher and each compressed tokenizer? What exact architecture, initialization, and training configuration are used for the reimplemented DC-Gen baseline?
  • How does the efficiency–quality tradeoff compare against the teacher using practical sampling schedules with substantially fewer than 1,000 steps?
  • Could the authors report paired reconstruction metrics such as PSNR, SSIM, and LPIPS, alongside rFID, to assess preservation of the input content?
  • Could the overlap between the bottom of Figure 1 and the following paragraph on page 2 be corrected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.