- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The author propose changing the reverse process of diffusion models, named SCDM. Under the intuition that in nature high-dimensional things often have a lower-dimensional behavior (e.g. the complex motion of a pendulum can be described by a simple equation), they propose a reverse process to allow the model implicitly learning such low-dimensional connections.
Specifically, during the reverse process the prediction is over a compressed intermediate representation , instead of over directly. During the reverse process is first “upsampled” (lifted) to an auxiliary overparametrized dimension, before being projected and compressed to some representation . The up-and-down procedure helps the method “find” the lower-dimensional behavior due to the redundant representations.
The results are promising on smaller physical datasets, however I have questions regarding this model’s applicability in real-world high-res cases such as images and videos.
Critical strengths
- The motivation and idea are intuitive and make sense.
- On the selected physical benchmarks used the results seem promising.
- Code is already available which I commend.
- The writing of the paper is very clear and concise.
- Some ablation experiments are very nice, for example Table 5 which shows it’s indeed the lifting responsible for the heavy lifting of the method, that it also helps DDPM and that without it SCDM isn’t as good. Also the interpretability figures help verify the latent space compression claims.
- The robustness to noise in the input data is nice.
Critical weaknesses
- There’s missing theoretical grounding. The original DDPM noise prediction is heavily grounded in theory, where is a parametrized from an approximation on the variational lower bound of the data log-likelihood. The move here in SCDMs to some arbitrary bottleneck isn’t done with any theoretical justification regarding as to how it affects ELBO minimization, posterior approximation error, or anything.
- From my understanding in L258 - you introduce softmax on the samples dimension. Specifically, this means that the representation is coupled across the samples in the batch - thus you can’t sample at inference with a batch of 1? In appendix D you investigate the effect of this batch size - you didn’t provide B=1 but you mention that indeed lower batch sizes are less stable (with only 20% stable samples at B=16). A generative model that cannot sample a single sample isn’t very practical from compute concerns, and it’s a bit weird since you cannot draw iid samples from the distribution.
- related: While that’s apparently what you do for the physical datasets, in L469 you say that for images you reshape the image samples such that now n isn’t the different samples, but rather features of the sample. So in the image domain you apply dot-product between image features and apply softmax - which seems like adding an attention block to the UNet architecture. Can it be then that the improvement for CIFAR is just that, since we know that attention helps in images? Does low-dimensional structure recovery here is then not what happens here?
- Not citing the original paper which proposed diffusion models is egregious (Sohl-Dickstein, Jascha, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. "Deep unsupervised learning using nonequilibrium thermodynamics." In International conference on machine learning, pp. 2256-2265. pmlr, 2015.).
- I’m concerned regarding the applicability to high-resolution data. Most experiments use very low-dimensions states (e.g. for x,y in physical systems), that are lifted to . The authors show results on CIFAR-10, where the results are lifted to . Is this method practical for real-world high-resolution images or videos? DDPMs have been shown to be very effective at capturing the distribution of images and videos, and their naïve applicability with LDMs help vastly with their applicability to high-res images and videos. In the paper, the authors just compare examine the image domain using CIFAR-10. However, CIFAR-10 is a small, unvaried dataset, highly clustered, where low-rank class-conditional features might easily be exploited by this structured approach. DDPMs are instead known for their robustness to a wider variety of images. The submission lacks evaluation on standard image benchmarks, even relatively low-res ones (ImageNet 64x64, CelebA). The limitations section don’t discusses this and it’s not enough.
- related: Table 7 shows throuput only on the small LA toy dataset, and claims that the runtime overhead of SCDM is acceptable (L742). However, the authors don’t say SCDM is only applicable to small physical data, they claim it’s for images as well - the overhead might not be acceptable there. Reporting throughput only on such a small dataset masks the VRAM used during lifting, and the real cost on real-world data. Wall-clock sampling time in actual seconds is needed, and on image data.
- related, results on CIFAR only include visual results for SCDM and we don’t see any comparison visually to DDPMs.
- related: In Fig 12 it would be better to have the same images shown for the different noise levels, and ordered so that we can see how the noise affects the generation.
- the value of noise added to the images from L461 is never mentioned in the text.
- Some weird experiments and presentations:
- Table 1 says RMAE of 24.59, for a lifting and . However, table 2 using has a reported RMAE of for the same LA data. What is this discrepancy?.
- Table 1 is missing confidence intervals (and is the only table without them, apart from the FID values in Tab3). Why? Some of the methods score double so margins here, and the Cis from the other tables suggest the margins between the different methods is smaller than the STDs.
- In table 5, the best MDN results are DDPM. In fact, lifting and projection helps the RMAE in DDPM as well, but shows that it causes the MDN to deteriorate significantly. Why does the MDN falters so much? If lifting and projection is truly what’s responsible for capturing the underlying lower-dimensional behavior, it should improved or preserve the MDN, no?
- Fig. 1 presents clean qualitative samples that demonstrate how SCDM outperform DDPMs, yet Fig .7 seems way less clean, revealing similar behvaiour as seen for DDPM in Fig1 - but for SCDM, on LH and PH (blobs outside the distribution). This induces suspicion of cherry picking. A direct comparison to DDPM for these cases in Fig 7 is needed to see how the methods really compare.
Minor:
- Some unclear writings: E.g. L34 “In the scenarios” - which scenarios?
- Fig2 – the reverse and forward arrows are opposite, since X_T is on the left.
- L197 – a priori.
- Table 1 would benefit from higher/lower is better arrows.
Questions for authors
My questions are basically the weaknesses:
- How is the theory affected by the compression bottleneck?
- What’s the deal with the discrepancy of Table 1 and 5 RMAE? What’s the in Tab1? What are the Cis?
- Why is the MDN so prone to degrading?
- How would you answer my concerns regarding the high-resolution? What is a real overhead of images, both on the small CIFAR-10 and for standard resolution of images (eg 256x256).
- Is it true that for the physical domains it would be numerically impractical to sample B=1?
- Can you explain if indeed you reshaped the features in the images domain such that there there’s no correlation between samples (like there is for physical domains) but now we have an attention block (and if so, how can this be thought of a fair comparison to a non-attention based DDPM model)?
- I’m contemplation the connection between this approach to LDMs. The compressed dimension can be thought of as the latent space of the VAE, however you don’t decode from the latent space, but rather generate directly in the final dimension based on the encoded space. Can this be thought of as similar to learning the encoder of the VAE while using the idea of latent-space-conditioned decoding of PiD ?
- Lu, Yifan, Qi Wu, Jay Zhangjie Wu, Zian Wang, Huan Ling, Sanja Fidler, and Xuanchi Ren. "PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion." arXiv preprint arXiv:2605.23902 (2026)
- How is this work related to Pai, Druv, Sam Buchanan, Ziyang Wu, Yaodong Yu, and Yi Ma. "Masked Completion via Structured Diffusion with White-Box Transformers." In ICLR. 2024., which show that under distribitions with low-dim structure denoising implements a projection like compression to this structure, and employ CRATE compression to masked auto encoders?
- How are the PIDM results so bad considering it requires explicit equations given?
11 comments
4. Scalability, Computational Overhead, and CIFAR-10 Evaluation
We thank the reviewer for these valuable comments.
Regarding scalability to high-resolution images, SCDM is applied to the U-Net bottleneck features rather than directly to the original image pixels. As described in Section 5.3, we reshape these features into , where and denote the channel and spatial dimensions, respectively. For CIFAR-10, the bottleneck features are lifted from 256 to 512 dimensions and projected onto a learned subspace with . Thus, the method is not restricted to low-dimensional physical states. To further investigate its scalability to higher-resolution images, we have also initiated experiments on the LSUN Church dataset, which are currently ongoing. Once the experiments are completed, we will report the results in the OpenReview comments and incorporate them into the revised manuscript.
Regarding computational overhead, we further evaluated the sampling efficiency of SCDM on CIFAR-10 using 1,000 denoising steps and a batch size of 16. Compared with the same backbone without the SCDM module, Full SCDM increases the sampling time only slightly, from 22.074 ± 0.025 s to 22.945 ± 0.102 s, corresponding to an overhead of approximately 3.95%. Meanwhile, both configurations exhibit nearly identical peak GPU memory consumption (0.285 GB), and the parameter count increases only marginally (35.87M vs. 35.88M). These results suggest that the proposed structural compression module introduces limited additional computational overhead during inference.
Regarding CIFAR-10 evaluation, we agree that direct visual comparisons with DDPM under matched noise conditions would provide a clearer assessment of generation quality. In our corruption protocol, 0%, 30%, 40%, 50%, and 60% of training images are selected for Gaussian corruption, with a fixed noise standard deviation of in the normalized pixel space .
5. Experimental Consistency and Presentation
Discrepancy in RMAE results between Tables 1 and 2 under the same experimental configuration. Missing confidence intervals in Table 1 and concerns about the statistical significance of performance differences. Deterioration of MDN %CE in Table 5 despite improvements in RMAE. Apparent inconsistencies between Figures 1 and 7, raising concerns about cherry-picking and the lack of direct DDPM comparisons.
Regarding the discrepancy between Tables 1 and 2, although the experimental configurations are identical (, ), the results may vary across different random seeds. The experiments in Table 2 were independently rerun as part of our ablation study, resulting in slight numerical variations. Specifically, the RMAE of 24.59 reported in Table 1 falls within the range of 25.22 ± 0.69 reported in Table 2. Therefore, this discrepancy is attributable to random-seed variability rather than differences in experimental settings.
Regarding the missing uncertainty estimates in Table 1, for clarity and readability, we report only the mean values in Table 1 to provide a concise overview of the performance comparisons. The variability across random seeds is further examined and reported as mean ± standard deviation in our subsequent experiments, including the ablation and sensitivity analyses.
Regarding the MDN results in Table 5, MDN %CE measures the center offset between the generated and real data distributions, reflecting their agreement in mean position rather than overall generation quality. Although DDPM achieves a lower MDN %CE (4.51 vs. 6.70), both methods exhibit relatively large standard deviations (2.90 and 3.73, respectively). Importantly, generation quality should be evaluated comprehensively across all three metrics. Full SCDM () achieves the best RMAE and %VFE, indicating improvements in other aspects of distributional fidelity. Therefore, these results reflect trade-offs among different distributional statistics rather than a uniform improvement across all metrics.
Regarding the qualitative results in Figures 1 and 7, the purpose of Figure 7 in the Appendix is to visualize the relationship between the generated samples of SCDM and the original data distributions across different physical datasets, rather than to provide direct comparisons with baseline methods. The qualitative comparison between SCDM and DDPM is presented in Figure 1, while quantitative comparisons with baseline methods are reported in Table 1.
6. Experimental Consistency and Presentation
Can you explain if indeed you reshaped the features in the images domain such that there there’s no correlation between samples (like there is for physical domains) but now we have an attention block (and if so, how can this be thought of a fair comparison to a non-attention based DDPM model)?
We acknowledge that the similarity-based operation in Eq. (10) resembles self-attention. However, unlike a standalone attention block, this operation is performed within the learned low-dimensional subspace as part of our coding-rate compression mechanism, aiming to capture and compress the underlying structural information. Moreover, SCDM and DDPM use the same underlying U-Net architecture and attention configuration, ensuring a consistent backbone for comparison. Our ablation studies in Appendix E further investigate the contributions of lifting, projection, and compression to the overall performance.
7. Connection to Latent Diffusion and PiD
I’m contemplation the connection between this approach to LDMs. The compressed dimension can be thought of as the latent space of the VAE, however you don’t decode from the latent space, but rather generate directly in the final dimension based on the encoded space. Can this be thought of as similar to learning the encoder of the VAE while using the idea of latent-space-conditioned decoding of PiD ?
We thank the reviewer for this insightful perspective. We agree that SCDM shares the general idea of leveraging compact latent representations to facilitate generation. However, unlike LDMs, which perform diffusion in a learned latent space, SCDM retains the reverse diffusion process in the original data space.
While PiD employs latent representations as conditioning signals for a pixel-space diffusion decoder, SCDM integrates low-dimensional structural compression directly into the denoising network. Specifically, at each denoising step, intermediate features are lifted and projected onto a learned low-dimensional subspace, where coding-rate compression is applied before noise prediction. These representations are jointly optimized through the DDPM denoising objective, without requiring a separately trained VAE encoder or an independent latent-conditioned decoder.
Therefore, although SCDM is conceptually related to latent-guided generation, its key distinction lies in incorporating low-dimensional structural constraints into the denoising process itself, rather than performing latent-to-pixel decoding as a separate generative stage.
8. Connection to CRATE-MAE
How is this work related to Pai, Druv, Sam Buchanan, Ziyang Wu, Yaodong Yu, and Yi Ma. "Masked Completion via Structured Diffusion with White-Box Transformers." In ICLR. 2024., which show that under distribitions with low-dim structure denoising implements a projection like compression to this structure, and employ CRATE compression to masked auto encoders?
Pai et al. develop an interpretable white-box Transformer framework from a structured diffusion perspective, in which each encoder layer progressively transforms image representations toward a parsimonious (e.g., compressed and sparse) representation, while the corresponding decoder layers progressively reconstruct the original image.
Our motivation is that the diffused data should be denoised under some low-dimensional structures, so that the recovered data could be robust. To this end, our primary goal is to leverage compression-based representation learning to improve diffusion models, allowing the generative process to be guided by the underlying low-dimensional structures and thereby produce more robust synthetic data. Our work is inspired by their coding-rate compression mechanism but differs in its objective and application. SCDM incorporates low-dimensional structural compression directly into the generative process of DDPMs. Specifically, the intermediate features are first lifted into a higher-dimensional space and projected onto a learned low-dimensional subspace. Coding-rate compression is then applied within this task-relevant subspace to suppress redundant variations and refine the representations before noise prediction.
1. Theoretical Justification of SCDM
We thank the reviewer for this important comment. From an information-theoretic perspective, the forward diffusion process progressively adds Gaussian noise, increasing the uncertainty and effective coding rate of the data distribution, as established in Theorem B.2 of [1]. Conversely, reverse denoising transforms a highly noisy, high-entropy sample toward a more structured and lower-entropy sample from the data distribution. This motivates us to introduce coding-rate compression before noise prediction to facilitate the DDPM denoising process.
Specifically, let ϕψ denote the lifting network, U denote the learned shared low-dimensional subspace, Cκ denote the coding-rate compression operator (see Equation 10), and ϵθ denote the denoising network. Accordingly, the standard DDPM denoising objective can be written in our framework as
minθ,ψ,UEϵ−ϵθ(Cκ(U⊤ϕψ(xt)),t)22.
Through this objective, the DDPM denoising objective jointly learns the lifting network and a shared low-dimensional subspace that retains information useful for denoising. Coding-rate compression is then applied within this task-relevant subspace to suppress redundant variations and further refine the representation before it is passed to the denoising network. Therefore, the two objectives play complementary roles: the DDPM denoising objective identifies which low-dimensional information is useful for noise prediction, while coding-rate compression removes redundancy within that representation.
2. Batch-Coupled Sampling and Low-Dimensional Structure Learning
Regarding batch dependence, we thank the reviewer for this observation. Our primary motivation is to identify and compress the underlying low-dimensional structure of data, thereby providing structural constraints for the generation process. In our physical-data experiments, the columns of Zt represent samples within a batch, and the similarity-based interactions in Eq. (10) are designed to capture their shared low-dimensional structure. During inference, we first independently sample a batch of initial noise vectors from the standard Gaussian distribution and then perform the reverse denoising process jointly, using the learned subspace basis U and denoising network, with similarity weights recomputed at each step. Consequently, our physical-data implementation adopts batch-coupled generation rather than independent sample generation. Individual samples can still be obtained from the jointly generated batch, corresponding to sampling from its marginal distribution. As already reported in Appendix D (Table 4), our batch-size sensitivity analysis over ten random seeds shows that smaller batches are more prone to numerical instability.
Regarding image generation, as described in Section 5.3, we reshape the U-Net bottleneck features into Xt∈RD×n, where D and n denote the channel and spatial dimensions, respectively. We agree that the similarity-based interactions have an attention-like form. However, unlike directly introducing a conventional attention module, SCDM first lifts the features into a higher-dimensional space and projects them onto a learned low-dimensional subspace, where structure-aware compression is performed. Thus, the similarity interactions operate on explicitly constrained low-dimensional representations rather than the original features. Our ablation studies (Appendix E) further demonstrate the complementary contributions of these components. Specifically, the DDPM denoising objective identifies the low-dimensional information relevant to noise prediction, while coding-rate compression reduces redundancy within the learned representations. Together, these two objectives encourage SCDM to preserve task-relevant information while learning compact, structured representations.
3. Citation of Foundational Diffusion Models
We thank the reviewer for pointing out this omission. We acknowledge the foundational contribution of Sohl-Dickstein et al. (2015), who introduced diffusion probabilistic models through nonequilibrium thermodynamics. We apologize for overlooking this important reference and have now added the citation in the Introduction.