acceptodds
Under review as a conference paper at ICLR 2027

Structure-aware Compression Diffusion Models

Abstract

The conventional reverse diffusion process focuses on the log-likelihoods based denoising, which may neglect the intrinsic laws in the natural world. In this work, we propose the Structure-aware Compression Diffusion Models (SCDM), with a novel reverse diffusion process that enforces the lifting and compression operation towards learning the low-dimensional structures of the original data. Our SCDM outperforms the conventional diffusion models on both physical datasets and image datasets. Besides, the experiments demonstrate that the low-dimensional structures learned by SCDM align well with scientific theories in several physical data generation tasks.

Primary area:
generative models
Keywords:
Diffusion Models, Representation Learning, Applications to Physical Sciences

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4xZmOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The author propose changing the reverse process of diffusion models, named SCDM. Under the intuition that in nature high-dimensional things often have a lower-dimensional behavior (e.g. the complex motion of a pendulum can be described by a simple equation), they propose a reverse process to allow the model implicitly learning such low-dimensional connections.

Specifically, during the reverse process the ε\varepsilon prediction is over a compressed intermediate representation Z~t\tilde{Z}_t, instead of over xtx_t directly. During the reverse process xtx_t is first “upsampled” (lifted) to an auxiliary overparametrized dimension, before being projected and compressed to some representation Z~t\tilde{Z}_t. The up-and-down procedure helps the method “find” the lower-dimensional behavior due to the redundant representations.

The results are promising on smaller physical datasets, however I have questions regarding this model’s applicability in real-world high-res cases such as images and videos.

Critical strengths
  • The motivation and idea are intuitive and make sense.
  • On the selected physical benchmarks used the results seem promising.
  • Code is already available which I commend.
  • The writing of the paper is very clear and concise.
  • Some ablation experiments are very nice, for example Table 5 which shows it’s indeed the lifting responsible for the heavy lifting of the method, that it also helps DDPM and that without it SCDM isn’t as good. Also the interpretability figures help verify the latent space compression claims.
  • The robustness to noise in the input data is nice.
Critical weaknesses
  • There’s missing theoretical grounding. The original DDPM noise prediction is heavily grounded in theory, where ε\varepsilon is a parametrized from an approximation on the variational lower bound of the data log-likelihood. The move here in SCDMs to some arbitrary bottleneck isn’t done with any theoretical justification regarding as to how it affects ELBO minimization, posterior approximation error, or anything.
  • From my understanding in L258 - you introduce softmax on the samples dimension. Specifically, this means that the representation is coupled across the samples in the batch - thus you can’t sample at inference with a batch of 1? In appendix D you investigate the effect of this batch size - you didn’t provide B=1 but you mention that indeed lower batch sizes are less stable (with only 20% stable samples at B=16). A generative model that cannot sample a single sample isn’t very practical from compute concerns, and it’s a bit weird since you cannot draw iid samples from the distribution.
    • related: While that’s apparently what you do for the physical datasets, in L469 you say that for images you reshape the image samples such that now n isn’t the different samples, but rather features of the sample. So in the image domain you apply dot-product between image features and apply softmax - which seems like adding an attention block to the UNet architecture. Can it be then that the improvement for CIFAR is just that, since we know that attention helps in images? Does low-dimensional structure recovery here is then not what happens here?
  • Not citing the original paper which proposed diffusion models is egregious (Sohl-Dickstein, Jascha, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. "Deep unsupervised learning using nonequilibrium thermodynamics." In International conference on machine learning, pp. 2256-2265. pmlr, 2015.).
  • I’m concerned regarding the applicability to high-resolution data. Most experiments use very low-dimensions uu states (e.g. u=2u=2 for x,y in physical systems), that are lifted to D=64D=64. The authors show results on CIFAR-10, where the results are lifted to D=512D=512. Is this method practical for real-world high-resolution images or videos? DDPMs have been shown to be very effective at capturing the distribution of images and videos, and their naïve applicability with LDMs help vastly with their applicability to high-res images and videos. In the paper, the authors just compare examine the image domain using CIFAR-10. However, CIFAR-10 is a small, unvaried dataset, highly clustered, where low-rank class-conditional features might easily be exploited by this structured approach. DDPMs are instead known for their robustness to a wider variety of images. The submission lacks evaluation on standard image benchmarks, even relatively low-res ones (ImageNet 64x64, CelebA). The limitations section don’t discusses this and it’s not enough.
    • related: Table 7 shows throuput only on the small LA toy dataset, and claims that the runtime overhead of SCDM is acceptable (L742). However, the authors don’t say SCDM is only applicable to small physical data, they claim it’s for images as well - the overhead might not be acceptable there. Reporting throughput only on such a small dataset masks the VRAM used during lifting, and the real cost on real-world data. Wall-clock sampling time in actual seconds is needed, and on image data.
    • related, results on CIFAR only include visual results for SCDM and we don’t see any comparison visually to DDPMs.
      • related: In Fig 12 it would be better to have the same images shown for the different noise levels, and ordered so that we can see how the noise affects the generation.
      • the value of noise added to the images σ\sigma from L461 is never mentioned in the text.
  • Some weird experiments and presentations:
    • Table 1 says RMAE of 24.59, for a lifting D=64D=64 and u=6u=6. However, table 2 using D=64D=64 has a reported RMAE of 25.22±0.6925.22\pm0.69 for the same LA data. What is this discrepancy?.
    • Table 1 is missing confidence intervals (and is the only table without them, apart from the FID values in Tab3). Why? Some of the methods score double so margins here, and the Cis from the other tables suggest the margins between the different methods is smaller than the STDs.
    • In table 5, the best MDN results are DDPM. In fact, lifting and projection helps the RMAE in DDPM as well, but shows that it causes the MDN to deteriorate significantly. Why does the MDN falters so much? If lifting and projection is truly what’s responsible for capturing the underlying lower-dimensional behavior, it should improved or preserve the MDN, no?
    • Fig. 1 presents clean qualitative samples that demonstrate how SCDM outperform DDPMs, yet Fig .7 seems way less clean, revealing similar behvaiour as seen for DDPM in Fig1 - but for SCDM, on LH and PH (blobs outside the distribution). This induces suspicion of cherry picking. A direct comparison to DDPM for these cases in Fig 7 is needed to see how the methods really compare.

Minor:

  • Some unclear writings: E.g. L34 “In the scenarios” - which scenarios?
  • Fig2 – the reverse and forward arrows are opposite, since X_T is on the left.
  • L197 – a priori.
  • Table 1 would benefit from higher/lower is better arrows.
Questions for authors

My questions are basically the weaknesses:

  • How is the theory affected by the compression bottleneck?
  • What’s the deal with the discrepancy of Table 1 and 5 RMAE? What’s the uu in Tab1? What are the Cis?
  • Why is the MDN so prone to degrading?
  • How would you answer my concerns regarding the high-resolution? What is a real overhead of images, both on the small CIFAR-10 and for standard resolution of images (eg 256x256).
  • Is it true that for the physical domains it would be numerically impractical to sample B=1?
  • Can you explain if indeed you reshaped the features in the images domain such that there there’s no correlation between samples (like there is for physical domains) but now we have an attention block (and if so, how can this be thought of a fair comparison to a non-attention based DDPM model)?
  • I’m contemplation the connection between this approach to LDMs. The compressed dimension can be thought of as the latent space of the VAE, however you don’t decode from the latent space, but rather generate directly in the final dimension based on the encoded space. Can this be thought of as similar to learning the encoder of the VAE while using the idea of latent-space-conditioned decoding of PiD ?
    • Lu, Yifan, Qi Wu, Jay Zhangjie Wu, Zian Wang, Huan Ling, Sanja Fidler, and Xuanchi Ren. "PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion." arXiv preprint arXiv:2605.23902 (2026)
  • How is this work related to Pai, Druv, Sam Buchanan, Ziyang Wu, Yaodong Yu, and Yi Ma. "Masked Completion via Structured Diffusion with White-Box Transformers." In ICLR. 2024., which show that under distribitions with low-dim structure denoising implements a projection like compression to this structure, and employ CRATE compression to masked auto encoders?
  • How are the PIDM results so bad considering it requires explicit equations given?
11 comments

1. Theoretical Justification of SCDM

How is the theory affected by the compression bottleneck?

We thank the reviewer for this important comment. From an information-theoretic perspective, the forward diffusion process progressively adds Gaussian noise, increasing the uncertainty and effective coding rate of the data distribution, as established in Theorem B.2 of [1]. Conversely, reverse denoising transforms a highly noisy, high-entropy sample toward a more structured and lower-entropy sample from the data distribution. This motivates us to introduce coding-rate compression before noise prediction to facilitate the DDPM denoising process.

Specifically, let ϕψ\phi_\psi denote the lifting network, UU denote the learned shared low-dimensional subspace, CκC_\kappa denote the coding-rate compression operator (see Equation 10), and ϵθ\epsilon_\theta denote the denoising network. Accordingly, the standard DDPM denoising objective can be written in our framework as

min⁡θ,ψ,UE∥ϵ−ϵθ(Cκ(U⊤ϕψ(xt)),t)∥22.\min_{\theta,\psi,U}\mathbb{E}\left\|\epsilon-\epsilon_\theta\left(C_\kappa\left(U^\top\phi_\psi(x_t)\right),t\right)\right\|_2^2.

Through this objective, the DDPM denoising objective jointly learns the lifting network and a shared low-dimensional subspace that retains information useful for denoising. Coding-rate compression is then applied within this task-relevant subspace to suppress redundant variations and further refine the representation before it is passed to the denoising network. Therefore, the two objectives play complementary roles: the DDPM denoising objective identifies which low-dimensional information is useful for noise prediction, while coding-rate compression removes redundancy within that representation.

2. Batch-Coupled Sampling and Low-Dimensional Structure Learning

The reviewer raises concerns about the batch dependence introduced by the softmax operation, particularly the feasibility of singleton sampling and the sensitivity to batch size. For image generation, the reviewer further asks whether the improvements on CIFAR-10 arise primarily from attention-like interactions rather than low-dimensional structure recovery.

Regarding batch dependence, we thank the reviewer for this observation. Our primary motivation is to identify and compress the underlying low-dimensional structure of data, thereby providing structural constraints for the generation process. In our physical-data experiments, the columns of ZtZ_t represent samples within a batch, and the similarity-based interactions in Eq. (10) are designed to capture their shared low-dimensional structure. During inference, we first independently sample a batch of initial noise vectors from the standard Gaussian distribution and then perform the reverse denoising process jointly, using the learned subspace basis UU and denoising network, with similarity weights recomputed at each step. Consequently, our physical-data implementation adopts batch-coupled generation rather than independent sample generation. Individual samples can still be obtained from the jointly generated batch, corresponding to sampling from its marginal distribution. As already reported in Appendix D (Table 4), our batch-size sensitivity analysis over ten random seeds shows that smaller batches are more prone to numerical instability.

Regarding image generation, as described in Section 5.3, we reshape the U-Net bottleneck features into Xt∈RD×nX_t\in\mathbb{R}^{D\times n}, where DD and nn denote the channel and spatial dimensions, respectively. We agree that the similarity-based interactions have an attention-like form. However, unlike directly introducing a conventional attention module, SCDM first lifts the features into a higher-dimensional space and projects them onto a learned low-dimensional subspace, where structure-aware compression is performed. Thus, the similarity interactions operate on explicitly constrained low-dimensional representations rather than the original features. Our ablation studies (Appendix E) further demonstrate the complementary contributions of these components. Specifically, the DDPM denoising objective identifies the low-dimensional information relevant to noise prediction, while coding-rate compression reduces redundancy within the learned representations. Together, these two objectives encourage SCDM to preserve task-relevant information while learning compact, structured representations.

3. Citation of Foundational Diffusion Models

Not citing the original paper

We thank the reviewer for pointing out this omission. We acknowledge the foundational contribution of Sohl-Dickstein et al. (2015), who introduced diffusion probabilistic models through nonequilibrium thermodynamics. We apologize for overlooking this important reference and have now added the citation in the Introduction.

4. Scalability, Computational Overhead, and CIFAR-10 Evaluation

We thank the reviewer for these valuable comments.

Regarding scalability to high-resolution images, SCDM is applied to the U-Net bottleneck features rather than directly to the original image pixels. As described in Section 5.3, we reshape these features into Xt∈RD×nX_t\in\mathbb{R}^{D\times n}, where DD and nn denote the channel and spatial dimensions, respectively. For CIFAR-10, the bottleneck features are lifted from 256 to 512 dimensions and projected onto a learned subspace with u=32u=32. Thus, the method is not restricted to low-dimensional physical states. To further investigate its scalability to higher-resolution images, we have also initiated experiments on the LSUN Church dataset, which are currently ongoing. Once the experiments are completed, we will report the results in the OpenReview comments and incorporate them into the revised manuscript.

Regarding computational overhead, we further evaluated the sampling efficiency of SCDM on CIFAR-10 using 1,000 denoising steps and a batch size of 16. Compared with the same backbone without the SCDM module, Full SCDM increases the sampling time only slightly, from 22.074 ± 0.025 s to 22.945 ± 0.102 s, corresponding to an overhead of approximately 3.95%. Meanwhile, both configurations exhibit nearly identical peak GPU memory consumption (0.285 GB), and the parameter count increases only marginally (35.87M vs. 35.88M). These results suggest that the proposed structural compression module introduces limited additional computational overhead during inference.

Regarding CIFAR-10 evaluation, we agree that direct visual comparisons with DDPM under matched noise conditions would provide a clearer assessment of generation quality. In our corruption protocol, 0%, 30%, 40%, 50%, and 60% of training images are selected for Gaussian corruption, with a fixed noise standard deviation of σ=0.1\sigma=0.1 in the normalized pixel space [0,1][0,1].

5. Experimental Consistency and Presentation

Discrepancy in RMAE results between Tables 1 and 2 under the same experimental configuration. Missing confidence intervals in Table 1 and concerns about the statistical significance of performance differences. Deterioration of MDN %CE in Table 5 despite improvements in RMAE. Apparent inconsistencies between Figures 1 and 7, raising concerns about cherry-picking and the lack of direct DDPM comparisons.

Regarding the discrepancy between Tables 1 and 2, although the experimental configurations are identical (D=64D=64, u=6u=6), the results may vary across different random seeds. The experiments in Table 2 were independently rerun as part of our ablation study, resulting in slight numerical variations. Specifically, the RMAE of 24.59 reported in Table 1 falls within the range of 25.22 ± 0.69 reported in Table 2. Therefore, this discrepancy is attributable to random-seed variability rather than differences in experimental settings.

Regarding the missing uncertainty estimates in Table 1, for clarity and readability, we report only the mean values in Table 1 to provide a concise overview of the performance comparisons. The variability across random seeds is further examined and reported as mean ± standard deviation in our subsequent experiments, including the ablation and sensitivity analyses.

Regarding the MDN results in Table 5, MDN %CE measures the center offset between the generated and real data distributions, reflecting their agreement in mean position rather than overall generation quality. Although DDPM achieves a lower MDN %CE (4.51 vs. 6.70), both methods exhibit relatively large standard deviations (2.90 and 3.73, respectively). Importantly, generation quality should be evaluated comprehensively across all three metrics. Full SCDM (u=6u=6) achieves the best RMAE and %VFE, indicating improvements in other aspects of distributional fidelity. Therefore, these results reflect trade-offs among different distributional statistics rather than a uniform improvement across all metrics.

Regarding the qualitative results in Figures 1 and 7, the purpose of Figure 7 in the Appendix is to visualize the relationship between the generated samples of SCDM and the original data distributions across different physical datasets, rather than to provide direct comparisons with baseline methods. The qualitative comparison between SCDM and DDPM is presented in Figure 1, while quantitative comparisons with baseline methods are reported in Table 1.

6. Experimental Consistency and Presentation

Can you explain if indeed you reshaped the features in the images domain such that there there’s no correlation between samples (like there is for physical domains) but now we have an attention block (and if so, how can this be thought of a fair comparison to a non-attention based DDPM model)?

We acknowledge that the similarity-based operation in Eq. (10) resembles self-attention. However, unlike a standalone attention block, this operation is performed within the learned low-dimensional subspace as part of our coding-rate compression mechanism, aiming to capture and compress the underlying structural information. Moreover, SCDM and DDPM use the same underlying U-Net architecture and attention configuration, ensuring a consistent backbone for comparison. Our ablation studies in Appendix E further investigate the contributions of lifting, projection, and compression to the overall performance.

7. Connection to Latent Diffusion and PiD

I’m contemplation the connection between this approach to LDMs. The compressed dimension can be thought of as the latent space of the VAE, however you don’t decode from the latent space, but rather generate directly in the final dimension based on the encoded space. Can this be thought of as similar to learning the encoder of the VAE while using the idea of latent-space-conditioned decoding of PiD ?

We thank the reviewer for this insightful perspective. We agree that SCDM shares the general idea of leveraging compact latent representations to facilitate generation. However, unlike LDMs, which perform diffusion in a learned latent space, SCDM retains the reverse diffusion process in the original data space.

While PiD employs latent representations as conditioning signals for a pixel-space diffusion decoder, SCDM integrates low-dimensional structural compression directly into the denoising network. Specifically, at each denoising step, intermediate features are lifted and projected onto a learned low-dimensional subspace, where coding-rate compression is applied before noise prediction. These representations are jointly optimized through the DDPM denoising objective, without requiring a separately trained VAE encoder or an independent latent-conditioned decoder.

Therefore, although SCDM is conceptually related to latent-guided generation, its key distinction lies in incorporating low-dimensional structural constraints into the denoising process itself, rather than performing latent-to-pixel decoding as a separate generative stage.

8. Connection to CRATE-MAE

How is this work related to Pai, Druv, Sam Buchanan, Ziyang Wu, Yaodong Yu, and Yi Ma. "Masked Completion via Structured Diffusion with White-Box Transformers." In ICLR. 2024., which show that under distribitions with low-dim structure denoising implements a projection like compression to this structure, and employ CRATE compression to masked auto encoders?

Pai et al. develop an interpretable white-box Transformer framework from a structured diffusion perspective, in which each encoder layer progressively transforms image representations toward a parsimonious (e.g., compressed and sparse) representation, while the corresponding decoder layers progressively reconstruct the original image.

Our motivation is that the diffused data should be denoised under some low-dimensional structures, so that the recovered data could be robust. To this end, our primary goal is to leverage compression-based representation learning to improve diffusion models, allowing the generative process to be guided by the underlying low-dimensional structures and thereby produce more robust synthetic data. Our work is inspired by their coding-rate compression mechanism but differs in its objective and application. SCDM incorporates low-dimensional structural compression directly into the generative process of DDPMs. Specifically, the intermediate features are first lifted into a higher-dimensional space and projected onto a learned low-dimensional subspace. Coding-rate compression is then applied within this task-relevant subspace to suppress redundant variations and refine the representations before noise prediction.

Reviewer_6cHuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCDM combines feature lifting, subspace projection, and compression-guided denoising. The results are promising, but stronger methodological justification and controlled evaluation are needed to support the claimed benefits.

Critical strengths
  1. The work explores learned structural priors without explicit physical constraints.
  2. Experiments span physical datasets and image generation under corrupted training data.
  3. The appendices include ablations, sensitivity analyses, runtime measurements, and failure cases.
Critical weaknesses
  1. Please clarify the contribution beyond CRATE [1] and LDM [2]. Comparisons with attention and bottleneck baselines with comparable capacity and compute, together with a CIFAR-10 ablation isolating compression, would help identify the source of the gains.
  2. The approximation from Eq. (8) to Eq. (9) requires clearer justification. Please explain its assumptions, clarify when Eq. (10) reduces coding rate, and compare it with the exact update in Eq. (8).
  3. The sample interactions in Eq. (10) may make generation dependent on batch composition. Please separate training and inference batch effects, test singleton and regrouped sampling, and clarify consistency with the DDPM [3] transition in Eq. (11).
  4. Feature correlations alone provide limited evidence of learning physical laws. Please strengthen Fig. 6 and Appendix I with held-out evaluations and simple feature baselines, justify the energy target, and assess physical consistency using governing equation residuals where applicable.
  5. Table 5 suggests metric trade-offs rather than uniform improvement. Please report uncertainty across runs and collapse rates. Specify the CIFAR-10 corruption strength, training and sampling budgets, and FID [4] evaluation protocol to make the comparisons reproducible.

References: [1] White-Box Transformers via Sparse Rate Reduction, NeurIPS, 2023. [2] High-Resolution Image Synthesis With Latent Diffusion Models, CVPR, 2022. [3] Denoising Diffusion Probabilistic Models, NeurIPS, 2020. [4] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NIPS, 2017.

Questions for authors

Please refer to the weakness part.

3 comments

1. Novelty Beyond CRATE and LDM

Please clarify the contribution beyond CRATE [1] and LDM [2]. Comparisons with attention and bottleneck baselines with comparable capacity and compute, together with a CIFAR-10 ablation isolating compression, would help identify the source of the gains.

Thank you for your valuable suggestion. We would like to clarify that SCDM is not a straightforward combination of CRATE and diffusion models. Our method is motivated by the information-theoretic interpretation of diffusion and introduces a structure-aware compression mechanism specifically designed for the denoising process.

From an information-theoretic perspective, the forward diffusion process progressively adds Gaussian noise, increasing the uncertainty and effective coding rate of the data distribution, as established in Theorem B.2 of [1]. Conversely, reverse denoising transforms a highly noisy, high-entropy sample toward a more structured and lower-entropy sample from the data distribution. This motivates us to introduce coding-rate compression before noise prediction to facilitate the DDPM denoising process.

Specifically, let ϕψ\phi_\psi denote the lifting network, UU denote the learned shared low-dimensional subspace, CκC_\kappa denote the coding-rate compression operator (see Equation 10), and ϵθ\epsilon_\theta denote the denoising network. Accordingly, the standard DDPM denoising objective can be written in our framework as

min⁡θ,ψ,UE∥ϵ−ϵθ(Cκ(U⊤ϕψ(xt)),t)∥22.\min_{\theta,\psi,U}\mathbb{E}\left\|\epsilon-\epsilon_\theta\left(C_\kappa\left(U^\top\phi_\psi(x_t)\right),t\right)\right\|_2^2.

Through this objective, the DDPM denoising objective jointly learns the lifting network and a shared low-dimensional subspace that retains information useful for denoising. Coding-rate compression is then applied within this task-relevant subspace to suppress redundant variations and further refine the representation before it is passed to the denoising network. Therefore, the two objectives play complementary roles: the DDPM denoising objective identifies which low-dimensional information is useful for noise prediction, while coding-rate compression removes redundancy within that representation.

Distinction from CRATE and LDM. Although our method draws inspiration from coding-rate-based representation learning, its formulation and optimization differ substantially from CRATE. CRATE is mainly designed for data containing multiple semantic classes, under the assumption that the learned features do not necessarily lie in a single shared subspace. It therefore introduces multiple subspaces to characterize features. By maximizing R−RcR-R_c, CRATE simultaneously expands the coding rate of the overall representation while compressing the representations within each class-specific subspace, thereby producing discriminative features suitable for downstream tasks such as classification.

During the design of our method, we also considered multiple subspaces. However, SCDM is not a direct insertion of the CRATE formulation into a diffusion model. Our objective differs fundamentally from that of CRATE. We assume that these samples share a common set of underlying dynamical laws. To avoid increasing the computational cost and the number of learnable parameters, we use a single subspace UU. We expect the denoising-relevant features to be organized within a shared low-dimensional subspace after nonlinear lifting. And to obtain a more compact representation, we apply coding rate descent to suppress redundant information. Moreover, our objective is to exploit the intrinsic features of the data to constrain the generative process, rather than to explicitly enforce sparse activations within the subspace. Therefore, the ℓ0\ell_0 sparsity term is not essential in our formulation.

2. Approximation and Coding-Rate Reduction

The approximation from Eq. (8) to Eq. (9) requires clearer justification. Please explain its assumptions, clarify when Eq. (10) reduces coding rate, and compare it with the exact update in Eq. (8).

We thank the reviewer for this insightful comment. The approximation from Eq. (8) to Eq. (9) follows the same derivation principle as CRATE [1] ( Appendix A.2, Eqs. (66)–(76)). Specifically, the exact coding-rate gradient in Eq. (8) is given by

∇ZtR(Zt)=unϵ2Zt(I+unϵ2Zt⊤Zt)−1.\nabla_{Z_t}R(Z_t)=\frac{u}{n\epsilon^2}Z_t\left(I+\frac{u}{n\epsilon^2}Z_t^\top Z_t\right)^{-1}.

Following CRATE, we apply the first-order Neumann expansion to the inverse matrix:

\approx I-\frac{u}{n\epsilon^2}Z_t^\top Z_t.$$ Substituting this approximation into Eq. (8), we obtain $$\nabla_{Z_t}R(Z_t) \approx\frac{u}{n\epsilon^2}Z_t -\left(\frac{u}{n\epsilon^2}\right)^2Z_t(Z_t^\top Z_t).$$ The Gram matrix $Z_t^\top Z_t$ captures pairwise correlations among projected representations. Following the same approximation strategy, we replace the unnormalized Gram matrix with its softmax-normalized similarity weights, yielding $$\nabla_{Z_t}R(Z_t) \approx\frac{u}{n\epsilon^2}Z_t -\left(\frac{u}{n\epsilon^2}\right)^2 Z_t\operatorname{softmax}(Z_t^\top Z_t),$$ which corresponds to Eq. (9). The softmax normalization introduces similarity-aware interactions among projected representations, following the design principle of CRATE. We emphasize that this normalization is a structure-motivated approximation rather than an exact algebraic consequence of the Neumann expansion. Regarding coding-rate reduction, Eq. (10) is given by $$\widetilde Z_t\approx Z_t\left[\left(1-\kappa\frac{u}{n\epsilon^2}\right)I+\kappa\left(\frac{u}{n\epsilon^2}\right)^2\operatorname{softmax}(Z_t^\top Z_t)\right].$$ A sufficient condition for reducing the coding rate is that the spectral norm of the transformation matrix in Eq. (10) is no greater than 1. Under this condition, $\widetilde Z_t\widetilde Z_t^\top\preceq Z_tZ_t^\top$, which implies $$R(\widetilde Z_t) =\frac12\log\det\left(I+\frac{u}{n\epsilon^2}\widetilde Z_t\widetilde Z_t^\top\right) \leq R(Z_t).$$ Since $\operatorname{softmax}(Z_t^\top Z_t)$ is row-stochastic, its spectral norm is bounded by $\sqrt n$. Therefore, $0\leq\kappa\frac{u}{n\epsilon^2}\leq1$ and $\frac{u}{n\epsilon^2}\sqrt n\leq1$ are sufficient to guarantee a non-increasing coding rate. ### 3. Batch Dependence and DDPM Consistency > The sample interactions in Eq. (10) may make generation dependent on batch composition. Please separate training and inference batch effects, test singleton and regrouped sampling, and clarify consistency with the DDPM [3] transition in Eq. (11). Regarding batch dependence, we thank the reviewer for this observation. Our primary motivation is to identify and compress the underlying low-dimensional structure of data, thereby providing structural constraints for the generation process. In our physical-data experiments, the columns of $Z_t$ represent samples within a batch, and the similarity-based interactions in Eq. (10) are designed to capture their shared low-dimensional structure. During inference, the learned subspace basis $U$ and denoising network remain fixed, while the similarity weights are recomputed from the current batch. Consequently, unlike standard DDPM with independent sample generation, our physical-data implementation introduces batch-dependent interactions. As reported in Appendix D (Table 4), our batch-size sensitivity analysis over ten random seeds shows that larger batch sizes improve numerical stability. For image generation, as described in Section 5.3, we reshape the U-Net bottleneck features into $X_t\in\mathbb{R}^{D\times n}$, where $D$ and $n$ denote the channel and spatial dimensions, respectively. The subspace projection operates along the channel dimension, while similarity interactions occur among spatial positions within each image. This allows SCDM to capture spatial structural relationships and improve generation quality. Therefore, the image-generation implementation remains consistent with the sample-wise DDPM reverse transition in Eq. (11), while the physical-data implementation retains its Gaussian form.

4. Physical Structure and Interpretability

Feature correlations alone provide limited evidence of learning physical laws.

As discussed in our manuscript, "The underlying physical processes are governed by a small number of low-dimensional structures and interactions." In many real-world scenarios, observed data are constrained by underlying low-dimensional structures and therefore often exhibit compressibility. Motivated by this observation, as stated in Section 1, "The intuition is that the diffused data should be denoised under some low-dimensional structures, so that the recovered data could be robust." Our primary goal is to leverage compression-based representation learning to improve diffusion models, allowing the generative process to be guided by the underlying low-dimensional structures and thereby produce more robust synthetic data, rather than explicitly discovering the governing physical equations underlying the observed data.

To provide complementary evidence beyond feature correlations, we conduct additional experiments on three datasets: Circle, Linear Oscillator (LA), and Pendulum (PA). We compare the conventional DDPM, Physics-Informed Diffusion Models (PIDM), and our SCDM. Unlike PIDM, which explicitly incorporates physical-equation residuals into its training objective, SCDM learns the data structure without access to these equations. We evaluate the generated samples using the root mean squared residual of the corresponding geometric or empirical state equation:

Physics RMSE=1N∑i=1N∥R(xi)∥2\mathrm{Physics\ RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left\|\mathcal{R}(x_i)\right\|^2}

The results are summarized below (mean ± standard deviation over three random seeds; lower is better).

DatasetDDPMPIDMSCDM
Circle1.026 ± 0.2980.486 ± 0.1790.518 ± 0.070
LA13.864 ± 5.4191.840 ± 0.9265.223 ± 0.902
PA0.163 ± 0.0380.110 ± 0.0290.065 ± 0.001

The results show that SCDM achieves physical consistency comparable to PIDM, despite not explicitly using the geometric constraint during training. These results provide additional empirical evidence supporting our intuition that the generative process can be guided by underlying low-dimensional structures without explicit physical supervision.

5. Training Stability and Experimental Reproducibility

Table 5 suggests metric trade-offs rather than uniform improvement. Please report uncertainty across runs and collapse rates. Specify the CIFAR-10 corruption strength, training and sampling budgets, and FID [4] evaluation protocol to make the comparisons reproducible.

Regarding training stability and collapse rates, numerical instability mainly occurs with small batch sizes or insufficient lifting dimensions, which limit sample interactions and representational capacity, respectively. As shown in Table 4, the stable ratio increases from 20% to 100% as the batch size increases from 16 to 128, with all runs remaining stable at larger batch sizes. Appendix K further reports experiments across 30 random seeds with different lifting dimensions, providing additional evidence that sufficiently large batch sizes and lifting dimensions are important for stable training and robust generation.

Regarding the metric trade-offs in Table 5, we acknowledge that individual components do not necessarily lead to uniform improvements across all metrics. Our ablation study aims to demonstrate their complementary contributions rather than consistent gains from each component independently. Specifically, lifting enriches the feature representation, subspace projection captures low-dimensional structure, and structure-aware compression further exploits structural relationships. These results highlight the importance of combining these components for improved overall generation quality, rather than relying on any single component.

Regarding CIFAR-10 reproducibility, we provide further details of our experimental protocol. We randomly select 0%, 30%, 40%, 50%, and 60% of the CIFAR-10 training images for corruption and add zero-mean Gaussian noise with a standard deviation of σ=0.1\sigma=0.1 in the normalized pixel space [0,1][0,1], followed by clipping to [0,1][0,1]. The models are trained for 800,000 steps with a batch size of 128, using the Adam optimizer with a learning rate of 2×10−42\times10^{-4}, 5,000 warmup steps, and an exponential moving average (EMA) decay rate of 0.9999. For sampling, we adopt the standard DDPM procedure with 1,000 denoising steps and a linear noise schedule ranging from β1=10−4\beta_1=10^{-4} to βT=0.02\beta_T=0.02. For evaluation, we generate 50,000 images and compute the Fréchet Inception Distance (FID) against cached statistics of the clean CIFAR-10 training set. The same evaluation protocol is used across model comparisons to ensure consistency and fairness.

Reviewer_yxpaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Structure-aware Compression Diffusion Models (SCDM). The main idea is to help diffusion models learn low-dimensional structures in the data, rather than only learning to remove noise. The method first maps noisy data into a larger feature space. It then projects the features into a smaller learned subspace and applies a compression step. The compressed features are used for denoising. The authors test the method on physical datasets and CIFAR-10. They report better generation results than standard diffusion models and argue that the learned features capture physical laws.

Critical strengths
  1. An interesting idea for learning physical structure. The paper tries to connect diffusion models with low-dimensional representations and physical laws. This is an important problem because good data generation should preserve the structure of real systems.

  2. Experiments on both physical and image data. The paper studies several physical systems and also tests the method on CIFAR-10. It examines different subspace sizes and uses correlation analysis to understand the learned features.

Critical weaknesses

-The evidence for learning physical laws is not convincing enough. The paper uses correlation heatmaps to argue that the learned subspace captures physical structure, while the displayed correlation between position and acceleration has the opposite sign. Also, the proposed state and energy quantities are largely determined by the original physical variables. Their correlation with learned features does not by itself show that the model has learned new physical structure beyond those variables.

Questions for authors
  1. Is the subspace matrix U constrained or normalized? If not, how does the method avoid the scale problem in the coding-rate objective?
  2. Can the authors show evidence that the learned subspace captures physical structure beyond correlations with the original variables?
1 comment

1. Subspace Matrix Normalization

Is the subspace matrix UU constrained or normalized? If not, how does the method avoid the scale problem in the coding-rate objective?

Thank you for your insightful comment. In our implementation, the subspace matrix UU is learnable and subjected to column-wise L2L_2 normalization during each forward pass. This ensures that each basis vector has approximately unit norm, preventing arbitrary scaling of the subspace basis and improving numerical stability in the coding-rate-based compression operation.

2. Physical Structure and Compressibility

Can the authors show evidence that the learned subspace captures physical structure beyond correlations with the original variables?

We thank the reviewer for this valuable suggestion. We would like to further clarify the motivation of SCDM and provide additional experimental evidence.

As discussed in our manuscript, “The underlying physical processes are governed by a small number of low-dimensional structures and interactions. " In many real-world scenarios, observed data are constrained by underlying low-dimensional structures and therefore often exhibit compressibility. Motivated by this observation, as stated in Section 1, “The intuition is that the diffused data should be denoised under some low-dimensional structures, so that the recovered data could be robust.” Our primary goal is to leverage compression-based representation learning to improve diffusion models, allowing the generative process to be guided by the underlying low-dimensional structures and thereby produce more robust synthetic data, rather than explicitly discovering the governing physical equations underlying the observed data.

In Section 5.2.3 (Subspace Structural Analysis) and Appendix I, we investigate the interpretability of the learned subspace by examining the correlations between projected features and quantities associated with known physical relationships. We observe strong correlations with certain state equations, suggesting that the learned representations retain information associated with the underlying physical structure.

To provide complementary evidence beyond feature correlations, we conduct additional experiments on three datasets: Circle, Linear Oscillator (LA), and Pendulum (PA). We compare the conventional DDPM, Physics-Informed Diffusion Models (PIDM), and our SCDM. Unlike PIDM, which explicitly incorporates physical-equation residuals into its training objective, SCDM learns the data structure without access to these equations. We evaluate the generated samples using the root mean squared residual of the corresponding geometric or empirical state equation:

Physics RMSE=1N∑i=1N∥R(xi)∥2\mathrm{Physics\ RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left\|\mathcal{R}(x_i)\right\|^2}

The results are summarized below (mean ± standard deviation over three random seeds; lower is better).

DatasetDDPMPIDMSCDM
Circle1.026 ± 0.2980.486 ± 0.1790.518 ± 0.070
LA13.864 ± 5.4191.840 ± 0.9265.223 ± 0.902
PA0.163 ± 0.0380.110 ± 0.0290.065 ± 0.001

The results show that SCDM achieves physical consistency comparable to PIDM, despite not explicitly using the geometric constraint during training. These results provide additional empirical evidence supporting our intuition that the generative process can be guided by underlying low-dimensional structures.

Comments on the paper

1 comment

Dear reviewers,

This is a kind reminder that the discussion with the authors ends on 18th Nov. Please take a moment to leave a comment to acknowledge the authors’ responses and engage in the discussion if you have further questions, particularly on points that may affect your assessment.

Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.