- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when the data distribution supports a shared denoising representation across Gaussian noise levels. It proves that a fixed frame diagonalizing the Bayes-denoiser Jacobian at every input at one positive noise level is equivalent to an affine orthogonal independent-component structure, enabling coordinatewise Bayes denoising in the same frame at all noise levels. It connects total correlation to integrated denoising regret and ideal product-model KL discrepancy, separates factorization and sharing costs, and derives finite-sample learning bounds. Experiments with independently trained denoisers, VAE-family representations, and CIFAR-10 latents support common-frame transfer and a finite-sample crossover between shared and noise-specific representations.
Critical strengths
- The paper offers an interesting ICA-based perspective on when denoisers can share a representation across noise levels, providing useful insight into the structure that enables such sharing.
- The paper distinguishes the cost of coordinatewise denoising from the additional cost of sharing a frame, helping explain when sharing can be beneficial and when it introduces bias.
Critical weaknesses
-
Limited validation on complex image data. CIFAR-10 images are only 32 × 32 pixels and are further compressed into 32-dimensional latents. Coordinate-shuffling controls also reduce dependence to better match the theoretical assumptions. While useful for testing the mechanism, these experiments leave applicability to unmodified, high-resolution images with richer structure and fine details unclear.
-
Uncertain reliability of a fixed frame across inputs and noise levels. Empirical frames are estimated from finite samples at one noise level, whereas the exact theorem requires the Jacobian condition to hold at every input. Although held-out transfer tests and controlled tying-bias experiments provide supporting evidence, they do not establish reliable frame recovery or suitability throughout denoising on complex image distributions.
-
Limited connection to practical architectures. Fixed orthogonal, coordinatewise denoising is considerably more restrictive than the nonlinear, interacting features used by practical diffusion networks. Comparisons with time-conditioned or noise-specialized architectures are absent, leaving the implications for practical sharing strategies, generation quality, efficiency, and robustness unclear.
-
Insufficiently demonstrated added explanatory value. Although the paper discusses related theory, the experiments primarily validate its own predictions. A clearer comparison with relevant ICA or covariance-based perspectives would help establish what additional insight the framework provides into cross-noise frame transfer and the benefits of sharing.
Questions for authors
-
Applicability to complex data: What do the results on the original, unshuffled latents establish about the usefulness of a shared frame? Which findings support extending this interpretation beyond low-dimensional representations of simple images, and which remain untested?
-
Computational implications: Does the shared-frame structure imply computational savings beyond reduced frame-estimation complexity? Please clarify whether lower model size, training cost, or inference cost follows from the analysis, or remains a potential future application.
-
Source-level sensitivity: Appendix I.5 evaluates multiple source noise levels for dSprites. How sensitive are the CIFAR-10 results to the choice of the source level, σ₀ = 0.6? In particular, is transfer to low-noise stages consistent across different source levels?
-
Connection to practical sharing: Which aspects of representation sharing in standard time-conditioned diffusion networks are captured by the fixed orthogonal, coordinatewise model? What insight does the analysis provide when useful denoising requires cross-coordinate interactions?
4 comments
Thank you for the clarifications and for adding Appendix L. The new analysis establishes a conditional transfer guarantee for targets at or above the source noise level. My concerns about practical applicability remain.
- Transfer to lower noise levels. The CIFAR-10 experiment fits the frame at σ₀ = 0.6 and reports a mean excess Jacobian residual of 0.001336 across three target levels. Reporting the excess residual for each target across several source noise levels would clarify whether it stays small when transferring to lower-noise targets on unmodified latents.
- Connection to denoising performance. The CIFAR-10 results measure the geometry of learned network Jacobians. Applying Corollary L.7 requires an independent bound on their error relative to the Bayes Jacobians. A controlled comparison with a standard time-conditioned denoiser on the same latents would help establish whether explicitly sharing a frame can achieve comparable denoising quality with fewer training samples.
I maintain my original rating of 2 because the practical benefit of explicitly sharing a frame in learned diffusion models remains insufficiently demonstrated.
We thank the reviewer for their thoughtful follow-up and for acknowledging the new theoretical guarantees in Appendix L. We appreciate the opportunity to clarify two important points.
Regarding lower-noise transfer, our CIFAR-10 experiment already evaluates a frame learned at on the original, unshuffled latents at , including two substantially lower-noise targets. The reported excess residual of is averaged over these three targets, relative to independently fitted target-level frames. Thus, the existing results already provide evidence of lower-noise transfer on dependent real-image latents. We agree that a target-specific breakdown across multiple source levels would further illuminate its robustness. Importantly, the exact theorem applies to any positive source level, while Appendix L provides complementary quantitative guarantees under explicit assumptions.
Regarding denoising performance, we would emphasize a distinction central to our contribution: sharing a representation does not require eliminating all cross-coordinate interactions. On the original CIFAR-10 latents, the fixed-frame residual is , yet the additional cross-noise transfer residual averages only . This suggests that a representation can remain relatively stable across noise levels even when substantial coordinate interactions persist. Our theory distinguishes these effects through factorization regret and tying cost, and Appendix K extends the framework to nonlinear, interacting denoisers within shared blocks.
Moreover, Section 6.2 directly evaluates denoising risk and demonstrates the finite-sample sharing–tying tradeoff. The CIFAR-10 experiments complement this by studying representation geometry in fully interacting, time-conditioned networks. We appreciate the proposed architectural comparison as a further test of sample efficiency.
Our central contribution is to establish when and why cross-noise representation sharing is supported by the Bayes-optimal denoising target itself, providing a mathematical foundation for investigating practical sharing strategies.
Thank you for the additional explanations. The results support cross-noise representation transfer in the studied settings, but without comparisons to standard time-conditioned diffusion models or other strong practical designs, it remains unclear whether this translates into practical benefits. I therefore maintain my current rating
We thank the reviewer for their careful reading and constructive feedback. Regarding presentation, we have revised the manuscript to improve clarity and organization, and invite the reviewer to consult the updated PDF.
Regarding applicability to complex data, modern high-resolution diffusion models often operate in learned latent spaces rather than directly in pixel space. Our paper characterizes when such latent distributions support a shared representation for the Bayes-optimal denoiser, the target of squared-error diffusion training. Our theory neither assumes linear encoders nor restricts the latent dimension. Importantly, coordinate shuffling is only an experimental control. We also evaluate the original dependent CIFAR-10 latents, whose fixed-frame residual is substantially larger than that of the shuffled controls (0.05356 versus 0.002372), yet whose additional cross-noise transfer error remains relatively small (0.001336). This suggests partial geometric stability even without approximate independence.
Regarding frame reliability, we have added Appendix L, which establishes quantitative approximate one-noise transfer guarantees using average Bayes-Jacobian defects, including blockwise extensions and a conditional held-out validation certificate for estimated frames. These results address imperfect diagonalization and finite-sample evaluation under explicit assumptions.
Regarding practical architectures, our denoisers are not necessarily linear: the coordinatewise functions can be arbitrarily nonlinear and noise-dependent. Their lack of cross-coordinate interactions is precisely the simplification justified by our theory. Moreover, Appendix K extends the characterization to independent blocks, permitting arbitrary nonlinear interactions within blocks. Our regret and tying-cost analysis quantifies the consequences of approximate sharing. Sections 6.1–6.2 investigate independently trained denoisers and shared-versus-untied frame estimation; our CIFAR experiments additionally examine time-conditioned networks.
Regarding computational implications, Section 5 establishes finite-sample benefits of estimating a shared frame, while Appendix K.2 demonstrates explicit computational savings for exact blockwise Gaussian-mixture denoising, including the possibility of independent or parallel block evaluation. We do not claim general reductions in neural-network size, training cost, or inference latency.
Regarding explanatory value beyond ICA and covariance, covariance eigenvectors remain unchanged under isotropic Gaussian corruption for every distribution, yet this does not imply shared Bayes-denoising coordinates (Appendix F.1). Beyond classical ICA, our framework establishes the one-noise Bayes-Jacobian characterization, its cross-noise consequences, and quantitative results for approximate sharing, tying bias, and independent blocks.
Regarding source-level sensitivity, σ0=0.6 was chosen as a representative level rather than optimized for transfer. The exact characterization applies at any positive noise level, and Appendix I.5 investigates multiple sources on dSprites. Additional CIFAR-10 source-level comparisons, particularly high-to-low noise transfer, remain valuable.
Finally, our paper is primarily theoretical: we characterize when representation sharing is justified by the Bayes-optimal target itself, rather than proposing a particular neural architecture or claiming improved end-to-end generation. Our experiments test the underlying mechanisms; validating practical architectural gains on larger spatial latents remains future work.