acceptodds
Desk rejected at ICLR 2027

Is Imitation All We Need for Generative Modeling? Spectral Entropy Guidance for Extrapolative Generation in Diffusion Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Distribution matching is a central principle of generative modeling, with models typically designed to reproduce a reference distribution as faithfully as possible. This raises the broader question of whether reference matching should be the sole objective of generative modeling. We study this question through *imaginative generation*, which seeks distributions satisfying prescribed properties while remaining as close as possible to a reference distribution. We instantiate this principle using representation-aware spectral entropy, with complementary formulations based on von Neumann and order- Rényi entropies. Both define convex constraint sets and admit a common projection-based interpretation. The spectral entropy of the data distribution defines a natural boundary, which we call the Entropy Wall. Below the wall, a spectral entropy requirement can, under suitable conditions, recover variation lost by a learned generator while reducing its discrepancy to the data distribution. Beyond the wall, the data no longer satisfies the prescribed entropy level, and the framework transitions to controlled spectral extrapolation. Under a KL-divergence anchor, we characterize the optimal targets through self-consistent spectral reweighting and derive *spectral entropy guidance*, a retraining-free mechanism for score-based and diffusion models. Our numerical experiments on synthetic and image-generation benchmarks indicate that the proposed guidance method could recover lost variation and enable controlled extrapolative generation beyond the spectral diversity of the data distribution.

Primary area:
generative models
Keywords:
Generative modeling, spectral entropy, guidance in diffusion models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tcLmOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper formulates generation as minimizing divergence from a reference distribution subject to a lower bound on the spectral entropy of a kernel covariance. It identifies the data entropy as an Entropy Wall: below this wall, the KL projection moves the reference closer to the data distribution, whereas above it, the target is explicitly extrapolative. The penalized problem yields a self-consistent exponential tilt, which motivates a retraining-free guidance method whose potential is fitted on an independent pilot set and then frozen. Across experiments on synthetic data, ImageNet-100, CelebA-HQ, FFHQ, and SDXL, the method generally increases diversity.

Critical strengths

S1. Population-level target with independent sampling. The proposed frozen pilot construction enables independent sampling, unlike methods such as Particle Guidance and SPARKE that couple samples at inference time. The treatment of the regime beyond the Entropy Wall, together with the rigorous spectral tilt characterization covering existence, uniqueness, a converse result, and mutual absolute continuity, appears novel. The paper also states its theoretical limitations clearly.

S2. Consistent repair behavior. With moderate guidance, both objectives generally reduce FID/KID and DINOv2 distributional distances while increasing recall across all three benchmarks, albeit at some cost in precision. The strongest ImageNet RKE configuration achieves the best FID/KID in the main comparison.

Critical weaknesses

W1. Relevant overlap with prior work is not adequately discussed.**

  • Exposing Diversity Bias in Deep Generative Models (Farnia et al., 2026, arXiv:2602.14682) is cited only in connection with the diversity gap. Its Section 6 defines the convex VNE superlevel set and proves the KL/Bregman projection inequality corresponding to the repair side of the present result. The current proposition is more general, and that work does not address the regime beyond the data entropy, but the main text should nevertheless credit and distinguish it.
  • Flow Density Control (FDC; De Santi et al., NeurIPS 2025, arXiv:2511.22640) studies KL-regularized distribution-level utilities, including spectral functionals, in the same broader problem class. The paper should position its fixed-point characterization relative to FDC, and attribute the Gibbs variational principle underlying Theorem 1.

W2. The baseline comparison lacks a common trade-off, and the evidence for imagination remains indirect.

  • IGA is swept over its guidance strength, whereas each baseline is shown at a single setting with no stated selection protocol. The ImageNet results expose a genuine trade-off: IGA is better on FID/KID and precision, while CADS has higher recall and entropy. At matched entropy on CelebA-HQ, IGA has lower FID and DINOv2 FD than SPARKE. Baseline sweeps on the same axes are needed for a fair comparison.
Questions for authors
  1. Which spectral stabilization method, pooling and clipping settings, sampler variant, sampling schedule, and CFG scale were used in each text-to-image experiment?
  2. Beyond the wall, different feature spaces can give conflicting indications of distance from the data. How should the guidance strength be selected without access to real data, and how were the text-to-image values chosen?

Minor:

  • Please use consistent notation for unsmoothed von Neumann entropy (H₀ in Tab. 3 versus H in Tabs. 1–2).
  • Information Projections Revisited (Csiszár and Matúš, IEEE Transactions on Information Theory, 2003), currently cited only in Appendix C.2, should also be cited with the corresponding main text.
Reviewer_BRHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper formulates diversity control for diffusion models as minimizing divergence from a reference distribution subject to a spectral entropy constraint. It derives a self-consistent target under KL divergence and an approximate diffusion guidance method. Experiments demonstrate diversity recovery and generation beyond the data’s spectral entropy on synthetic and common image generation benchmarks.

Critical strengths
  • The paper is well motivated and well-written with excellent illustrations and organization.
  • The paper defines an explicit target distribution. Equation 14 shows that the exact KL projection improves agreement with the data when the data distribution sassifies the constraint.
  • The experiments informatively evaluate distributional agreement and entropy. They show that the proposed method IGA-RKE improves Inception FID and recall together on ImageNet, and evaluation outside the guidance representation helps evaluate whether gains extend beyond the optimized features.
Critical weaknesses
  • The main theoretical contribution concerns an exact optimizer show potential depends on its own covariance. The implementation instead fits a pilot potential and approximates the conditional expectation in Equation 24 with a denoised prediction. The experiments do not measure the resulting fixed-point error or discrepancy from the optimizer. Therefore, it remains unclear whether the proposed distribution-level formulation explains the practical gains over existing diversity guidance.
  • The evidence for useful extrapolation is limited. Exceeding the data’s spectral entropy does not distinguish meaningful new variation from distortions. In Table 1, stronger IGA-VNE guidance increase recalls but reduces precision from 67.0 to 42.6 and worsens FID from 9.0 to 10.14. The text-to-image results include quality-weighted diversity scores, but do not directly resolve this tradeoff.
Questions for authors
  • On a tractable problem, how closely does the implemented sampler match the exact optimizer?
  • At matched image quality or precision, does guidance beyond the entropy threshold improve semantic diversity over the baselines, including closely related spectral guidance methods?
Reviewer_UFGcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to promote diversity among generated samples by formulating generative modelling as a constrained optimisation problem with an entropy lower bound controlled by ρ\rho. It explores two entropy measures: von Neumann entropy and order-2 Rényi entropy. To maintain numerical stability when computing gradients, the authors introduce a smooth spectral entropy. The spectral entropy of the data distribution defines a natural boundary, termed the “Entropy Wall”. Below this boundary, a spectral entropy requirement can, under suitable conditions, recover variation lost by a learned generator while reducing its discrepancy from the data distribution. Beyond this boundary, the data distribution no longer satisfies the prescribed entropy level, and the framework transitions to controlled spectral extrapolation.The paper then goes on to also show empirical success on diffusion models,

Critical strengths
  1. The use of von Neumann entropy as a diversity metric for guided generation of samples for diffusion models and subsequent derivation of the smooth von Neumann entropy is, at least to me, novel and interesting.
  2. The connection between the choice of Rényi-2 entropy as the diversity measure and SPARKLE is a good observation.
  3. The theory is sound.
Critical weaknesses
  • Novelty concerns: The core idea of incorporating an entropy term into generative modelling has established precedents, such as Maximum Entropy Generators for Energy-Based Models (Kumar et al., 2019). Related Rényi-based objectives have also been explored in variational inference, for example in Rényi Divergence Variational Inference (Li and Turner, 2016). The paper would benefit from better positioning its novelty at the start

  • Experimental validation: It is unclear whether the method is broadly applicable. The empirical results support its effectiveness for diffusion models, but experiments with other types of generative models are needed to substantiate the framework’s value beyond diffusion. Furthermore, a more informative synthetic experiment would use a (5 \times 5) grid of Gaussian components to demonstrate the method’s effectiveness. Using components with unequal mixture weights would provide a more challenging setting and help clarify the method’s effects.

Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.