- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to represent a pre-trained diffusion model with a small, finite batch of samples, and this is different from generating a batch of diverse samples. The proposed method combines profiled spectral MMD, log-determinant coverage, and Stein quadrature to jointly determine the sample locations and their weights. Experiments are done on controlled mixture, 4 images datasets, and a planning task to validate the method in support coverage.
Critical strengths
The problem that this paper aims to address is interesting: using a finite number of samples in a batch to represent a model's distribution. The paper also makes it clear that representativeness is not equivalent to diversity.
Critical weaknesses
- Some citations are missing: First paragraph in introduction section needs more citations to support every statement, or they need to be shown in the paper. It looks very vague to me to simply state them. Equation (2) also needs a citation since this is not a new definition.
- Insufficient justification and unclear connection between equations and claims: this paper repeatedly moves from math equations to some strong conclusions without transition and explanations, and without citations to support their claims. For example, Equations (6) defines a kernel, and it suddenly jumps to the effects of . Same gaps happens at Equations (9) to (11), (12) to (13). Equation (16) states the correction term, but has no evidence to show this. I think they need clear assumptions, transitions and citations to support or differentiate the existing statements and this paper's contributions, given some clarifications in appendix.
- Section 3.3 and 3.4 is ambiguous in presentation: "Profiled" is not well defined before its first usage; Equation (8) has negative entropy included, and simply saying we can optimize the weights; At the beginning, the weights are used to measure how well the current support represents , and then at the end it is used as guidance; I don't see a direct connection between Equation (10) and model's support coverage. The coherence in section 3.3 to 3.7 is implicit, and it reads more as a sequence of definitions than coherent methodology. Some interpretations of equations are treated as conclusion and I find them confusing.
- The experiments did not evaluate on the sampling quality. The samples can have good coverage but remain low quality. The first experiment is only 1-D or 2-D, and the second experiment is still at small-scale. The third experiment is still in low dimension.
Notation-wise: is not defined in section 3.1; the paper uses to denote weights in equation 1, and then it changes to in section 3.3, and it changes to in section 3.5, which is confusing. Given the fact that this paper involves many notations, this is necessarily being fixed.
Questions for authors
- Function f is not well-defined when introducing MMD.
- How's equation 14 explicitly defined? There is no definition for each and no ablation study for each term in this equation. I can see some motivations and intuitions but role of each term is not clear empirically.
- The proposed method needs training, but I find some training details are missing: including architectures, optimizer, GPU resources, computational cost. Please report them.
- As mentioned above in weaknesses, the experiments did not assess sampling quality. This also determine the validity of the method.
2 comments
Official Comment by Authors
1. Citations in the Introduction and Eq. (2)
- Particle-based control and sampled planning motivate the Introduction (Chua et al., 2018, and Antonoglou et al., 2022, cited in Section 2). We will cite them there with Karras et al. (2022, in the bibliography) and Janner et al. (2022). The IID miss probability is for a region of mass .
- Appendix A derives Eq. (2) as the backward generator of the auxiliary Langevin diffusion. Nadler et al. (2005) give the spectral background (cited in Sections 2 and 3.2). We will cite Pavliotis (2014) at Eq. (2).
2. From spectral geometry to guided samples and weights
- Eq. (6). Appendix A shows that a normalized eigenfunction's gradient energy is its eigenvalue. The ratio of higher- to lower-energy kernel weights is , so increasing attenuates faster modes (Section 3.2).
- Eqs. (9)-(11). Appendix B gives feature normalization. Eq. (10) rewards feature volume, as in DPPs. DiverseFlow uses Euclidean distances in predicted-output embeddings (Morshed and Boddeti, 2025, cited in Section 2). Our slow spectral features vary little within a mode. An uncovered mode can add a direction and increase volume. For orthogonal unit region indicators with counts , the log-det rises when moving a duplicate to an empty region.
- Eqs. (12)-(13). Appendix C expands squared feature mismatch as a Gram quadratic and averages positive-noise Stein tests into endpoint constraints. Eq. (12) fits the masses, and Eq. (13) uses them on the fixed samples (Liu and Lee, 2017, cited in Section 2).
- Eq. (16). Appendix E derives the Euler update in Eq. (37), whose correction gives local descent at fixed noise. Figure 6 shows guidance toward a missing mode and Table 1 ablates components. In new N20 image runs (100 seeds, two fixed fits), guidance and IID use the same initial noise and equal final weights. Guidance lowers embedding/latent squared MMD 5.9%/7.1% on CIFAR-10 and 26.1%/10.6% on EuroSAT.
3. Profiling and coverage in the sampler
- Eq. (8) profiles by minimizing over weights on fixed noisy locations (van Leeuwen and Aravkin, 2021). Positive entropy favors spread and ensures a unique interior solution. The envelope identity lets us differentiate with those weights fixed (Milgrom and Segal, 2002). Eq. (11) adds coverage and Eq. (16) moves particles. Final quadrature infers clean-sample masses from spectral moments and optional Stein tests.
- Across 40 paired seeds and two fixed fits, removing profiling raises CIFAR-10 N80 embedding/latent error 3.3%/2.6%. Removing coverage raises CIFAR-10 N40 latent error 0.7%.
- The finite kernel gives small Gram and Newton systems, and guidance can run alongside denoising. On CIFAR-10 N20, reducing guidance from 63 to 16 steps cut guidance evaluations 75%, with embedding/latent error up 0.5%/0.7% over 50 paired seeds.
4. Sampling quality and scale (also follow-up 4)
- In new experiments we scored every image uniformly with a held-out InceptionV3 encoder against 20,000 model reference images per dataset. At N20/N80 on CIFAR-10 and EuroSAT, guided precision was within 1.2 percentage points of IID. Guided density point estimates were lower in all four comparisons.
- The main image experiments use 32- or 128-dimensional latents. Appendix G.2 evaluates direct 784-dimensional pixel-space MNIST. Section 4.3 tests downstream action-value estimation with 16 successors per MCTS call.
Notation and follow-up 1. Eq. (1)'s is a unit point mass, and lists the masses of the returned measure. Eq. (8)'s is for guidance, and Eq. (12)'s supplies the output masses. In MMD, ranges over the unit RKHS ball (Gretton et al., 2012, cited in Section 2).
Follow-up 2. Eq. (14). Appendix A derives energy minimization under centering and orthonormality. The off-diagonal term favors the eigenbasis, where . Appendix D computes feature mean and covariance and estimates with Jacobian-vector probes at fixed noise. In 20 matched fits, removing covariance yields 3.95x/1.81x EuroSAT N20 embedding/latent error. Removing energy raises EuroSAT N20/N80 embedding error 47.0%/22.1%. Retaining mean and off-diagonal penalties reduces held-out mean norms 78-86% and RMS cross-gradient cosine 8.1%/23.5% on CIFAR-10/EuroSAT, respectively.
Follow-up 3. Training settings and cost. The rank-16 network uses three SiLU hidden layers of width 256 or 64. Adam uses learning rate for 5,000 updates with batch size 1,024 and fresh Gaussian corruption of 20,000 model outputs. FP32 training uses one RTX 6000 Ada.
Given no response to the review, I choose to maintain my score.