acceptodds
Under review as a conference paper at ICLR 2027

Finite-Budget Representative Sampling from Diffusion Models

Abstract

Diffusion models are usually sampled with independent denoising trajectories, but under tight inference budgets an iid batch can duplicate common outcomes and miss important probability regions. We formulate finite-budget sampling as the construction of a representative empirical measure, separating two goals: placing support across density-separated regions and assigning weights for accurate expectation estimation. Without access to the original data, we learn a kernel from model samples that treats samples connected through high-density regions as similar and use it during sampling to guide the batch toward distribution matching and nonredundant coverage. We then assign quadrature weights to the sampled points for expectation estimation. Experiments show that, under limited sampling budgets, our method improves support coverage and produces weights that better represent the model distribution. When used in Monte Carlo tree search, the resulting weighted batches yield more accurate action-value estimates.

Primary area:
generative models
Keywords:
Diffusion models, representative sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4QTaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to represent a pre-trained diffusion model with a small, finite batch of samples, and this is different from generating a batch of diverse samples. The proposed method combines profiled spectral MMD, log-determinant coverage, and Stein quadrature to jointly determine the sample locations and their weights. Experiments are done on controlled mixture, 4 images datasets, and a planning task to validate the method in support coverage.

Critical strengths

The problem that this paper aims to address is interesting: using a finite number of samples in a batch to represent a model's distribution. The paper also makes it clear that representativeness is not equivalent to diversity.

Critical weaknesses
  1. Some citations are missing: First paragraph in introduction section needs more citations to support every statement, or they need to be shown in the paper. It looks very vague to me to simply state them. Equation (2) also needs a citation since this is not a new definition.
  2. Insufficient justification and unclear connection between equations and claims: this paper repeatedly moves from math equations to some strong conclusions without transition and explanations, and without citations to support their claims. For example, Equations (6) defines a kernel, and it suddenly jumps to the effects of τ\tau. Same gaps happens at Equations (9) to (11), (12) to (13). Equation (16) states the correction term, but has no evidence to show this. I think they need clear assumptions, transitions and citations to support or differentiate the existing statements and this paper's contributions, given some clarifications in appendix.
  3. Section 3.3 and 3.4 is ambiguous in presentation: "Profiled" is not well defined before its first usage; Equation (8) has negative entropy included, and simply saying we can optimize the weights; At the beginning, the weights are used to measure how well the current support represents pp, and then at the end it is used as guidance; I don't see a direct connection between Equation (10) and model's support coverage. The coherence in section 3.3 to 3.7 is implicit, and it reads more as a sequence of definitions than coherent methodology. Some interpretations of equations are treated as conclusion and I find them confusing.
  4. The experiments did not evaluate on the sampling quality. The samples can have good coverage but remain low quality. The first experiment is only 1-D or 2-D, and the second experiment is still at small-scale. The third experiment is still in low dimension.

Notation-wise: θ\theta is not defined in section 3.1; the paper uses ww to denote weights in equation 1, and then it changes to α\alpha in section 3.3, and it changes to ww in section 3.5, which is confusing. Given the fact that this paper involves many notations, this is necessarily being fixed.

Questions for authors
  1. Function f is not well-defined when introducing MMD.
  2. How's equation 14 explicitly defined? There is no definition for each λ\lambda and no ablation study for each term in this equation. I can see some motivations and intuitions but role of each term is not clear empirically.
  3. The proposed method needs training, but I find some training details are missing: including architectures, optimizer, GPU resources, computational cost. Please report them.
  4. As mentioned above in weaknesses, the experiments did not assess sampling quality. This also determine the validity of the method.
2 comments

Given no response to the review, I choose to maintain my score.

AuthorsOpenReview ↗

Official Comment by Authors

1. Citations in the Introduction and Eq. (2)

  • Particle-based control and sampled planning motivate the Introduction (Chua et al., 2018, and Antonoglou et al., 2022, cited in Section 2). We will cite them there with Karras et al. (2022, in the bibliography) and Janner et al. (2022). The IID miss probability is (1−p)N(1-p)^N for a region of mass pp.
  • Appendix A derives Eq. (2) as the backward generator of the auxiliary Langevin diffusion. Nadler et al. (2005) give the spectral background (cited in Sections 2 and 3.2). We will cite Pavliotis (2014) at Eq. (2).

2. From spectral geometry to guided samples and weights

  • Eq. (6). Appendix A shows that a normalized eigenfunction's gradient energy is its eigenvalue. The ratio of higher- to lower-energy kernel weights is e−τ(λh−λl)e^{-\tau(\lambda_h-\lambda_l)}, so increasing τ\tau attenuates faster modes (Section 3.2).
  • Eqs. (9)-(11). Appendix B gives feature normalization. Eq. (10) rewards feature volume, as in DPPs. DiverseFlow uses Euclidean distances in predicted-output embeddings (Morshed and Boddeti, 2025, cited in Section 2). Our slow spectral features vary little within a mode. An uncovered mode can add a direction and increase volume. For orthogonal unit region indicators with counts ncn_c, the log-det ∑clog⁡(1+nc/ϵ)\sum_c\log(1+n_c/\epsilon) rises when moving a duplicate to an empty region.
  • Eqs. (12)-(13). Appendix C expands squared feature mismatch as a Gram quadratic and averages positive-noise Stein tests into endpoint constraints. Eq. (12) fits the masses, and Eq. (13) uses them on the fixed samples (Liu and Lee, 2017, cited in Section 2).
  • Eq. (16). Appendix E derives the Euler update in Eq. (37), whose correction gives local descent at fixed noise. Figure 6 shows guidance toward a missing mode and Table 1 ablates components. In new N20 image runs (100 seeds, two fixed fits), guidance and IID use the same initial noise and equal final weights. Guidance lowers embedding/latent squared MMD 5.9%/7.1% on CIFAR-10 and 26.1%/10.6% on EuroSAT.

3. Profiling and coverage in the sampler

  • Eq. (8) profiles by minimizing over weights on fixed noisy locations (van Leeuwen and Aravkin, 2021). Positive entropy favors spread and ensures a unique interior solution. The envelope identity lets us differentiate with those weights fixed (Milgrom and Segal, 2002). Eq. (11) adds coverage and Eq. (16) moves particles. Final quadrature infers clean-sample masses from spectral moments and optional Stein tests.
  • Across 40 paired seeds and two fixed fits, removing profiling raises CIFAR-10 N80 embedding/latent error 3.3%/2.6%. Removing coverage raises CIFAR-10 N40 latent error 0.7%.
  • The finite kernel gives small Gram and Newton systems, and guidance can run alongside denoising. On CIFAR-10 N20, reducing guidance from 63 to 16 steps cut guidance evaluations 75%, with embedding/latent error up 0.5%/0.7% over 50 paired seeds.

4. Sampling quality and scale (also follow-up 4)

  • In new experiments we scored every image uniformly with a held-out InceptionV3 encoder against 20,000 model reference images per dataset. At N20/N80 on CIFAR-10 and EuroSAT, guided precision was within 1.2 percentage points of IID. Guided density point estimates were lower in all four comparisons.
  • The main image experiments use 32- or 128-dimensional latents. Appendix G.2 evaluates direct 784-dimensional pixel-space MNIST. Section 4.3 tests downstream action-value estimation with 16 successors per MCTS call.

Notation and follow-up 1. Eq. (1)'s δz\delta_z is a unit point mass, and ww lists the masses of the returned measure. Eq. (8)'s ασ⋆\alpha_\sigma^\star is for guidance, and Eq. (12)'s w⋆w^\star supplies the output masses. In MMD, ff ranges over the unit RKHS ball (Gretton et al., 2012, cited in Section 2).

Follow-up 2. Eq. (14). Appendix A derives energy minimization under centering and orthonormality. The off-diagonal term favors the eigenbasis, where Dab=λbδabD_{ab}=\lambda_b\delta_{ab}. Appendix D computes feature mean and covariance and estimates D=E[JJ⊤]D=\mathbb E[JJ^\top] with Jacobian-vector probes at fixed noise. In 20 matched fits, removing covariance yields 3.95x/1.81x EuroSAT N20 embedding/latent error. Removing energy raises EuroSAT N20/N80 embedding error 47.0%/22.1%. Retaining mean and off-diagonal penalties reduces held-out mean norms 78-86% and RMS cross-gradient cosine 8.1%/23.5% on CIFAR-10/EuroSAT, respectively.

Follow-up 3. Training settings and cost. The rank-16 network uses three SiLU hidden layers of width 256 or 64. Adam uses learning rate 5×10−45\times10^{-4} for 5,000 updates with batch size 1,024 and fresh Gaussian corruption of 20,000 model outputs. FP32 training uses one RTX 6000 Ada.

Reviewer_DnbcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the problem of generating a small batch of diffusion-model samples that accurately represents the model distribution. Rather than relying on independent sampling or optimizing diversity alone, the authors jointly consider sample locations and quadrature weights. They construct a learned kernel based on spectral properties of the model distribution, guide sampling using distribution-matching and coverage objectives, and subsequently estimate weights using spectral and Stein-based constraints. The method is evaluated on synthetic distributions, image-generation benchmarks, and Monte Carlo tree search.

Critical strengths

a) The paper makes an important observation that generating diverse samples is not necessarily equivalent to constructing a representative approximation of the target distribution. A diverse batch may overrepresent rare outcomes, while reweighting alone cannot recover missing modes. The proposed separation between support placement and quadrature weighting addresses these two issues simultaneously and provides a motivation for the method.

b) Rather than measuring diversity using Euclidean distances or generic embeddings, the method learns features associated with the Langevin operator of the model distribution. The motivation is that slowly varying eigenfunctions can distinguish regions separated by low-density bottlenecks while treating samples within well-connected regions as similar.

c) Their experiments provide a useful ablation of quadrature weighting, MMD guidance, and the coverage objective. These results support the claim that modifying sample locations and assigning better weights can offer complementary benefits.

Critical weaknesses

I don't have any concrete weaknesses to mention, so I would just mention them as questions:

a) Section 4.2 states that classifiers are used only for evaluation, but Appendix G explicitly acknowledges that some reported image results used classifier-based development metrics for hyperparameter selection. Selecting hyperparameters using classifier information may favor precisely the semantic distinctions measured by the evaluation metrics, could the authors clarify the setting?

b) In the image and planning experiments, the proposed method uses optimized quadrature weights, while the competing methods use uniform weights. Consequently, it is unclear how much of the advantage arises from generating better support locations rather than simply assigning better weights to the generated samples. Table 1 demonstrates that reweighting iid samples alone can substantially reduce distributional errors. Could the authors clarify if they have any additional experiments or ablations for this?

Questions for authors

Please see above.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

(a) Classifier use in evaluation and hyperparameter selection

  • Classifier use in the submission. Appendix G details how classifier metrics are used offline for evaluation and, in some image settings, hyperparameter selection.

  • Guidance also lowers error with random pixel features. We evaluated guided and IID batches against the same 20,000 reference images from the frozen model. To measure squared MMD, we approximated a Gaussian kernel with random Fourier features of the decoded pixels. These random features were reserved for evaluation. The sampler uses learned spectral features for guidance. With equal weights for every image, guided batches had lower average error at every tested batch size. Each percentage compares guidance to IID at the same batch size, over 100 paired seeds and two fixed feature fits per dataset.

    Dataset20 images40 images80 images
    CIFAR-1017.2%16.1%8.2%
    EuroSAT18.9%14.1%13.3%

(b) Support placement and the effect of final quadrature weights

  • Table 1 separates placement and weighting on the mixtures. Section 3.1 defines the output as a weighted measure for estimating expectations. On the 2D mixture at N12, Table 1 reports mode-mass errors of 0.374 for ordinary IID, 0.100 for IID with quadrature, and 0.051 for guidance with quadrature. At least one mode is missing in 36% of batches for both IID rows and 12% for guidance with quadrature.

  • New image experiments vary locations and weights separately. On CIFAR-10 and EuroSAT, we run each frozen model from the same initial noise with guidance and with ordinary IID sampling. We evaluate each resulting set under uniform weights 1/N1/N and under the same quadrature objective and constraints. We use 100 paired seeds and average two fixed feature fits within each seed per dataset.

    Effect of changing the locations. At N20 with uniform weights, guided samples reduce embedding/latent squared MMD by 5.9%/7.1% on CIFAR-10 and 26.1%/10.6% on EuroSAT relative to ordinary IID. Applying identical quadrature to both sets gives reductions of 8.0%/8.7% and 20.9%/11.4%, respectively. All eight pointwise 95% paired intervals favor guidance.

    Effect of changing the weights. Holding the 80 guided EuroSAT locations fixed, quadrature lowers embedding/latent squared MMD by 14.7%/0.2% relative to uniform weights on those same samples. The table reports all tested sizes. Each cell gives embedding/latent squared MMD divided by ordinary IID error at the same N.

    DatasetNIID + quadratureGuided + uniformGuided + quadrature
    CIFAR-10201.022 / 1.0200.941 / 0.9290.940 / 0.931
    CIFAR-10401.028 / 1.0240.965 / 0.9410.965 / 0.947
    CIFAR-10801.016 / 1.0090.985 / 0.9750.990 / 0.981
    EuroSAT200.913 / 1.0100.739 / 0.8940.723 / 0.895
    EuroSAT400.871 / 1.0110.785 / 0.9130.737 / 0.917
    EuroSAT800.808 / 0.9990.866 / 0.9470.738 / 0.944

    Guided sets grow in groups of 20 with earlier locations fixed.

  • A separate ablation measures the effect of the final masses directly. On identical EuroSAT N80 locations from 40 paired trajectories and two fixed fits, replacing optimized final weights with uniform weights raises embedding/latent error by 20.1%/0.8%.

  • Planning evaluates the complete weighted sampler. Section 4.3 tests how the weighted set affects action-value estimates in MCTS, with 16 successors per diffusion call under a fixed call budget.

Reviewer_RzEvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the finite-sample-budget representative sampling problem and proposes to use a system of sampling heuristics based on a spectral kernel MMD criteria constructed from the Langevin generator.

Critical strengths

The paper studies an interesting and important problem, and the central idea to use the Langevin-operator induced kernel MMD looks promising and may lead to a principled design if properly developed.

Critical weaknesses
  • The paper does not seem to follow a core design principle, but choose to stack a long list of heuristic components across 5 subsections of Section 3, with both offline preparation and a complex during-sampling system that requires both repetitive solving of a weight optimization and evaluation of coverage log-det, gradient of the energy, and a final quadrature weight solver. Each of the components introduces a new set of hyperparameters and computational complexity. It is not theoretically (and empirically) justified whether the system has been optimized for achieving the goal.

  • The presentation also makes the core contribution of the paper hidden and unclear. It is a bit surprising that the Introduction has zero citations to sufficiently motivate and provide background knowledge about the field. Many non-common concepts appear early and are central to the paper, yet I find no formal definition at all, such as “density connectivity”. Further, many components in the Method section do not look completely new, yet without sufficient credit to the sources, it is not clear how to determine and separate out the real contribution of the paper from established results.

Questions for authors
  • For the image experiments, you demonstrate more competitive performance at a fixed number of samples, but I am concerned about the computational efficiency and the limited budget range with N≤30N \leq 30.

    • Because the proposed framework requires repeated weight optimization (Sec. 3.3), log-det and additional gradient evaluation (Sec. 3.4 and 3.7), and further require computing a final quadrature weight (Sec. 3.5), I doubt whether computational-wise the method is indeed more efficient in generating the same number of samples as vanilla iid sampling.

    • Further, if iid or any baseline can be much faster, then according to your own plots (Fig. 2 and 3) it seems that I can simply double the number of samples generated to reach comparable performance as your proposed method. I believe reporting a fair computational time against the baselines would be crucial. Or maybe you have a reasonable justification that we must preserve such a small set of samples in certain applications.

  • Can you defend that defining the global problem with respect to MMD and your chosen set of metrics are proper? While MMD has clear statistical meaning and can be characteristic metric in the ideal case, the finite embedding-based approach can omit various distributional differences, and evaluation performed in the selected representation can give your approach intrinsic advantages over baselines because you optimize the metric while the other works do not. Missing-class and class-TV metrics look much narrower as a distributional metric as well.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Summary. Diversity describes spread. A representative batch must also capture probability masses. We make an early contribution by coordinating sample locations and masses in diffusion generation for expectation estimation at a fixed output count.

1. Principled two-stage construction

  • One objective, two stages. Offline, we learn and calibrate the frozen model's density geometry at each noise level. Online, we guide particles with coverage and spectral mismatch minimized over temporary weights, then assign final masses on the fixed support. Combining guidance terms follows Particle Guidance (Corso et al., 2024) and DiverseFlow (Morshed and Boddeti, 2025). Appendices C and E derive the final weighting and sampling update.
  • The loss comes from the spectral problem. Appendix A minimizes Dirichlet energy under centering and orthonormality. The mean and covariance penalties relax those constraints. The off-diagonal penalty encourages an eigenbasis in which heat weights act mode by mode.
  • New ablations. In 20 fits on CIFAR-10 and EuroSAT, removing covariance yields 3.95x/1.81x EuroSAT embedding/latent error at N20. Removing energy raises its embedding error 47.0%/22.1% at N20/N80. The mean penalty reduces held-out mean norms 78-86%. The off-diagonal term reduces RMS cross-gradient cosine 8.1%/23.5% on CIFAR-10/EuroSAT. In a 40-paired-seed sampling study, removing profiling raises CIFAR-10 N80 embedding/latent error 3.3%/2.6%, and removing coverage raises N40 latent error 0.7%. In the submitted 1D mixture ablation, adding coverage reduces the missed-mode rate from 24% to 4% at N12.

2. Contribution, attribution and density connectivity

  • The Introduction identifies the contribution as jointly generating and weighting a finite diffusion batch, including sequential extension with fixed old locations. Section 2 credits kernel quadrature, diffusion maps, neural eigensolvers, determinant diversity and Stein weighting. We will also place background citations in the Introduction.
  • Section 3.2 defines the auxiliary Langevin operator. At a fixed noise level, diffusion distance compares its transition laws from two starting points (Coifman and Lafon, 2006, cited in Section 2). Appendix A shows that variation costs more in high-density regions. Slow eigenfunctions therefore vary little within connected regions and can separate them across low-density bottlenecks.

3. Compute, wall time and sample budget

  • Why compare at equal counts? Particle filtering repeatedly scores each particle, and stochastic MCTS branches over sampled successors at each rollout. In time-sensitive or edge settings, a small returned set limits this recurring work. Screening a larger proposal pool also requires a reliable screening rule.
  • Why guidance can be efficient. Appendix D's finite kernel uses precomputed heat weights. Gauss-Jordan solves small profile Newton systems, and coverage uses a Gram matrix sized by the smaller of sample count and feature rank. Guidance gradients through the small feature network can run alongside denoising. At CIFAR-10 N20, going from 63 to 16 guided steps cut guidance evaluations 75%, with embedding/latent error up 0.5%/0.7% over 50 paired seeds.
  • Same compute cap. We count modeled arithmetic for denoising, decoding, feature gradients and solves, with a multiply-add counted as two operations. Under a 17.42B-operation EuroSAT cap, guided N20 uses 16.62B and uniform IID21 uses 17.00B. Guided embedding/latent squared MMD is 23.6%/6.3% lower over 100 paired seeds and two fixed feature fits.
  • Comparable wall time. On EuroSAT with batches of ten, guided N20 takes 43.63 ms versus 20.94 ms for IID20. IID40 takes 41.80 ms and has lower errors on both metrics. The added guidance cost is a smaller fraction of runtime as denoising gets slower, provided feature costs stay similar.
  • Samples needed at comparable error. With larger EuroSAT batches, the first tested IID counts matching both mean errors of 20/40/80 guided samples were 28/55/108, or 28.6%/27.3%/25.9% fewer samples for guidance. This matters when the downstream output count is fixed.

4. Kernel expressiveness and evaluation

  • MMD measures worst-case expectation error over the kernel's unit RKHS ball. A compact representation can miss differences outside that function family. The relevant question is how well the kernel represents the functions needed downstream.
  • Figure 3 evaluates latent and classifier-embedding MMD alongside class TV. Appendix G.2 adds direct pixel-space MNIST, where at N20 we improve all three reported class-allocation and coverage metrics over the tested baselines. Using a pixel kernel held out of tuning and identical quadrature on both sets at N20, squared MMD falls 15.8% on CIFAR-10 and 11.4% on EuroSAT. These gains also appear with equal weights on both sets.

Thank you for your response. However, I find it unusually elaborate, with an overwhelming amount of granular numerical and implementation details that do not necessarily translate into a clearer understanding of the central scientific questions. I would have appreciated a more focused discussion and clear explanations reflecting the authors' own reasoning about the issues I raised. Given that I also see no evidence of submitted revision, I am maintaining my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.