acceptodds
Under review as a conference paper at ICLR 2027

Schedule-Free Denoising Diffusion Sampling using Time Scores

Abstract

Annealing-based neural samplers amortize sampling from unnormalized target distributions by transporting samples along a family of intermediate smoothed densities connecting a tractable source distribution to the target. Existing approaches typically rely on a predefined noise schedule, prescribing the progression through the intermediate densities independently of the current sample state. To remedy this, we propose Dual Score Diffusion Sampling (DSDS), which instead treats the diffusion time, particularly its associated noise varaince, as a stochastic variable and lifts the sampling problem to the joint space. In this lifted space, both variables evolve jointly through stochastic dynamics, which allows the sampler to adaptively navigate the joint probability space. Across synthetic and image-generation experiments, we show that this leads to improved robustness, while remaining competitive with schedule-based samplers.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Langevin Sampling, Denoising Score Matching

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_BssEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose DSDS and its inertial variant, iDSDS, sampling methods that treat the noise variance as a stochastic variable and sample it jointly with the data. They further develop a Fisher preconditioner based on VE and VP kernels. The ablations study the method’s sensitivity to the initial noise variance, the variance parametrisation, and preconditioning.

Critical strengths

The paper combines spatial and temporal scores with a Fisher-informed preconditioner in a log-variance parametrisation. This addresses the different scaling of the data and noise-variance coordinates through a corruption-kernel-based construction. The proposed implementation of the joint dynamics is tractable, and the Gaussian-mixture experiments with oracle scores allow sampling behaviour to be studied independently of model approximation error. The ablation studies help explain how prior selection and parametrisation influence sampling performance.

Critical weaknesses

I believe that the main contribution of this paper is algorithmic. The authors recognise that joint sampling has already been proposed by Kim & Ye (2023) and Qin & Risteski (2024), and that dual score matching also appears in prior work, as stated on lines 94–100, 137–141, and 277–283. Hence, the primary contribution is the preconditioning construction and its integration into the Langevin samplers (both overdamped and inertial), rather than the introduction of joint sampling itself. The theory supports the design, yet the guarantee provided in Proposition 3.1 does not establish faster convergence or account for discretisation and approximation errors. The paper should then demonstrate a practical advantage.

However, the claimed robustness improvements do not sufficiently demonstrate such an advantage. The comparative sensitivity experiment is conducted only on a one-dimensional Gaussian mixture. The robustness improvement requires a specially picked log-Gaussian prior and leads to slower convergence from the standard high-noise initialisation. The prior selected for the remaining experiments, i.e., the Pareto prior, remains sensitive to initialisation, similarly to ALD (see Figure 3 and the paragraph beginning on line 351). I read this as a speed–robustness trade-off, rather than a simultaneous improvement in robustness and sampling performance across the evaluated tasks. Moreover, the sampler hyperparameters are tuned separately for each experimental setting and, in the ablations, for each sampling budget (lines 288–291; lines 347–350). Though this is not inherently problematic, these experiments do not show that replacing a prescribed schedule with a noise prior reduces the overall tuning burden.

Likewise, the image-generation examples provide only partial evidence of an advantage. Indeed, iDSDS performs well on MNIST at 100 NFE, but its FID is worse than both EM and PC at every CIFAR-10 evaluation budget in Table 1b. Even at a generous budget of 2,000 NFE, its FID is 94, compared with 28.6 for PC and 68 for ALD. These numbers must also be weighed against the overhead of time-score learning, whose training cost and approximation error are acknowledged by the authors (lines 484–485). It also remains an open question whether the method achieves an advantage in wall-clock time.

I thank the authors for being transparent about their limitations and their contribution relative to prior work. Overall, the limited evidence of improved robustness, reduced tuning, or computational benefit leaves the practical significance of the algorithmic extension unestablished. This, combined with the narrow theoretical contribution, motivates my rejection.

Questions for authors
  1. For the Optuna tuning, what was the number of trials and what was the objective? Were the tuning done for each NFE?
  2. What precisely constitutes one NFE for each method, including the predictor and corrector evaluations in PC, and the density evaluations in Gibbs-update? Appendix D.3.3 provides that obtaining the score requires a forward and backward pass while Appendix F.1 says that the baselines discard the time score, is it computed and discarded or not computed at all? Could the authors provide the wall clock sampling time for each method on the same hardware?
  3. Could you please reconcile the statement on lines 427-428 that iDSDS outperforms ALT for every CIFAR-10 NFE budget in both FID and NLL (Table 1b)? The ALD outperforms iDSDS in terms of FID at 2000NFE.
3 comments
AuthorsOpenReview ↗

Addressing weaknesses part I

Thank you for your constructive review and the interesting points brought up. We have conducted several additional experiments to address your questions and concerns.

W1. Conceptual novelty and algorithmic contributions.

Conceptual novelty: We believe that directly generating clean samples from joint-space dynamics is a conceptual contribution. Prior work considered joint-space sampling for warm-starting (Kim & Ye, 2023) or theoretical mixing analysis (Qin & Risteski, 2024), rather than direct clean-sample generation. Moreover, Qin & Risteski's concrete mixing-time guarantees focus on Gaussian mixtures, whereas we additionally demonstrate joint-space sampling on image data.

Algorithmic contributions: Our contributions go beyond the joint-space formulation. The ablations in Fig. 4 show that existing design choices do not directly translate into efficient clean-sample generation. In particular, Euclidean preconditioning and Gibbs time updates correspond to key aspects of Qin & Risteski (2024) and Kim & Ye (2023), respectively, highlighting the importance of our proposed modifications. We further introduce an underdamped joint-space Langevin sampler with state-dependent preconditioning.

Practical advantage: Our preliminary updated results (see W3) indicate that iDSDS outperforms ALD across the evaluated CIFAR-10 budgets and substantially narrows the gap to EM.

W2. Speed–robustness trade-off and tuning burden.

Speed–robustness trade-off: We agree that our results demonstrate a speed–robustness trade-off rather than a simultaneous improvement. The purpose of our robustness experiment was to highlight the flexibility of the framework: (i) different priors induce different sampling dynamics, and (ii) even the same prior can exhibit different behavior depending on its hyperparameters. We do not expect a single prior to be optimal for both fast sampling and robustness.

We would like to clarify that the log-normal prior was not specifically constructed for robustness but is a natural generalization of the Pareto prior, replacing a constant log-variance score with a linear one. Furthermore, Fig. 4 demonstrates that clean-sample generation is possible with different priors, not only Pareto. Exploring more expressive priors remains an interesting direction for future work.

Tuning burden: We acknowledge that replacing a prescribed schedule with a noise prior does not eliminate hyperparameter tuning. Our contribution is to replace a fixed noise-level trajectory with state-dependent stochastic dynamics, rather than to claim a reduced tuning burden.

W3. Sampling results.

iDSDS sampling performance on CIFAR-10: Upon revisiting the Optuna tuning results, we discovered that the optimal step size had reached the lower bound of the search range. We retuned and are currently reevaluating the results over multiple seeds. The table shows the preliminary results for a single seed.

Table: CIFAR-10 sampling performance. FID and NLL (mean ± std) at different NFE budgets. Lower is better. Updated iDSDS results are from a single seed.

Method500 FID ↓500 NLL ↓1000 FID ↓1000 NLL ↓2000 FID ↓2000 NLL ↓
ALD144.43 ± 0.895.99 ± 0.46101.37 ± 0.615.46 ± 0.7367.72 ± 0.204.68 ± 0.86
DSDS341.16 ± 0.148.90 ± 0.3398.20 ± 0.705.62 ± 0.4868.67 ± 0.774.89 ± 0.63
iDSDS (updated)65.204.5647.303.7743.323.11
EM46.68 ± 0.283.62 ± 0.5442.99 ± 0.233.45 ± 0.5241.85 ± 0.523.39 ± 0.51
PC38.40 ± 0.423.89 ± 0.5831.22 ± 0.243.70 ± 0.5328.58 ± 0.313.50 ± 0.55

Overhead of time score-learning: The main computational overhead arises from choosing an energy-based model (EBM) over a score-based model (SBM), rather than from learning the additional time score. The following table compares training times for denoising score matching (DSM) and dual score matching (DDSM) using both architectures. For the SBM, we adapted the backbone by using one output channel instead of 16 and removing the energy head.

Table: Training runtimes for denoising score matching (DSM) and dual score matching (DDSM) using energy-based (EBM) and score-based (SBM) models on MNIST with an NVIDIA Titan V.

ModelDSMDDSM
EBM24h 51m24h 50m
SBM3h 6m3h 8m

The results show that DDSM introduces negligible additional training time compared with DSM for the same architecture.

AuthorsOpenReview ↗

Addressing weaknesses part II and questions

W3. Sampling results. (continued)

wall-clock sampling time: We additionally measured the average wall-clock sampling time per sample and NFE on identical hardware and with the same batch size.

Table: Average wall-clock sampling time per sample and function evaluation on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU.

DatasetALDEMPCDSDSiDSDS
MNIST (µs/sample/NFE)83.66 ± 0.0183.79 ± 0.2583.69 ± 0.0083.76 ± 0.0083.79 ± 0.01
CIFAR-10 (ms/sample/NFE)1.57 ± 0.061.60 ± 0.111.56 ± 0.111.43 ± 0.051.45 ± 0.04

The sampling time is dominated by model evaluation. Consequently, DSDS and iDSDS introduce negligible additional sampling overhead compared with the baselines.

Overhead of preconditioning To further assess the efficiency of our closed-form Fisher preconditioner, we compare its runtime against Euclidean (no preconditioner) and curvature-aware preconditioning on the 1D GMM.

Table: Inference runtimes (ns/sample/NFE) for different preconditioners on a 1D GMM using an NVIDIA Titan V.

EuclideanFisherCurvature
29.00 ± 7.8931.38 ± 2.491768 ± 84

The Fisher preconditioner has comparable runtime to no preconditioning, while curvature-aware preconditioning is substantially more expensive. Note that in the GMM setting, the Hessian is available in closed form (see Appendix A.5), so the curvature-aware preconditioner does not require numerical Hessian estimation. For general distributions, estimating the Hessian would introduce additional computational overhead.

W4.

We thank the reviewer for highlighting these concerns and agree that practical significance should be supported by convincing empirical evidence. We believe our work makes meaningful conceptual and algorithmic contributions through direct clean-sample generation in the joint space and the proposed Fisher-informed Langevin samplers. We hope that the revised experiments, runtime comparisons, and clarifications presented above provide stronger evidence of the practical relevance of our approach.

Q1. Optuna Tuning.

Hyperparameters were tuned separately for each NFE budget. We used 30 Optuna trials for samplers with one hyperparameter (ALD, PC) and 50 trials for samplers with two or more hyperparameters (DSDS: 2, iDSDS: 3).

The objective was evaluated across multiple seeds and defined as objective=metricmean+0.1⋅metricstd\mathrm{objective}=\mathrm{metric} _{\mathrm{mean}}+0.1\cdot\mathrm{metric} _{\mathrm{std}}, where the metric is the (sliced) W2W_2 distance for GMMs and FID with feature dimension 64 for MNIST and CIFAR-10. We will add a detailed description of the tuning procedure to the manuscript.

Q2. NFE clarifications.

We define one NFE as one forward and backward pass through the energy model to obtain ∇(x,t)Uθ(x∣t)\nabla_{(x,t)}U_{\theta}(x|t). The baselines require only ∇xUθ(x,t)\nabla_x U_\theta(x,t). Our implementation computes the full gradient ∇(x,t)Uθ(x∣t)\nabla_{(x,t)}U_\theta(x|t) and discards the temporal component. Wall-clock sampling times are reported in W3.

Regarding the NFE accounting:

  • PC sampler: Both predictor and corrector steps require one NFE each. Thus, at 500 NFE, PC visits 250 noise levels, compared with 500 for EM and ALD.

  • Gibbs update: Additional density evaluations (forward passes) are excluded from the NFE count. Kim & Ye (2023) use a separate lightweight noise-prediction network, whose evaluation cost they consider negligible relative to the score network. In our implementation, we instead evaluate the energy at candidate noise levels and sample from a categorical distribution. We retain this NFE convention to avoid penalizing the Gibbs baseline for our alternative implementation, while acknowledging that its additional computation is not captured by NFE.

We will clarify these details in the revised manuscript.

Q3. 'reconcile the statement on lines 427-428 that iDSDS outperforms ALT for every CIFAR-10 NFE budget in both FID and NLL'

We thank the reviewer for identifying this inconsistency. The original statement was incorrect for the results in Table 1b. With the updated iDSDS results (see W3), the claim now holds for the evaluated NFE budgets. We will update the table and corresponding discussion accordingly.

I thank the authors for their response, which noticeably strengthens the paper’s empirical case. Although my concerns about the paper’s theoretical aspects remain, I believe its practical significance is sufficient to justify changing my recommendation to “weak accept.”

Reviewer_qRecOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Dual Score Diffusion Sampling (DSDS), a diffusion-based sampler that treats diffusion time (or, equivalently, the associated noise variance) as a random variable during generation. Instead of fixing a prescribed noise schedule, the sampler simulates preconditioned Langevin dynamics directly in the joint data-noise distribution p(x∣t)p(t)p(\mathbf{x} \mid t)p(t) and discards the sampled tt, effectively drawing from the marginal px(x)=∫p(x∣t)p(t) dtp_{\mathbf{x}}(\mathbf{x}) = \int p(\mathbf{x} \mid t)p(t) \, \mathrm{d}t . The prior p(t)p(t) is chosen to favor small tt in order to make clean data likely in px(x)p_{\mathbf{x}}(\mathbf{x}), which is formalized as a bound on W2(px,pdata)\mathcal W_2(p_{\mathbf{x}}, p_{\text{data}}). The authors justify different preconditioning schemes by analyzing the geometry of the (x,t)(x,t) target space and finally show that their samplers empirically show competitive performance on synthetic and image generation benchmarks.

Critical strengths
  • Clear presentation and well-communicated results.
    • The paper is generally easy to follow and the author's main findings and insights are clearly presented.
  • The main idea of directly generating clean samples via the data-noise space is interesting and novel.
    • The core idea of sampling in the augmented (x,t)(x,t)-space (and discarding the sampled tt) is conceptually and algorithmically interesting, as a suitable prior p(t)p(t) may improve mixing times when sampling from complex, highly multimodal target distributions.
  • The use of a Fisher preconditioner is well-justified.
    • The scale analysis of the conditional spatial and temporal scores (under the VE process) naturally justify using the inverse Fisher metric to compensate the scale mismatch.
  • Competitive empirical performance on synthetic benchmarks and low-NFE MNIST.
    • iDSDS performs favorably on synthetic benchmarks (1D and 2D GMMs), and outperforms common diffusion samplers on 100-NFE MNIST, while showing empirical robustness to the choice of initial noise variance.
Critical weaknesses
  • Experimental evaluation lacks interesting synthetic cases.
    • While the experimental evaluation include 1D and 2D GMMs, it would be interesting to see the sample quality and convergence of DSDS and iDSDS on higher-dimensional GMMs---in particular because the smoothing error in Eq. 14 depends on the dimension dd.
    • All GMMs use Gaussians with isotropic covariances. An ablation with less well-conditioned covariance matrices would strengthen the empirical evaluation.
  • DSDS and iDSDS lose to traditional samplers on CIFAR-10.
    • PC and EM substantially outperform DSDS and iDSDS in terms of FID on CIFAR-10. In fact, while larger NFE improves the performance of DSDS, the FID score of iDSDS become worse at 2000 NFE.
Questions for authors
  • DSDS and iDSDS rely on energy-based models that also expose meaningful time-derivatives of the energy, which is why these models are trained with dual score matching. Can the authors elaborate on the computational costs compared to regular score matching?
  • Since energy values are available, MH corrections could be used to correct discretization bias. Have the authors thought about how to extend DSDS to incorporate such corrections?
2 comments
AuthorsOpenReview ↗

Addressing weaknesses and questions

Thank you for your constructive review and the interesting points brought up. We have conducted several additional experiments to address your questions and concerns.

W1. Additional synthetic experiments

Additional experiments on higher-dimensional GMM simplices. We computed the full convergence curves for the GMM simplices described in Sec. E.2 (scaling across dimensions) for DSDS with 500 steps.

The table below addresses the question of sample quality by comparing the final sliced-W2W_2 distances with the theoretical Bounds from eq. (14) for dimensions d=8d=8 to d=512d=512.

Dimension (d)8163264128256512
W₂ ± std (×10⁻²)0.66 ± 0.040.78 ± 0.020.85 ± 0.031.08 ± 0.021.22 ± 0.011.36 ± 0.011.56 ± 0.01
Bound (×10⁻²)3.214.536.388.9912.6517.8925.37

We will add two plots to the manuscript: (i) the full convergence curves and (ii) the results from the table above.

Experiments with anisotropic Gaussians. We are currently using available compute to reevaluate the imaging results. If resources permit, we will also conduct additional experiments on anisotropic GMMs.

W2. Sampling performance

Updated iDSDS performance on CIFAR-10: Upon revisiting the tuning results, we realized that Optuna had reached the lower bound of the step-size search range. We retuned and are currently reevaluating the results. The table shows the preliminary results for a single seed.

Table: CIFAR-10 sampling performance. FID and NLL (mean ± std) at different NFE budgets. Lower is better. Updated iDSDS results are from a single seed.

Method500 FID ↓500 NLL ↓1000 FID ↓1000 NLL ↓2000 FID ↓2000 NLL ↓
ALD144.43 ± 0.895.99 ± 0.46101.37 ± 0.615.46 ± 0.7367.72 ± 0.204.68 ± 0.86
DSDS341.16 ± 0.148.90 ± 0.3398.20 ± 0.705.62 ± 0.4868.67 ± 0.774.89 ± 0.63
iDSDS (updated)65.204.5647.303.7743.323.11
EM46.68 ± 0.283.62 ± 0.5442.99 ± 0.233.45 ± 0.5241.85 ± 0.523.39 ± 0.51
PC38.40 ± 0.423.89 ± 0.5831.22 ± 0.243.70 ± 0.5328.58 ± 0.313.50 ± 0.55

Clarifying performance relative to reverse-SDE samplers.

We acknowledge that EM and PC achieve better FID on CIFAR-10, even with the updated iDSDS results. However, the gap has narrowed substantially and we would like to clarify two points:

(i) As demonstrated by Song et al. (2021, Score-Based Generative Modeling through Stochastic Differential Equations), reverse-SDE sampling can outperform annealed Langevin dynamics (SMLD). Since DSDS extends ALD, we consider ALD the most direct baseline. We included EM and PC as stronger reference methods for transparency. With our updated results, we aim to demonstrate that DSDS can narrow the gap to reverse-SDE samplers.

(ii) We acknowledge that more efficient higher-order samplers exist, particularly for reverse-SDE and probability-flow ODE formulations. Our goal is not to outperform these specialized solvers, but to investigate the potential of schedule-free joint-space sampling.

Q1. Computational cost.

The main computational overhead comes from choosing an energy-based model (EBM) over a score-based model (SBM). The following table compares training times for denoising score matching (DSM) and dual score matching (DDSM) using both architectures. For the SBM we trained our backbone with slight modifications (1 output channel instead of 16 and no energy head)

Table: Training runtimes for denoising score matching (DSM) and dual score matching (DDSM) using energy-based (EBM) and score-based (SBM) models on MNIST with an NVIDIA Titan V.

ModelDSMDDSM
EBM24h 51m24h 50m
SBM3h 6m3h 8m

Q2. MH correction.

Incorporating an MH correction is a natural extension enabled by our energy-based parameterization, which we would like to explore in future work. For the state-dependent preconditioner, the unreflected Gaussian proposal density is q(z′∣z)=N(z′;μ(z),2ηA(z))q(z'\mid z)=\mathcal N(z';\mu(z),2\eta A(z)) with μ(z)=z+η[A(z)s(z)+∇⋅A(z)].\mu(z)=z+\eta[A(z)s(z)+\nabla\cdot A(z)]. The main complication arises from boundary reflection: multiple unconstrained Gaussian proposals can map to the same reflected state z′z'. Consequently, the reflected proposal density is no longer Gaussian and must account for all such proposals.

Thank you for your rebuttal and the additional experiments. My questions have been adequately answered. I will thus keep my accept score.

Reviewer_4SE6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends annealed Langevin sampling to the joint space of (x,t)(x, t), treating the VE noise variance as a stochastic variable governed by a prior p(t), and applies preconditioned (over or underdamped) Langevin dynamics to p(x∣t)p(t)p(x | t) p(t). The time score is estimated via dual score matching. The central technical advance is a Fisher-information-based preconditioner, derived from the corruption kernel, which becomes constant in the tt-direction after a log-variance reparameterization using a Pareto prior. Experiments on GMMs and MNIST/CIFAR-10 demonstrate that the inertial variant (iDSDS) outperforms ALD and is competitive with EM/PC samplers at low NFE.

Critical strengths
  • The core question whether a joint (x,t)(x, t) chain can directly generate clean samples is natural and previously unexplored. Prior works by Kim and Ye, and Qin and Risteski, leveraged the lifted space only for warm starts or theoretical analysis, not for direct sampling.

  • The section on geometry is clear and accurate.

  • The inertial variant is thoughtfully constructed, incorporating a Riemannian mass, specular reflection, and position-only discretization.

  • The appendix contains sufficient detail for reproducibility.

  • The paper provides extensive ablation studies, and the parameter-transfer experiment in E.4.2 is an appropriate control.

Critical weaknesses
  • The claim of being “schedule-free” is overstated. The sampler still determines how quickly it progresses through t: the prior score, τ-mobility, and step size collectively set a velocity in the noise direction, making this choice analogous to specifying a schedule.

  • The method is an analog of simulated tempering and inherits its associated challenges. Sections 3.1 and 3.2 are not diffusion-specific; replacing tt with inverse temperature recasts the approach as simulated tempering with continuous Langevin updates in the auxiliary variable. The paper should make this connection explicit and address the standard criticism: a tempering chain only aids inference when it resides at the target, and can only access new modes by returning to the reference and backtracking. For complex multimodal distributions, this results in long excursions, frequent backtracking, or chains stuck at intermediate noise levels, reducing the fraction of useful samples per unit computation. ALD avoids this issue: each chain traverses the full path and can be debiased with SMC, whereas the joint chain lacks a comparable correction. The experiments sidestep this regime (short chains, sharp prior), leaving open questions about DSDS performance when mode discovery genuinely requires adding a lot of noise.

  • The paper does not clearly delineate what aspects of the method generalize. It would be helpful to distinguish three layers: (1) elements that apply to any annealing path, results specific to any Gaussian-kernel diffusion path and (3) those unique to VE.

  • Algorithms 3 and 4 use a fixed number of steps L without a stopping rule, making run-time deterministic while the noise level at readout—and thus sample quality—is random. This trade-off is suboptimal for a method targeting p(x∣tmin)p(x | t_min): if the chain does not reach tmint_min within LL steps, the output sample remains noisy. How can you ensure that the particles dont just bounce around in a random walk fashion?

  • While the preconditioner is cheap, each NFE requires forward and backward passes through the energy model, and training involves double backpropagation and an additional loss function. The computational cost of tuning per NFE budget is also unreported. The claim of “negligible overhead” in Section 5.2 should be qualified accordingly.

Questions for authors

Questions and discussion

  • The time score ∂_t log p(x | t) is the classical Fisher score of the family {p(· | t)} with respect to t, and the Fisher information used in the preconditioner is its second moment. Is there an information-geometric interpretation for the space-time score?

  • Is it necessary to learn the time score? For any diffusion path, the marginals satisfy the Fokker-Planck equation; for VE, ∂tlogp=12∇⋅s+12∥s∥2\partial_t log p = \frac{1}{2}\nabla\cdot s + \frac{1}{2}\|s\|^2. Thus, any pretrained score network can provide the time score using a single Hutchinson JVP per step.

  • While the abstract frames the method as an annealing-based neural sampler for unnormalized targets, the paper does not actually operate in that setting. In the GMM experiments, exact scores along the VE path are used (i.e., inference-time control, no learning), while the image experiments involve learning a model from data and sampling from it. Have you tried an inference-time control setting, where we have a pre-trained diffusion model and its cores are given, but the marginal law is not?

3 comments
AuthorsOpenReview ↗

Addressing weaknesses part I

Thank you for your constructive review and the interesting points brought up. We have conducted several additional experiments to address your questions and concerns.

W1. Notion of schedule-free sampling

It is correct that the drift (determined by the preconditioner and joint score) and stepsize govern the noise-level evolution (alongside the stochastic contribution). However, this is fundamentally different from prescribing a noise schedule. In ALD, the noise levels are fixed a priori and shared across all samples, independently of their current states. In contrast, DSDS evolves the noise variable stochastically based on the current state (x,t)(x,t). Consequently, the visited noise levels are determined dynamically and differ across sample trajectories, as illustrated in Fig. 2 (right) by the nonzero standard deviation of the noise trajectories.

W2. Tempering vs. annealing.

Thank you for highlighting this connection. We agree that the joint Langevin dynamics in Eqs. (12)–(13) also apply when considering an inverse temperature β\beta instead of the noise variance tt. However, there is an important distinction between simulated tempering and diffusion-based annealing. Simulated tempering applies a pointwise power transformation, pβ(x)∝p(x)βp_\beta(x)\propto p(x)^\beta, preserving the locations of the modes. In contrast, diffusion-based annealing convolves the density with a Gaussian, pt(x)=(p∗N(0,tI))(x)p_t(x)=(p*\mathcal N(0,tI))(x), which smooths the distribution and can shift or merge modes (see Fig. 1 in Akhound-Sadegh et al. Progressive Inference-Time Annealing of Diffusion Models for Sampling from Boltzmann Densities). We acknowledge that repeated excursions to higher noise levels may reduce sampling efficiency. We will clarify the connection to simulated tempering, the differences between the annealing paths, and this potential limitation.

W3. Generality of the framework.

The joint target construction (Sec. 2.1) and Langevin dynamics (Eqs. 12–13) apply to general annealing paths, independently of the corruption process. For Gaussian diffusion paths, the score-scaling bounds and Fisher-informed preconditioner admit kernel-dependent expressions. In particular, the VE diffusion admits simple closed-form expressions, with similarly tractable results for VP diffusion under a data-normalization assumption. We will revise the paper to make this more explicit.

W4 Stopping Criterion

We agree that a fixed number of sampling steps does not guarantee that the final noise level is close to t_{\min}. We use a fixed sampling budget to enable controlled comparisons at equal NFE, while the current implementation returns the final data state irrespective of the noise level. Two possible modifications could address this limitation: i) Terminate the trajectory when the noise variable first reaches a sufficiently low threshold ii) If the trajectory terminates at a higher noise level, apply a final denoising step using Tweedie's formula to estimate the corresponding clean sample. We will discuss these alternatives as potential extensions.

W5 Computational overhead

Training cost The main computational overhead arises from choosing an energy-based model (EBM) over a score-based model (SBM), rather than from learning the additional time score. The table below compares training runtimes for denoising score matching (DSM) and dual score matching (DDSM). For the SBM, we adapted the backbone to use one output channel instead of 16 and removed the energy head.

Table: Training runtimes for denoising score matching (DSM) and dual score matching (DDSM) using energy-based (EBM) and score-based (SBM) models on MNIST with an NVIDIA Titan V.

ModelDSMDDSM
EBM24h 51m24h 50m
SBM3h 6m3h 8m

These results show that DDSM introduces negligible additional training cost relative to DSM within the same architecture.

Preconditioning overhead Our claim of negligible overhead in Sec. 5.2 refers specifically to the closed-form Fisher preconditioner, not the overall computational cost. The following table compares the runtime of different preconditioners.

Table: Inference runtimes (ns/sample/NFE) for different preconditioners on a 1D GMM using an NVIDIA Titan V.

EuclideanFisherCurvature
29.00 ± 7.8931.38 ± 2.491768 ± 84

The Fisher preconditioner has comparable runtime to Euclidean preconditioning (i.e. no preconditioning), while curvature-aware preconditioning is substantially more expensive. Note that in the GMM setting, the Hessian is available in closed form (see appendix A.5), so the curvature-aware preconditioner does not require numerical Hessian estimation. For general distributions, estimating the Hessian would introduce additional computational overhead.

AuthorsOpenReview ↗

Addressing weaknesses part II and questions

W5 Computational overhead (continued)

Sampling time The following table reports wall-clock sampling time per sample and NFE under identical hardware and batch-size conditions.

Table: Average wall-clock sampling time per sample and function evaluation on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU.

DatasetALDEMPCDSDSiDSDS
MNIST (µs/sample/NFE)83.66 ± 0.0183.79 ± 0.2583.69 ± 0.0083.76 ± 0.0083.79 ± 0.01
CIFAR-10 (ms/sample/NFE)1.57 ± 0.061.60 ± 0.111.56 ± 0.111.43 ± 0.051.45 ± 0.04

All samplers exhibit comparable per-NFE sampling times, indicating that joint-space updates introduce little additional runtime beyond model evaluation.

Tuning cost. Optuna tuning is dominated by sampling and score evaluations. In our experiments, metric evaluation and optimization overhead account for only a small fraction of the total tuning time. We will clarify the tuning procedure and computational costs in the revised manuscript.

Q1. information-geometric Interpretation.

In our joint-space formulation, both xx and tt are stochastic sampling coordinates. The spatial and temporal scores describe how the joint log-density changes along these coordinates.

The Fisher metric characterizes the local sensitivity of the distribution to changes in these coordinates. Large Fisher information indicates directions of high sensitivity, whereas small Fisher information indicates less sensitive directions. Its inverse provides a principled preconditioner that adapts the sampling dynamics to this local geometry (Titsias, 2023, Optimal Preconditioning and Fisher Adaptive Langevin Sampling ).

Since the Fisher metric of the joint target p(x,t)p(x,t) is generally intractable, we instead use the tractable metric induced by the Gaussian corruption kernel and noise prior q(x,t∣x0)=q(x∣x0,t)p(t)q(x,t\mid x_0)=q(x\mid x_0,t)p(t).

Q2 and Q3. Learning versus estimating the time score.

Thank you for highlighting this interesting alternative. Estimating the time score via the Fokker–Planck equation would allow DSDS to use pretrained score models without additional training. The main trade-offs are increased inference cost due to Hutchinson trace estimation and the associated estimation error, which must be compared with the approximation error of a learned time score.

We conducted preliminary MNIST experiments using two separately trained score-based models: DSM with Hutchinson time-score estimation and DDSM with a directly learned time score. At 200 NFE, DSDS achieved an FID of 9.63 with DDSM, compared with 69.50 using DSM and Hutchinson estimation. The inference runtimes are reported below (additionally stating the EBM)

Table: Inference runtimes [ms/sample/NFE] for denoising score matching (DSM) and dual score matching (DDSM) using energy-based (EBM) and score-based (SBM) models on MNIST with an NVIDIA Titan V.

ModelDSMDDSM
EBM7.460.47
SBM3.290.14

Direct time-score evaluation is considerably faster than Hutchinson estimation. While these preliminary results demonstrate the feasibility of inference-time control, achieving comparable sample quality requires further investigation.

We will discuss this alternative and its trade-offs in the revised manuscript.

Thank you to the authors for the clarifications. I found the answers to be insightful. I think the paper would be improved by including a discussion of the geometry and a comparison of the various scores and training. I have raised my score.

Comments on the paper

1 comment

We thank all reviewers for their time, effort, and constructive feedback. In response, we have uploaded a revised manuscript with all substantive changes highlighted for ease of reference:

  • Updated image-generation results for iDSDS based on the revised hyperparameter tuning range (Tables 1, 9, 10).
  • Extended GMM experiments to higher dimensions (8–512; Sec. E.2).
  • Quantified computational overhead of sampling (Table 2), preconditioning (Sec. E.4.3), and energy-model parameterization and dual score matching (Sec. D.3.3).
  • Expanded discussions on simulated tempering (Sec. 4), framework generality (Sec. 3.4), limitations (Sec. 7), and hyperparameter tuning (Sec. D.4).

Below, we provide detailed responses to each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.