- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose DSDS and its inertial variant, iDSDS, sampling methods that treat the noise variance as a stochastic variable and sample it jointly with the data. They further develop a Fisher preconditioner based on VE and VP kernels. The ablations study the method’s sensitivity to the initial noise variance, the variance parametrisation, and preconditioning.
Critical strengths
The paper combines spatial and temporal scores with a Fisher-informed preconditioner in a log-variance parametrisation. This addresses the different scaling of the data and noise-variance coordinates through a corruption-kernel-based construction. The proposed implementation of the joint dynamics is tractable, and the Gaussian-mixture experiments with oracle scores allow sampling behaviour to be studied independently of model approximation error. The ablation studies help explain how prior selection and parametrisation influence sampling performance.
Critical weaknesses
I believe that the main contribution of this paper is algorithmic. The authors recognise that joint sampling has already been proposed by Kim & Ye (2023) and Qin & Risteski (2024), and that dual score matching also appears in prior work, as stated on lines 94–100, 137–141, and 277–283. Hence, the primary contribution is the preconditioning construction and its integration into the Langevin samplers (both overdamped and inertial), rather than the introduction of joint sampling itself. The theory supports the design, yet the guarantee provided in Proposition 3.1 does not establish faster convergence or account for discretisation and approximation errors. The paper should then demonstrate a practical advantage.
However, the claimed robustness improvements do not sufficiently demonstrate such an advantage. The comparative sensitivity experiment is conducted only on a one-dimensional Gaussian mixture. The robustness improvement requires a specially picked log-Gaussian prior and leads to slower convergence from the standard high-noise initialisation. The prior selected for the remaining experiments, i.e., the Pareto prior, remains sensitive to initialisation, similarly to ALD (see Figure 3 and the paragraph beginning on line 351). I read this as a speed–robustness trade-off, rather than a simultaneous improvement in robustness and sampling performance across the evaluated tasks. Moreover, the sampler hyperparameters are tuned separately for each experimental setting and, in the ablations, for each sampling budget (lines 288–291; lines 347–350). Though this is not inherently problematic, these experiments do not show that replacing a prescribed schedule with a noise prior reduces the overall tuning burden.
Likewise, the image-generation examples provide only partial evidence of an advantage. Indeed, iDSDS performs well on MNIST at 100 NFE, but its FID is worse than both EM and PC at every CIFAR-10 evaluation budget in Table 1b. Even at a generous budget of 2,000 NFE, its FID is 94, compared with 28.6 for PC and 68 for ALD. These numbers must also be weighed against the overhead of time-score learning, whose training cost and approximation error are acknowledged by the authors (lines 484–485). It also remains an open question whether the method achieves an advantage in wall-clock time.
I thank the authors for being transparent about their limitations and their contribution relative to prior work. Overall, the limited evidence of improved robustness, reduced tuning, or computational benefit leaves the practical significance of the algorithmic extension unestablished. This, combined with the narrow theoretical contribution, motivates my rejection.
Questions for authors
- For the Optuna tuning, what was the number of trials and what was the objective? Were the tuning done for each NFE?
- What precisely constitutes one NFE for each method, including the predictor and corrector evaluations in PC, and the density evaluations in Gibbs-update? Appendix D.3.3 provides that obtaining the score requires a forward and backward pass while Appendix F.1 says that the baselines discard the time score, is it computed and discarded or not computed at all? Could the authors provide the wall clock sampling time for each method on the same hardware?
- Could you please reconcile the statement on lines 427-428 that iDSDS outperforms ALT for every CIFAR-10 NFE budget in both FID and NLL (Table 1b)? The ALD outperforms iDSDS in terms of FID at 2000NFE.
3 comments
Addressing weaknesses part II and questions
W3. Sampling results. (continued)
wall-clock sampling time: We additionally measured the average wall-clock sampling time per sample and NFE on identical hardware and with the same batch size.
Table: Average wall-clock sampling time per sample and function evaluation on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU.
| Dataset | ALD | EM | PC | DSDS | iDSDS |
|---|---|---|---|---|---|
| MNIST (µs/sample/NFE) | 83.66 ± 0.01 | 83.79 ± 0.25 | 83.69 ± 0.00 | 83.76 ± 0.00 | 83.79 ± 0.01 |
| CIFAR-10 (ms/sample/NFE) | 1.57 ± 0.06 | 1.60 ± 0.11 | 1.56 ± 0.11 | 1.43 ± 0.05 | 1.45 ± 0.04 |
The sampling time is dominated by model evaluation. Consequently, DSDS and iDSDS introduce negligible additional sampling overhead compared with the baselines.
Overhead of preconditioning To further assess the efficiency of our closed-form Fisher preconditioner, we compare its runtime against Euclidean (no preconditioner) and curvature-aware preconditioning on the 1D GMM.
Table: Inference runtimes (ns/sample/NFE) for different preconditioners on a 1D GMM using an NVIDIA Titan V.
| Euclidean | Fisher | Curvature |
|---|---|---|
| 29.00 ± 7.89 | 31.38 ± 2.49 | 1768 ± 84 |
The Fisher preconditioner has comparable runtime to no preconditioning, while curvature-aware preconditioning is substantially more expensive. Note that in the GMM setting, the Hessian is available in closed form (see Appendix A.5), so the curvature-aware preconditioner does not require numerical Hessian estimation. For general distributions, estimating the Hessian would introduce additional computational overhead.
W4.
We thank the reviewer for highlighting these concerns and agree that practical significance should be supported by convincing empirical evidence. We believe our work makes meaningful conceptual and algorithmic contributions through direct clean-sample generation in the joint space and the proposed Fisher-informed Langevin samplers. We hope that the revised experiments, runtime comparisons, and clarifications presented above provide stronger evidence of the practical relevance of our approach.
Q1. Optuna Tuning.
Hyperparameters were tuned separately for each NFE budget. We used 30 Optuna trials for samplers with one hyperparameter (ALD, PC) and 50 trials for samplers with two or more hyperparameters (DSDS: 2, iDSDS: 3).
The objective was evaluated across multiple seeds and defined as , where the metric is the (sliced) distance for GMMs and FID with feature dimension 64 for MNIST and CIFAR-10. We will add a detailed description of the tuning procedure to the manuscript.
Q2. NFE clarifications.
We define one NFE as one forward and backward pass through the energy model to obtain . The baselines require only . Our implementation computes the full gradient and discards the temporal component. Wall-clock sampling times are reported in W3.
Regarding the NFE accounting:
-
PC sampler: Both predictor and corrector steps require one NFE each. Thus, at 500 NFE, PC visits 250 noise levels, compared with 500 for EM and ALD.
-
Gibbs update: Additional density evaluations (forward passes) are excluded from the NFE count. Kim & Ye (2023) use a separate lightweight noise-prediction network, whose evaluation cost they consider negligible relative to the score network. In our implementation, we instead evaluate the energy at candidate noise levels and sample from a categorical distribution. We retain this NFE convention to avoid penalizing the Gibbs baseline for our alternative implementation, while acknowledging that its additional computation is not captured by NFE.
We will clarify these details in the revised manuscript.
Q3. 'reconcile the statement on lines 427-428 that iDSDS outperforms ALT for every CIFAR-10 NFE budget in both FID and NLL'
We thank the reviewer for identifying this inconsistency. The original statement was incorrect for the results in Table 1b. With the updated iDSDS results (see W3), the claim now holds for the evaluated NFE budgets. We will update the table and corresponding discussion accordingly.
I thank the authors for their response, which noticeably strengthens the paper’s empirical case. Although my concerns about the paper’s theoretical aspects remain, I believe its practical significance is sufficient to justify changing my recommendation to “weak accept.”
Addressing weaknesses part I
Thank you for your constructive review and the interesting points brought up. We have conducted several additional experiments to address your questions and concerns.
W1. Conceptual novelty and algorithmic contributions.
Conceptual novelty: We believe that directly generating clean samples from joint-space dynamics is a conceptual contribution. Prior work considered joint-space sampling for warm-starting (Kim & Ye, 2023) or theoretical mixing analysis (Qin & Risteski, 2024), rather than direct clean-sample generation. Moreover, Qin & Risteski's concrete mixing-time guarantees focus on Gaussian mixtures, whereas we additionally demonstrate joint-space sampling on image data.
Algorithmic contributions: Our contributions go beyond the joint-space formulation. The ablations in Fig. 4 show that existing design choices do not directly translate into efficient clean-sample generation. In particular, Euclidean preconditioning and Gibbs time updates correspond to key aspects of Qin & Risteski (2024) and Kim & Ye (2023), respectively, highlighting the importance of our proposed modifications. We further introduce an underdamped joint-space Langevin sampler with state-dependent preconditioning.
Practical advantage: Our preliminary updated results (see W3) indicate that iDSDS outperforms ALD across the evaluated CIFAR-10 budgets and substantially narrows the gap to EM.
W2. Speed–robustness trade-off and tuning burden.
Speed–robustness trade-off: We agree that our results demonstrate a speed–robustness trade-off rather than a simultaneous improvement. The purpose of our robustness experiment was to highlight the flexibility of the framework: (i) different priors induce different sampling dynamics, and (ii) even the same prior can exhibit different behavior depending on its hyperparameters. We do not expect a single prior to be optimal for both fast sampling and robustness.
We would like to clarify that the log-normal prior was not specifically constructed for robustness but is a natural generalization of the Pareto prior, replacing a constant log-variance score with a linear one. Furthermore, Fig. 4 demonstrates that clean-sample generation is possible with different priors, not only Pareto. Exploring more expressive priors remains an interesting direction for future work.
Tuning burden: We acknowledge that replacing a prescribed schedule with a noise prior does not eliminate hyperparameter tuning. Our contribution is to replace a fixed noise-level trajectory with state-dependent stochastic dynamics, rather than to claim a reduced tuning burden.
W3. Sampling results.
iDSDS sampling performance on CIFAR-10: Upon revisiting the Optuna tuning results, we discovered that the optimal step size had reached the lower bound of the search range. We retuned and are currently reevaluating the results over multiple seeds. The table shows the preliminary results for a single seed.
Table: CIFAR-10 sampling performance. FID and NLL (mean ± std) at different NFE budgets. Lower is better. Updated iDSDS results are from a single seed.
Overhead of time score-learning: The main computational overhead arises from choosing an energy-based model (EBM) over a score-based model (SBM), rather than from learning the additional time score. The following table compares training times for denoising score matching (DSM) and dual score matching (DDSM) using both architectures. For the SBM, we adapted the backbone by using one output channel instead of 16 and removing the energy head.
Table: Training runtimes for denoising score matching (DSM) and dual score matching (DDSM) using energy-based (EBM) and score-based (SBM) models on MNIST with an NVIDIA Titan V.
The results show that DDSM introduces negligible additional training time compared with DSM for the same architecture.