- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates and proposes a new method for diffusion sampling, generalizing the existing stochastic (SDE) and deterministic (PF-ODE) samplers. Their method jointly learns the drift and noise variance parameters that parameterize the solver. The solver is tested on several benchmarks and outperforms existing, fixed and learnable, approaches in the majority of settings.
Critical strengths
- Impact
- The proposed method shows strong performance on several benchmarks, indicating a clear advancement over existing approaches.
- Clarity of writing
- Outside of the complexity of the content, the paper is clearly presented, with very helpful use of stylization and well-executed figures and tables.
Critical weaknesses
None, but there are minor weaknesses detailed in the questions.
Questions for authors
Clarifications:
- Section 4 and Figure 2 nicely illustrate the role of . I would be interested to see what values of you observed in the runs reported in Section 5. Of particular interest would be whether follows the trade-off that the theory predicts, i.e., more stochasticity where distribution mismatch is large and the score is reliable.
- Class conditioning dramatically improves generative performance for diffusion models and potentially affects different sampling methods differently. You only report 2 class-conditional benchmarks, even though CIFAR-10 and ImageNet64 also offer labels. Do you have additional results on those?
- FID can have significant run-to-run variation, which is especially relevant for small FID values (<3) and allows for seed selection. It appears that all FID numbers are individual runs, as opposed to, e.g., averages of 3. Please clarify how you handled seed noise.
Future work:
- The pixel models use an Inception loss and are then evaluated with FID, in the same feature space used for the loss. This is a problematic confound that distorts the results. Do the results change when decoupling these, e.g., using FD_DINOv2?
- FID and FD_DINOv2 often yield qualitatively different results, even without the aforementioned confounding, and FID has several known problems [1]. Showing that the RD-Solver beats existing approaches on most benchmarks with FDD as well would improve the evaluation.
[1] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models, Stein et al., 2023
Reply to rebuttal
Thank you for the detailed and thorough response. The clarifications on the learned stochasticity coefficients, the class-conditional setting, and the seed variance fully answer my questions. I think the new analyses in Appendices C.3–C.5 improve the paper.
My one remaining suggestion concerns the evaluation metric. The FD-DINOv2 results in the rebuttal are, in my view, a very valuable addition. I would encourage the authors to report FD-DINOv2 in as many places in the paper as possible, alongside FID. FID has well-documented limitations [Stein et al., 2023], and FD-DINOv2 tracks perceptual quality more closely. I believe it should be the standard metric for evaluating generative performance and that FID is still around mostly for historical reasons. This is of course especially helpful when feature spaces are used for a loss. On CIFAR-10, the improvement over iPNDM was inflated under FID, as they are noticeably smaller under FD-DINOv2. The gains remain clear, but reporting both metrics gives readers a more faithful picture of the method’s advantage. It would also make the paper a stronger reference for future work.
If not already planned, I would add the remaining rebuttal results, namely the class-conditional CIFAR-10 results, the seed study, and the DINOv2-loss ablation, to the appendix.
Overall, I remain positive about this submission and maintain my score.