- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposed DDMC:Denoising Diffusion Monte Carlo. It combines MCMC with diffuson models and run MCMC on the path space of the learned diffusion. The rejection is calculated by the forward-backward path weight. The method was evaluated on a large range of tasks, showing its effectiveness.
Critical strengths
The experiment in this paper is comprehensive: GMM, particles and BNNs. This show their proposed method can be applied to a variety of targets. The writing of the paper is also easy to follow and well-structured.
The proposed i-SIR parallel proposals and path tuning might be potential in inspiring future works.
Critical weaknesses
- The novelty is limited and missing the most relevant related work. The idea of applying MCMC to path is also standard. For example [1] (eq, 6 and the line below eq 6: Reweighing/resampling methods utilized in this context are e.g. Importance Sampling or Neural MCMC].); [2]. The main contribution of this paper is to apply this to diffusion models. Another highly related work is accelerated parallel tempering (APT) [3]. APT (APT-diff) propose to run a MCMC chain on ladder of paths, while DDMC proposed to run a MCMC chain on the entire paths: DDMC is just the 1 ladder (most vanilla) version of APT in this sense. However, the paper did not mention or compare with them.
[1] Stochastic Normalizing Flows, NeurIPS.
[2] Advanced MCMC Methods for Sampling on Diffusion Pathspace.
[3] Accelerated Parallel Tempering via Neural Transports. ICLR.
- One concern I have is the scalability of the proposed method: running MCMC on path is known to be hard. This is because the path largely augment the space and can have much higher variance (hence higher rejec rate). Therefore, how does the method scale with dimension is not super clear to me. When clear evidence supporting my concern is LJ13 to LJ55, the proposal acceptance largely decrease. (also see my question)
Questions for authors
-
Can you try to see how the rejection increase as the dimension scales?
-
Standard MCMC can easily tune step size to increase acceptance. What is the analog in this method?
7 comments
Sorry for the confusion about my earlier point regarding APT. I agree that APT and DDMC are different in their respective settings, and that DDMC is more flexible.
I also realized that I had misunderstood part of the proximal step. Thank you for answering my questions and clarifying this. I reevaluated the paper and changed my score for now.
I have one follow-up question: How is tuned? For larger , the acceptance rate tends to be low, while smaller values make the proposals less exploratory.
DiGS has a similar issue. I remember that they use two different values of (at least in some of their experiments). When the paper first came out, I checked their acceptance rates and found that almost all accepted moves occurred at the smaller . This may not be a major problem for DiGS (although the overall acceptance rate could be somewhat misleading), since their computational cost is not dominated by diffusion-network evaluations. Here, however, rejected proposals here could waste many expensive network calls.
I also checked the PT baseline settings in G3 and think the baseline could be improved in two ways:
- PT should attempt swaps at every iteration rather than waiting for 50 MALA steps between swaps. This would not require additional energy evaluations.
- Instead of using a geometric temperature ladder, following Syed et al., PT could use the first few rounds to adapt the ladder so that the swap rejection rates are approximately equal across adjacent temperature pairs.
Thank you for your response and for taking the time to re-evaluate our work. We are glad the clarification of the proximal step was helpful, and we are grateful to you for bringing SNF, Beskos et al. (2013), and APT to our attention. Situating our work in the appropriate context with prior work is very important and these references are necessary for accurately presenting our work to future readers.
Tuning of . In this work our goal is to primarily study values of that facilitate global proposals. Although smaller values have higher acceptance and require fewer diffusion steps, they are unlikely to provide jumps between regions separated by large barriers, which are a key limitation of existing MCMC techniques. We set using heuristics similar to [a], [b], so that the VE-SDE noise spans across the state space. While this level of does enable cross-region jumps in our experiments, we noticed that there is still non-negligible dependence on the clean state and that the method is noticeably more likely to propose jumps to the same basin than we would observe for an independence sampler (which can be approached in our framework by using much larger ). The most useful way to tune given a sampling objective is to perform a sweep of different values and see which one provides the most efficient way to measure the quantity of interest. For example, in our post-run analysis of the LJ-38 medium temperature experiment (see Appendix D.2), we found that setting a smaller value of than we used in the original run (going from 2.0 to 0.7) might lead to a boost in efficiency of occupancy estimation. In the actual LJ-38 experiment, we decided to use the value 2.0 that is used by iDEM after we had observed round-trips between the funnels at this value in early development experiments before our final training and sampling run.
On the cost of rejected proposals. We see the cost of rejected DDMC proposals as part of the mixing problem rather than as waste. Local samplers also spend most (sometimes all) of their compute on moves that do not change the metastable region, and in PT an entire ladder advances together to gradually produce each cold-rung state whose funnel identity is independent of its past, so what matters in every case is the cost per independent crossing. In DDMC, rejected global proposals are part of the independent crossing cost. In practice we recommend a global neural proposal only in situations like LJ-38 below melting, where local samplers like MALA fail to mix and PT is known to be slow.
PT Baseline. Thank you for your suggestions on improving our PT baseline. Making a fair wall-clock comparisons for MCMC and deep learning methods is difficult, and we devoted considerable effort to tuning the PT baseline. Our ladder was chosen from a screen that varied the number of rungs, the hottest temperature, the rung placement, and the number of local steps per swap round. The compute needed for a converged occupancy estimate varied by less than a factor of two across the configurations we tried, which suggests the baseline is in the right ballpark. Nonetheless, we expect there is still room for improvement in the PT tuning. Attempting swaps after every local step is a helpful suggestion and worth trying in future PT baselines. We did not use the adaptive spacing of Syed et al., but our geometric ladder already gives approximately uniform swap acceptance across adjacent pairs (Appendix G.3).
We hope this addresses your remaining questions, and we thank you again for the careful reading.
[a] Song and Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019.
[b] Song and Ermon. Improved Techniques for Training Score-Based Generative Models. NeurIPS 2020.
We thank the reviewer for directing us towards important prior work.
W1, related work. Our revision now cites and discusses SNF, APT and Beskos et al. (2013). The DDMC acceptance ratio is a realization of the forward/backward path ratio of NCMC, SNF, SPS, and APT that we formulate as an MH correction of the proximal sampler. This proximal sampler correction was introduced concurrently by MAD-Path. We discuss similarities and differences with APT below.
DDMC vs. One-Rung APT. DDMC is closely related but not equivalent to one-rung Diff-APT. DDMC is a Metropolis-within-Gibbs framework that corrects bias in a learned diffusion used as a proximal sampler, while one-rung Diff-APT is a transport between two distributions which evolve independently between swaps. In particular, the noisy state of DDMC is a forward process sample conditioned on a paired target sample at each step, which is not true of one-rung Diff-APT. The two coincide as σmax→∞ in which case both become independence samplers. We observe that a finite σmax can provide better mixing than increasingly large σmax (Appendix D.2, Table 7). One-rung Diff-APT must directly model the noisy state marginal at finite σmax using an EBM, which is a limitation not shared by DDMC.
Motivation of DDMC vs. APT. APT contains the single-transport correction as a special case it neither investigates nor recommends. Its premise is that a learned transport is too weak to be accepted on its own, so the ladder exists to split the transport into short hops that PT can tolerate, and its round-trip analysis treats the per-rung rejection as the quantity to keep small. Every APT experiment uses N≥4 rungs and nothing in the paper indicates that one transport level could be viable. DDMC starts from the opposite premise, that a standard denoiser trained on local MCMC data can be accurate enough to serve as a direct global proposal, and the evidence of the paper is that this holds in a variety of cases. DDMC is not a special case that APT covers but a design in the regime APT is built to avoid, and showing that a direct global proposal can work is our contribution.
Score vs. EBM Diffusion. The rungs of Diff-APT are the noised marginals of the target, and PT needs each rung as an unnormalized density for its local moves and swap weights. Diff-APT must therefore parametrize the noised marginals with an energy-based model, take its gradient as the score of the transport, and train it by score matching on its own approximate samples. The swap acceptance then depends on the learned rung densities being right as well as on the transport, which is a source of error that the transport alone does not have, and a score that is constrained to be the gradient of an energy is more restrictive to train and to scale than an unconstrained denoiser. DDMC never evaluates a noised marginal density. The target enters only at the path endpoints, the intermediate distributions belong to the model, and the model can be trained by standard denoising regression instead of EBM diffusion.
Comparison with APT. To our knowledge no APT code or models have been released, so we reproduced its d=100 scaling study with exact analytic densities and score, an upper bound on any trained Diff-APT, and compared wall-clock per effective sample on one GPU at each method's saturating batch (Appendix F.3, Table 12). DDMC trained on mis-weighted MALA data produces an effective sample 2.4x faster than analytic Diff-APT at its best paper setting K=5, and 4.3x faster than a hypothetical Diff-APT whose trained model reproduces the analytic one. This adds important context about the viability of direct global proposals beyond what is presented in the APT paper.
W2 and Q1, acceptance against dimension. Please refer to the newly added Appendix H.1 Table 15 and the general response about scaling with dimension. Global proposals for high-dimensional targets are an active research area, and the requirement of a model that is accurate per coordinate applies to the whole class rather than to DDMC alone. Within this class DDMC scales well: it is an exact sampler at the scale of LJ-55, where recent neural samplers struggle to match the target energy spectrum (Table 2, Appendix F.2), and it attains useful acceptance at d=550 on the BNN.
Q2, the step-size analog. σmax is the proximal sampler's step size: it sets how far a proposal can move, and larger values cross between regions but attain lower acceptance. The same applies to DDMC, which is a correction for a learned proximal sampler. Appendix D.2 (Table 7) scans it on LJ-38 and finds the accepted cross-funnel rate peaks at 0.7, 2.1x the rate at the production value of 2.0, so the main-text LJ-38 results are a lower bound for the same model. This analysis was done after sampling was completed and is not part of the cost comparison.