- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes PD-CDS, a guided sampling method for diffusion models that allows for the incorporation of various constraints during the sampling process. Like many other approaches, PD-CDS is training-free; it applies guidance to the posterior mean estimate, , during diffusion sampling. A key distinction of this method is that the step size, , is derived from a dual problem and updated automatically. In our experiments—which utilized a simple unconditional image generation diffusion model—we addressed basic inverse problems in imaging as well as the generation of human face images that satisfy specific, user-defined conditions. In the latter task, PD-CDS demonstrated superior performance compared to other existing methods.
Critical strengths
In the task of generating samples of human faces that satisfy two specific criteria, this approach demonstrated strong performance compared to other methods. This suggests that similar techniques could potentially be applied in other domains as well.
Critical weaknesses
Weakness 1: Although the authors claim to have automated the hyperparameter , the method still requires to determine . Furthermore, hyperparameter tuning can effectively be performed indirectly by varying the definition of . Providing an ablation study—specifically regarding or the different definitions of —to demonstrate the algorithm's robustness would significantly strengthen the case for its utility.
Weakness 2: In Table 1, it appears that TV regularization could potentially be applied to other algorithms as well; however, no such comparison was conducted. Additionally, it would be beneficial to demonstrate the utility of the proposed method across a broader range of domains, beyond relatively simple experiments like those using the CelebA-HQ or FFHQ datasets.
Questions for authors
-
How did you choose ? I wonder whether is a sensitive hyperparameter.
-
What would happen if you define in other ways? For example, l2-norm-square vs l2-norm
5 comments
Weakness 2a
Indeed, TV regularization could also be added to other methods. Yet, it is important to highlight that this is not as straightforward and natural as in the case of PD-CDS. To underscore this point, we conducted experiments incorporating TV guidance into DPS and RED-Diff on motion and nonlinear deblurring (see table below). Notice that while TV regularization improves the performance of DPS on both tasks, its effect on RED-Diff is less regular. This is to be expected given that these methods are not designed to naturally accommodate multiple constraints. Overall, PD-CDS achieves better metrics on both tasks.
| Task | Method | FID | LPIPS |
|---|---|---|---|
| Motion deblurring | PD-CDS | 56.41 | 0.075 |
| PD-CDS + TV | 46.12 | 0.061 | |
| DPS | 85.67 | 0.169 | |
| DPS + TV | 71.94 | 0.158 | |
| RED-Diff | 63.94 | 0.117 | |
| RED-Diff + TV | 60.04 | 0.109 | |
| Nonlinear deblurring | PD-CDS | 35.38 | 0.150 |
| PD-CDS + TV | 29.65 | 0.107 | |
| DPS | 284.71 | 0.626 | |
| DPS + TV | 276.10 | 0.605 | |
| RED-Diff | 110.64 | 0.339 | |
| RED-Diff + TV | 130.41 | 0.346 |
Weakness 2b
We acknowledge that evaluating PD-CDS in different domains would better showcase its utility, but disagree that the experiments provided are “relatively simple.” Image generation and restoration are standard benchmarks for evaluating diffusion-based (guidance) methods and are used across our baselines. Our experiments also go beyond typical settings by simultaneously enforcing up to four heterogeneous constraints, a significant additional challenge compared to single-constraint tasks. Finally, note that PD-CDS is being applied to two tasks that are often tackled by different algorithms (conditional generation and inverse problems), as evidenced by the different sets of baselines we deployed. Nevertheless, if the reviewer had a specific additional experiment in mind, we would gladly consider it (within the rebuttal time constraints).
I appreciate the authors for the thoughtful response.
The ablation results in Table 10 show that several step sizes give similar performance in one setting. However, I still think this alone does not show that choosing the step size is easy or that the method is generally insensitive to it.
For Weakness 2a, the authors report experiments on motion deblurring and nonlinear deblurring. These are only two of the five settings in Table 1 (which is not updated in the revision), and they are also the settings where PD-CDS already performs best. This makes the comparison feel selective, rather than a fair check of how TV regularization affects the other methods. It would be more convincing to add TV regularization to the other methods in the remaining three settings and update Table 1 with the results.
For Weakness 2b, I thought like DIV2K was a natural choice. Many image restoration works use DIV2K. It would be much more great if the authors could do that cause, if so, they could insist generalizability.
For those reasons (especially concerns about generalizability), I would like to maintain my score.
P.S.) I think the abbreviation for 'assumption' is 'asm,' which looks better than 'ass.'
We thank the reviewer for their insightful comments. We address each of their concerns point by point below. Additionally, we are preparing a revised version of the manuscript including these new experiments and comments (we thought it preferable to give priority to the responses). We will let them know as soon as the revised version is ready.
Question 1/ Weakness 1
The step size η is indeed a hyperparameter of PD-CDS, as we note in the manuscript (line 232), but it remains a scalar hyperparameter regardless of the number of constraints or diffusion steps (e.g., the 4 constraints in Fig. 1). This is a key advantage of PD-CDS: by tuning a single step size, we obtain a sample-specific, dynamic guidance schedule. This is in contrast to conventional guidance methods that require tuning the relative weight of each constraint jointly with their method-specific hyperparameters, a procedure that is challenging even for a moderate number of constraints and that does not necessarily yield better results (see answer to Question 1.1 to reviewer
aS9Q).In practice, we observe that PD-CDS is not particularly sensitive to η. As the reviewer suggested, we conducted an ablation study of η on the Smile + brightness experiments. The results are summarized below (performance is evaluated on 50 images):
These preliminary results indicate that a broad range of η achieves competitive results. In fact, PD-CDS outperforms the baselines for almost all step size values. We observe a similar robustness in the experiments suggested by reviewer
Zaeu. There, we change the number of diffusion steps and obtain similar performance using the same η (see response to Weakness 2). We will include these results in the revised manuscript to showcase the robustness of PD-CDS.Question 2 / Weakness 1
We believe there is a fundamental difference between gi and η. Indeed, the constraint functions gi are part of the problem definition, namely, part of the optimization problem (PI) (line 143). The step size η, on the other hand, is a hyperparameter of Algorithm 1 which targets (PI). Thus, changing gi changes the optimization problem itself, including its feasibility set and geometry, therefore affecting any algorithm used to solve it. In our view, this is fundamentally different from tuning an algorithmic hyperparameter such as η.
That being said, we agree that the choice of gi can also affect the algorithm. As requested, we have therefore conducted new experiments on nonlinear and motion deblurring with different gi. The results are summarized below.
As expected, changing gi changes the problem and therefore the performance of the solution. Nevertheless, all the constraints are largely satisfied (relative violation less than 1), except for nonlinear deblurring. Yet, note that the constraint tolerance is set to 0.01, so that a 5% violation corresponds to an excess of 0.0005 on the original intensity scale. Since the expectation is estimated over 100 images, we cannot rule out that the final distribution is feasible. Interestingly, the ℓ1- and the ℓ2-norm formulations do not consistently outperform one another (ℓ1 is better for motion and ℓ2 for nonlinear deblurring). This is an important advantage of PD-CDS, as it can accommodate a wide variety of gi (and their combinations).