- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Probability-Flow Distillation (PFD). PFD first moves particles using probability-flow ODE of the current particle distribution, rather than just adding Gaussian noise on it. Then, PFD leverages the differnece between the target score and the particle score for optimization, which follows VSD. Main experiments cover image generation on CelebA (10K samples) and text-to-3D (only 22 samples).
Critical strengths
-
This paper studies an interesting problem. PFD tries to calculate the score differences along a determinstic trajectory, without the randomness of adding a gaussian noise.
-
Experiments are reasonably controled. PFD and VSD are both tested under same time quote and times of score evaluation. The text-to-3D experiment is also conducted under the same prior model, representation, and optimizer. CelebA's experiment covers 4 seeds and 10K samples.
Critical weaknesses
-
My biggest concern is that the performance gains over VSD is very slight. In table 1 the quality metirics are basically the same, and PFD's FID is even worse. In table 2, the ImageReward improves but the CLIP score is also basically the same. And both methods takes 2 hours which is significantly slow. It's also very hard to tell the difference in figure 2 & 3 (except figure 2's right subfigure, which clearly shows the determinstic nature of PFD). I believe that PFD makes the distillation more determinatic, but why neither the quality nor efficiency seems to be benefited from this change.
-
My second-biggest concern is that the 3D generation quality comparison is conducted on only 22 samples, making these experiments less convencing compared to CelebA's experiment on image generation.
-
The paper's "mode-seeking" claim for prior methods is questionable. Methmatically, equation 13 can be considered as optimizing the "mean" across different noise levels, but it's methmatically different from the "mode" of clean distribution . For example, there are many samples around -1 and +1, so there are 2 modes, but the global optimization direction will be pointed to 0, which is clearly not a mode. This issue may make the methmatical logics of PFD less convencing.
Questions for authors
See the weaknesses please.
3 comments
Dear Reviewer ZY2C,
We would be grateful to know whether our response above and the revised manuscript address your concerns. If there are further experiments that could be completed within the remaining discussion period, it would be very helpful to hear about them now, so that we can report the results before it ends.
Gentle Reminder
Dear Reviewer ZY2C,
As the author–reviewer discussion period is closing soon, we would be very grateful if you could share any further concerns you may have, so that we can respond promptly before the deadline.
We thank the reviewer for their time and thoughtful comments, which helped us state PFD's advantage over VSD more precisely here. We address each weakness below.
1. Performance gain over VSD.
PFD and VSD differ only in how xt is computed from x0. They share the same fixed point, so given enough steps the FIDs of both reach similar values, which is why they differ only slightly in Table 1. What the deterministic coupling (x0,xt) buys over the stochastic one is a faster rate of convergence. Theorem 5 proves this for general targets in the small-time regime. Extending it beyond this is difficult for general targets and we leave it to future work, but Prop. 6 does so for isotropic Gaussians, where Fig. 2 also confirms it empirically. To show that this also holds beyond the Gaussian case, we now report the comparison for CelebA generation in the table below, which gives FID every 20 iterations (lower is better). FID is computed on 5k samples against 5k real images, so it is not directly comparable to Table 1. For reference, the FID between two disjoint sets of 5k real images is 5.7. PFD reaches each FID level in fewer iterations than VSD at the same per-step cost. For example, PFD reaches an FID of 46.4 at iteration 120, a level VSD reaches only at about iteration 240, that is, with about half the compute. We have added this result, with a plot of FID against iteration, to Sec. 5.2 of the revised manuscript (Fig. 5).
2. Scale of the text-to-3D evaluation.
We acknowledge that 22 prompts is a relatively small evaluation, and we now state this as a limitation in the conclusion. A single run takes about 2 GPU-hours (1 for SDS), so evaluating four methods on 22 prompts took about 154 GPU-hours. Since only one 3D representation fits in GPU memory at a time, these runs had to be done sequentially, which set the scale of the evaluation within our compute budget.
3. Where SDS converges.
We agree that ''modes'' is imprecise here, and we used the term only to follow prior work, where SDS is often associated with ''mode-seeking'' and ''mode collapse''. Eq. 13 does average over noise levels, but in the reviewer's example the particles do not all go to 0. Those starting near +1 end at a single point on the positive side, slightly pulled towards 0, and those near −1 at its mirror image, so the particle distribution collapses into two spikes with no spread. These spikes lie close to the modes of p0 when the modes are far apart. Whether two modes merge depends on the distance between them relative to the noise level, and since distances typically grow with the dimension, such merging is rare in high dimensions. We refer the reviewer to the 20-D MoG in Fig. 3 (bottom row, panel b), where SDS collapses the particles onto single points close to the modes of p0. To be technically precise, the revision replaces "collapses onto the modes of p0" with "collapses onto point masses near the modes of p0". Either way the spread of p0 is lost, which is the failure of SDS that Sec. 3.1 points out. PFD does not have this problem, since its fixed point is p0 itself.
We hope these clarifications and the new convergence results address the reviewer’s concerns, and we would be glad to discuss any remaining questions.