- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose to improve the inpainting performance of one-step diffusion models using a multi-stage approach: first, the reconstruction loss is applied to different parts of output (by learning to reconstruct the noise , the clean latent code , and the pixel-space image ). Second, the inpainting model output is further refined with a residual U-Net. Finally, multiple realisations of the inpainting are averaged to produce the final output.
Critical strengths
- The method is easy to understand, and the main figure is clear.
- The tackled problem is interesting, as the coherence between the inpainted region and the original region has been an important topic, and generating in one step makes inpainting models usable in real editing tools.
Critical weaknesses
- The experiments need to be deepened. Multiple one-step diffusion model baselines are cited in the related works, but the only diffusion-based baseline is StrDiffusion, and it is not single-step if I am not mistaken. The authors introduce multiple one-step/few-step diffusion models without comparing against them (starting at line 62). Also, the authors never compare the speed of their model against any other model.
- The authors state in the paper that the inpainting is one-step, but the method relies on multiple inferences to produce an average (Figure 1 C). This does not really make the model one-step if multiple inferences are required.
- The justification for the independent-sample inference is weak (Section 2.5, line 295). Fluctuations are necessary to have diversity in the outputs, which is important for an inpainting task.
- The Generator-only (without Residual U-Net) performance (Table 4) is on par with GraphFill (25.80 dB vs. 25.73 dB), making the claim about reconstruction-space adaptation weak.
- I would like the authors to describe the different terms of Equation (13) in more detail. Aren't the first and second terms measuring the exact same error, but weighted differently (taking into account the mask size in the second term)? What is the contribution of the first term in this case? Could they just be merged? Either way, the authors need to describe each term and its relative utility.
- I would like to know how the K sample averaging impacts the diversity of the inpainting. From the paragraph at line 295, it seems that the averaging might get rid of part of the diversity.
- The error needs to be detailed for the different types of masks. Results for different mask shapes, such as those in RePaint [a], need to be reported.
- In Table 5, the authors detail inpainting metrics for different K (= NFE); they also need to compare the performance for K diffusion steps.
Overall, I think that the manuscript needs further work before being submitted again. [a] RePaint: Inpainting using Denoising Diffusion Probabilistic Models
Questions for authors
- Is the initial assumption well motivated? While at t=999 the velocity prediction has a lower theoretical reconstruction-error amplification than the noise prediction, it should also be taken into account that the noise may be easier to predict at this same t=999: since the noisy input is mostly dominated by the noise , the noise will be easier to predict than the velocity. This might make the motivation overstated.