- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This manuscript addresses the challenge of diffusion posterior sampling (DPS) in ill-conditioned inverse problems (such as severe blur and heavy downsampling). The authors propose a "posterior continuation" framework based on two coordinated principles:
- Noise-conditioned frequency exposure: An intermediate posterior likelihood schedule that restricts measurement consistency to low-frequency bands at high diffusion noise levels and progressively expands to full-band consistency as noise decreases.
- Post-refinement Haar-domain detail commitment: A multi-resolution commitment mechanism operating after inner Langevin refinement, which fully accepts coarse-scale corrections while gating high-frequency detail corrections relative to the diffusion predictor using an exposure-linked schedule.
The proposed framework is evaluated across four linear inverse problems ( super-resolution, motion deblurring, Gaussian deblurring, and random/box inpainting) on FFHQ and ImageNet at , with further extensions to real-world blur and latent diffusion (Stable Diffusion v1.5). The method demonstrates significant improvements over competitive baselines, notably outperforming prior art by over PSNR on motion deblurring.
Critical strengths
- Clear Conceptual Intuition:
The conceptual separation between measurement exposure (which frequencies should guide intermediate refinement) and detail commitment (how much refined high-frequency detail should actually be retained relative to the prior) is well-reasoned and addresses a genuine failure mode in existing diffusion inverse solvers. - Compelling Performance:
The empirical results across super-resolution, inpainting, and deblurring are consistently strong. The improvements are particularly striking on severely ill-conditioned tasks, where the proposed sampler beats established baselines by large margins while visibly eliminating the severe ringing and spurious frequency artifacts common in prior methods. - Thorough Ablation Analysis:
The authors do a good job isolating the contributions of each individual design choice. The ablations convincingly demonstrate that frequency continuation in the loss and staged commitment in the wavelet domain are complementary, and that simpler alternatives (like pure frequency re-weighting without spatial wavelet gating) fail to achieve the same stability.
Critical weaknesses
-
Substantial Computational Overhead from Inner Langevin Iterations:
While the diffusion backbone's evaluation cost is kept relatively standard (, via outer steps with 5 PF-ODE steps per iteration), the inner Langevin refinement requires steps at every outer stage. This means the sampler evaluates the measurement gradient —involving the forward operator, its adjoint, and 2D FFT operations— times per image. This results in heavy wall-clock sampling times ( per image on FFHQ and on ImageNet). While the authors show that a low-budget configuration () is feasible (Tables 8 and 9), the reconstruction quality drops noticeably compared to the full setting. -
Theoretical Gap Between Diagonalization Assumptions and Practical Degradations:
The core error-attenuation guarantees in Appendix A.5 (Corollaries A.5 and A.8) strictly require that the forward operator and the spectral projection mask are simultaneously diagonalizable in the Fourier basis. While this holds for circular convolutions/blur, it naturally does not hold for operators like spatial inpainting (where spatial masking mixes Fourier frequencies) or non-periodic boundary conditions. While the authors briefly acknowledge in Section A.3.2 that masking mixes frequencies, the paper currently lacks formal bounds or characterizations of gradient mismatch for operators that do not share the Fourier eigenbasis. Clarifying the theoretical boundaries of the framework would strengthen the presentation. -
Heuristic Dual-Domain Representation (Fourier vs. Wavelet):
The method relies on two different frequency tools: it filters the measurement loss in the Fourier domain (Eq. 4–6), but then switches to Haar wavelets to update image details after refinement (Eq. 11–13). While the authors test a few wavelet types in Table 3, the paper does not clearly explain why it mixes both Fourier and wavelets instead of using one consistent representation. For example, is a wavelet used for the update step because it works locally on image patches and avoids the wavy "ringing" artifacts that Fourier filtering can cause? Adding some plain intuition for this design choice would make the paper much clearer.
Questions for authors
-
Inpainting and Extension to Non-Linear Operators:
As noted in Section A.3.2, inpainting breaks the theoretical assumption of simultaneous Fourier diagonalizability because the spatial mask mixes frequencies. Despite this, your empirical results on random and box inpainting are quite strong. Since the method appears robust to violations of this assumption, did you try applying your framework to non-linear forward operators ? (non-linear blur, etc.) -
Fairness of Budget & Pareto Analysis:
In Table 1, the baseline methods use different sampling parameters (e.g., DAPS uses Langevin steps, whereas your method uses ).- (a) Matched Budget: If DAPS is evaluated with the exact same inner refinement budget ( steps), how much of the performance gap remains?
- (b) Pareto Frontier: To better highlight the efficiency–quality trade-off, would it be possible to provide a Pareto curve (e.g., PSNR vs. Runtime or PSNR vs. Total Sampling Steps) comparing your method against a strong baseline (such as DAPS or DPS) on at least one representative task and dataset (e.g., FFHQ motion deblurring)? This would make the computational comparison much clearer.
-
Implementation in Latent Diffusion (Stable Diffusion v1.5):
In Section A.3.3 and Table 5, you apply the method to Stable Diffusion v1.5 at . Could you briefly clarify how the Fourier residual filtering and Haar commitment are executed? Are they applied directly inside the latent space, or is the intermediate estimate decoded to pixel space at each Langevin step?
6 comments
Official Comment Part 2
For both our method and DAPS, the standard configuration uses five PF-ODE denoiser evaluations per outer stage, corresponding to predictor evaluations; the inner budgets are and , respectively. For the supplementary comparison in Figure 5, DAPS uses , and DPS uses respaced DDPM steps. Our method is evaluated at multiple computational budgets, and each plotted point retains its recorded total sampling NFE rather than being inferred from a shared outer-stage grid. In particular, the 36.91 dB result uses 1200 NFEs, comprising 200 additional network evaluations relative to the 1000-NFE configuration (a 20% increase); it is not an equal-NFE comparison with the 1000-NFE baselines. The 33.96 dB result retains its reported budget of 150 sampling NFEs. The sweep specifies seed 42 and one run per budget. Appendix A.4.3 separately discloses untimed warm-up calls retained in some exported DPS counts; the main-text NFE comparisons use sampling-only counts.
Runtime is batch-amortized sampling cost in GPU-seconds per image, obtained by summing synchronized sampling times across GPU workers and dividing by the total reconstructed-image count. It includes sampling-region work but excludes model loading, observation/initial-noise generation, warm-up, and final metric evaluation. The method-specific batching configurations and the distinction from isolated-image or parallel end-to-end latency are stated in Appendix A.4.3.
Within this benchmark, our method achieves 33.96 dB at 150 sampling NFEs, compared with the best evaluated DAPS result of 31.84 dB at 2000 NFEs and DPS result of 27.31 dB at 1000 NFEs. It reaches 36.70, 36.91, and 36.93 dB at 27.0, 56.0, and 117.0 s/image, respectively. These results support the quality–cost advantage over the evaluated configurations under the disclosed execution conditions, not a universal runtime ordering or a repeated-run significance claim.
3. Theory beyond simultaneous Fourier diagonalization (Weakness 2; Question 1 — inpainting)
The shared-basis assumption supports the frequency-wise corollaries, not execution of the sampler. We have added the following fixed-stage results to Appendix A.6.1 to clarify what remains valid without it. All vector and operator norms are Euclidean and spectral norms.
General linear operators. Fix and the scheduled weights. Let be the real residual-space projector induced by the conjugate-symmetric Fourier mask, including inverse transformation. Define
For the same pair of clean estimates with difference , differentiation gives
Since ,
Equations (61)–(62) provide these identities and bounds. The matrix bound follows from . It applies to non-periodic blur but bounds worst-case sensitivity, not pointwise improvement for every error direction. The score-error form additionally requires .
Binary masks. For zero-filled inpainting, . Because and are contractions,
Proposition A.13, Eqs. (63)–(64), proves non-increase of total mismatch for box or random masks without commutation, but not strict or per-frequency improvement. Also, : weighting does not create measurements of missing pixels.
For fixed and finite-second-moment errors, with and ,
Lemma A.14 and Eqs. (65)–(66) establish this second-moment identity, including conditioning on a random mask. It follows by expanding the quadratic form. includes bias; no isotropy is required. The mask inequality also holds in expectation, including over random masks. These results compare the same estimates, not different trajectories or final PSNR. The empirical inpainting evidence is reported in Table 1 for random masks and Appendix A.3.2, Table 5, and Figure 9 for box masks.
Official Comment Part 3
4. Fourier weighting versus Haar commitment (Weakness 3)
The revised Secs. 3.2–3.3 describe their respective roles, and Appendix A.6.3 adds the correction-magnitude and reconstruction-risk analysis below. Fourier weighting controls residual frequencies on the measurement grid, including the lower-resolution grid for super-resolution. Haar commitment controls image-space corrections in localized coarse/detail coordinates.
For an orthonormal Haar transform with compatible normalization and boundary handling, let
Then , and orthonormality gives
Proposition A.16, Eqs. (78)–(79), formalizes this non-expansiveness result. It limits correction magnitude, not ground-truth error. For fixed pre-fusion distributions with finite second moments, predictor detail error and correction satisfy
Proposition A.17, Eqs. (80)–(81), gives this risk identity and the corresponding fixed-stage oracle gate. The identity requires no independence assumption and explains the balance between suppressing uncertain changes and retaining useful corrections. It does not make Eq. (13) globally optimal.
Haar locality motivates the representation but does not guarantee artifact-free images or prove Haar uniquely optimal. Sec. 4.3, Table 2, and Figure 8 separate the two interventions. Table 3 compares the tested bases and a measurement-reweighting alternative; Appendix A.3.5 and Figure 12 provide qualitative comparisons of the wavelet bases. That alternative is not post-refinement Fourier gating, so we do not claim to exclude all Fourier-only designs.
5. Extension to blind bilinear inverse problems (Question 1 — nonlinear operators)
Did the authors apply the framework to nonlinear forward operators?
Yes: Appendix A.3.3 and Figure 10 report blind-deblurring examples on FFHQ and ImageNet under the BlindDPS setting (Chung et al., 2023a). Both the image and blur kernel are unknown during reconstruction. The ground-truth kernel is used to generate the observation, not supplied to the reconstruction algorithm. The revised text explicitly states the unknown-kernel protocol in Appendix A.3.3 and the Figure 10 caption.
The forward map is . It is linear in either variable separately but nonlinear in the joint variable . In particular,
The cross term distinguishes the joint blind problem from known-kernel linear deblurring. The experiment uses the official BlindDPS motion-blur setting: 64×64 kernels, intensity 0.5, and Gaussian measurement noise in normalized image coordinates. Exposure weights the residual computed with the estimated kernel; commitment controls the image correction after refinement. These operations and the joint nonlinearity are explained in Appendix A.3.3. Figure 10 provides qualitative comparisons with BlindDPS and ground truth.
Connection to the nonlinear analysis (Appendix A.6.4, Proposition A.18, Eqs. (82)–(84)). Appendix A.3.3 already identifies the applicability of this local result to the bilinear map; the following joint-variable calculation makes that specialization explicit. For fixed stage parameters, let be the residual-space orthogonal frequency projector and define
The joint measurement gradient is
Finite-dimensional convolution is bilinear, so its Jacobian is Lipschitz and is bounded on bounded neighborhoods. With a bounded residual on such a convex neighborhood, Proposition A.18 in Appendix A.6.4 applies to and yields
Here, bounds the Jacobian norm, is its Lipschitz constant, and bounds the residual. This is a local sensitivity statement, not a convergence or reconstruction-quality guarantee. Figure 10 supplies the empirical evidence for this blind extension. The fixed-linear Fourier-wise results are not transferred unchanged to the joint problem.
Official Comment Part 1
Thank you for recognizing the design and experimental results. All manuscript references below refer to the revised manuscript.
1. Computational overhead and reduced-budget evidence (Weakness 1)
The default uses 1000 diffusion evaluations and 40,000 inner updates. In pixel space, the predictor is fixed within refinement, so inner steps do not require denoiser calls, but operator/adjoint evaluations and Fourier transforms still incur cost. Haar commitment occurs once per outer stage. Latent refinement additionally differentiates through the decoder. We therefore report runtime rather than equating NFE with total cost.
Appendix A.3.4 and Table 8 report intermediate budgets on FFHQ motion deblurring. At (N,T)=(200,100), our method achieves 36.86/0.945/0.050 in approximately 110 s/image; at (200,50) it achieves 35.81/0.943/0.047 in 70 s/image, versus 36.69/0.940/0.054 in 130 s/image for the default. These are useful operating points beyond the aggressive low-budget settings in Tables 12–13, not a claim of universal Pareto dominance. The revised discussion of computational cost and the supplementary quality–efficiency comparison appear in Sec. 4.2; Appendices A.4.2–A.4.3 distinguish NFE from measurement/refinement costs and specify the separate Figure 5 protocol and timing convention.
2. Matched DAPS budget and quality–runtime comparison (Weakness 1; Question 2(a)–(b))
(a) Matched budget (Question 2(a)). At N=200,T=100, our Table 8 result is 36.86 dB, compared with 29.66 dB for DAPS in Table 1: a 7.20 dB FFHQ motion-deblurring difference at equal update counts. The tables below additionally report DAPS at T=200. All configurations use N=200 and 1000 NFE; T=100/200 corresponds to 20,000/40,000 inner updates. Equal counts do not imply equal runtime. Appendix A.4, “DAPS”, specifies the N=200,T=100 baseline. The additional DAPS T=200 results below are reported in this response; they are not tabulated in the current revised manuscript.
FFHQ:
ImageNet:
At T=200, our method exceeds DAPS on all three metrics across the four tasks on both datasets. Increasing DAPS's inner budget does not consistently improve its results. We retain the T=100 comparison, including its lower Gaussian-deblurring LPIPS, rather than selecting only the less favorable baseline configuration.
The DAPS-at-100 and default-method runtimes come from Table 1, whose caption specifies motion deblurring; they are not separate timing measurements for all four tasks. The supplementary curves below use a jointly re-evaluated benchmark with different hardware, image selection, and timing/batching conventions. Their absolute timings should not be combined with those of Tables 1 and 8.
(b) Quality–runtime comparison (Question 2(b)). Section 4.2 and Figure 5 compare our method with DAPS and DPS. Appendix A.4.2 specifies the shared 100-image FFHQ-256 subset, 61×61 motion-blur kernel with intensity 0.5, and observation noise σy=0.05. Appendix A.4.3 adds the execution budgets, NFE accounting, and timing/batching details for the H200 runs. This supplementary benchmark is separate from Tables 1 and 8.