- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates a computational efficiency bottleneck in solving inverse problems with diffusion models. The proposed PASEO framework efficiently incorporates a learned degradation operator without backpropagating through the denoising network. The method is well motivated. Experimental results achieves perceptual reconstruction quality while reducing runtime and memory overhead.
Critical strengths
The paper addresses a well-motivated computational bottleneck in diffusion-based inverse problems. Existing approaches often face a trade-off between the flexibility of degradation and computational efficiency. The proposed PASEO framework tackle the problem by extending the proxy-based formulation to nonlinear learned operators through local Gaussian approximation and perturb-and-solve sampling.
Critical weaknesses
- The framework builds upon existing approaches such as DING and SILO. Perturb-and-Solve sampling, local linearization, and conjugate gradient optimization are established techniques. The paper needs to further clarify what methodological innovations are introduced beyond a composition of prior arts.
- The paper proves posterior sampling exactness only under the setting of linear operators and exact solvers. However, practical method should rely on nonlinear learned operators, local linear approximations, and finite conjugate gradient iterations. There lacks adequate analysis regarding the extent to which these approximations preserve the properties of the target posterior distribution.
- The approach requires training a latent-space degradation operator separately for each combination of degradation task and generative prior. Moreover, its performance degrades on JPEG restoration, suggesting that the method is sensitive to local approximations from the learned operator. More challenging nonlinear degradation tasks are recommended to verify the method’s applicability and robustness.
Questions for authors
1.Could the authors clarify the methodological advances of PASEO beyond combining DING, SILO,and existing perturb-and-solve techniques? 2. How do local linearization and truncated CG iterations affect posterior sampling accuracy, particularly given that fewer CG iterations sometimes outperform exact solutions in perceptual quality? 3. Could the authors further clarify the fairness of computational comparisons when the numbers of denoiser evaluations differ across methods, and provide matched-budget comparisons?
3 comments
Response (2/2)
Question: effect of linearization and truncated CG
As the reviewer rightly points out, the proofs hold only for a linear operator and an exact solve. It does not say that the full sampler follows the exact posterior.
How large is the linearization error? We ran PASEO with the settings of the paper on 30 FFHQ test images. In each of the 28 sampling steps, the denoiser gives an estimate of the clean latent. PASEO linearizes the operator at , with Jacobian , and the CG correction moves the estimate to . We compare the real change of the operator output along this step with the change that the linear approximation predicts:
Average over all steps and images:
| Prior | SR ×4 | SR ×8 | Gaussian blur | Inpainting | JPEG | Nonlinear blur |
|---|---|---|---|---|---|---|
| SD3.5-medium | 44% | 36% | 27% | 6% | 94% | 36% |
| RV-v5.1 | 38% | 47% | 42% | 25% | 90% | 42% |
So the linear approximation of the operator that PASEO uses mispredicts the change of the operator output by 25% to 47% on most cases. It is best on inpainting with SD3.5 (6%) and almost useless on JPEG (90% and 94%).
Does this distort the samples? To measure this, we generate a latent with the prior, and create a measurement in latent space as , where is the noise level that the sampler assumes. For this measurement the learned operator is exactly the true degradation, so the test isolates the sampler. We draw 20 samples for each . If the samples follow the posterior, behaves like one more sample. Then, on average, the distance between two samples equals the distance between a sample and . We report the ratio of the two. A ratio of 1 is correct, and below 1 means that the samples are too similar to each other.
| Prior | SR ×4 | SR ×8 | Gaussian blur | Inpainting | JPEG | Nonlinear blur |
|---|---|---|---|---|---|---|
| SD3.5-medium | 1.00 | 1.02 | 1.03 | 1.06 | 1.05 | 0.99 |
| RV-v5.1 | 1.01 | 0.98 | 0.98 | 1.06 | 0.98 | 0.97 |
The ratio of the default PASEO is between 0.97 and 1.06 on all twelve cases, also on JPEG. So the linear approximation is not exact, but the samples of PASEO are still as diverse as they should be: neither too similar to each other nor too spread out.
What does truncated CG change? We repeat the same test with different numbers of CG iterations, for both anchors (SD3.5). We also add a blur that is applied directly to the latent. It is exactly linear, so it has no linearization error.
| Anchor | CG iterations | Learned Gaussian blur | Exactly linear blur |
|---|---|---|---|
| PASEO† | 2 | 0.48 | 0.98 |
| PASEO† | 16 | 0.94 | 1.10 |
| PASEO† | 50 | 1.00 | 1.04 |
| PASEO | 2 | 1.03 | 1.05 |
| PASEO | 16 | 1.08 | 1.08 |
| PASEO | 50 | 1.04 | 1.03 |
With the arc anchor of PASEO, the ratio stays close to 1 for any number of iterations. With the original anchor of PASEO†, the version our proof covers, 2 iterations are enough on the exactly linear blur, but on the learned operator the samples are too similar (0.48), and 16 or more iterations are needed. So stopping CG early is safe with the arc anchor, which is our default.
Why can fewer CG iterations look better? We tested this with the blur that is applied directly to the latent, so the linearization plays no role. A blur removes the fine detail of the latent, so the measurement says almost nothing about it. With 50 CG iterations, the sampler still tries to recover this detail from the measurement, and ends up amplifying noise: the samples contain up to 26 times more fine detail than the true latent. With 2 iterations they contain the same amount (1.1 times). This false detail is what makes the images worse. When we replace only the fine detail of the 50-iteration samples by that of the 2-iteration samples, LPIPS goes from 0.88 back to 0.08.
We added these results to Section 4.2 and Appendix F.1.3 of the revised paper (in purple).
Dear, 3FoE, thank you once more for your time!
We tried to resolve the issues you have mentioned within given time frame and we hope you find them valuable. If you think there should be more clarifications, please just let us know, we are eager to do!
Dear Reviewer 3FoE,
We did not want to bother you but author-reviewer discussion closes today, so we just wanted to briefly summarize how we addressed each of your concerns.
-
Novelty beyond DING and SILO. DING has a closed-form update only for inpainting, and SILO back-propagates through the denoiser at every step. PASEO removes both limits: it supports a learned operator using only Jacobian–vector products of a small network, and the denoiser is never differentiated.
-
Effect of linearization and truncated CG. We measured this directly. The sample-diversity ratio of PASEO stays between 0.97 and 1.06 on all twelve task–prior cases (1 is ideal), including JPEG and the new nonlinear deblurring task. And stopping CG provides better results as does not over optimize tiny artifacts that hurt visual metrics.
-
Matched-budget comparison. We reran all SD3.5 baselines at the same 53 NFEs. PASEO is the best on four of the five tasks.
-
As you recommended, we also added a more challenging nonlinear task: we trained new operators for nonlinear deblurring for both priors, and PASEO is the best among the compared methods with both priors.
We hope this answers your questions. If so, we would be grateful if you could reconsider your rating. If any concern remains, please tell us and we will answer right away.
Best regards,
Authors
We thank the reviewer 3FoE for the careful reading, for finding the method well motivated and critically analyzing it. This vastly helps us improve the clarity and rigor of our work. Below we answer each point.
Question: what is new beyond DING and SILO
We agree that each ingredient is known. However, PASEO is not a direct combination of them in inverse problems field. Let us explain why.
DING avoids back-propagation through the denoiser, but its latent update has a closed form only for inpainting, where the operator diagonal matrix - mask. For a full matrix there is no closed form in DING. To support it each step would need a full covariance matrix, which is too large to build or invert. And let alone having a learned operator instead of a mask. SILO on the other hand supports learned operators, but it back-propagates through the denoiser and the operator at every step.
PASEO removes both limits. We write the DING step with a learned operator as a Gaussian inference problem. We then sample it by perturb-and-solve, using only products with the Jacobian of the small operator network. No matrix is built, and the denoiser is never differentiated. We also prove that this sample is exact for a linear operator. As a result, PASEO is the fastest method and uses the least memory in Table 1. Finally, we show that PASEO works with both a diffusion prior (RV-v5.1) and a flow-matching prior (SD3.5-medium). DING is shown only on SD3.5, a flow-matching model, and SILO only on SD-v1.5 and RV-v5.1, which are diffusion models.
Question: matched-budget comparison
To make the comparison matched, we reran the SD3.5 baselines at 53 NFEs on 300 FFHQ test images. We did not run on full validation set (1k images) due to computational constraints and limited time as some baselines run very slowly. The table shows LPIPS (lower is better).
At the same budget PASEO is the best on four of the five tasks. On JPEG, ReSample is better, as it is in Table 1. We added this comparison to Section 4.2 of the revised paper (Table 5, in purple).
Response (1/2)