acceptodds
Under review as a conference paper at ICLR 2027

Perturb-and-Solve: Efficient Learned-Operator Conditioning for Latent Diffusion Inverse Problems

Abstract

Latent diffusion models serve as powerful priors for solving inverse problems in image restoration, such as deblurring, inpainting, and super-resolution. **Current methods have a trade-off between generality and efficiency.** Solvers that are restricted to a fixed set of degradation operators are fast and efficient. Methods that support arbitrary degradation operators are slow and require gradients through the diffusion network. To break this bottleneck, we introduce PASEO (**P**erturb-**A**nd-**S**olve for **E**fficient **O**perator conditioning), a method that uses a small (1M parameters) learned network to degrade diffusion model predictions in latent space. **PASEO supports learned degradation operators without back-propagating through the diffusion network.** We efficiently sample reconstructions from an approximate posterior by combining the diffusion model's prediction with the observed image. We do this by adding noise and solving linear equations based on a local linear approximation of the learned network, without building or inverting large covariance matrices. Across super-resolution, deblurring, and inpainting on FFHQ and COCO, PASEO achieves strong perceptual quality while running up to 19 faster and using up to 34% less peak memory than the tested baselines, with the same or fewer model evaluations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Inverse Problems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3FoEOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates a computational efficiency bottleneck in solving inverse problems with diffusion models. The proposed PASEO framework efficiently incorporates a learned degradation operator without backpropagating through the denoising network. The method is well motivated. Experimental results achieves perceptual reconstruction quality while reducing runtime and memory overhead.

Critical strengths

The paper addresses a well-motivated computational bottleneck in diffusion-based inverse problems. Existing approaches often face a trade-off between the flexibility of degradation and computational efficiency. The proposed PASEO framework tackle the problem by extending the proxy-based formulation to nonlinear learned operators through local Gaussian approximation and perturb-and-solve sampling.

Critical weaknesses
  1. The framework builds upon existing approaches such as DING and SILO. Perturb-and-Solve sampling, local linearization, and conjugate gradient optimization are established techniques. The paper needs to further clarify what methodological innovations are introduced beyond a composition of prior arts.
  2. The paper proves posterior sampling exactness only under the setting of linear operators and exact solvers. However, practical method should rely on nonlinear learned operators, local linear approximations, and finite conjugate gradient iterations. There lacks adequate analysis regarding the extent to which these approximations preserve the properties of the target posterior distribution.
  3. The approach requires training a latent-space degradation operator separately for each combination of degradation task and generative prior. Moreover, its performance degrades on JPEG restoration, suggesting that the method is sensitive to local approximations from the learned operator. More challenging nonlinear degradation tasks are recommended to verify the method’s applicability and robustness.
Questions for authors

1.Could the authors clarify the methodological advances of PASEO beyond combining DING, SILO,and existing perturb-and-solve techniques? 2. How do local linearization and truncated CG iterations affect posterior sampling accuracy, particularly given that fewer CG iterations sometimes outperform exact solutions in perceptual quality? 3. Could the authors further clarify the fairness of computational comparisons when the numbers of denoiser evaluations differ across methods, and provide matched-budget comparisons?

3 comments

We thank the reviewer 3FoE for the careful reading, for finding the method well motivated and critically analyzing it. This vastly helps us improve the clarity and rigor of our work. Below we answer each point.

 

Question: what is new beyond DING and SILO

We agree that each ingredient is known. However, PASEO is not a direct combination of them in inverse problems field. Let us explain why.

DING avoids back-propagation through the denoiser, but its latent update has a closed form only for inpainting, where the operator diagonal matrix - mask. For a full matrix there is no closed form in DING. To support it each step would need a full covariance matrix, which is too large to build or invert. And let alone having a learned operator instead of a mask. SILO on the other hand supports learned operators, but it back-propagates through the denoiser and the operator at every step.

PASEO removes both limits. We write the DING step with a learned operator as a Gaussian inference problem. We then sample it by perturb-and-solve, using only products with the Jacobian of the small operator network. No matrix is built, and the denoiser is never differentiated. We also prove that this sample is exact for a linear operator. As a result, PASEO is the fastest method and uses the least memory in Table 1. Finally, we show that PASEO works with both a diffusion prior (RV-v5.1) and a flow-matching prior (SD3.5-medium). DING is shown only on SD3.5, a flow-matching model, and SILO only on SD-v1.5 and RV-v5.1, which are diffusion models.

 

Question: matched-budget comparison

To make the comparison matched, we reran the SD3.5 baselines at 53 NFEs on 300 FFHQ test images. We did not run on full validation set (1k images) due to computational constraints and limited time as some baselines run very slowly. The table shows LPIPS (lower is better).

TaskPASEOReSampleGML-DPSLDPSPSLD
SR ×40.1800.2230.4020.4540.525
SR ×80.3270.3940.4470.4630.460
Gaussian blur0.1970.2390.4650.4780.457
Inpainting0.1170.2230.5450.5480.502
JPEG q100.3420.2050.4340.447–

At the same budget PASEO is the best on four of the five tasks. On JPEG, ReSample is better, as it is in Table 1. We added this comparison to Section 4.2 of the revised paper (Table 5, in purple).

 

Response (1/2)

Response (2/2)

 

Question: effect of linearization and truncated CG

As the reviewer rightly points out, the proofs hold only for a linear operator and an exact solve. It does not say that the full sampler follows the exact posterior.

 

How large is the linearization error? We ran PASEO with the settings of the paper on 30 FFHQ test images. In each of the 28 sampling steps, the denoiser gives an estimate z\mathbf{z} of the clean latent. PASEO linearizes the operator at z\mathbf{z}, with Jacobian J\mathbf{J}, and the CG correction moves the estimate to z′\mathbf{z}'. We compare the real change of the operator output along this step with the change that the linear approximation predicts:

error=∥Hψ(z′)−Hψ(z)−J(z′−z)∥∥J(z′−z)∥\text{error} = \frac{\lVert \mathbf{H}_\psi(\mathbf{z}') - \mathbf{H}_\psi(\mathbf{z}) - \mathbf{J}(\mathbf{z}' - \mathbf{z})\rVert}{\lVert \mathbf{J}(\mathbf{z}' - \mathbf{z})\rVert}

Average over all steps and images:

PriorSR ×4SR ×8Gaussian blurInpaintingJPEGNonlinear blur
SD3.5-medium44%36%27%6%94%36%
RV-v5.138%47%42%25%90%42%

So the linear approximation of the operator that PASEO uses mispredicts the change of the operator output by 25% to 47% on most cases. It is best on inpainting with SD3.5 (6%) and almost useless on JPEG (90% and 94%).

 

Does this distort the samples? To measure this, we generate a latent z⋆\mathbf{z}^\star with the prior, and create a measurement in latent space as y=Hψ(z⋆)+r ξ\mathbf{y} = \mathbf{H}_\psi(\mathbf{z}^\star) + r\,\xi, where rr is the noise level that the sampler assumes. For this measurement the learned operator is exactly the true degradation, so the test isolates the sampler. We draw 20 samples for each y\mathbf{y}. If the samples follow the posterior, z⋆\mathbf{z}^\star behaves like one more sample. Then, on average, the distance between two samples equals the distance between a sample and z⋆\mathbf{z}^\star. We report the ratio of the two. A ratio of 1 is correct, and below 1 means that the samples are too similar to each other.

PriorSR ×4SR ×8Gaussian blurInpaintingJPEGNonlinear blur
SD3.5-medium1.001.021.031.061.050.99
RV-v5.11.010.980.981.060.980.97

The ratio of the default PASEO is between 0.97 and 1.06 on all twelve cases, also on JPEG. So the linear approximation is not exact, but the samples of PASEO are still as diverse as they should be: neither too similar to each other nor too spread out.

 

What does truncated CG change? We repeat the same test with different numbers of CG iterations, for both anchors (SD3.5). We also add a blur that is applied directly to the latent. It is exactly linear, so it has no linearization error.

AnchorCG iterationsLearned Gaussian blurExactly linear blur
PASEO†20.480.98
PASEO†160.941.10
PASEO†501.001.04
PASEO21.031.05
PASEO161.081.08
PASEO501.041.03

With the arc anchor of PASEO, the ratio stays close to 1 for any number of iterations. With the original anchor of PASEO†, the version our proof covers, 2 iterations are enough on the exactly linear blur, but on the learned operator the samples are too similar (0.48), and 16 or more iterations are needed. So stopping CG early is safe with the arc anchor, which is our default.

 

Why can fewer CG iterations look better? We tested this with the blur that is applied directly to the latent, so the linearization plays no role. A blur removes the fine detail of the latent, so the measurement says almost nothing about it. With 50 CG iterations, the sampler still tries to recover this detail from the measurement, and ends up amplifying noise: the samples contain up to 26 times more fine detail than the true latent. With 2 iterations they contain the same amount (1.1 times). This false detail is what makes the images worse. When we replace only the fine detail of the 50-iteration samples by that of the 2-iteration samples, LPIPS goes from 0.88 back to 0.08.

We added these results to Section 4.2 and Appendix F.1.3 of the revised paper (in purple).

 

 

Dear, 3FoE, thank you once more for your time!

We tried to resolve the issues you have mentioned within given time frame and we hope you find them valuable. If you think there should be more clarifications, please just let us know, we are eager to do!

Dear Reviewer 3FoE,

We did not want to bother you but author-reviewer discussion closes today, so we just wanted to briefly summarize how we addressed each of your concerns.

  1. Novelty beyond DING and SILO. DING has a closed-form update only for inpainting, and SILO back-propagates through the denoiser at every step. PASEO removes both limits: it supports a learned operator using only Jacobian–vector products of a small network, and the denoiser is never differentiated.

  2. Effect of linearization and truncated CG. We measured this directly. The sample-diversity ratio of PASEO stays between 0.97 and 1.06 on all twelve task–prior cases (1 is ideal), including JPEG and the new nonlinear deblurring task. And stopping CG provides better results as does not over optimize tiny artifacts that hurt visual metrics.

  3. Matched-budget comparison. We reran all SD3.5 baselines at the same 53 NFEs. PASEO is the best on four of the five tasks.

  4. As you recommended, we also added a more challenging nonlinear task: we trained new operators for nonlinear deblurring for both priors, and PASEO is the best among the compared methods with both priors.

We hope this answers your questions. If so, we would be grateful if you could reconsider your rating. If any concern remains, please tell us and we will answer right away.

Best regards,

Authors

Reviewer_xw72OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper introduces a method for solving inverse problems (e.g., super-resolution, deblurring, inpainting) with latent diffusion priors that do not require backpropagating through a denoiser. Typically, given a forward model y=A(x)+ny = \mathcal{A}(x) + n, the sampling process requires computing the gradient of the likelihood log⁡p(y∣xt)\log p(y \mid x_t) (and subsequently the gradient of the denoiser). The proposed method combines two prior methods: a learned operator Hψ≈E∘A∘DH_\psi \approx \mathcal{E} \circ \mathcal{A} \circ \mathcal{D} models image degradations and a frozen noise prediction removes the need for denoiser gradients. Only HψH_\psi is differentiated, which enables backpropagation-free sampling.

Critical strengths
  • The proposed idea has clear computational advantages and reasonable trade-offs; i.e., the fixed noise prediction x^1′\hat{x}_1^\prime avoids computing denoiser gradients at the cost of approximation errors for non-linear operators.
  • Experimental results support the claimed runtime and memory advantages of the approach and demonstrate competitive performance against the tested baselines and image quality metrics.
Critical weaknesses
  • The paper is generally difficult to follow. Heavy use of color-coding and boldface is quite distracting and the proposed approach is not presented clearly (e.g., ideas / definitions which are core to the main method are tangled into the background / related works text in Sec. 2).
Questions for authors
  • I'd suggest testing additional non-linear operators (other than JPEG) to provide a sense of generality regarding the reported performance of the method (i.e., how well does the approximate posterior perform on other non-linear operators)?
  • Sections describing the main method (and related works) should be presented more clearly
3 comments

We thank the reviewer xw72 for the careful reading, and for recognizing the runtime and memory advantages of PASEO. We are sorry that the paper was difficult to follow. Your comments on the presentation are very helpful to us.

 

More nonlinear operators

We agree, and we have added nonlinear deblurring. To make the response clearer, could you please refer to the "Nonlinear blur" row in the Table 1 of our response to Reviewer aPvp (the response before)

 

Presentation

Thank you for this honest comment. We agree. We tried to keep the text technical and show how the method is built, and at the same time accessible. Since PASEO depends on two prior works DINO and SILO some core definitions ended up in Section 2 (related works). We thought colors and boldface would make it easier to follow the text. We have already reduced the color-coding in the revised paper to two variables. We are afraid a proper rewrite needs more time than the rebuttal period allows. In the final version we will rewrite the method section and move the core definitions out of Section 2.

I thank the authors' for their response, and, after reading through their comments, I am inclined to maintain my current rating.

Dear XW72 reviewer,

Thank you for your quick response!

We appologize for our a short response, the intention only was to keep it clean and clear, not to dismiss the requests. We hope we at least parcially have resolved the issues you have mentioned. If you think there is anything else that needs to be fixed within the time left, we are eger to do. Please, just let us know. Thank you once more for the time you have devoted to improve this paper.

Reviewer_aPvpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors tackle posterior sampling for inverse problems in diffusion models. The proposed method combines ideas from prior related work by using a noisy proxy state to avoid backpropagating through the diffusion model while only needing to train a degradation operator. Experiments are demonstrated on FFHQ and COCO for tasks like super-resolution and inpainting.

Critical strengths
  1. The idea of generating posterior samples by avoiding backprop through the diffusion model and only through a learned degradation operator is quite appealing. Classical methods like DPS require Jacobian computations at each diffusion step which is quite expensive.

  2. Experiments are demonstrated on large-scale models like SD3.5 which speaks to the scalability of the method.

Critical weaknesses
  1. Lack of experimental results: I have the following concerns:

-- The inverse problem space is crowded, and I think some strong baselines are missing from this paper. For instance, DiffPIR [1], NDTM [2], and DAPS [3] are strong baselines that are omitted. DAPS is compared with in Table 25 in the Appendix, but its results look off to me. For instance, it achieves better performance on SRx8 compared to SRx4. However, the former is a more difficult task than the latter.

-- Can the authors demonstrate results on some non-linear inverse problems, such as non-linear deblurring?

  1. On the degradation operator:

-- The method requires training a degradation operator for each corruption process. What is the wall-clock time to train these degradation operators, and how does it compare to fine-tuning diffusion models [4,5]?

-- In this case, the point of comparison should rather be other diffusion fine-tuning methods I mentioned and not just test-time scaling methods. Can the authors elaborate more on this?

-- Is it possible to learn the joint degradation operator for multiple degradations at once?

-- Can the authors comment on the generalizability of these degradations to other rewards except inverse problems? I know the paper specifically tackles inverse problems but this aspect of PASEO would be good to discuss in the paper.

References:

[1] Denoising Diffusion Models for Plug-and-Play Image Restoration, Zhu et al.

[2] Variational Control for Guidance in Diffusion Models, Pandey et al.

[3] Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing, Zhang et al.

[4] Hierarchical Variational Policies for Reward-Guided Diffusion, Pandey et al.

[5] DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised h-transform, Denker et al.

Questions for authors

See weaknesses

5 comments

We thank the reviewer aPvp for the time and the careful reading. The suggested corrections were very helpful and gave us a clear direction to improve the paper. Below we answer each point.

 

Weakness: missing baselines

We agree. We have added DiffPIR and NDTM on FFHQ with both priors, on the five tasks of Table 1.

On nonlinear tasks: in the original submission we followed the validation protocol of SILO, where JPEG is the only nonlinear one, so we did not think of adding more. As requested, we now added nonlinear deblurring and tested all three methods on it.

Evaluations on 1k images are expensive, so within the rebuttal period we prioritized FFHQ and this one nonlinear task. We are afraid that results on COCO and on further nonlinear tasks, such as phase retrieval and HDR, will not be ready in time. We will include them in the final version of the paper.

Table 1: Metric LPIPs ↓\downarrow

RV-v5.1SD3.5-m
TaskPASEODiffPIRNDTMPASEODiffPIRNDTM
SR ×40.1390.2450.2250.1820.2200.217
SR ×80.2120.2500.3440.3300.2820.354
Gaussian blur0.1620.2400.2550.2000.3090.267
Inpainting0.1440.2680.3430.1180.2180.353
JPEG q100.2580.2090.1600.3500.2360.149
Nonlinear blur0.1650.2420.1850.1300.2790.223
Avg. time (s/image)172399011219125
Avg. memory (GiB)3.65.77.216.918.922.7

PASEO has the best LPIPS in 9 of 12 cells. It is also the fastest and uses the least memory in all 12. To be fair, the baselines are better on JPEG with both priors, and DiffPIR is better on SR ×8 with SD3.5. The full results are in Table 1 of the revised paper and in Appendix G.6 and G.7, marked in purple.

 

Weakness: DAPS on SR ×4 and SR ×8

The reviewer is right, and we are grateful for the catch. This was an unintentional tuning error on our side. After the fix, DAPS reaches LPIPS 0.186 on SR ×4 (it was 0.681). As expected, this is now better than its 0.362 on SR ×8. We also reran SR ×8 with the corrected setup and found no issue there: its LPIPS changes only from 0.368 to 0.362. PASEO scores 0.208 and 0.344 on the same tasks, and it is 19× and 25× faster. We have updated Table 33.

 

Question: joint operator

Thank you for pointing out the joint operator idea. We have tested it, but decided not to include it in the paper, since its quality was worse on most tasks than training each operator separately. We have now added it in purple: Section 4.2 and Table 4. FFHQ with RV-v5.1.

Task (PSNR)Per-taskJoint
SR ×427.9825.84
SR ×826.7425.72
Gaussian blur27.6826.88
Inpainting21.7621.01
JPEG q1024.6024.63

 

Question: training time against fine-tuning

We let our operator train about 25 hours on one GPU. Fine-tuning with DEFT [5] takes 9.5 hours (using default training setting) with RV and 11 hours with SD3.5 on FFHQ. So full training of our operator is about 2.5× longer.

To make comparison we tested a checkpoint of our operator after 8 hours, which is less than the DEFT training time. The comparison is on nonlinear deblurring, in LPIPS (lower is better).

MethodTraining timeRV-v5.1SD3.5-medium
DEFT9.5 h / 11 h0.3660.240
PASEO, early checkpoint8 h0.1970.144
PASEO, full training25 h0.1630.129

With less training time, PASEO is already better than DEFT with both priors. DEFT also needs more memory at sampling: 5.7 GiB against 4.0 GiB with RV, and 18.9 GiB against 17.2 GiB with SD3.5.

On the implementation. DEFT is a pixel-space method, and its official code does not work with latent diffusion models. So we implemented it ourselves for both latent priors. We kept the training recipe and the 100 sampling steps of the DEFT paper. Its data-consistency term is computed in pixel space and passed back to the latent through the VAE decoder.

We could not add the second suggested method [4], since it has no public code.

 

Question: rewards other than inverse problems

PASEO is not limited to degradations. The sampler only needs a differentiable network that maps the latent to some output, and a target for that output. In this paper the network models a degradation and the target is the measurement. The same sampler could be used with other networks, for example a depth or edge estimator, to generate images that follow a given depth map or sketch. Rewards that are a single score to maximise, such as an aesthetic score, do not fit this form directly, and we leave them to future work. We have not run these experiments; we added this discussion to the paper (Section 5, in purple).

Dear aPvp Reviewer,

Thank you once more for your time and effort you have put to review this paper!

We tried to resolve the problmes you have mentioned within given time frame. We hope you find them valuable. If you think there should be any more additional clarifications provided, please just let us know, we are eager to do!

Dear Reviewer aPvp,

We did not want to bother you but author-reviewer discussion closes today, so we just wanted to briefly summarize how we addressed each of your concerns.

  1. Missing baselines. We added DiffPIR and NDTM on FFHQ with both priors. PASEO has the best LPIPS in 9 of 12 cells, and is the fastest (11–17 s per image against 90–239 s) with the lowest memory in all 12.

  2. DAPS results. You were right. It was a tuning error on our side. After the fix, DAPS reaches 0.186 LPIPS on SR ×4 (was 0.681), and the SR ×4 / SR ×8 ordering is now as expected.

  3. Nonlinear problems. We trained new latent operators for nonlinear deblurring, one for each prior (about 25 GPU-hours each), and evaluated all methods on it. PASEO is the best with both priors (0.165 and 0.130 LPIPS).

  4. Comparison with fine-tuning. DEFT's official code does not support latent diffusion, so we re-implemented it and trained it ourselves for both priors (9.5 h and 11 h). With less training time (8 h), PASEO reaches 0.197 / 0.144 LPIPS against 0.366 / 0.240.

  5. Joint operator and other rewards. We trained a new single operator jointly on all five tasks (Table 4) and added a discussion of other rewards (Section 5).

We hope this resolves your main concerns. If so, we would be grateful if you could reconsider your rating. If any concern remains, please tell us and we will answer right away.

Best regards,

Authors

Comments on the paper

1 comment

Dear reviewers,

We thank you reviewers for your time, effort and for careful reading of the paper, we appreciate it very much. Your comments helped us improve our paper. We tried to answer every point, with new experiments where possible. Here is a short summary of what we added; the details are in the individual responses.

 

  • New baselines: DiffPIR and NDTM on FFHQ, with both priors.
  • New nonlinear task: nonlinear deblurring, as all three reviewers asked.
  • DAPS: we fixed a tuning error on our side and updated its results.
  • Training cost: a comparison with diffusion fine-tuning (DEFT).
  • Joint operator: one operator trained for all five tasks.
  • Linearization and truncated CG: we measured how much they affect the samples.
  • Matched budget: all SD3.5 baselines rerun at the same number of NFEs as PASEO.
  • Presentation: less color-coding now; a rewrite of the method section in the final version.

 

Across the new comparisons, PASEO keeps the best LPIPS on most tasks and remains the fastest method with the lowest memory use.

Results on COCO for the new baselines, and further nonlinear tasks, we are afraid will not be ready within the rebuttal period due to limited computational resources, however we are going to include them in the final version.

All changes in the revised paper are marked in purple.

If there anything else to clarify, show or to explain, we are more than happy to do so. We hope, we have resolved problems you have mentioned.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.