acceptodds
Under review as a conference paper at ICLR 2027

DiffOps: Efficiently Solving Diffusion Inverse Problems with Operator-Aware Optimization

Abstract

Diffusion models provide expressive priors that can improve reconstruction from incomplete or noisy measurements for ill-posed inverse problems. However, existing methods often apply a uniform measurement-correction procedure across diverse inverse problems, despite substantial differences in measurement models and numerical requirements. To address this limitation, we introduce DiffOps, a flexible framework that integrates operator-aware optimization into diffusion-based reconstruction. In this framework, diffusion first provides learned prior information to guide reconstruction where measurements alone cannot fully determine the signal, after which numerical corrections exploit the available operator structure to enforce measurement consistency. As a result, classical operator-selection principles can be integrated directly within diffusion iterations according to the numerical demands of each specific problem. Through experiments on both image restoration and scientific imaging tasks, we demonstrate that DiffOps achieves competitive or superior reconstruction quality and robustness while requiring fewer diffusion-network evaluations.

Primary area:
optimization
Keywords:
diffusion models, inverse problems, operator-aware optimization, numerical solvers, computational imaging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TX1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

DiffOps combines pretrained diffusion priors with numerical solvers selected according to the measurement operator and noise model for image restoration and scientific imaging. At each noise level, it generates a clean-image reference, holds it fixed during measurement correction initialized from that reference, and then adds noise for the next level. The framework supports spectral solvers, conjugate gradients, Gauss–Newton, ensemble methods, and Poisson EM, with staged DR→GD correction for phase retrieval. Experiments on FFHQ and four scientific imaging tasks demonstrate its applicability and practical value.

Critical strengths
  1. The framework addresses the limitations of uniform measurement correction by selecting solvers based on operator structure and noise characteristics, making effective use of established numerical methods.
  2. The paper covers diverse inverse problems and supports the rationale for operator-aware adaptation through mechanism analyses, baseline comparisons, and solver ablations.
  3. The supplementary material includes parameter settings and implementation code, facilitating reproduction and further extension of the framework.
Critical weaknesses
  1. The presentation and layout require further refinement. (1) Sections 2 and 3 could distinguish existing methods, approximation assumptions, and new designs more clearly. (2) Some configurations are inconsistent: raw MRI uses CG×2 in Table S7 but CG×20 inner iterations in Table S13. (3) ''Principled probabilistic imaging using diffusion models as plug-and-play priors'' is cited twice in the references. (4) On page 12, the DDNM first author’s name is misspelled as "Yinhui Wang" instead of "Yinhuai Wang." (5) Some figures and tables also have excessive surrounding whitespace and insufficient explanatory detail.
  2. The practical gains from solver selection have not been sufficiently isolated. Most deterministic solvers achieve similar reconstruction quality on linear FFHQ tasks. A comparison against a single-solver baseline, such as gradient descent across all tasks, using the same diffusion prior, sampling schedule, tuning budget, and hardware would better establish the independent quality or efficiency gains from operator-aware adaptation.
  3. Some image panels lack visual polish, limiting the overall aesthetic quality of the presentation.
  4. The methodological novelty of the proposed approach needs to be further clarified. Its main contribution lies in integrating established solvers into diffusion reconstruction based on operator characteristics, with task-specific solver selection and parameter tuning.
Questions for authors
  1. How is the aggregate quality score in Figure 3 defined? PSNR and SSIM have different preference directions from LPIPS and FID, and their numerical scales differ. Directly averaging them and subtracting a standard deviation therefore does not ensure that higher values indicate better reconstruction. Please specify the metrics, direction adjustments, normalization, and standard-deviation definition for each task.
  2. For new inverse problems, solver selection and parameter configuration still rely on users’ expertise in numerical optimization, operator structure, and noise models, raising the barrier to use and limiting transferability to new tasks. Could the authors provide explicit solver-selection criteria and parameter-setting guidelines based on operator structure, available derivative information, and noise models?
  3. The applicability of the framework to latent diffusion models remains unverified. Unlike the latent extension demonstrated by DAPS [a], incorporating a decoder into DiffOps generally makes the composite operator nonlinear, even when the original measurement operator is linear. How should solver selection be adapted, and can the original quality and efficiency advantages be retained?

[a] Improving diffusion inverse problem solving with decoupled noise annealing. CVPR 2025.

4 comments

We thank the reviewer for the careful reading and the concrete presentation comments. Please also see the general response.

W1 (presentation). All items are fixed in the revision:

  • Table S13 now lists CG×2 for raw MRI (Supp. L939); Tables S7/S8 (Supp. L687, L714) and the released preset use 2.

  • The duplicate Wu et al. entry is merged (L645–650).

  • "Yinhuai Wang" is corrected (L638).

  • Figures now span the full line width with less surrounding whitespace, captions give more detail, and the image panels are polished.

The paper states what DiffOps shares with prior work and where it differs (L116–117, L126–127, L194–196). In short:

  • Shared form. The correction objective has the same decoupled form, with an anchor at the denoiser output, as in DAPS, DiffPIR and related methods.

  • Different role. In those methods the anchor approximates the diffusion prior and keeps the correction consistent with the diffusion trajectory. In DiffOps it is a geometric local constraint around the denoiser output (L194–196), which turns each noise level into a standalone classical problem. That is why classical solver selection applies.

  • What follows. The diffusion signal-to-noise ratio enters only through λt=β/σt2\lambda_t=\beta/\sigma_t^2, which sets the weight, step size and conditioning of whichever solver is used. Solvers can therefore be combined across noise levels to match the stages of the diffusion, e.g., DR at high noise and GD at low noise for phase retrieval.

W2 (single-solver control). The paper contains this control for all seven FFHQ operators (Table S4, Supp. L541; Table S5, Supp. L559; Table 2, L408–419), with the same prior, schedule and hardware. On linear operators the solvers tie, as the analysis predicts for a strictly convex quadratic. On HDR, nonlinear blur and phase retrieval, GD loses by 1.6 dB, 2.2 dB and 22 points of success.

The same control on the scientific tasks, with A10 runtimes:

TaskOperator-matched solverGD
Black hole, 10% coverageL-BFGS: 23.90 dB, χcp2\chi^2_{\rm cp} 1.25, 2.9 s23.07 dB, χcp2\chi^2_{\rm cp} 2.60, 4.0 s
Simulated MRICG: 32.86 dB32.52 dB
Inverse scatteringCG×5: 34.78 dB in 4.3 sGD×10: 34.63 dB in 6.0 s

Raw MRI ties at two iterations, where correction depth, not the solver, is the limit.

W4 (novelty). Our core contribution is the framework outlined in W1: the geometric anchor reduces each noise level to a standalone classical subproblem under a fully decoupled formulation, allowing the correction mechanism to be tailored to the operator structure and adapted across noise levels. W2 demonstrates the resulting gains: our matched solver outperforms baselines when the forward operator violates default correction assumptions, matches them otherwise, and improves overall sampling efficiency. Whereas existing methods such as DDS, ReSample, and DPS adhere strictly to the prior manifold via a single, rigid correction mechanism, DiffOps readily accommodates operator classes none of them evaluate—including zero-Jacobian clipping, network blur, closure quantities, exact Poisson likelihoods, and staged non-convex corrections. Furthermore, our formulation naturally extends to physics-based forward models and neural operators by treating data correction as a general non-linear inverse problem.

Q1 (Figure 3). Each panel's vertical axis is the mean normalized reconstruction-quality score: an equal-weight average of that task's quality metrics, without standard-deviation subtraction. Each metric is first direction-corrected so that higher is better (LPIPS, FID and relative L2L_2 error are reversed), then min–max normalized across the methods in that panel. Heights are comparable only within a panel. Metrics: FFHQ PSNR, SSIM, LPIPS, FID; MRI and microscopy PSNR, SSIM; inverse scattering PSNR, SSIM, relative L2L_2; black hole shift-aligned and 20 µas beam-blurred PSNR. The revision states this definition in the main text. The original text mentioned one standard deviation because we initially intended to include a standard-deviation indication in the visualization. However, the final figure displays the mean scores, and the corresponding text was inadvertently left unchanged. We will remove the outdated reference to one standard deviation and explicitly state the correct definition in the Figure 3 caption (L340).

Q2 (guidelines). The selection rule is stated in §2 (L137–160) and §3.2 (L235–250), with the solver families in Table S1 (Supp. L068) and the per-task choices in Table S13 (Supp. L929); in short:

Operator structureSolver
Linear, FFT-diagonalizable normal operatorWiener (closed form)
Linear, matrix-free A\mathcal{A} and A∗\mathcal{A}^*CG
Smooth nonlinear, usable JacobianGauss–Newton or L-BFGS
Zero or unavailable JacobianEKI
Poisson counts, nonnegative operatorEM
Nonconvex magnitude dataDR, then GD

In the submitted minimal-tuning configuration, β\beta is 0 for noiseless data and otherwise shared per noise model, with β=2\beta=2 across Gaussian FFHQ operators; inner iterations are fixed per solver family (Table S13). Sharing β\beta is a default choice motivated by relative insensitivity to moderate variations, not a restriction. The additional operator-specific selection of β\beta and the solver on 100 validation images assesses sensitivity and potential gains from more extensive tuning; its solver choices agree with the table: CG for SR and inpainting and Wiener for motion blur.

Q3 (latent priors). All experiments use pixel-space priors as in the FFHQ and InverseBench protocols for all baselines. With a latent prior the composite operator A∘D\mathcal{A}\circ\mathcal{D} is nonlinear even for linear A\mathcal{A}. That places it in DiffOps's nonlinear branch: Gauss–Newton or L-BFGS with the decoder Jacobian, or EKI if the decoder is treated as a black box. Alternatively, the correction can run as in pixel space and be re-encoded and decoded, as in ReSample. However, we also additionally validate DiffOps-Latent (GN) under the same evaluation protocol and operators used in our other image-restoration tasks on FFHQ256, 100 validation images. At each noise level, we approximately solve

min⁡zft(z),ft(z)=∥y−A(D(z))∥222σy2+λt2∥z−z^0∥22,λt=βσt2.\min_z f_t(z),\qquad f_t(z)=\frac{\|y-\mathcal{A}(\mathcal{D}(z))\|_2^2}{2\sigma_y^2}+\frac{\lambda_t}{2}\|z-\hat z_0\|_2^2,\qquad \lambda_t=\frac{\beta}{\sigma_t^2}.

We use Levenberg–Marquardt initialized at the denoised latent estimate z^0\hat z_0. Each step uses 20 conjugate-gradient iterations to approximately solve

(J⊤Jσy2+(μ+λt)I)δ=−∇zft(z),J=∂(A∘D)∂z,\left(\frac{J^\top J}{\sigma_y^2}+(\mu+\lambda_t)I\right)\delta=-\nabla_z f_t(z),\qquad J=\frac{\partial(\mathcal{A}\circ\mathcal{D})}{\partial z},

where μ\mu is the Levenberg–Marquardt damping parameter. We compare 50 and 100 NFE with LatentDAPS at 1K/4K NFE. Entries are PSNR↑ / SSIM↑ / LPIPS↓; the best value for each metric within each operator is bold.

OperatorLatentDAPS (1K/4K NFE)L-GN, 50 NFEL-GN, 100 NFE
Gaussian blur27.93 / 0.764 / 0.23428.98 / 0.821 / 0.20029.09 / 0.825 / 0.195
4× SR27.48 / 0.801 / 0.18229.10 / 0.832 / 0.18529.17 / 0.834 / 0.179
Motion blur27.00 / 0.814 / 0.28328.98 / 0.820 / 0.19429.12 / 0.828 / 0.188
Box inpainting23.99 / 0.802 / 0.19424.06 / 0.823 / 0.17224.53 / 0.841 / 0.147
Nonlinear blur28.11 / 0.713 / 0.23527.87 / 0.810 / 0.20728.85 / 0.830 / 0.185
HDR25.94 / 0.751 / 0.22324.82 / 0.831 / 0.19125.22 / 0.835 / 0.187

At 100 NFE, DiffOps-Latent improves SSIM and LPIPS mostly. These results support DiffOps's applicability to latent diffusion priors and show that its correction framework can improve latent-space reconstruction with substantially fewer diffusion-network evaluations.

We add this to the conclusion as future validation, but comprehensively evaluating latent priors is beyond this submission.

I thank the authors for their detailed responses and for addressing some of my concerns. However, the manuscript still requires further improvement, and I am therefore inclined to maintain my current score at this stage.

Thank you for reviewing our responses; we have further improved our manuscript and uploaded the revised PDF.

Reviewer_MBZPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The authors propose DiffOps, a decoupled diffusion inverse solver where, at each noise level, the denoiser estimate anchors a Tikhonov subproblem which is solved for a few iterations using a classical solver, chosen per task prior to renoising. The authors claim that matching the inner solver to the operator gives competitive or better reconstructions at fewer NFEs. DiffOps is evaluated on FFHQ with seen operators, as well as five different scientific tasks. There, we see DiffOps perform competitively against contemporary competitors.

Critical strengths

S1) Evaluation is robust, with many operators and noise models.

S2) DR->GD staging for phase retrieval is a clear improvement over compared baselines.

S3) Overall the paper is well-written, and it is easy to follow.

Critical weaknesses

W1) The method is anticipated by work which is not discussed. The subproblem Eqs. 7/8 is the DAPS/DAPS++/DCDP/DiffPIR correction step, and DDfire [1] already solves exactly the subproblem by SVD when available and CG otherwise inside a few-NFE DDIM loop. DDfire also has an EP extension to phase retrieval, and it has been applied to multi-coil MRI [2]. Similarly, DDS [3] is another uncited work which has been applied to MRI and solves the subproblem via CG. These methods should be cited, discussed, and compared to.

W2) The results do not support the competitive or superior claim. On FFHQ, DiffOps is best on 2 of 6 operators. Similarly, on the five scientific tasks, DiffOps wins one metric per task, often by a small margin. The conclusion's "quality, reliability, or consistency on most tasks" picks a different metric for each task!

W3) Efficiency comparison should similarly be expanded to include other NFE-efficient methods (DDfire, DiffPIR).

W4) Some of the evaluation is seemingly unfair. You tuned DiffOps for each task, but did you tune competitors fairly for all tasks? This is unclear.

W5) The analysis is thin and reporting is incomplete. The "analytical analysis" is a typical step-size bound, a known null-space property of CG, and a 2D toy. Also, PSNR/SSIM are promised for FFHQ but absent fron Table 1 and the Figure 3 y-axis is undefined for mixed-scale metrics.

[1] M. Bendel, S. K. Shastri, R. Ahmad, and P. Schniter. Solving inverse problems using diffusion with iterative colored renoising. Transactions on Machine Learning Research, 2025.

[2] M. Bendel, P. Schniter, and R. Ahmad. MRI reconstruction using diffusion with iterative colored renoising (DDfire). In Proc. ISMRM-ISMRT Annual Meeting, Cape Town, 2026.

[3] H. Chung, S. Lee, and J. C. Ye. Decomposed diffusion sampler for accelerating large-scale inverse problems. In International Conference on Learning Representations (ICLR), 2024.

Questions for authors

Q1) How does DiffOps compare to DDfire and DDS at matched NFE on the shared FFHQ operators, on phase retrieval, and on fastMRI? Does replacing β\beta with DDfire's residual-based estimate improve DiffOps?

Q2) Is there a rule (even a soft heuristic one, or some intuition) for mapping operator properties to solver, β\beta, and inner budget? Or were these chose ad-hoc per task?

Q3) Why do DAPS and PnP-DM collapse on confocal microscopy? What likelihood were they run under?

Q4) How is β\beta calibrated from the diffusion prior, and what does adjusting it to enable comparison mean for raw MRI?

5 comments

We thank the reviewer for the expert review and the pointers to DDfire and DDS. Please also see the general response.

W1 (DDS, DDfire; Eqs. 7/8). Eqs. 7/8 (L143, L190) have the same form as the DAPS/DAPS++/DCDP/DiffPIR correction step, as §3 states (L164). The difference is their role:

  • In those methods, the anchor approximates the diffusion prior and keeps the correction consistent with the diffusion trajectory.

  • In DiffOps, it is a geometric local constraint around the denoiser output (L194–196), which turns each noise level into a standalone classical problem. The diffusion signal-to-noise ratio enters only through λt=β/σt2\lambda_t=\beta/\sigma_t^2.

This is what makes classical solver selection applicable. It also lets solvers be combined across noise levels, e.g., DR→GD for phase retrieval.

DDS and DDfire solve the same linear-Gaussian subproblem, and the revision cites both in §2 (general response, point 1). On the shared linear operators the methods tie, which is what our analysis predicts.

The difference lies outside that subproblem. DiffOps selects a different numerical correction where the operator requires one, and it evaluates clipping, non-GLM nonlinear operators, exact Poisson likelihoods and staged nonconvex correction.

W2. DiffOps is a framework for choosing the correction by operator structure; it does not aim to win every metric on every task. The claim is "competitive or superior … with fewer network evaluations", and the per-task evidence behind it is:

  • phase retrieval: 94% vs 79% success;

  • black hole: closure χ2\chi^2 closest to the noise level;

  • Gaussian blur and HDR: best LPIPS/FID;

  • MRI, black hole and inverse scattering: faster than DPS, DAPS, RED-diff and PnP-DM on the same GPU.

On motion and nonlinear blur, DiffOps trails DAPS/DAPS++ in LPIPS/FID while using a quarter to a twentieth of their network calls.

W3 / Q1 (efficiency; DDfire and DDS at matched NFE). DiffPIR is in Table 1 at 100 NFE (L372). New matched comparisons, with the same operators, measurements and checkpoint:

SettingDiffOpsDDfire
Gaussian blur, 50 NFE28.03 dB, LPIPS 0.159, 1.6 s28.22 dB, 0.159, 2.8 s
4× SR, 50 NFE28.02 dB, LPIPS 0.175, 1.8 s28.09 dB, 0.175, 2.6 s
Gaussian blur, DDfire at 100 / 1000 NFE28.03 dB in 1.6 s (50 NFE)27.85 dB in 5.5 s / 27.70 dB in 55 s
Phase retrieval, 100 NFE74% success≤ 14%
Phase retrieval, 1000 NFE95% success≤ 62%

Residual-based weight. DDfire's residual-based weight also improves DiffOps when plugged into it: FID 43.3 → 41.1 on Gaussian blur and 47.0 → 44.0 on SR.

DDS on fastMRI. Our baselines focus on general-purpose solvers, mostly the InverseBench suite plus general methods. That is why DDS, a method specialized for MRI and CT, was not in the submission. Its correction, CG on an anchored quadratic, is an instance of DiffOps's linear branch. We therefore ran it on our protocol: same A10, slices, operator and prior, with its weight γ\gamma and its CG steps per level tuned per setting on validation slices.

  • Simulated MRI. DDS improves as the CG depth per level grows: 34.66 / 35.56 / 36.62 / 37.50 dB at 50 / 100 / 200 / 400 steps. This is the inner-budget lever of our framework. Compared at equal runtime with the same lever, DiffOps leads: 34.85 vs 34.66 dB at about 10.5 s per slice, 35.98 vs 35.56 at about 17 s, and 37.03 vs 36.62 at about 30 s. At the default budgets of about 100 network calls, DDS (200 CG steps per level, 30 s) reaches 36.62 dB and DiffOps (20 steps, 5.5 s) 32.86 dB.

  • Raw k-space. In this new DDS-matched comparison, the two are within 0.2 dB (DDS 28.27 vs DiffOps 28.09 dB, at 4.95 vs 4.4 s per slice), and deeper CG helps neither. The original Table 3 evaluation reports DiffOps at 28.02 dB and 3.8 s per slice.

W4 (tuning). The submitted DiffOps hyperparameters (Table S13, Supp. L929) were set on small pilot sets of 1–5 images per task before the full runs (one shared FFHQ set, outside the test set). The submission minimizes tuning by sharing β=2\beta=2 across Gaussian FFHQ operators, motivated by relative insensitivity to moderate variations. Optional operator-specific selection of β\beta and the solver on 100 held-out images assesses sensitivity and potential gains from more extensive tuning; it improves test LPIPS and FID on all six FFHQ operators without test images for selection. A separate held-out check returns the submitted configuration for raw MRI (see the general response, point 3, and the table in our reply to Reviewer Zn2D, Q2). Baselines on MRI, black hole and scattering use InverseBench's validation-tuned configurations.

The submitted minimal-tuning settings follow these default rules (per-task values in Table S13, Supp. L929):

  • the solver, chosen by operator;

  • β\beta by noise model: 2 for Gaussian FFHQ, 1 for Poisson and scientific tasks, 0 for noiseless data, and 10−310^{-3} for raw MRI under InverseBench's σy=1\sigma_y = 1;

  • inner iterations, fixed per solver family.

W5. The analysis is there to explain which correction fits which operator, and its predictions hold. The zero Jacobian at saturated pixels predicts that gradient and Gauss–Newton steps stall on HDR (Supp. L178; Table S5, Supp. L559; Fig. S7). Prop. S2 (Supp. L145) predicts that β\beta matters only for noisy data (Table S3, Supp. L512). Prop. S1 (Supp. L129) rests on the same Krylov-subspace argument as DDS; we use it to show what the warm start supplies.

PSNR/SSIM for every FFHQ operator are reported for DiffOps in Table S2 (Supp. L440). Baseline PSNR/SSIM, computed from the same reconstructions as Table 1:

Method (NFE)Gauss. blur4× SRBox inpaint.Motion blurHDRNonlin. blur
DAPS++ (100)27.55 / .75927.96 / .76524.84 / .81629.29 / .80726.02 / .81326.86 / .748
DAPS (1000)28.59 / .76228.43 / .75924.54 / .74230.54 / .81627.13 / .82727.73 / .741
DPS (1000)22.10 / .62019.96 / .56523.63 / .69820.32 / .56221.45 / .66721.87 / .622
DDRM (20)25.94 / .74427.58 / .79325.65 / .766———
DDNM (100)—28.02 / .80622.07 / .847———
DiffPIR (100)24.62 / .68026.22 / .73521.43 / .57926.62 / .739——
DCDP27.83 / .77926.15 / .62521.64 / .71226.03 / .749——
DiffOps (50)28.02 / .77628.01 / .77923.56 / .80527.99 / .76927.55 / .84426.48 / .691

The Figure 3 axis is the mean normalized reconstruction-quality score, obtained by equally averaging direction-corrected, per-panel min–max-normalized quality metrics without standard-deviation subtraction (FFHQ: PSNR, SSIM, LPIPS, FID; MRI and microscopy: PSNR, SSIM; scattering: PSNR, SSIM, relative L2L_2; black hole: aligned and 20 µas blurred PSNR; data-fit measures excluded); the revision states this definition in the main text.

Q2 (selection rule). The rule is stated in §2 (L137–160) and §3.2 (L235–250). Table S1 (Supp. L068) lists the solver families and their updates, and Table S13 (Supp. L929) the solver used for each task. In short:

Operator structureSolver
Linear, FFT-diagonalizable normal operatorWiener (closed form)
Linear, matrix-free A\mathcal{A} and A∗\mathcal{A}^*CG
Smooth nonlinear, usable JacobianGauss–Newton or L-BFGS
Zero or unavailable JacobianEKI
Poisson counts, nonnegative operatorEM
Nonconvex magnitude dataDR, then GD

In the submitted minimal-tuning configuration, β\beta is 0 for noiseless data and otherwise shared per noise model. This is a default choice, not a mathematical restriction. The additional operator-specific validation selection assesses sensitivity and potential gains from more extensive tuning; its solver choices agree with the rule: CG for SR and inpainting and Wiener for motion blur.

Q3 (Poisson baselines). Both drops come from how the data term behaves at low photon counts.

  • DAPS overfits the Poisson noise. Its annealed Langevin step must run enough MCMC iterations, with a small enough step size, to sample the noise-level marginal that DAPS's derivation assumes. These iterations cannot be cut, nor the step size enlarged, without leaving that regime, and at 10 photons they fit the noise.

  • PnP-DM. We adapted PnP-DM to the same algorithm family as DiffOps: a Poisson negative log-likelihood with a Richardson–Lucy (EM) update. To prevent fitting the noise, the adaptation used an SNR floor, below which the update took the previous output as its target instead of the observation. At low photon counts this ignores the observation in the late levels. Keeping the observation at every level makes PnP-DM a full adaptation of our correction, and its accuracy improves substantially. For example, on confocal microscopy at 10 photons (Table 3, L464) it goes from 14.43 to 18.94 dB (DiffOps: 22.25 dB), and in several other microscopy settings it exceeds DiffOps. It still needs 27× the computation of DiffOps (154 vs 5.8 s per image).

Q4 (β\beta; L432–433, Supp. L664–666). The raw k-space noise level is σy≈0.04\sigma_y \approx 0.04, which in our convention gives β=1\beta = 1. InverseBench's likelihood fixes σy=1\sigma_y = 1, so the same weight βσy2\beta\sigma_y^2 corresponds to β≈10−3\beta \approx 10^{-3} (1.6×10−31.6\times10^{-3} vs 10−310^{-3}). One constant β\beta per noise model is our default minimal-tuning choice, not a restriction; Table S2 shows that PSNR changes by at most 0.17 dB for β∈[2,4]\beta \in [2, 4], motivating this shared setting. The additional operator-specific validation experiment assesses sensitivity and potential gains from more extensive tuning. The revision replaces "calibrated from the diffusion prior" (L209) with "set per noise model" for the default configuration.

Reviewer_Zn2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, authors propose DiffOps, a framework for solving inverse problems with pretrained diffusion priors and operator-aware numerical corrections. At each noise level, a denoised estimate initializes the measurement-consistency solve and serves as the center of a quadratic regularization term. The inner solver is selected according to the forward operator and observation model, including spectral methods, conjugate gradients, nonlinear optimization, and ensemble methods. Authors evaluate the framework on FFHQ restoration tasks, Fourier phase retrieval, and scientific imaging problems including MRI, black-hole imaging, Poisson microscopy, and inverse scattering. The results examine reconstruction quality, measurement consistency, and diffusion-network evaluations.

Critical strengths
  • The paper is written clearly and generally contextualizes the approach well within the decoupled diffusion and classical optimization literature. The distinction between the diffusion reference and the numerical correction is easy to follow.
  • I appreciate the comprehensive evaluation across linear and nonlinear operators, scientific imaging applications, and a broad selection of competing methods. The supplementary solver, regularization, and noise-level ablations help explain when the different corrections are useful. The phase-retrieval results are particularly encouraging: the mixed DR-to-GD strategy reaches 94% success at 1000 NFEs, compared with 79% for DAPS++, and the solver comparison supports combining the two correction strategies. The simulated-MRI results in Table S7 are also strong. Reporting measurement fidelity alongside image-quality metrics makes these comparisons more informative.
Critical weaknesses
  • My main concern is the extent of the methodological contribution beyond existing optimization-based diffusion solvers. The paper acknowledges that decoupling and the quadratic correction objective already appear in prior work. ReSample [1] is another closely related method missing from the main discussion: it initializes data-consistency optimization from a denoised estimate and maps the corrected result back to a noisy state. Its appendix also considers pixel-space optimization and CG-based correction. The latent-space formulation and resampling scheme differ, but the overlap deserves a direct discussion of what DiffOps adds beyond using different classical solvers within this existing pattern.
  • The experimental gains are mixed, and the comparison protocol makes their significance difficult to assess. Section 4 and Appendix F use recommended task-specific settings for the baselines without describing validation-based tuning for the exact datasets and noise conventions. Repository recommendations do not necessarily transfer, so this can disadvantage competing methods. Table 1 is broadly competitive with DAPS++ rather than consistently better: DiffOps improves Gaussian blur and HDR but has worse LPIPS/FID on the other four tasks. Using 50 rather than 100 NFEs is promising, but inner optimization is excluded from NFE and timings use different hardware. The supplement strengthens the evidence in some settings, but gains on raw scientific measurements remain mixed; for example, raw-MRI misfit is nearly identical to RED-diff while PSNR and SSIM are lower. I would need a clearer comparison of quality against total computation to judge the practical advantage.
  • Please see the questions below as well.
Questions for authors

Questions and Suggestions:

  • Could the authors discuss the similarities and differences with ReSample [1], including its pixel-space optimization and CG-based correction? Which aspect of DiffOps provides a capability or benefit beyond the existing denoise–optimize–re-noise structure?
  • Were the recommended baseline hyperparameters validated for the exact settings in this paper? How were DiffOps' beta and inner-solver budgets selected? I would recommend tuning both DiffOps and the baselines on a separate validation split and reporting the final numbers on the test set.
  • Could the authors compare DiffOps and DAPS++ at matched computational budgets on the same GPU, including the inner optimization cost? This would help distinguish the benefit of operator-aware correction from differences in diffusion and solver budgets.
  • How are the different quality metrics normalized and combined in Figure 3? In particular, how are metrics with different scales and preferred directions handled before computing the mean-minus-standard-deviation score?

References:

[1] Song, B., Kwon, S. M., Zhang, Z., Hu, X., Qu, Q., and Shen, L. Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency. ICLR 2024.

4 comments

We thank the reviewer for the detailed review and for the encouraging comments on the evaluation and the phase-retrieval results. Please also see the general response.

W1 / Q1 (ReSample). ReSample enforces hard data consistency in latent space without an anchor term (initialization and early stopping). Its appendix adds pixel-space optimization and a CG pseudo-inverse step for CT. It is evaluated on linear operators and nonlinear blur.

DiffOps builds on the same decoupled skeleton, as §3 states (L164). Its anchor plays a different role: it is a geometric local constraint around the denoiser output (L194–196), not an approximation of the diffusion prior, so each noise level becomes a standalone classical problem. What it adds is choosing the correction by operator structure, with evidence that a fixed correction fails where the operator breaks it. Inside our own pipeline, a fixed gradient correction reaches 52% success on phase retrieval at 100 NFE, while the staged DR→GD correction reaches 74% (Table 2, L408–419). The black-hole, MRI and scattering controls in the general response show the same pattern. We add ReSample, DDS and DDfire to §2.

W2 (Table 1 vs DAPS++). Our claim is "competitive or superior reconstruction quality … while requiring fewer diffusion-network evaluations" (abstract, L022–024), and Table 1 (L363–375) shows exactly that:

  • DiffOps is competitive with DAPS++ and best on Gaussian blur and HDR.

  • It uses 50 network calls against DAPS++'s 196, and takes 4.8× less time on the same GPU (1.62 vs 7.85 s).

  • With optional operator-specific selection of β\beta and the solver on 100 held-out images, DiffOps improves further on all six operators (see Q2).

Raw MRI in the original evaluation (Table 3, L462): DiffOps has the lowest data misfit (31.58) and a higher SSIM than RED-diff (0.661 vs 0.626); RED-diff has the higher PSNR (28.74 vs 28.02). On one A10, DiffOps takes 3.8 s per slice and RED-diff 45 s.

Q2 (tuning). For MRI, black hole and inverse scattering, the baselines use InverseBench's configurations, which InverseBench selected per algorithm and task on a validation set. Our reruns match the published numbers within 0.3 dB.

The submitted DiffOps hyperparameters (Table S13, Supp. L929) were set on small pilot sets of 1–5 images per task before the full runs; the FFHQ operators share one pilot set outside the FFHQ test set. The submitted minimal-tuning default uses one β\beta per noise model, with β=2\beta=2 shared across Gaussian FFHQ operators because performance is relatively insensitive to moderate variations. This is a design choice, not a restriction. Following your recommendation, we additionally assessed sensitivity and potential gains from more extensive tuning through operator-specific selection on 100 held-out FFHQ images, disjoint from the 100 test images:

  • grid: β∈{1,2,4,8}\beta \in \{1, 2, 4, 8\} × two solvers per operator (the submitted one and the one our selection rule prescribes);

  • selection metric: LPIPS.

On the test images, the selected configurations improve on the submitted ones for every operator, at equal or lower cost:

OperatorSelected solver, β\betaTest LPIPS (submitted → selected)Test PSNR change
Gaussian blurWiener, 40.158 → 0.156−0.01 dB
4× SRCG, 40.175 → 0.170−0.14 dB
Box inpaintingCG, 80.140 → 0.133+0.20 dB
Motion blurWiener, 40.184 → 0.173−0.12 dB
HDREKI, 40.156 → 0.154+0.17 dB
Nonlinear blurGauss–Newton, 80.209 → 0.189+0.07 dB

FID also decreases on all six operators, by 0.3 to 6.1. These optional configurations are separate from the submitted shared-β\beta setting and use no test images for selection. For raw MRI, a separate held-out check on InverseBench validation slices re-selects the submitted configuration (CG×2, β=10−3\beta = 10^{-3}). Inner budgets are fixed per solver family: 20 iterations for iterative solvers, 5 for EKI, 2 for Gauss–Newton, 1 EM step and 10 for L-BFGS. Raw MRI uses CG×2, as in Tables S7/S8 (Supp. L687, L714); Table S13 listed 20 and is corrected in the revision (Supp. L939).

Q3 (matched compute on the same GPU). On one A10, with wall-clock including every inner solve:

TaskDiffOpsBaseline
Gaussian blur1.62 s (50 network calls)DAPS++ 7.85 s (196 calls); DAPS 90.6 s
Nonlinear blur31.2 sDAPS++ 48.8 s
Phase retrieval, 1000 NFE38.9 sDAPS++ 80.3 s; DAPS 55.0 s; DPS 71.5 s

For the original evaluation, the revision states the hardware at L323–324: DiffOps runs on an A10, and the baselines on an A10 or a faster A100, so any hardware difference favors the baselines. The new runtime comparisons above use the same A10.

Q4 (Figure 3, L340; axis described at L323–324 and L343–344). Each panel's vertical axis is the mean normalized reconstruction-quality score: an equal-weight average of that task's metrics, without standard-deviation subtraction. Each metric is processed in two steps:

  1. Direction correction: higher is better for every metric (LPIPS, FID and relative L2L_2 error are reversed).

  2. Min–max normalization across the methods shown in that panel: worst 0, best 1.

The metrics are:

  • FFHQ: PSNR, SSIM, LPIPS and FID;

  • MRI and microscopy: PSNR and SSIM;

  • inverse scattering: PSNR, SSIM and relative L2L_2 error;

  • black-hole imaging: shift-aligned PSNR and 20 µas beam-blurred PSNR.

Data-fit measures (misfit, closure χ2\chi^2) are not included. The original text mentioned one standard deviation because we initially intended to include a standard-deviation indication in the visualization. However, the final figure displays the mean scores, and the corresponding text was inadvertently left unchanged. We will remove the outdated reference to one standard deviation and explicitly state the correct definition in the Figure 3 caption (L340).

I would like to thank the authors for their time and responses. My questions and concerns regarding baseline tuning are resolved. I also appreciate the numbers for the same-GPU runtime comparisons.

Unfortunately, I don't find the arguments for the technical novelty of DiffOps beyond ReSample (or beyond DDS and DDFire as Reviewer MBZP noted) convincing enough. The main contribution is indeed the use of operator-specific solvers. While phase-retrieval results seem to benefit from this, the remaining results are mixed in terms of image metrics (Tables 1 and 3). I do acknowledge that DiffOps compares favorably with the baselines in terms of NFE.

For these reasons, I intend to maintain my score.

Thank you for the follow-up and for confirming that the tuning and runtime questions are resolved. The main claim of our paper is that matching the correction algorithm to the operator structure yields substantial efficiency gains while maintaining performance comparable to existing solvers, and achieving superior results where a fixed correction fails:

  • Accuracy gains on nonlinear operators: With the prior, schedule, and NFE held fixed, replacing our matched solver with standard gradient descent drops performance by 1.6 dB on HDR and 2.2 dB on nonlinear blur (Tables S4 to S5), lowers the phase-retrieval success rate, and more than doubles the closure χ2\chi^2 on black-hole imaging (from 1.25 to 2.60). On FFHQ, DiffOps achieves these results with significantly fewer network evaluations than the baselines.
  • Analytical grounding for solver selection: The paper theoretically derives which correction fits each operator. Specifically, we show how $\lambda_t $ governs conditioning and step size, why conjugate gradient (CG) initialized at the denoiser output preserves its nullspace component, when β\beta matters, and why gradient and Gauss-Newton steps stall at clipped pixels. To the best of our knowledge, this is the first systematic treatment of solver selection based on operator structure in diffusion-based reconstruction, and our rule generalizes to operator classes beyond those evaluated here.
  • Competitiveness on linear-Gaussian tasks: We do not expect to achieve a new state-of-the-art on all linear-Gaussian tasks, since all methods ultimately solve the same convex subproblem. However, as InverseBench and the spread of top entries in Table 3 confirm, no single existing method dominates across all operator classes, which means users have to test multiple solvers blindly without a selection rule. DiffOps prescribes the right correction directly from the operator's structure and remains competitive across every task we evaluated.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Relation to related works and solver choice

We thank the reviewers for their careful reading. Three questions recur, and we answer them with new experiments. The same-GPU runtime comparisons below use one NVIDIA A10, the GPU used for DiffOps in the paper; other supplementary experiments may use an A100.

1. Relation to DDS, DDfire and ReSample. DDS and DDfire share the anchored linear-Gaussian subproblem that DiffOps solves in its linear branch: the denoised estimate anchors a quadratic that CG or an SVD solves. All three methods follow the broader decoupled diffusion-based inverse-problem paradigm, but their correction mechanisms differ:

  • DDS: CG with a constant weight;

  • DDfire: CG/SVD with a residual-based weight and colored renoising, plus EP for phase retrieval;

  • ReSample: hard data consistency in latent space without an explicit anchor term.

DiffOps starts from the classical inverse problem and adds the diffusion prior to it. The anchor is a geometric local constraint around the denoiser output (L194–196), not an approximation of the diffusion prior, so each noise level becomes a standalone classical problem whose solver is selected by operator structure. The diffusion signal-to-noise ratio enters only through λt=β/σt2\lambda_t=\beta/\sigma_t^2, so solvers can also be combined across noise levels to match the stages of the diffusion. It also evaluates operator classes none of them evaluate: zero-Jacobian clipping (HDR), network blur, interferometric closure quantities, an exact Poisson likelihood, and staged nonconvex correction (phase retrieval).

New head-to-head results on our protocol (same operators, measurements and checkpoint):

  • Gaussian blur and 4× SR at 50 NFE. DiffOps and validation-tuned DDfire are at parity: PSNR 28.03 vs 28.22 and 28.02 vs 28.09; LPIPS 0.159 vs 0.159 and 0.175 vs 0.175. DiffOps is faster: 1.6–1.8 s vs 2.6–2.8 s per image. Parity is what a shared subproblem predicts. Plugging DDfire's residual-based weight into DiffOps lowers FID to 41.1 (Gaussian blur) and 44.0 (SR), so DiffOps can adopt that estimate directly.

  • Fourier phase retrieval. DiffOps reaches 74% success vs ≤ 14% for DDfire-PR at 100 NFE, and 95% vs ≤ 62% at 1000 NFE.

  • MRI (DDS). DDS is specialized for MRI and CT, and its CG correction is an instance of our linear branch. It improves with deeper CG per level, which is the inner-budget lever of our framework. At equal runtime DiffOps leads on simulated MRI by 0.2–0.4 dB (e.g., 37.03 vs 36.62 dB at about 30 s per slice); on raw k-space the two are within 0.2 dB in this new DDS-matched comparison (DiffOps: 28.09 dB, 4.4 s per slice; DDS: 28.27 dB, 4.95 s per slice).

2. Does the solver choice matter? On linear-Gaussian operators every solver minimizes the same strictly convex objective, so the solvers tie (Table S4, Supp. L541), as our analysis predicts. Where the operator breaks the default correction, the choice matters. With the same prior, schedule and GPU:

TaskOperator-matched solverGD
Black hole, 10% coverageL-BFGS: 23.90 dB, χcp2\chi^2_{\rm cp} 1.2523.07 dB, χcp2\chi^2_{\rm cp} 2.60 (GD×30: 23.57 dB at 3.3× the time)
Simulated MRICG: 32.86 dB32.52 dB
Inverse scatteringCG×5: 34.78 dB in 4.3 sGD×10: 34.63 dB in 6.0 s
HDREKI: 26.97 dB25.41 dB
Nonlinear blurGauss–Newton: 26.23 dB23.99 dB
Phase retrieval (100 NFE)DR→GD: 74% success52%

HDR and nonlinear blur: Table S5 (Supp. L559); phase retrieval: Table 2 (L408–419).

3. Hyperparameters and compute. The submitted hyperparameters (Table S13, Supp. L929) were set on small pilot sets of 1–5 images per task before the full runs: one set shared by the FFHQ operators, outside the FFHQ test set, and a task-specific set for each scientific problem. The submission follows a minimal-tuning principle: one shared β\beta per noise model, with β=2\beta=2 across Gaussian FFHQ operators, motivated by relative insensitivity to moderate β\beta variations. This is a default design choice, not a mathematical restriction of DiffOps.

As Reviewer Zn2D recommended, we additionally assessed sensitivity and potential gains from more extensive tuning by selecting β∈{1,2,4,8}\beta\in\{1,2,4,8\} and one of two solvers per operator on 100 held-out FFHQ images using LPIPS, without test images for selection. These optional operator-specific configurations are separate from the submitted shared-β\beta setting. They improve on the submitted ones in test LPIPS and FID for all six FFHQ operators, at equal or lower cost (e.g., motion blur LPIPS 0.184 → 0.173). They also pick the solver our selection rule prescribes. For raw MRI a separate held-out check returns the submitted configuration. Both settings use no test images for hyperparameter selection.

Same-GPU timing (A10, wall-clock including every inner solve):

  • Gaussian blur: DiffOps 1.62 s per image, DAPS++ 7.85 s, DAPS 90.6 s.

  • Raw MRI (original Table 3 evaluation): DiffOps 28.02 dB at 3.8 s per slice, RED-diff 45 s, DAPS 113 s.

  • Network calls: DAPS++'s 100-NFE setting makes 196 calls, counted the way we count DiffOps's 50.

Revision. We will make these changes:

  • §2: cite DDS, DDfire and ReSample;

  • Figure 3 caption (L340): define the vertical axis as the mean normalized reconstruction-quality score, without standard-deviation subtraction;

  • L209: describe β\beta as set per noise model in the default minimal-tuning configuration;

  • L323–324: state the hardware (DiffOps on an A10; baselines on an A10 or a faster A100 from the supplementary experiments);

  • conclusion

  • the fixes listed by Reviewer TX1x.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.