acceptodds
Under review as a conference paper at ICLR 2027

FANO: Anchored Forward-Noise Optimization for Diffusion Inverse Problems

Abstract

Pre-trained diffusion models have become a widely adopted zero-shot prior for image inverse problems, among which annealed decoupling methods achieve leading reconstruction performance on nonlinear tasks. We find, however, that annealed methods that directly optimize the denoiser input exhibit *radial drift* in later layers: within-layer optimization drives the noise-component norm of away from its typical value, pushing the denoiser out of its high-performance regime and degrading reconstruction quality. To address this, we propose FANO (Anchored Forward-Noise Optimization). At each annealing layer, FANO fixes the previous denoiser estimate as an anchor and optimizes only the forward noise , whose distribution is known, instead of , thereby preserving the forward-diffusion structure of the denoiser input. The standard Gaussian prior on concentrates its norm near the typical value, and deviations degrade denoiser performance. We accordingly decompose in polar coordinates into a direction and a radial magnitude, and apply a soft potential to regulate the radius, keeping in the high-density region of the forward distribution. On multiple linear and nonlinear inverse problems, FANO matches the strongest current baselines on linear tasks and achieves consistent improvements on nonlinear tasks.

Primary area:
generative models
Keywords:
Diffusion models, image inverse problems, nonlinear inverse problems, zero-shot image restoration, diffusion priors, annealed optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_81MFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new diffusion-based inverse-problem solver, termed FANO, built on an annealing framework. Its key insight is to optimize the injected noise at each annealing step, rather than directly optimizing x0x_0 or xtx_t. To exploit the inherent prior on the noise norm, the authors introduce a chi-square-based regularization term into the optimization objective, constraining the optimized noise from deviating too far from the typical Gaussian shell. The performance of FANO is evaluated on FFHQ and ImageNet.

Critical strengths

I find the identified phenomenon of radial drift interesting and useful. This observation provides a convincing explanation for why SITCOM may deviate from the manifold, and makes the core motivation of the paper particularly clear. I also find the idea of optimizing the injected noise, rather than directly optimizing x0x_0 or xtx_t, to provide a meaningful insight for diffusion-based inverse problems. In addition, the proposed norm prior and the resulting optimization procedure appear well motivated, though there are some questions about the weight as discussed below.

Critical weaknesses
  1. Please at least place the title and author information at the top of the first page.

  2. I noticed that the weight λrad\lambda_{\text{rad}} used in the paper is generally very small, and is even set to zero for two tasks. This makes me question how much the proposed norm prior actually contributes to the performance. Only for random inpainting do the authors use λrad=1\lambda_{\text{rad}}=1, where the prior appears to play a meaningful role. However, in most other cases, the weight is below 10−210^{-2}. From Figure C.1, I also observe that λrad=0\lambda_{\text{rad}}=0 generally achieves reasonably good performance, while increasing λrad\lambda_{\text{rad}} can even lead to performance degradation on some tasks. Therefore, I find it difficult to assess how necessary it is to constrain the optimized noise to remain near the typical Gaussian shell, or whether optimizing the noise ϵ\epsilon itself is already sufficient to achieve substantial improvements.

  3. Although the authors appear to have selected several recent methods as baselines, I would like to see comparisons with other representative approaches. For example, DDNM (NFE=100) is a simple yet effective inverse-problem solver, and a comparison in terms of both reconstruction quality and computational efficiency would be valuable. Since the paper uses a pixel-space diffusion model, DDNM should be directly applicable to at least the linear inverse problems considered here. In addition, some more recent methods, such as [1], also consider the noise distribution from a related perspective. Comparing against such methods would help better position FANO within the existing literature.

    I also find the comparison with INO somewhat unusual, as INO appears to focus on settings with imperfect generative priors, such as those arising from shortened sampling trajectories, which are not the primary settings considered in this paper.

  4. Several recent studies have also recognized the importance of accounting for the noise prior at each step of diffusion-based inverse-problem solving, such as [2]. A more detailed discussion of these works, or additional experimental comparisons where applicable, would help clarify the novelty and contribution of FANO relative to existing approaches.

[1] Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction. ICML, 2026.

[2] Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior. ICLR, 2026.

Questions for authors

I am curious about what Theorem 1 is intended to establish. In particular, part (ii) appears to state that the objective value at a non-optimal point is necessarily larger than the optimal value, which seems rather straightforward.

I understand the authors' argument that if the optimal noise corresponds to a radius r≠1r \neq 1, forcibly projecting it onto the unit sphere may lead to suboptimal loss value. However, I feel that this observation and the motivation for imposing a Gaussian prior are somewhat conflict. Directly optimizing the measurement-consistency objective ∣A(f(xt))−y∣2|A(f(x_t))-y|^2 can naturally drive the noise away from the Gaussian prior, which is precisely the issue underlying the radial drift observed in SITCOM. This motivates introducing a Gaussian prior to regularize the optimization.

On the other hand, a high-dimensional Gaussian distribution is inherently concentrated near a thin spherical shell. For typical image dimensions (e.g., 3×256×2563\times256\times256), approximating the radial distribution with a spherical-shell constraint seems quite reasonable to me. This brings me back to my earlier concern: why does the method already achieve strong performance when λrad\lambda_{\text{rad}} is close to zero? I would appreciate a more thorough discussion of the actual contribution and necessity of the proposed norm prior, particularly in distinguishing its effect from the benefit of optimizing the noise itself.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 81MF (1/5)

Thank you for the positive assessment and the specific questions. Your main question is how much the norm prior contributes, given that λrad\lambda_{\mathrm{rad}} is generally very small (W2, Q2). You also suggested additional baselines such as DDNM and a discussion of recent work such as DMAP, and asked what Theorem 1 (Proposition 1 in our submission) is meant to establish (W3, W4, Q1). The revision is uploaded; our point-by-point replies follow.

Unless stated otherwise, results are on 100 FFHQ images with measurement noise σy=0.05\sigma_y=0.05 and are reported as PSNR / LPIPS. Changes in the revision are marked in blue, and each reply ends with the location of the changes, including line numbers.

The main new results are:

  • An ablation separates optimizing the noise from the norm prior (the radial potential in the paper): the improvement comes mainly from optimizing in the noise coordinate, while the prior controls the remaining radial deviation and mainly improves LPIPS; the values of λrad\lambda_{\mathrm{rad}} are of the same order of magnitude as the weight given by the Bayesian posterior (W2, Q2).
  • We added DDNM: FANO has higher PSNR in seven of the eight comparisons; DDNM is 8 to 28 times faster but applies only to linear tasks whose operator admits an SVD (W3).
  • The revision replaces Proposition 1 with the coordinate-scale law and the late-layer update bound (Q1).

W1. Title and author information at the top of the first page

Done as requested. In the submission, the teaser figure floated above the title on the first page; in the revision, the title and author information are at the top of the first page.

AuthorsOpenReview ↗

Response to Reviewer 81MF (2/5)

W2. λrad\lambda_{\mathrm{rad}} is small in general and zero for two tasks: does the improvement come from the norm prior or from optimizing the noise itself?

As you suspect, the improvement comes mainly from optimizing in the noise coordinate, which is the core design of FANO. FANO-Euc with λrad=0\lambda_{\mathrm{rad}}=0, which only changes the search variable, is already 1.15 dB (HDR) and 1.12 dB (nonlinear deblurring) better than SITCOM (Appendix C.3). The norm prior handles the radial deviation that remains in the noise coordinate, and it does have an effect: without it, the radius on the two linear deblurring tasks shrinks to 0.83 and 0.88 in the late layers, and on HDR it expands to 1.22 at the final layer; with the prior it returns to about 1 and to 1.07, respectively. The effect shows mainly in perceptual quality: LPIPS improves on almost every image for Gaussian deblurring and on nearly 90% of the images for motion deblurring, while PSNR changes by at most 0.14 dB. The table separates the two parts: going from FANO-Euc (λrad=0\lambda_{\mathrm{rad}}=0) to FANO (λrad=0\lambda_{\mathrm{rad}}=0) adds only the polar decomposition, and going on to FANO (default λrad\lambda_{\mathrm{rad}}) adds the prior.

GaussianMotionRandomPRHDRNL
FANO-Euc (λrad=0\lambda_{\mathrm{rad}}=0)30.64 / 0.22533.03 / 0.16733.99 / 0.12232.54 / 0.17229.18 / 0.17130.80 / 0.182
FANO (λrad=0\lambda_{\mathrm{rad}}=0)30.76 / 0.23433.15 / 0.17234.02 / 0.12032.12 / 0.18829.76 / 0.15930.85 / 0.184
FANO (default λrad\lambda_{\mathrm{rad}})30.65 / 0.22533.04 / 0.16734.07 / 0.12031.98 / 0.18729.88 / 0.15730.83 / 0.183

A small value of λrad\lambda_{\mathrm{rad}} does not mean that the prior is weak. Writing the injected noise ee as a radius rr and a direction uu (∥e∥=d−1 r\lVert e\rVert=\sqrt{d-1}\,r), the negative log-density of the standard Gaussian prior in these coordinates is (d−1)(r2/2−log⁡r)(d-1)(r^2/2-\log r). The negative log-posterior at layer tt is

−log⁡p(r,u∣y)=∥A(Dt(xt))−y∥22σy2+(d−1)(r22−log⁡r)+const-\log p(r,u\mid y)=\frac{\lVert \mathcal A(D_t(x_t))-y\rVert^2}{2\sigma_y^2}+(d-1)\Big(\frac{r^2}{2}-\log r\Big)+\text{const}

while FANO's per-layer objective is

∥A(Dt(xt))−y∥2+λrad (d−1)(r22−log⁡r−12)\lVert \mathcal A(D_t(x_t))-y\rVert^2+\lambda_{\mathrm{rad}}\,(d-1)\Big(\frac{r^2}{2}-\log r-\frac12\Big)

Multiplying the first expression by 2σy22\sigma_y^2 gives the second with λrad=2σy2\lambda_{\mathrm{rad}}=2\sigma_y^2, which is 0.005 at σy=0.05\sigma_y=0.05. In the main table, λrad\lambda_{\mathrm{rad}} for Gaussian deblurring, motion deblurring, phase retrieval, HDR, and nonlinear deblurring lies in 0.0015–0.01, exactly this order of magnitude. The potential is also multiplied by d−1≈2×105d-1\approx2\times10^5, so a small numerical value of λrad\lambda_{\mathrm{rad}} does not mean a weak prior.

As for the three tasks with λrad\lambda_{\mathrm{rad}} equal to 0 or 1 that you mention, box inpainting uses 0 and random inpainting uses 1. On both inpainting tasks the radius barely moves, so the value of λrad\lambda_{\mathrm{rad}} hardly matters (over the whole sweep in Appendix C.2, the PSNR of random inpainting changes by only 0.05 dB). For SR, λrad\lambda_{\mathrm{rad}} was selected by PSNR and is therefore 0; with λrad>0\lambda_{\mathrm{rad}}>0, LPIPS improves on all 30 images of the sweep while PSNR drops by 0.06–0.09 dB.

In the revision, the ablation table above is in Appendix C.2 (Table 8, lines 1081–1085).

AuthorsOpenReview ↗

Response to Reviewer 81MF (3/5)

W3. DDNM (quality and efficiency); comparison with DC-PnPDP; is the comparison with INO appropriate?

We added DDNM as suggested, and FANO has higher PSNR in seven of the eight comparisons. DC-PnPDP targets medical image reconstruction, and we discuss it in the related work; INO is the existing method closest to FANO.

We use DDNM+ (100 steps), the variant for noisy measurements, with 100 images per dataset and the same measurements as Table 1. DDNM has higher PSNR only on ImageNet box inpainting; in LPIPS, DDNM is lower on SR (both datasets) and on FFHQ box inpainting. In efficiency, DDNM is 8 to 28 times faster, but it relies on an SVD of the operator and cannot be applied to motion deblurring (whose kernel uses reflective boundaries) or to the three nonlinear tasks. Quality and runtime:

SR ×4GaussianBoxRandom
DDNM, FFHQ30.37 / 0.20328.92 / 0.22923.89 / 0.14131.12 / 0.167
FANO, FFHQ30.68 / 0.23230.65 / 0.22525.33 / 0.15634.07 / 0.120
DDNM, ImageNet26.81 / 0.30425.89 / 0.34120.62 / 0.21227.26 / 0.240
FANO, ImageNet27.03 / 0.32127.01 / 0.31720.40 / 0.20629.94 / 0.157
Time per image (s): DDNM / FANO4.0 / 803.4 / 943.2 / 273.2 / 36

As you note, DC-PnPDP also looks at the distribution of the noise, but at a different property. It recovers the dual variable in a PnP diffusion + ADMM framework to remove the steady-state bias and turns the structured residual in the denoiser input into approximately white noise; it is concerned with whether the noise spectrum is white, whereas we are concerned with the noise norm. It targets linear CT and MRI reconstruction with medical images and dedicated priors, a setting that differs from our natural-image evaluation, so we did not compare with it experimentally and discuss it in the related work instead.

Regarding INO, you are right that its starting point is imperfect generative priors. We still compare with it because it is the existing method closest to FANO: it also optimizes in the noise space and uses a shell constraint. We use its recommended configuration.

In the revision, the DDNM results are in Appendix E.4 (new, Table 19, lines 1478–1487), and Section 2.4 adds the discussion of DC-PnPDP (lines 200–202).

W4. Recent work such as DMAP also accounts for the noise prior at each step

Thank you for pointing out DMAP (Xu et al., ICLR 2025). It starts from an idea similar to FANO's, namely that the noise norm should be taken into account at every step. The difference is that DMAP uses a hard projection, whereas FANO uses the noise as the search variable and keeps a soft radial degree of freedom; in addition, DMAP guides each reverse step, whereas FANO uses annealed decoupling.

In each reverse step, DMAP runs several gradient-descent steps on xx and, after each step, projects onto the sphere of radius d σt\sqrt d\,\sigma_t centered at the transition mean; the paper also proposes training a lightweight conditional score estimator. FANO instead re-noises from the anchor at each layer and searches in the noise coordinate.

You suggest experimental comparisons where applicable. The public implementation of DMAP is built on Stable Diffusion 2 (a latent model) and cannot load the pixel-space models we use; porting it would require rewriting its latent-space derivation and interfaces, so we did not compare with it directly. Instead, we added the same kind of hard shell projection to SITCOM (projection after every update) and tested it on HDR and nonlinear deblurring, where radial drift is pronounced. It reaches 28.26 on HDR and 29.75 on nonlinear deblurring, 0.23 and 0.07 dB above SITCOM without the projection (Appendix D.1) and 1.6 and 1.1 dB below FANO. Fixing the radius at 1 in the noise coordinate (FANO-Sphere) is also 0.82 dB below FANO on HDR. A hard constraint therefore helps less than changing the search variable and adding a soft constraint.

In the revision, Section 2.2 introduces DMAP (lines 139–140) and Section 2.4 contrasts it with FANO (lines 198–200), and the results of SITCOM with a hard shell projection are in Appendix C.3 (new, lines 1124–1130 and Table 10).

Reviewer_VQVXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FANO addresses image inverse problems using pretrained diffusion models. To mitigate late-stage radial drift, it fixes the previous reconstruction as an anchor and optimizes the forward noise with a soft radial constraint. Experiments cover eight linear and nonlinear tasks on FFHQ and ImageNet.

Critical strengths
  1. The paper provides a detailed analysis of radial drift and proposes a targeted remedy.
  2. FANO is simple and applicable to both linear and nonlinear inverse problems.
  3. The evaluation includes several recent baselines and shows notable gains on nonlinear deblurring, HDR, and phase retrieval.
Critical weaknesses
  1. The choice to fix the anchor and optimize the noise needs further justification. Could the authors compare FANO with an alternative that holds a sampled Gaussian noise vector e\mathbf{e} fixed and optimizes the anchor with a quadratic penalty toward the previous reconstruction xˉ\bar{\mathbf{x}}, e.g., min⁡vˉtℓy(Dt(atvˉt+σte))+μ∣vˉt−xˉ∣22\min_{\bar{\mathbf{v}}_t} \ell_y\big(D_t(a_t\bar{\mathbf{v}}_t+\sigma_t\mathbf{e})\big)+\mu|\bar{\mathbf{v}}_t-\bar{\mathbf{x}}|_2^2? Reporting reconstruction quality and measurement residuals under matched computational budgets would help assess this design choice.
  2. The evaluation is limited to pixel-space diffusion models. The current experiments do not establish whether FANO extends effectively to latent diffusion models. Following the latent-space evaluation settings of methods such as SITCOM [a] and DAPS [b], the authors could include representative inverse problems to examine whether radial drift also occurs in latent space and whether the proposed radial control yields consistent reconstruction improvements.
  3. A comparison with SITCOM augmented only by the radial potential would strengthen the analysis. FANO-Euc already evaluates direct noise optimization with the radial potential and without polar decomposition. However, it uses FANO’s within-layer objective and therefore does not directly establish whether adding radial control to the original SITCOM objective would mitigate late-stage degradation and yield comparable gains. I suggest retaining SITCOM’s original measurement loss and Euclidean regularizer, adding only the same radial potential applied to the equivalent noise defined relative to a fixed anchor. This variant should be compared with the original SITCOM, FANO-Euc, and FANO under matched computational budgets and appropriately tuned radial regularization coefficient λrad\lambda_{\mathrm{rad}}.

[a]. SITCOM: Step-wise triple-consistent diffusion sampling for inverse problems. ICML 2025.

[b]. Improving diffusion inverse problem solving with decoupled noise annealing. CVPR 2025.

Questions for authors

Appendix C.2 shows that some tasks are relatively insensitive to λrad\lambda_{\mathrm{rad}}, with radial regularization yielding only marginal gains or the best performance occurring at λrad=0\lambda_{\mathrm{rad}}=0. Could the authors report results without the radial potential across all tasks, keeping all other settings and computational budgets unchanged, and explain this task-dependent behavior?

10 comments
AuthorsOpenReview ↗

Response to Reviewer VQVX (1/5)

Thank you for proposing three control experiments and one question. All of the experiments are done and the revision is uploaded; our point-by-point replies follow.

Unless stated otherwise, results are on 100 FFHQ images with measurement noise σy=0.05\sigma_y=0.05 and are reported as PSNR / LPIPS. All three control experiments use HDR and nonlinear deblurring: radial drift occurs mainly on these two nonlinear tasks (on linear tasks SITCOM's final radius is 1.00–1.11; see Appendix E.3), and they are also the tasks of the ablation in Section 4.3. Changes in the revision are marked in blue, and each reply ends with the location of the changes, including line numbers.

The main new results are:

  • The variant that fixes the noise and optimizes the anchor has 0.58 dB higher PSNR than FANO on HDR but worse SSIM and LPIPS; on nonlinear deblurring it is worse on all three metrics (PSNR 1.39 dB lower). The variant is sensitive to μ\mu and shows the same degradation as SITCOM in the last two layers (W1).
  • In latent space, SITCOM's final radius increases to 1.4–1.5 and it degrades in the late layers on HDR; FANO keeps the radius at 1 and shows almost no late-layer degradation. Without the radial potential, FANO's PSNR is 0.57 dB lower on HDR and 1.52 dB lower on nonlinear deblurring (W2).
  • Adding only the radial potential to SITCOM improves it by 0.63 dB on HDR and 0.94 dB on nonlinear deblurring and almost removes the late-layer degradation (W3).
  • Without the radial potential, the differences between tasks depend on how the measurement constrains the high-frequency energy of the denoised output: the radius shrinks on blur-type tasks, stays almost unchanged on inpainting and phase retrieval, and expands on HDR (Question).
AuthorsOpenReview ↗

Response to Reviewer VQVX (2/5)

W1. Why fix the anchor and optimize the noise; comparison with fixing the noise and optimizing the anchor with a quadratic penalty

We fix the anchor and optimize the noise so that each layer's update matches the noise level of that layer. On HDR, the variant achieves 0.58 dB higher PSNR than FANO; however, its SSIM and LPIPS are worse, it is sensitive to μ\mu, and it shows the same late-layer degradation as SITCOM. On nonlinear deblurring it is worse on all three metrics (PSNR 1.39 dB lower).

This design follows the coarse-to-fine way in which diffusion models generate images: at high noise levels they first determine the low-frequency global structure, and high-frequency details are added only as the noise level decreases (Rissanen et al., 2023; Falck et al., 2025). By the layer at noise level σt\sigma_t, the earlier layers have already fixed the low-frequency structure, and this layer should modify only the high-frequency details that correspond to σt\sigma_t. With the anchor fixed and only the noise optimized, each step changes xtx_t in proportion to σt\sigma_t, which matches this coarse-to-fine division of labor. The variant you suggest behaves the other way: each step changes xtx_t in proportion to ata_t, so the steps grow in the late layers. The quadratic penalty limits the displacement of the anchor, but in the late layers even a small displacement is large relative to σt\sigma_t, so the penalty does not restrain the update. Change of the denoiser input xtx_t per inner step for the three approaches:

MethodChange of xtx_t per stepFrom the first to the last layer
SITCOMγ\gammaunchanged
Fixed noise, optimized anchorγat\gamma a_tgrows by more than 100×
FANOησt\eta\sigma_tshrinks by about 100×

For the experiment, we implemented this variant with the objective you gave and compared it with FANO under the same budget; μ\mu was selected for each task from {0, 0.1, 1, 10} on the first 20 images, and 0.1 was selected for both tasks. The variant's final radius rises to 2.1 (HDR) and 2.2 (nonlinear deblurring), and its PSNR and LPIPS worsen in the last two layers, the same phenomenon as SITCOM's late-layer degradation; FANO's final radius is 1.07 and 1.02. Reconstruction quality on 100 images (PSNR / SSIM / LPIPS) and the final-layer measurement residual you requested:

HDRNL deblurringFinal measurement residual (HDR / NL)
Fixed noise, optimized anchor (μ=0.1\mu=0.1)30.46 / 0.876 / 0.17929.44 / 0.793 / 0.2460.049 / 0.048
FANO29.88 / 0.878 / 0.15730.83 / 0.873 / 0.1830.054 / 0.050

The variant is also sensitive to μ\mu. FANO, in contrast, uses the same learning rate on all eight tasks; with λrad=0\lambda_{\mathrm{rad}}=0 its PSNR differs from the default by at most 0.14 dB, and with λrad→∞\lambda_{\mathrm{rad}}\to\infty (radius fixed at 1) it is 0.82 dB lower on HDR and at most 0.2 dB lower on the other tasks. PSNR of the variant for different μ\mu (first 20 images):

μ\mu00.1110
HDR28.0330.2327.0323.02
NL deblurring28.8028.8626.0721.40

In the revision, the full comparison with this variant and its sensitivity to μ\mu are in Appendix C.4 (new, lines 1142–1168), and Section 3.2 points to it (lines 259–260).

AuthorsOpenReview ↗

Response to Reviewer VQVX (3/5)

W2. Latent diffusion models: does radial drift occur in latent space, and is the radial control equally effective?

You asked whether radial drift occurs in latent space and whether FANO's radial control is effective there. Both answers are yes. SITCOM's radius drifts to 1.4–1.5 and its PSNR drops by 0.94 dB in the late layers on HDR; FANO keeps the radius at 1 and shows no late-layer degradation. Without the radial potential, FANO's PSNR drops by 0.57 dB on HDR and 1.52 dB on nonlinear deblurring, confirming that the potential is even more important in latent space than in pixel space. In overall quality, FANO has higher PSNR (by 0.33 dB on HDR and 0.05 dB on nonlinear deblurring) while SITCOM achieves lower LPIPS. The two differ mainly in the radius and in the late-layer behavior, in three respects.

  1. SITCOM's radial drift causes late-layer degradation on HDR. On HDR, SITCOM's mean PSNR drops by 0.94 dB in the late layers, and more than half of the images lose over 1 dB from their peak to the final layer; on nonlinear deblurring the radius also deviates, but the PSNR barely drops. FANO's mean PSNR is highest at the final layer on both tasks, and almost no image shows such a drop. Without the potential there is likewise no late-layer degradation, so the noise coordinate itself is what prevents it.
  2. The radial potential determines the quality of the final solution. Without it, the radius deviates from 1 from the first layer on and ends at 1.10 on HDR and 0.59 on nonlinear deblurring. This deviation is not a late-layer drift and causes no late-layer drop, but the reconstruction is worse than FANO's from the early layers on: the final PSNR is 0.57 and 1.52 dB lower, and LPIPS worsens on nearly every image. In pixel space, by contrast, removing the potential changes PSNR by at most 0.14 dB (see the Question).
  3. Two differences between latent and pixel space explain these observations. First, the noise level of the last layer is about 0.04 in latent space versus 0.01 in pixel space. By the coordinate-scale law, the bound on the radial deviation for fixed-size steps on xtx_t is proportional to γ/σt\gamma/\sigma_t, consistent with SITCOM drifting less in latent space (final radius 1.4–1.5 versus 2–3 in pixel space). Second, the measurements remain in pixel space (about 2×1052\times10^5 values), whereas the noise has only about 1.2×1041.2\times10^4 dimensions, so the weight of the data term relative to the radial potential is about 16 times that in pixel space. Without the potential the radius is pushed away from 1 more easily, which makes the potential more important in latent space.

Following your suggestion, we ran HDR and nonlinear deblurring with the latent diffusion model LDM-VQ4, using the same images and measurements as in the pixel-space experiments and the same computational budget for both methods. Results on 100 images (PSNR / SSIM / LPIPS):

HDRNL deblurringFinal radius (HDR / NL)
SITCOM26.98 / 0.850 / 0.21129.66 / 0.844 / 0.2301.522 / 1.431
FANO (λrad=0\lambda_{\mathrm{rad}}=0)26.74 / 0.818 / 0.29728.20 / 0.804 / 0.3061.102 / 0.591
FANO27.31 / 0.836 / 0.26029.72 / 0.846 / 0.2451.003 / 1.002

The revision adds these experiments to Appendix E.5 (new, lines 1508–1526).

Reviewer_WFdUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines SITCOM, an annealed-decoupling solvers for zero-shot inverse problems with pixel-space diffusion priors. The authors find that in the final outer iterations the PSNR drops while the measurement residual keeps decreasing. They attribute this to radial drift: the norm of the noise component in the denoiser's input moves away from its typical Gaussian value. They propose FANO, which fixes the previous signal estimate and optimizes only the noise added to it, with a soft penalty on its norm.

Critical strengths
  1. The paper identifies a specific late outer-iteration failure of SITCOM (PSNR drops) and links it to an increase in the denoiser's input noise norm.

  2. The proposed FANO method is simple to understand and implement.

  3. The presented experiments show that FANO is competitive on linear tasks and improves on nonlinear ones.

Critical weaknesses
  1. Limited novelty relative to existing noise-geometry constraints.
    “Radial drift” is an instance of the "manifold deviation" identified by DSG (Yang et al., 2024) and others: measurement-driven updates push the denoiser input off the concentration shell of the Gaussian, degrading the denoiser. The paper should emphasize this relation rather than presenting radial drift as a new diagnosis.

  2. The proposed method integrates a shell constraint on noise (similar to DSG and others) with the SITCOM annealed-decoupling scheme, making the contribution somewhat incremental.

  3. In Figure 2, present also LPIPS per annealing iteration. Report these trends also for other tasks (e.g. linear, in the appendix). I also suggest presenting the results for the noiseless cases in figures (e.g. in the appendix) instead of only mentioning numbers in the text.

  4. Since xt=atxˉ+σtex_t=a_t\bar{x}+\sigma_t e, optimizing over ee implicitly rescales the step size w.r.t. xtx_t by a factor that shrinks with σt\sigma_t. On the other hand, SITCOM uses a constant step size γ\gamma. So the issue may be solved or greatly mitigated by scaling SITCOM step size γt=γσt\gamma_t = \gamma \sigma_t. This should be carefully examined, where γ\gamma is likely to be tuned differently than in SITCOM default.

  5. The radius optimization does not seem necessary.
    This can be observed when comparing FANO in Table 1 with the FANO-Sphere variant in Table 5 (perhaps except for HDR).
    To facilitate the comparison, please add the two ablated versions to Table 1. Alternatively, add FANO and the variant FANO-Euc to Table 5 and report there also LPIPS.

  6. Regarding the experiments, can the authors explain the gaps between the re-run baselines and their published results? For example, under the same phase retrieval setting (σy=0.05\sigma_y=0.05, 100 validation images, best of 4 runs), the re-run SITCOM is 1.5 dB (FFHQ) and 5.0 dB (ImageNet) below its published PSNR results.
    State (possibly in the appendix) the hyperparameters used for each competitor.

  7. The proposed method requires about N x K (1500 in your implementation) evaluations and differentiations through the model, which is computationally heavy. You need to report runtime and compare it with all the other competitors in the main body of the paper.

  8. For the linear problems, you should compare with simpler methods that do not require many steps or backpropagating through the model, and provide strong metrics, such as:
    DDRM, NeurIPS 2022, https://arxiv.org/abs/2201.11793
    DDPG, CVPR 2024, https://arxiv.org/abs/2312.16519
    For the nonlinear problems compare with:
    DiffRGD, ECCV 2026, https://arxiv.org/abs/2606.28417
    which you consider only for linear problems, even though it can be applied to any differentiable A.

  9. It would be useful to examine also lower observation noise level, e.g., σy=0.01\sigma_y=0.01 as in SITCOM's appendix.

Questions for authors

See above. Including examining the contribution of the implicit step size decay and the necessity of optimizing the radius.

9 comments
AuthorsOpenReview ↗

Response to Reviewer WFdU (1/5)

Thank you for the detailed comments and the concrete experimental suggestions. Your two main concerns are how radial drift relates to the manifold deviation of DSG (W1, W2) and whether scaling SITCOM's step size by σt\sigma_t would already be enough (W4). You also asked for additional figures, an ablation of the radius, the hyperparameters of the compared methods, runtimes, additional baselines, and a lower measurement noise level (W3, W5–W9). All of these experiments are done and the revision is uploaded; our point-by-point replies follow.

Unless stated otherwise, results are on 100 FFHQ images with measurement noise σy=0.05\sigma_y=0.05 and are reported as PSNR / LPIPS. Changes in the revision are marked in blue, and each reply ends with the location of the changes, including line numbers.

The main new results are:

  • FANO-Euc with λrad=0\lambda_{\mathrm{rad}}=0 (only the search variable is changed to the noise; the radial potential is switched off) outperforms SITCOM with a shell constraint by 0.92 dB on HDR and 1.05 dB on nonlinear deblurring (W2).
  • Scaling SITCOM's step size by σt\sigma_t, as you suggest, makes each update to xtx_t shrink with σt\sigma_t. With this change SITCOM improves by 1.16 dB on HDR and 1.04 dB on nonlinear deblurring, which shows that the late-layer degradation comes from a mismatch between the step size and the noise level; a theorem in the revision bounds the radial deviation as a function of σt\sigma_t (W4).
  • Under a matched budget, FANO and SITCOM take about the same time, and FANO has higher PSNR than the newly added baselines (DDRM, DDPG, and DiffRGD) and than SITCOM at σy=0.01\sigma_y=0.01 (W7–W9).

W1. Radial drift and the manifold deviation of DSG

We agree with you that radial drift and the manifold deviation identified by DSG belong to the same family of phenomena: in both, the denoiser input is pushed off the concentration shell of the Gaussian. The revision now states this relation explicitly. However, the two arise at different stages, have different causes, and call for different strategies. We trace radial drift to the search coordinate used in within-layer optimization, with both empirical and theoretical evidence.

First, they arise at different stages. DSG addresses a single guidance update per reverse step, an update that pushes the sample off the shell. Our paper addresses the within-layer optimization of annealed decoupling: once the anchor is fixed, the many optimization steps within one layer gradually move the radius of the forward noise away from its typical value.

Second, the causes differ. In the late, low-noise layers of SITCOM, the measurement residual keeps decreasing while the reconstruction quality drops. We attribute this failure to radial drift with four pieces of evidence and identify the source of the drift theoretically:

  • Layerwise tracking: the quality drop and a marked increase of the noise radius (relative to the fixed anchor) occur in the same late layers.
  • Noiseless control: without measurement noise the degradation persists and even becomes worse, so overfitting to the measurement noise cannot explain it.
  • Controlled intervention: taking the final state of the last layer of a SITCOM run and correcting only the radius, with the anchor and the direction unchanged, recovers part of the lost reconstruction quality. This separates the effect of the radius from changes in the anchor and the direction, so the evidence is interventional rather than only correlational.
  • Theory: the coordinate-scale law (Theorem 1) shows that, for a search over xtx_t with a fixed step size, the bound on the radial deviation grows as σt\sigma_t decreases. The drift therefore comes from the search coordinate, consistent with its concentration in the late layers; a search in the noise coordinate has no such amplification (see W4).

Third, the strategies differ. DSG fixes the radius at its typical value. FANO instead searches over the noise and regulates the radius with a soft potential. Adding a shell projection to SITCOM improves it by only 0.23 dB on HDR and 0.07 dB on nonlinear deblurring, whereas changing only the search variable improves it by 1.15 and 1.12 dB (see W2). Within FANO, a DSG-style hard constraint corresponds to FANO-Sphere, and freeing the radius improves HDR by a further 0.82 dB.

In the revision, Section 2.4 (lines 198–200) states how FANO differs from DSG, and Appendix C.3 (lines 1126–1127) notes how the shell-projection experiment corresponds to DSG.

AuthorsOpenReview ↗

Response to Reviewer WFdU (2/5)

W2. Is FANO a shell constraint on the noise combined with SITCOM's annealed decoupling?

FANO does include radial control, but its core contribution, and the source of most of the improvement, is changing the inner optimization variable to the forward noise. The two methods share only the outer framework of annealed decoupling (DAPS and FAST-DIPS also belong to this family). They differ within each layer, in two ways.

First, the search variables differ. SITCOM searches directly over the denoiser input xtx_t; FANO fixes the anchor and searches over the noise ee through xt=atxˉ+σtex_t=a_t\bar x+\sigma_t e, treating direction and radius separately. As explained in W1 and W4, SITCOM's late-layer degradation stems from its search coordinate, that is, from a step size that does not match the noise level. FANO changes exactly this source instead of adding a module on top of SITCOM.

Second, they control the radius differently. The shell constraint you mention is not the core of FANO: by default FANO uses a soft radial potential, and a hard shell is only the special case λrad→∞\lambda_{\mathrm{rad}}\to\infty.

To test the combination you describe directly, we projected the equivalent noise back onto the shell after every SITCOM update and compared the methods under the same budget on HDR and nonlinear deblurring. We chose these two tasks because radial drift occurs mainly on nonlinear tasks (on linear tasks SITCOM's final radius is 1.00–1.11; see W3), and they are also the tasks of the ablation in Section 4.3. FANO-Euc and SITCOM are compared on all 8 tasks in Table 7 of Appendix C.1.

The shell constraint improves SITCOM by only 0.23 dB on HDR and 0.07 dB on nonlinear deblurring, whereas FANO-Euc with λrad=0\lambda_{\mathrm{rad}}=0, which only changes the search variable, is 0.92 and 1.05 dB better than SITCOM with the shell constraint. On top of this, FANO adds the polar decomposition and the radial potential and gains another 0.70 dB on HDR. Results (PSNR / LPIPS):

HDRNL deblurring
SITCOM28.03 / 0.21629.68 / 0.237
SITCOM + shell constraint (projection after every update)28.26 / 0.19229.75 / 0.221
FANO-Euc (λrad=0\lambda_{\mathrm{rad}}=0)29.18 / 0.17130.80 / 0.182
FANO29.88 / 0.15730.83 / 0.183

In the revision, this comparison is in Appendix C.3 (new, lines 1124–1130 and Table 10).

W3. Per-layer LPIPS in Figure 2; curves for other tasks and for the noiseless case

All three additions you suggested are in the revision. The new figures show that SITCOM's LPIPS worsens together with its PSNR, that the degradation becomes more severe without measurement noise, and that radial drift occurs mainly on nonlinear tasks.

Per-layer LPIPS is now part of Figure 2(a). To make the late-layer behavior visible, panel (a) starts at annealing layer 10; before that, the curves of the two methods nearly coincide. SITCOM's LPIPS worsens in the last two layers together with its PSNR, whereas both metrics of FANO improve monotonically up to the final layer.

For other tasks, the appendix adds layerwise curves for linear tasks. On these tasks SITCOM's final-layer PSNR does not drop and its final radius stays within 1.00–1.11, so radial drift occurs mainly on nonlinear tasks.

As suggested, the noiseless case is now also shown in figures (first 40 FFHQ images). Without measurement noise SITCOM degrades even more, so the degradation is not overfitting to the measurement noise; under the same conditions FANO shows no final-layer drop, with final radius 1.01 (nonlinear deblurring) and 1.06 (HDR). On the same 40 images, the change of SITCOM from its best layer to the final layer:

SITCOMPSNR: peak → finalLPIPS: best → finalFinal radius
NL deblurring, σy=0.05\sigma_y=0.05−0.8 dB0.204 → 0.2533.0
NL deblurring, noiseless−3.4 dB0.176 → 0.3886.5
HDR, σy=0.05\sigma_y=0.05−0.3 dB0.194 → 0.2192.4
HDR, noiseless−0.5 dB0.182 → 0.2174.0

In the revision, per-layer LPIPS is in Figure 2(a) (caption, line 184), and Section 3.1 now notes that LPIPS worsens at the same time (line 218). The layerwise curves for the noiseless case and for the linear tasks are in Appendix E.3 (new, lines 1443–1451; Table 18 and Figure E.3 at lines 1405–1431, Figure E.4 at lines 1458–1475). Section 3.1 also states that radial drift is most pronounced on nonlinear tasks (lines 248–250), so it no longer suggests that all tasks degrade in the same way.

AuthorsOpenReview ↗

Response to Reviewer WFdU (3/5)

W4. Optimizing over ee rescales the step size on xtx_t by σt\sigma_t; is γt=γσt\gamma_t=\gamma\sigma_t enough for SITCOM?

Your derivation is correct: optimizing over ee amounts to multiplying the step size on xtx_t by σt\sigma_t, and step-size scaling does remove most of SITCOM's late-layer degradation. Step-size scaling validates the central claim of our analysis: the late-layer degradation stems from the mismatch between step size and noise level. FANO goes further by treating direction and radius separately, which gives another 0.69 dB on HDR.

First, with the scaled step size, SITCOM's Adam updates on xtx_t are equivalent to the Adam updates of FANO-Euc (λrad=0\lambda_{\mathrm{rad}}=0) on the noise, and each update changes xtx_t in proportion to σt\sigma_t. We implemented this in the official SITCOM code with step size γσt\gamma\sigma_t (and Adam's stability constant converted accordingly). It improves SITCOM by 1.16 dB on HDR and 1.04 dB on nonlinear deblurring, brings the final radius back to about 1, and gives almost the same results as FANO-Euc (λrad=0\lambda_{\mathrm{rad}}=0) on all five tasks. The late-layer degradation therefore comes from the mismatch between the step size and the noise level.

Second, FANO goes beyond step-size scaling. The noise coordinate allows direction and radius to be treated separately, with the direction optimized on the sphere and the radius optimized on its own and regulated by the soft potential. On HDR, FANO is 0.69 dB better than SITCOM with the scaled step size (LPIPS 0.157 vs. 0.171).

Third, the coordinate-scale law in the revision (Theorem 1) shows that, with KK Adam steps within one layer, a search over xtx_t with learning rate γ\gamma satisfies ∣rK−r0∣≤CKγ/σt|r_K-r_0|\le C_K\gamma/\sigma_t, while a search in the noise coordinate with learning rate η\eta satisfies ∣rK−r0∣≤CKη|r_K-r_0|\le C_K\eta, with constants independent of σt\sigma_t. The first bound grows as σt\sigma_t decreases, consistent with the degradation being concentrated in the late layers; the second has no such amplification by construction. With step size γσt\gamma\sigma_t, the first bound becomes CKγC_K\gamma and no longer grows as σt\sigma_t shrinks, which is why step-size scaling works. The late-layer update bound (Theorem 2) further shows that FANO's per-layer change of the estimate is at most a constant multiple of σt\sigma_t.

PSNR on five tasks:

SR ×4GaussianMotionHDRNL
SITCOM, fixed step size γ\gamma (Appendix D.1)30.6130.0932.6228.0329.68
SITCOM, step size γσt\gamma\sigma_t30.5930.6433.0329.1930.72
FANO-Euc (λrad=0\lambda_{\mathrm{rad}}=0)30.5830.6433.0329.1830.80
FANO30.6830.6533.0429.8830.83

In the revision, the coordinate-scale law and the late-layer update bound replace the former Proposition 1 (Section 3.2, lines 307–338; proofs in Appendix A.2, lines 726–807), and the step-size scaling experiment is in Appendix C.3 (new, lines 1114–1121; Table 9 at lines 1089–1094).

W5. Is the radius optimization necessary (FANO vs. FANO-Sphere)?

Your observation is correct: freeing the radius does not help on every task. On tasks whose measurements lose amplitude information (most clearly HDR), however, freeing the radius is necessary, and on the other tasks it costs nothing. FANO-Sphere is a special case of FANO, corresponding to an infinite weight on the radial potential with the radius fixed at 1, so freeing the radius adds a single scalar to the optimization.

On HDR, FANO is 0.82 dB better than FANO-Sphere (0.52 dB on ImageNet), with LPIPS 0.157 vs. 0.167. The extra error of a fixed radius concentrates on the saturated pixels, whose MSE is 13% higher: the measurements of these pixels carry no amplitude information, so the amplitude has to come from the noise. On box inpainting, FANO is also better, by 0.19 dB (FFHQ) and 0.23 dB (ImageNet). On the remaining tasks, FANO's PSNR is lower only on Gaussian deblurring (by 0.01 dB) and phase retrieval (by 0.08 dB), both within run-to-run variation, and its LPIPS is equal or better except on SR (+0.005). Results on all eight tasks:

SR ×4GaussianMotionBoxRandomPRHDRNL
FANO-Sphere30.57 / 0.22730.66 / 0.22533.03 / 0.16625.14 / 0.15634.06 / 0.12132.06 / 0.18829.06 / 0.16730.72 / 0.183
FANO-Euc30.58 / 0.22730.64 / 0.22533.03 / 0.16724.98 / 0.16033.99 / 0.12232.54 / 0.17229.18 / 0.17130.80 / 0.182
FANO30.68 / 0.23230.65 / 0.22533.04 / 0.16725.33 / 0.15634.07 / 0.12031.98 / 0.18729.88 / 0.15730.83 / 0.183

Following your second suggestion, the revision adds FANO and FANO-Euc to the table in Appendix C.1 (Table 5 in the submission, Table 7 in the revision, lines 1027–1037) and reports LPIPS.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers and the AC for their reviews and the discussion. The reviewers found that radial drift convincingly explains why SITCOM may deviate from the manifold (81MF), that our analysis is detailed and the remedy targeted (VQVX), and that FANO is simple to implement and gives clear gains on nonlinear tasks (WFdU, VQVX). During the discussion, VQVX reported no further questions and raised the score, 81MF found our explanations helpful and kept the rating of weak acceptance, and WFdU found that the new experiments address several of the requests; for the remaining step-size concern (W4), we added a latent-space comparison. The table summarizes the concerns and revisions (changes in blue in the revised PDF).

Concern (reviewer)What we didResultWhere
Is FANO SITCOM plus a shell constraint; relation to DSG and DMAP (WFdU W1–W2, 81MF W4)Added a hard shell projection to SITCOM; discussed DSG and DMAP in the related workFANO-Euc, which only changes the search variable, beats shell-constrained SITCOM by 0.92 / 1.05 on HDR / nonlinear deblurring (NL); the shell alone adds only 0.23 / 0.07Sec. 2.2, 2.4; App. C.3
Is scaling SITCOM's step size by σt\sigma_t enough? (WFdU W4 and follow-up)Ran the official SITCOM code with step size γσt\gamma\sigma_t, also in latent space (LDM-VQ4); stated the mechanism in the abstract and introductionPixel space: the late-layer degradation almost disappears (+1.16 / +1.04); this equals FANO-Euc, and FANO is still 0.69 higher on HDR. Latent space: on NL, 1.27 below fixed-step SITCOM and 1.33 below FANO; on HDR, on par with FANOAbstract, Sec. 1; App. C.3, E.5
What Theorem 1 establishes (81MF Q1)Replaced Proposition 1 with two new theoremsFor a search over xtx_t, the bound on the within-layer radial deviation grows as 1/σt1/\sigma_t; in the noise coordinate it does not depend on σt\sigma_tSec. 3.2; App. A.2
Is the radius optimization necessary? (WFdU W5, 81MF Q2)Compared FANO with FANO-Sphere (radius fixed at 1) on all eight tasksFANO is 0.82 higher on HDR (0.52 on ImageNet) and 0.19 / 0.23 higher on box inpainting (FFHQ / ImageNet); it is slightly lower only on Gaussian deblurring and phase retrieval (0.01 / 0.08)App. C.1
Contribution of the radial potential; small λrad\lambda_{\mathrm{rad}} (VQVX Q and follow-up, 81MF W2, Q2)Removed the potential on all tasks; analyzed the radius; added visual comparisonsPixel space: PSNR changes by at most 0.14, and the potential mainly improves LPIPS; latent space: removing it lowers PSNR by 0.57 / 1.52; λrad\lambda_{\mathrm{rad}} is of the same order as the Bayesian posterior weightApp. C.2, E.6
Why fix the anchor and optimize the noise? (VQVX W1)Implemented the opposite variant (fixed noise, optimized anchor) with tuned μ\mu0.58 higher PSNR on HDR but worse SSIM and LPIPS, and 1.39 lower PSNR on NL; sensitive to μ\mu, and degrades in the last two layers like SITCOMSec. 3.2; App. C.4
Latent diffusion models (VQVX W2)HDR and NL with LDM-VQ4Radial drift also occurs: SITCOM's radius drifts to 1.4–1.5 and its PSNR drops by 0.94 in the late layers on HDR; FANO keeps the radius at 1 and has 0.33 / 0.05 higher PSNR, while SITCOM has lower LPIPSApp. E.5
SITCOM with only the radial potential (VQVX W3)Kept SITCOM's objective and added only the same potential+0.63 / +0.94, and the late-layer degradation almost disappearsApp. C.3
More baselines (WFdU W8, 81MF W3)DDRM, DDPG, and DDNM on linear tasks; DiffRGD on nonlinear tasksFANO has higher PSNR than DDRM, DDPG, and DiffRGD, and than DDNM in 7 of 8 comparisons; the SVD-based methods are faster but do not apply to motion deblurring or nonlinear tasksApp. E.4
Runtime (WFdU W7)Timed ten methods on the same type of GPUUnder a matched budget, FANO and SITCOM take about the same timeSec. 4.2; App. D.3
Lower measurement noise, σy=0.01\sigma_y=0.01 (WFdU W9)Gaussian deblurring, HDR, NLPSNR 0.19 / 1.99 / 4.04 higher than SITCOM, with lower LPIPS on all three; SITCOM's final radius reaches 2.2–6.6App. E.1
Per-layer LPIPS; other tasks and the noiseless case (WFdU W3)Added per-layer LPIPS to Fig. 2(a) and curves for linear tasks and the noiseless caseSITCOM's LPIPS worsens with its PSNR in the late layers, more so without noise; radial drift occurs mainly on nonlinear tasksSec. 3.1; App. E.3
Gap to SITCOM's published results; hyperparameters (WFdU W6)Listed all hyperparameters of every compared methodSettings follow SITCOM's official code; our phase retrieval result exceeds two independent reproductions with that codeApp. B.6

In addition, the title and authors now appear at the top of page 1 (81MF W1), and the related work discusses DC-PnPDP (81MF W3). Details are in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.