acceptodds
Under review as a conference paper at ICLR 2027

WHERE THE NOISE HIDES: BLIND DENOISING FROM THE SCORE JACOBIAN’S SPECTRUM

Abstract

Blind denoising, i.e., recovering a clean signal when the noise level is unknown, is a long-standing challenge in signal processing and machine learning. Differentiating Tweedie's formula ties the Jacobian of the score function to the posterior covariance of the clean signal, and hence to the noise variance. However, conventional methods built on it assume the true score function, ignoring the approximation error that neural network training introduces. In this work, we target the noise estimation and denoising problem by analysing the effect of that error on the spectrum. We first show that when the clean signal is supported on a low-dimensional manifold, the true score Jacobian has a cluster of eigenvalues at displaced only slightly by signal manifold curvature, making the smallest eigenvalue an optimal noise estimator with the exact score. Using neural tangent kernel theory, we illustrate that score matching perturbs the learned score Jacobian with zero mean initialization randomness, even for a network that interpolates its training samples exactly. In an idealized case the resulting distribution of the Jacobian noise eigenvalues is symmetric about for any covariance of the perturbation, implying that the centre of the eigenvalue cluster is robust to learning perturbation while the smallest eigenvalue drifts to underestimate noise. A practical algorithm, JAcobian-spectrum Noise Estimation (JANE), is hence proposed to estimate from the leftmost mode of the eigenvalue density, with no knowledge of the noise level, the signal dimension, or a clean reference. Across six dataset configurations and different noise levels, JANE attains % mean relative error in estimation against % for Noise2Score and % for UNSURE, and matches an oracle step-size denoising to within dB PSNR in all dataset and noise level settings.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
blind denoising, noise estimation, score Jacobian, eigenvalue spectrum

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3nWuOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a method called JANE for blind image denoising. Specifically, under the assumption of additive white Gaussian noise, they show that the noise variance σ2\sigma^2 can be read directly from the eigenvalues of the analytical score Jacobian, defined as the Hessian of log⁡p(y)\log p(\boldsymbol{y}). Provided that the score s(y)=∇log⁡p(y)s(y) = \nabla \log p(\boldsymbol{y}) can be learned, this yields a method for blind denoising since E(x∣y)=y+σ2s(y)\mathbb{E}(x | y) = y + \sigma^2 s(y) according to Tweedie's formula. However, the authors notice that the eigenvalues of the analytical score Jacobian do not match the ones from the learned one and propose a theory to explain this phenomenon, as well as a pratical method to estimate σ2\sigma^2 from the eigenvalues of the leaned score Jacobian anyway. In the end, comparisons are made with Noise2Score and UNSURE for image denoising, showing that JANE estimates more accurately σ2\sigma^2 and so provides better reconstructions.

Critical strengths

Noise2Score (2022) showed that the noise variance σ2\sigma^2 could be estimated directly from a single noisy image yy, provided that the score s(y)=∇ylog⁡p(y)s(y) = \nabla_y \log p(y) is learned. However, it was not clear why it worked, the arguments being based only on intuitions. This paper derives a theory to explain why the noise variance σ2\sigma^2 does hide in the score, and more precisely in the spectrum of its Jacobian. I think this is a piece of theoretical contribution, even though JANE is to be seen only as a proof of concept, as in practice, I don't think it will adopted due to its very high computational burden.

Critical weaknesses

Major:

  1. In your experiments, you consider denoisers of the form fα(y)=y+αs(y)f_\alpha(y) = y + \alpha s(y), where α∈R\alpha \in \mathbb{R} is the step size that needs to be estimated and for which the optimal value is σ2\sigma^2 (assumed unknown) according to Tweedie's formula. You propose three different ways to estimate α\alpha: your method JANE, Noise2Score (that requires computing s(y)s(y) and s(y+τh)s(y+\tau h)), and the oracle parameter that you call Genie MMSE where α^=arg⁡min⁡α∥fα(y)−x∥22\hat{\alpha} = \arg \min_\alpha \Vert f_\alpha(y) - x \Vert_2^2. But you should also compare with methods that can estimate α\alpha by directly estimating the noise level σ\sigma (see for example [a, b]).

  2. For estimating σ2\sigma^2, JANE needs to compute the eigenvalues of \nabla_y s_\hat{\theta}(y_i) for i∈Si \in \mathcal{S}, where S\mathcal{S} is a sufficiently large subset of the training images. I believe that the computational cost is tremendous, all that for just estimating σ2\sigma^2... Indeed, for a 512x512 image yy, I am not even sure that computing \nabla_y s_\hat{\theta}(y) is feasible. Therefore, JANE is only applicable to very small images. Could you please provide a time comparison between all competing methods? Also, Noise2Score (2022) estimates σ2\sigma^2 with only one single image at inference, while JANE requires a subset S\mathcal{S} for estimation, it is an important difference that needs to be highlighted.

  3. The noise assumptions are very restrictive: JANE is tested for additive Gaussian noise with constant σ\sigma. What happens if the noise level is non-constant and varies for each image in the training set as in [c]?

  4. Section 4 is particularly hard to follow. To be honest, I did not understand it. I think a complete rewrite is necessary to make it clear. In particular, some terms are obscur to me:

    • Line 268, what is the residual kernel K~\tilde{K} exactly?
    • Line 269, what is the cross-output derivative covariance Σ∂mm′\boldsymbol{\Sigma}_\partial^{mm'}? What does mm and m′m' even represent?
    • Line 270, what does vec⁡(Δ)\operatorname{vec}(\Delta) stand for?
    • Line 282, what is the subset S\mathcal{S}? Is it the one defined in line 377?
    • Some sentences are enigmatic such as "For independent readouts the blocks are diagonal; with weight sharing across output positions, as in convolutional networks"...

[a] Stanislav Pyatykh, Jurgen Hesser, and Lei Zheng. Image noise level estimation by principal component analysis. IEEE TIP, 2013.

[b] Guangyong Chen, Fengyuan Zhu, and Pheng Ann Heng. An efficient statistical method for image noise level estimation. ICCV, 2015.

[c] Sebastien Herbreteau and Etienne Meunier. Divergence-free neural networks with application to image denoising. ICLR, 2026.

Minor:

  1. You could specify that the λi\lambda_i are the eigenvalues of Hy\mathbf{H}_\mathbf{y} in equation (6). Also, for improved clarity, please explain that λi≥−1/σ2\lambda_i \geq - 1 / \sigma^2 line 147 comes from the fact that νi≥0\nu_i \geq 0 since a covariance matrix always exhibits non-negative eigenvalues.

  2. Missing reference for practical learning of s(y)s(y) (notably used in Noise2Score): Lim, J. H., Courville, A., Pal, C., & Huang, C. W. AR-DAE: Towards unbiased neural entropy gradient estimation. ICML, 2020.

  3. Please add the PSNR values of the denoised images in Figure 3.

Questions for authors
  1. Line 050, you say that UNSURE can estimate σ2\sigma^2 from a learned score network. Although I am familiar with UNSURE, I don't see what you mean. Can you explain how UNSURE estimates σ2\sigma^2? Because if it could, it would have the same performance as Noise2Score, but it does not seem to be the case (see [c]). If I am not mistaken, line 090, you mention that it is learned from the score norm but (1nEy∥∇ys(y)∥22)−1(\frac{1}{n}\mathbb{E}_y \Vert \nabla_y s(y) \Vert_2^2)^{-1} only estimates the Lagrange multiplier η^\hat{\eta}, not σ2\sigma^2.

  2. Why choosing σ∈\sigma \in {12, 25, 51, 77}? It seems weird. Usually, people use σ∈\sigma \in {15, 25, 50, 75}.

  3. In order to denoise images of size higher than 64x64, I wonder if it is possible to estimate the value of σ2\sigma^2 on randomly chosen patches of size 32x32 of a single noisy image yy (to form the subset S\mathcal{S})? Once σ2\sigma^2 is estimated, you could simply output y + \hat{\sigma}^2 s_\hat{\theta}(y). Indeed, I think the computational burden of JANE is mainly on the estimation of σ2\sigma^2. If you could provide PSNR results on standard test datasets such as the ones used in [c], it would strengthen the paper.

  4. Why is UNSURE so bad on Figure 3? Do you have explanations for that? What if you use the method recommended by the authors of UNSURE, that is searching for a saddle point? (the authors of UNSURE indeed state in their paper that they obtained slightly worse reconstruction results in their experiments with the score-approach compared to the saddle-point approach).

  5. Line 406, what does "the color configuration uses an ensemble of 32 networks" mean?

[c] Sebastien Herbreteau and Etienne Meunier. Divergence-free neural networks with application to image denoising. ICLR, 2026.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 3nWu (1/7)

Thank you very much for your time and review! Please also see our general response above, which summarizes all revisions and the main results. We hope that our clarifications and additional experimental results have adequately addressed your concerns. We would greatly appreciate it if you could reconsider your initial assessment and update your rating accordingly.

Major 1. In your experiments, you consider denoisers of the form fα(y)=y+αs(y)f_\alpha(y) = y + \alpha s(y), where α∈R\alpha\in\mathbb{R} is the step size that needs to be estimated and for which the optimal value is σ2\sigma^2 (assumed unknown) according to Tweedie's formula. You propose three different ways to estimate α\alpha: your method JANE, Noise2Score (that requires computing s(y)s(y) and s(y+τh)s(y+\tau h)), and the oracle parameter that you call Genie MMSE where α^=arg⁡min⁡α∥fα(y)−x∥2\hat\alpha = \arg\min_\alpha \|f_\alpha(y) - x\|^2. But you should also compare with methods that can estimate α\alpha by directly estimating the noise level σ\sigma (see for example [a, b]).

We thank the reviewer for the suggestion. During the revision we have extended the evaluation to broader classes of blind denoising methods, and the classical noise-level estimators of Pyatykh et al. (2013) [a] and Chen et al. (2015) [b] are now among the benchmarks. We have cited both papers in the revised manuscript: Chen et al. in the related work (Section 2.1), and both in the experimental setup of Section 6 and in Appendix D.1, where their settings are described. They are run on the same noisy images and plugged into the same Tweedie step with the same score network, so that only the noise estimate differs. Averaged over the 48 settings, JANE's error in σ2\sigma^2 is lower by a factor of 8.88.8 than that of Chen et al. and 9.39.3 than that of Pyatykh et al., and its PSNR is on average at least 1.11.1 dB higher (Tables 1, 12 and 13). JANE's PSNR is equal to or higher than both in 47 of the 48 settings. Both estimators rely on local patch statistics, which can fail badly. On the channel data, for example, Chen et al. is on average 6.96.9 dB below the oracle. On colour images, the estimate of Chen et al. is closer to σ2\sigma^2 than JANE's in 14 settings, but JANE still denoises equally well or better, with an equal or higher PSNR in 13 of them.

Changed text: Section 2.1 (citation of Chen et al.); Section 6 (setup, Table 1 and results); Appendix D.1; Appendix D.4 (Tables 12 and 13).

AuthorsOpenReview ↗

Response to Reviewer 3nWu (2/7)

Major 2. For estimating σ2\sigma^2, JANE needs to compute the eigenvalues of ∇ys^θ(yi)\nabla_y \hat s_\theta(y_i) for i∈Si \in S, where SS is a sufficiently large subset of the training images. I believe that the computational cost is tremendous, all that for just estimating σ2\sigma^2... Indeed, for a 512x512 image yy, I am not even sure that computing ∇ys^θ(y)\nabla_y \hat s_\theta(y) is feasible. Therefore, JANE is only applicable to very small images. Could you please provide a time comparison between all competing methods? Also, Noise2Score (2022) estimates σ2\sigma^2 with only one single image at inference, while JANE requires a subset SS for estimation, it is an important difference that needs to be highlighted.

We thank the reviewer for the comment. Indeed, the eigendecomposition of the full Jacobian was one of the most important limitations of our original design, and it limited the previous submission to small images. During the revision we have addressed this in three ways.

  • Scalable variants. We have included JANE-crops, which builds the spectrum from the Jacobian blocks of random crops of each image and follows your suggestion in Question 3, and JANE-SLQ, which approximates the spectrum by stochastic Lanczos quadrature from Jacobian–vector products. Neither of them forms the full Jacobian. With JANE-SLQ, building the spectrum costs 2.12.1 to 2.32.3 GPU-seconds per 256×256256\times256 image. Due to the limited time and computational resources of the revision, we have not been able to finish the evaluation on 512×512512\times512 images. However, the cost of JANE-SLQ per image is mm Jacobian–vector and vector–Jacobian products plus O(m2D)\mathcal{O}(m^2D) operations (Table 7), and for a convolutional network both terms grow only linearly with the signal dimension DD. A 512×512512\times512 image has four times as many pixels as a 256×256256\times256 one, so we expect about 8 to 9 GPU-seconds per image, that is, about 15 GPU-minutes for 100 pooled images. This is still far below the cost of training the score network, which already takes more than 12 GPU-hours at 256×256256\times256 (Table 3).
  • Time comparison (Table 3, and every configuration in Table 15). Training dominates the cost of JANE-SLQ and of every baseline. The score network is trained once and shared by JANE and Noise2Score, so considering the training time, the overall time cost of JANE-SLQ is within 4% of that of Noise2Score on every configuration. The per-image estimation of JANE-SLQ is still slower than that of Noise2Score, but with training included the total time of the two methods is similar. For Neighbor2Neighbor, Noise2Self and UNSURE, the cost is their own training, from a few minutes to about 18 hours depending on the method and configuration.
  • A single image. JANE does not need a subset, and it works even with a single image's eigenspectrum, which we now state explicitly. Estimated from each image alone, on the six small configurations, JANE's average error in σ2\sigma^2 is 3.63.6%, lower by a factor of 2.22.2 than that of Noise2Score, which also uses a single image at inference. Its estimate is closer to σ2\sigma^2 than that of Noise2Score in 23 of 24 settings, and its PSNR is on average within 0.030.03 dB of denoising with the true σ2\sigma^2 (Table 14). Pooling more observations further reduces the spread of the estimate (Table 9).

Changed text: Section 5; Table 3 and the last two paragraphs of Section 6; Appendix D.4 (Tables 14 and 15); Appendix E.

AuthorsOpenReview ↗

Response to Reviewer 3nWu (3/7)

Major 3. The noise assumptions are very restrictive: JANE is tested for additive Gaussian noise with constant σ\sigma. What happens if the noise level is non-constant and varies for each image in the training set as in [c]?

We thank the reviewer for the comment. The per-image noise setup is a new scenario for us, and we have tried our best to be close to the standard workflow in [c] within the limited revision time and computational resources. In the revised manuscript, we have added experiments where the noise level of each image is drawn uniformly from [5,50][5,50] (Table 2). JANE works in this setting without change, because each image is denoised with the estimate from its own spectrum, even though per-image noise violates the noise model of our analysis. JANE's PSNR is on average 0.150.15 dB below that of the oracle that knows every σi\sigma_i. Averaged over the six configurations, it is at least 1.11.1 dB higher than every baseline except Noise2Score (Table 2). Noise2Score is comparable in PSNR, and JANE estimates the noise more accurately, with a mean error of 11.911.9% against 14.314.3% (11.311.3% for JANE-SLQ).

Our noise range starts at 5 rather than 0 for the following reason. By Equation (6), the noise eigenvalues of the score Jacobian lie at about −1/σ2-1/\sigma^2, so they diverge as σ→0\sigma\to0: in the directions without signal, the noisy samples concentrate at a single point. When σ>0\sigma>0 and all images share the same noise level, pooling the spectra of many images still locates the peak regardless of the width of the spectrum. With per-image noise, however, each image has to be estimated from its own spectrum, and for a noise level close to 0 it is very hard to locate a peak from the limited number of eigenvalues of one image, spread over such a wide range. As long as the noise level is bounded away from 0, the width of the spectrum stays bounded and the peak can be located.

We also agree that CENSURE [c] should be evaluated in this scenario. Due to the limited time and computational resources of the revision, we have not been able to finish this evaluation. We have therefore added it, together with the standard training and test sets of [c], to the evaluation plan in the limitations (Appendix E), and have cited this insightful work.

Changed text: Section 6 (paragraph on single-image and per-image estimation); Table 2; Appendix E.

Reviewer_zLbdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper estimates the unknown noise level from noisy observations and then using this estimate for denoising. It analyzes how curvature and errors in the Jacobian of the learned score function affect the estimation. Based on the analysis, it proposes JANE, which estimates the noise level by finding the leftmost mode of the eigenvalue density

Critical strengths
  1. The paper analyzes how curvature and errors in the learned score Jacobian affect noise estimation. It gives a bound on the curvature error and studies the effect of random initialization on the learned Jacobian. This helps explain why using the minimum eigenvalue can fail and provides motivation for the proposed method.
Critical weaknesses
  1. The theory does not fully justify the choice of the leftmost peak. Proposition 2 proves symmetry under idealized conditions, but it does not prove that the noise eigenvalue distribution is unimodal. A symmetric distribution can have two peaks on either side of the center, so the leftmost peak may not be at λ0\lambda_0. It leaves a gap in the justification of the main estimation rule.

  2. The theory does not provide a finite-sample error bound for the peak estimated from a single trained network’s pooled eigenvalues. It also does not study how the peak location and noise estimation error change with the number of pooled samples within the same setting.

Questions for authors
  1. Under what conditions does pooling more samples reduce noise estimation error? Could authors provide a bound relating the required sample size to this error?

  2. Could authors show how the peak location and noise estimation error change as the number of pooled samples increases, while keeping the trained network fixed? This would help assess how sensitive JANE is to the sample size and whether the chosen sample sizes are sufficient.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zLbd (1/2)

Thank you very much for your time and review! Please also see our general response above, which summarizes all revisions and the main results. We hope that our clarifications and additional experimental results have adequately addressed your concerns. We would greatly appreciate it if you could reconsider your initial assessment and update your rating accordingly.

Weakness 1. The theory does not fully justify the choice of the leftmost peak. Proposition 2 proves symmetry under idealized conditions, but it does not prove that the noise eigenvalue distribution is unimodal. A symmetric distribution can have two peaks on either side of the center, so the leftmost peak may not be at λ0\lambda_0. It leaves a gap in the justification of the main estimation rule.

We thank the reviewer for this insightful comment, which has helped us consolidate the theory. In the revised manuscript, the choice of the leftmost peak is justified in three steps.

  • Proposition 2(i) shows that, when the other factors are set aside, the initialization-dependent random part of the Jacobian error perturbs the noise eigenvalues into a law that is symmetric about λ0=−1/σ2\lambda_0=-1/\sigma^2, for any covariance of the perturbation. Proposition 2(ii) states that if this law has a single peak, the peak is at λ0\lambda_0. This connects noise estimation to peak finding. Since the signal eigenvalues lie above −1/σ2-1/\sigma^2 by Equation (6), the noise cluster is the leftmost one, so its peak is the leftmost peak of the spectrum (Section 5).
  • Remark 3 uses a loose relaxation to explain why a single peak is expected. To first order, each noise eigenvalue is perturbed by its own centred Gaussian, so the law of the noise eigenvalues is a mixture of Gaussians with a common centre λ0\lambda_0. Such a mixture is symmetric and has a single peak at λ0\lambda_0.
  • Since this loose relaxation might not be exact in practical scenarios, we have validated the property on every constant-noise-variance spectrum used in the paper. Appendix C.6 (Figure 4) shows the spectra from which JANE estimates the noise level in all 48 combinations of configuration and noise level, small and 256×256256\times256 images. In every one of them, the noise side has exactly one peak, and it is the peak JANE identifies and locates. In the toy model of Appendix C.7, where the true Jacobian is known, the mode and the median of the noise cluster differ by at most 0.210.21%, as expected for a symmetric single-peaked law.

Changed text: Section 4 (Proposition 2 and Remark 3); Appendix C.6 (Figure 4); Appendix C.7.


Weakness 2. The theory does not provide a finite-sample error bound for the peak estimated from a single trained network's pooled eigenvalues. It also does not study how the peak location and noise estimation error change with the number of pooled samples within the same setting.

Thanks for the comment. We agree that it is important to understand how pooling affects the located peak, both to assess the reliability of JANE and to choose the number of pooled observations. During the revision, we have studied this in the same setting, with the trained network fixed (Appendix D.3, Table 9). For each of the six small configurations, the default subset is split into disjoint blocks of ∣S∣|\mathcal{S}| observations, from a single observation up to the full subset, and JANE is applied to each block. Since σ^2=−1/μ^\hat\sigma^2=-1/\hat\mu, the error in σ^2\hat\sigma^2 directly measures the error of the located peak. The mean error over the blocks shows how accurate a peak estimated from ∣S∣|\mathcal{S}| observations is, and the standard deviation of σ^2/σ2\hat\sigma^2/\sigma^2 across the blocks shows how much the estimate varies from one subset of that size to another. Together they serve as an empirical finite-sample error bound for the located peak.

  • With a single observation, the mean error is 2.862.86% averaged over the configurations, and the standard deviation across blocks is 2.32.3% on average.
  • With 10 observations, the mean error is 2.272.27%, and the standard deviation is 0.370.37% on average.
  • From 100 observations on, the mean error is 2.172.17% to 2.212.21%, and the standard deviation is 0.140.14% on average with 100 observations, falling to 0.050.05% with 1000.

Pooling therefore mainly reduces the variance of the estimate, while beyond about 100 observations the mean error barely changes, as it reaches the remaining bias of the noise cluster. This also gives a simple rule for choosing ∣S∣|\mathcal{S}|: about 100 observations are enough, which is the number JANE-crops and JANE-SLQ pool in Table 1. Even a single observation already gives a usable estimate (Table 14).

Changed text: Appendix D.3 (Table 9).

AuthorsOpenReview ↗

Response to Reviewer zLbd (2/2)

Question 1. Under what conditions does pooling more samples reduce noise estimation error? Could authors provide a bound relating the required sample size to this error?

Thanks for the question. Pooling helps when the observations share the same noise level. In that case each observation's noise eigenvalues have a symmetric law with the same centre λ0\lambda_0, and the pooled spectrum is a mixture of these laws. The mixture keeps the centre and the single peak, and pooling only reduces the sampling error of the located peak. This reduction saturates once that sampling error is small compared with the remaining bias of the cluster, which in our experiments happens at about 100 observations (Table 9). Regarding the bound, we have measured the error band of the located peak directly, with the network fixed (Table 9). The standard deviation of σ^2/σ2\hat\sigma^2/\sigma^2 across disjoint blocks of ∣S∣|\mathcal{S}| observations is on average 2.32.3% with a single observation, 0.370.37% with 10 observations and 0.140.14% with 100 observations, while the mean error on the six small configurations settles at about 2.22.2%. For a given ∣S∣|\mathcal{S}|, this band can serve as an empirical finite-sample error bound. In terms of the required sample size, from 10 observations on the mean error is within about 0.10.1 percentage points of its limit, and about 100 observations also reduce the standard deviation to 0.140.14% on average.

Changed text: Appendix C.5 (paragraph on pooling after the proof of Proposition 2); Appendix D.3 (Table 9).


Question 2. Could authors show how the peak location and noise estimation error change as the number of pooled samples increases, while keeping the trained network fixed? This would help assess how sensitive JANE is to the sample size and whether the chosen sample sizes are sufficient.

Thanks for the suggestion, which we have followed. With the trained network fixed, we have split the default subset of each of the six small configurations into disjoint blocks of ∣S∣|\mathcal{S}| observations and applied JANE to each block (Table 9). The mean error falls from 2.862.86% with one observation to 2.212.21% with 100, and the standard deviation across blocks from 2.32.3% to 0.140.14% on average. Beyond about 100 pooled observations the error barely changes, and the default subsets of exact JANE (1,000 to 5,000 observations) are well above that, while JANE-crops and JANE-SLQ use 100. The same holds for the two new variants (Tables 10 and 11): averaged over the configurations, the error of JANE-SLQ stays between 2.272.27% and 2.602.60% from 10 pooled images on, and that of JANE-crops between 2.872.87% and 2.892.89% over 1 to 64 crops per image. JANE and all its variants also work from a single observation. Estimated from each image alone, its PSNR is on average within 0.030.03 dB of denoising with the true σ2\sigma^2 (Table 14).

Changed text: Appendix D.3 (Tables 9 to 11); Appendix D.4 (Table 14).

Thank you for the detailed response and additional experiments. The sample-size sensitivity analysis addresses my concern about the effect of pooling more observations. Some of my theoretical concerns, including unimodality and finite-sample error bounds, are addressed mainly through empirical evidence.

One remaining concern is that observations share the same trained network, so their Jacobian errors may contain a common bias. While Proposition 2 establishes symmetry over network initializations, this does not necessarily hold for a fixed network. Therefore, pooling more observations may reduce variance without eliminating this bias.

AuthorsOpenReview ↗

Follow-up Response to Reviewer zLbd

Thank you very much for the follow-up, and for acknowledging our revisions and additional experiments. We agree that, because all pooled observations come from one trained network, pooling more of them reduces the variance of the estimate but cannot remove an offset that these observations share.

To measure this offset, we have added Table 17 (Appendix D.4) to the revised manuscript. For each of the six small configurations and each σ∈15,25,50,75\sigma\in\\{15,25,50,75\\}, we trained four score networks with different random seeds on the same noisy images, keeping the noise realization, the pooled subset S\mathcal{S} and all other training settings fixed. The four seeds include the one used for all other results in the paper. We applied JANE to each network exactly as in the paper, and Table 17 reports the mean error ∣σ^2/σ2−1∣|\hat\sigma^2/\sigma^2-1| over the four seeds and the standard deviation of σ^2/σ2\hat\sigma^2/\sigma^2 across them.

The results show that each single trained network gives nearly the same estimate, so the part of the offset that depends on the trained network is small. Averaged over the 24 settings, the standard deviation across seeds is 0.220.22 %, compared with a mean error of 2.192.19 %. In every setting it is at most about one fifth of the mean error, and typically about one tenth. In addition, the network used in the paper is a typical one. Averaged over the 24 settings, its error is 2.182.18 %, compared with 2.192.19 % for the mean over the four seeds. The reported accuracy therefore depends little on which network is trained.

The remaining part of the offset is shared by all networks trained in the same way on the same data. It corresponds to the systematic residual of training described in Section 4. Since every result in the paper is obtained from a single trained network, this shared offset is already reflected in all reported numbers, such as the average error over the 48 settings and the comparisons with the baselines in Section 6. Its effect on denoising is small, as JANE's PSNR is on average within 0.010.01 dB of that of Genie (Section 6).

We hope that this additional experiment addresses your remaining concern. If so, we would greatly appreciate it if you could kindly consider updating your rating. Thank you very much again for your valuable time and the insightful comments!

Reviewer_HWPqOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces JANE a method for blind denoising that estimates unknown additive Gaussian noise variance from the eigenvalue spectrum of a learned score network’s Jacobian. The theoretical analysis connects the score Jacobian to posterior covariance and argues that noise-related eigenvalues cluster around −1/σ2-1/\sigma^2. An idealized perturbation analysis motivates estimating noise variance from the leftmost mode of the pooled eigenvalue distribution. The method is evaluated across six configurations covering CIFAR-10, ImageNet-derived patches, CUB bird images, and a synthetic wireless-channel dataset. The work combines theoretical analysis with empirical evaluation, although the practical method relies on assumptions about the eigenvalue density that are not fully guaranteed by the theory.

Critical strengths
  1. The connection between noise variance, posterior covariance, and the score-Jacobian spectrum provides an interesting theoretical foundation for blind noise estimation.
  2. JANE achieves lower reported variance-estimation error than Noise2Score and UNSURE, with denoising performance close to the oracle step-size baseline. 3, The method does not require clean reference images or prior knowledge of the signal dimension.
Critical weaknesses
  1. The estimator relies on the leftmost mode of the pooled eigenvalue distribution, but the theoretical analysis does not guarantee that this mode is consistently identifiable in practical settings. The sensitivity to density estimation and peak-selection parameters is not sufficiently established.
  2. Computing the full Jacobian eigen decomposition has cubic complexity in the Jacobian dimension, substantially limiting applicability to high-resolution images. The paper acknowledges this limitation but does not evaluate scalable approximations.
  3. Although the experiments cover six configurations, the breadth of validation is limited, particularly for color images, where an ensemble is used. The evaluation also leaves open how well JANE compares with broader classes of blind-denoising methods.
Questions for authors
  1. How sensitive is JANE to the KDE bandwidth, peak threshold, and other mode-selection parameters? Do the results remain stable across reasonable choices?
  2. Can the authors quantify how deterministic residuals, signal–noise block coupling, and deviations from sign symmetry affect the estimated noise variance?
  3. Have approximate or iterative spectral methods been evaluated to reduce the computational cost of full Jacobian eigen decomposition?
  4. How does JANE perform on a broader range of color-image datasets and against blind-denoising methods that do not explicitly estimate noise variance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HWPq (1/4)

Thank you very much for your time and review! Please also see our general response above, which summarizes all revisions and the main results. We hope that our clarifications and additional experimental results have adequately addressed your concerns. We would greatly appreciate it if you could reconsider your initial assessment and update your rating accordingly.

Weakness 1. The estimator relies on the leftmost mode of the pooled eigenvalue distribution, but the theoretical analysis does not guarantee that this mode is consistently identifiable in practical settings. The sensitivity to density estimation and peak-selection parameters is not sufficiently established.

We thank the reviewer for this comment. During the revision we have consolidated the theory and added a sensitivity analysis, which address the two parts of the comment.

Section 4 now starts by stating that the violation of the eigenvalue bound in Section 3 is caused by the unconstrained Jacobian of the learned score, and analyzes the Jacobian error by separating the systematic residual of training from the random part that depends on the initialization. Proposition 2 shows that, when the other factors are set aside, the random part perturbs the noise eigenvalues into a law that is symmetric about −1/σ2-1/\sigma^2. If this law has a single peak, the peak is therefore at −1/σ2-1/\sigma^2, and noise estimation becomes peak finding, which is the general operation of JANE. Remark 3 uses a loose relaxation to explain why a single peak is expected. To first order, each noise eigenvalue receives its own centred Gaussian perturbation, and a mixture of Gaussians with a common centre has a single peak. Since this relaxation might not be exact in practical scenarios, we have validated the property on every constant-noise spectrum used in the paper (Appendix C.6). In all 48 combinations of configuration and noise level, the noise side has exactly one peak, and it is the peak JANE identifies and locates.

We have added a sensitivity analysis of every parameter of JANE (Appendix D.3). Table 8 changes each parameter of the peak locator on its own: the number of histogram bins (250 to 1000), the histogram smoothing (1 to 5 bins), the peak-height threshold (1% to 15%), the window around the coarse peak (25 to 100 bins), and the KDE bandwidth (halved and doubled). The mean estimation error over the six small configurations stays between 2.162.16% and 2.232.23%, against 2.182.18% at the default setting. The threshold and the smoothing leave the error unchanged to two decimals. Therefore, the same settings are used for all configurations, and none of them is tuned for a particular dataset.

Changed text: Section 4 (Proposition 2 and Remark 3); Appendix C.6 (Figure 4); Appendix D.3 (Table 8).

AuthorsOpenReview ↗

Response to Reviewer HWPq (2/4)

Weakness 2. Computing the full Jacobian eigen decomposition has cubic complexity in the Jacobian dimension, substantially limiting applicability to high-resolution images. The paper acknowledges this limitation but does not evaluate scalable approximations.

We thank the reviewer for the comment. Indeed, the eigendecomposition of the full Jacobian was one of the most important limitations of our original design. During the revision, we have investigated scalable approximations, and have included two new variants of JANE as a part of our work (Section 5):

  • JANE-crops builds the eigenspectrum from the principal blocks of the full-image Jacobian at KK random crops of each image (for example 32×3232\times32 pixels), so that its computational complexity is determined only by the crop size and number, not by the dimension of the whole image.
  • JANE-SLQ uses stochastic Lanczos quadrature to approximate the spectrum with fewer degrees of freedom, using only Jacobian–vector and vector–Jacobian products, so that neither the Jacobian nor its eigendecomposition is formed. Section 5 explains the algorithm step by step, and why the moments of the approximation match those of the exact spectrum in expectation.

Both variants run on 256×256256\times256 images, in grayscale and in colour (D=196,608D=196,608), and neither needs the spectra of all images. For example, in Table 1 they pool only 100 images. JANE-SLQ costs 2.12.1 to 2.32.3 GPU-seconds per 256×256256\times256 image, so the 100 pooled images take less than 4 GPU-minutes, and considering the training of the score network, its overall time cost is comparable with that of Noise2Score (Table 3).

Both variants keep the accuracy of JANE, compared with exact JANE and with the baselines:

  • Against exact JANE. On the small images, where the exact spectrum is available, the error of JANE-SLQ averaged over the configurations is 2.272.27% to 2.602.60% from 10 pooled images on, against 2.182.18% for exact JANE (Appendix D.3, Table 10), and over the 48 settings its mean PSNR is within 0.010.01 dB of that of JANE.
  • Against the baselines. On the 24 settings at 256×256256\times256 (Tables 1, 12 and 13), the average error in σ2\sigma^2 of JANE-crops is 2.92.9%. This is lower by a factor of 1.61.6 than that of Noise2Score, the strongest baseline, and by 2.72.7 to 9.69.6 than those of the others. Its PSNR is on average within 0.010.01 dB of the oracle. The average error of JANE-SLQ is 3.43.4%, lower by a factor of 1.41.4 than that of Noise2Score and by 2.32.3 to 8.28.2 than those of the others. Its PSNR is on average within 0.020.02 dB of the oracle.

In the revised manuscript, we have added both variants, introduced their algorithms in Section 5, and evaluated their performance and sensitivity in Section 6 and Appendix D.3.

Changed text: Section 5 (JANE-crops and JANE-SLQ); Table 3 and its paragraph in Section 6; Appendix D.2 (Table 7); Appendix D.3.


Weakness 3. Although the experiments cover six configurations, the breadth of validation is limited, particularly for color images, where an ensemble is used. The evaluation also leaves open how well JANE compares with broader classes of blind-denoising methods.

We thank the reviewer for the comment, and agree with both points. During the revision we have extended the evaluation in both directions.

Breadth and colour. We have extended the evaluation from six to twelve configurations, adding ImageNet, CUB-200 and Waterloo Exploration at 256×256256\times256 in grayscale and in colour, and have replaced the ensemble of the colour configuration with a single network everywhere. Over the 48 settings, JANE's PSNR (exact JANE on the small images and JANE-crops on the 256×256256\times256 images) is on average within 0.010.01 dB of the oracle, and this also holds on the colour configurations alone (Tables 1, 12 and 13).

Other blind denoisers. We have added five benchmarks: the noise-level estimators of Chen et al. (2015) and Pyatykh et al. (2013), which are plugged into the same Tweedie step with the same score network, and Neighbor2Neighbor, Noise2Self and the saddle-point version of UNSURE, which train their own network and do not need σ2\sigma^2. Averaged over the 48 settings, the noise estimation error of JANE is 2.52.5%, about half that of the strongest baseline, Noise2Score, and its PSNR is at least 0.80.8 dB higher than those of these three blind denoisers (Tables 1, 12 and 13). JANE matches or exceeds the PSNR of every blind baseline in 42 of the 48 settings.

Changed text: Section 6 (setup, Table 1 and results); Appendix D.1; Appendix D.4 (Tables 12 and 13).

AuthorsOpenReview ↗

Response to Reviewer HWPq (3/4)

Question 1. How sensitive is JANE to the KDE bandwidth, peak threshold, and other mode-selection parameters? Do the results remain stable across reasonable choices?

Thanks for the question. Yes, the results remain stable. We have added a sensitivity analysis of every parameter of JANE in the revised manuscript. Each parameter of the peak locator is changed on its own over a wide range: the number of histogram bins (250 to 1000), the histogram smoothing (1 to 5 bins), the peak-height threshold (1% to 15%), the window around the coarse peak (25 to 100 bins), and the KDE bandwidth (halved and doubled). Over all these changes, the mean estimation error on the six small configurations stays between 2.162.16% and 2.232.23%, against 2.182.18% at the default setting. The number of crops of JANE-crops and the number of Lanczos steps of JANE-SLQ change the error averaged over the configurations by less than 0.50.5 percentage points once at least 10 images are pooled.

Changed text: Appendix D.3 (Tables 8, 10 and 11).


Question 2. Can the authors quantify how deterministic residuals, signal–noise block coupling, and deviations from sign symmetry affect the estimated noise variance?

We thank the reviewer for this question. Section 4 now separates the systematic residual of training from the random part that depends on the initialization, and lists the factors that Proposition 2 sets aside: the curvature of the signal, the coupling between the noise block and the signal directions, and the systematic residual. Please note that these factors cannot be separated on images, because that requires the true score Jacobian. We have therefore quantified them in the analytical-score toy of Section 3 (a 2-torus in R4\mathbb{R}^4), where the true Jacobian is known in closed form and the training target is exact. For each noise level and data seed, we trained 16 networks that share their training data and differ only in their initialization. Their average error gives the deterministic residual, and each network's deviation from that average gives the random part. The results over six noise levels and ten data seeds are as follows.

  • The random part alone, added to the true Jacobian, changes the estimate by at most 0.050.05 percentage points.
  • The deterministic residual changes it by −0.59-0.59 to +0.45+0.45 percentage points, depending on the noise level.
  • With every term present, a single network's estimate differs from the estimate from the true Jacobian by −0.51-0.51 to +0.39+0.39 percentage points, depending on the noise level.
  • The coupling with the signal directions changes it by less than 0.0010.001 percentage points.
  • Flipping the sign of the random part changes it by at most 0.040.04 percentage points on average, and the estimates from the mode and the median of the noise cluster differ by at most 0.210.21 percentage points.

Results show that none of these factors moves the peak estimate by more than 0.60.6 percentage points, while the minimum eigenvalue is off by −1.2-1.2% to −15.6-15.6% on the same networks because of the perturbation in Proposition 2. The peak estimate is therefore stable with respect to all the mentioned factors.

Changed text: Last paragraph of Section 4; Appendix C.7 (Tables 4 and 5).

AuthorsOpenReview ↗

Response to Reviewer HWPq (4/4)

Question 3. Have approximate or iterative spectral methods been evaluated to reduce the computational cost of full Jacobian eigen decomposition?

Thanks for the question. Following this suggestion, we have investigated iterative spectral methods during the revision, and have included JANE-SLQ as a part of our work. It approximates the eigenvalue density by stochastic Lanczos quadrature, with m=100m=100 Lanczos steps and one probe per image, using only Jacobian–vector and vector–Jacobian products, so that neither the Jacobian nor its eigendecomposition is formed. Appendix D.3 reports its sensitivity to mm and to the number of pooled images (Tables 10 and 11). Across the 48 settings of Tables 1, 12 and 13, its PSNR is on average within 0.020.02 dB of the oracle, and its average error in σ2\sigma^2 is 2.92.9%, lower by a factor of 1.71.7 to 8.08.0 than those of the baselines. Considering the training of the score network, its overall time cost is comparable with that of Noise2Score (Table 3).

Changed text: Section 5 (JANE-SLQ); Section 6; Appendix D.3 (Tables 10 and 11).


Question 4. How does JANE perform on a broader range of color-image datasets and against blind-denoising methods that do not explicitly estimate noise variance?

Thanks for the question. During the revision, we have extended the evaluation to three 256×256256\times256 colour datasets in addition to CUB-32C, all with a single network, and have added Neighbor2Neighbor, Noise2Self and the saddle-point version of UNSURE, which do not estimate σ2\sigma^2. On the colour configurations, JANE's PSNR is the best or tied for the best in 13 of the 16 settings. Compared with each of these three methods separately, JANE's PSNR is on average at least 0.40.4 dB higher, and it is equal or higher than that of Neighbor2Neighbor and of Noise2Self in 15 of the 16 settings each, and than that of UNSURE in all of its settings.

Changed text: Section 6; Appendix D.4 (Tables 12 and 13).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revision and Response to the Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the area chair and all reviewers for the valuable time and insightful reviews. The comments have allowed us to revise the manuscript thoroughly, and to extend our design significantly to support larger and more practical use cases. A summary of the revisions is listed below; all changed text is highlighted in blue in the revised PDF.

  • Scalable JANE variants (Section 5). The reviewers pointed out that the eigendecomposition of the full Jacobian has high computational complexity, and kindly suggested exploring iterative spectral methods or random crops. Following these suggestions, we have added two variants. JANE-crops builds the spectrum from the Jacobian blocks of random crops, so its cost is set by the crop size and number, not by the image size. JANE-SLQ approximates the spectrum by stochastic Lanczos quadrature without forming the Jacobian. Considering the training of the score network, the total time cost of JANE-SLQ is comparable to that of Noise2Score, and JANE can now be evaluated on 256×256256\times256 colour images.
  • Single-image and per-image noise estimation (Section 6, Appendix D). The reviewers pointed out that Noise2Score estimates the noise level from a single image at inference, whereas JANE pooled the spectra of a subset of images, and asked how JANE performs when the noise level varies from image to image, as in [c] (Herbreteau and Meunier, 2026). During the revision, we have shown that JANE and its variants can estimate the noise level from the eigenspectrum of a single image. Estimated from each image alone on the small configurations, JANE's PSNR is on average within 0.030.03 dB of denoising with the true σ2\sigma^2, and its average error in σ2\sigma^2 is 3.63.6%, lower by a factor of 2.22.2 than that of Noise2Score, which also uses a single image at inference. Since every image can then be denoised with its own estimate, JANE also handles per-image noise. With noise levels drawn uniformly from [5,50][5,50], its PSNR is on average 0.150.15 dB below that of the oracle and higher than that of every baseline.
  • Theory (Section 4, Appendix C). We have rewritten Section 4 in a clearer way. It now starts from the cause of the violation in Section 3, namely that training fits the values of the score at the training samples but leaves its Jacobian unconstrained. It then separates the systematic residual of training from the random part that depends on the initialization. Proposition 2 and Remark 3 show that the random part perturbs the noise eigenvalues into a symmetric distribution with a single peak, which connects noise estimation to peak finding. Since Remark 3 relies on a loose relaxation, we have checked the empirical spectra of all 48 evaluation cases of the paper, and found that the noise eigenvalue distribution has a single peak in every one of them. We also quantified in a toy model all factors that Proposition 2 sets aside, and found that each of them has only a minor effect on the peak estimate.
  • More benchmarks and datasets (Section 6, Appendix D). We have added five benchmarks, the noise-level estimators of Chen et al. (2015) and Pyatykh et al. (2013), and Neighbor2Neighbor, Noise2Self and the saddle-point version of UNSURE. The evaluation now has twelve configurations, including ImageNet, CUB-200 and Waterloo Exploration at 256×256256\times256 in grayscale and colour, all at σ∈{15,25,50,75}\sigma\in\{15,25,50,75\} and all with a single network (no ensemble).
  • Sensitivity (Appendix D.3). Every parameter of the peak locator, the number of crops, the number of Lanczos steps and the number of pooled images are varied. The estimate is insensitive to the parameters of the peak locator, the number of crops and the number of Lanczos steps. A single image already gives a usable estimate, and pooling more images reduces the error further, with little gain beyond about 100 images. The same settings are used for all configurations.
  • Clarity and corrections. We have corrected the description of UNSURE (the score-based baseline is now called UNSURE (score)), defined all symbols of the NTK analysis, removed the ensemble, and added PSNR values to Figure 3.

Main results. In Tables 1, 12 and 13, averaged over the 48 settings (12 configurations at 4 noise levels), JANE's error in σ2\sigma^2 is 2.52.5%, lower by a factor of 2.02.0 to 9.39.3 than those of the baselines. Its PSNR is on average within 0.010.01 dB of the oracle step size. (As in Table 1, JANE denotes exact JANE on the small images and JANE-crops on the 256×256256\times256 images.)

Detailed responses to each comment are posted below. We hope that our clarifications and additional experimental results have adequately addressed your concerns. We would greatly appreciate it if you could reconsider your initial assessment and update your rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.