acceptodds
Under review as a conference paper at ICLR 2027

Learning to Recorrupt: Noise Distribution Agnostic Self-Supervised Image Denoising

28 views

Abstract

Self-supervised image denoising methods have traditionally relied on architectural constraints, pseudo-pair constructions, or specialized loss functions to avoid the trivial identity mapping. Among these, approaches such as Noisier2Noise or R2R create training pairs by adding synthetic noise to noisy images. While effective, these recorruption-based approaches require precise knowledge of the noise distribution, which is often unavailable. We present Learning to Recorrupt (L2R), a self-supervised framework that does not require exact knowledge of the noise distribution. Our method introduces a learnable recorruptor jointly optimized with the denoiser through a min–max saddle-point objective. The proposed method achieves state-of-the-art performance among methods without prior knowledge of the noise distribution across unconventional and heavy-tailed noise distributions, such as log-gamma and Laplace, as well as spatially correlated noise, while obtaining competitive results on real-world denoising benchmarks characterized by signal-dependent noise.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-supervised image denoising, learnable recorruption, uknown noise statistics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3p15OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Learning to Recorrupt (L2R), a self supervised image denoising framework for settings where the exact noise distribution is unknown. The main idea is to replace the fixed recorruption process used in previous recorruption based methods with a learnable recorruptor. The denoiser and recorruptor are jointly optimized through a min max objective. The learned recorruptor is designed to capture auxiliary noise statistics that make the self supervised objective approximate the corresponding supervised denoising objective. The paper also establishes connections between L2R, GR2R, and UNSURE, and introduces different recorruptor parameterizations for independent, spatially correlated, and signal dependent noise. Experiments on synthetic noise, SIDD, and Cryo EM data evaluate the proposed framework under a range of noise conditions.

Critical strengths
  1. The paper provides a principled way to learn the recorruption process instead of assuming that the exact noise distribution is known in advance. The min max formulation gives a clear connection between the learned recorruption and the supervised denoising objective.
  2. The experimental study is relatively thorough. The paper evaluates different noise structures, analyzes the recorruptor design and optimization strategy, and includes additional studies on training stability and learned noise statistics.
Critical weaknesses
  1. Limited methodological novelty relative to GR2R and UNSURE. The proposed method is closely connected to existing recorruption based objectives. The paper itself describes L2R as a learned extension of GR2R and a learned generalization of UNSURE. The main change is to replace a fixed or analytically specified recorruption mechanism with a learnable neural model that is optimized jointly with the denoiser. This is useful, but I am not fully convinced that the resulting methodological advance is large enough for ICLR.

  2. The claim of noise distribution agnostic denoising appears stronger than the actual assumptions of the method. Although L2R does not require the exact noise distribution or its parameters, the recorruptor architecture still needs to reflect the broad structure of the corruption process. Different designs are used for independent, spatially correlated, and signal dependent noise. The appendix also states that the recorruptor architecture acts as a user specified structural prior. Therefore, the method is not fully agnostic to the noise structure, and the current wording may overstate its generality.

Questions for authors
  1. Since L2R is presented as a learned extension of GR2R and is also closely related to UNSURE, could the authors clarify what new learning principle is introduced beyond replacing the fixed recorruption mechanism with a learned one?
  2. The method uses different recorruptor structures for independent, correlated, and signal dependent noise. How should a practitioner select the appropriate recorruptor when the noise structure itself is unknown?
  3. Could the authors clarify the practical failure cases when the chosen recorruptor class does not contain a compatible noise transformation? How sensitive is the method to such model mismatch?
  4. Could the authors provide comparisons with more recent self supervised denoising methods to better position L2R with respect to the current state of the field?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 3p15 (1/2)

W1.1 "Limited methodological novelty relative to GR2R and UNSURE. The proposed method is closely connected to existing recorruption based objectives. The paper itself describes L2R as a learned extension of GR2R and a learned generalization of UNSURE. The main change is to replace a fixed or analytically specified recorruption mechanism with a learnable neural model that is optimized jointly with the denoiser."

Q1 "Since L2R is presented as a learned extension of GR2R and is also closely related to UNSURE, could the authors clarify what new learning principle is introduced beyond replacing the fixed recorruption mechanism with a learned one?"

We thank the reviewer for this important question. We would like to clarify why making the recorruption learnable is not a simple replacement:

  • Learning hh requires a new objective. Naively optimizing the recorruption within the GR2R loss fails (see the table below), since the loss does not specify what hh should learn; it requires our carefully designed min–max objective (Eq. 18) and L2R loss. Moreover, the L2R formulation goes beyond UNSURE, which is limited to Gaussian and Poisson–Gaussian families, by covering complex noise distributions.
  • First method to handle complex unknown noise distributions without blind spots. To our knowledge, no previous self-supervised method achieves this with a single loss and a standard denoiser.
  • Theoretical link to the oracle. We connect the learned recorruptor to the GR2R oracle recorruption: when the learned hh matches it, L2R reduces to GR2R (Eq. 10). Learning trades this exact knowledge for a saddle-point problem, at a small cost in practice: 30.77 dB for L2R vs. 31.14 dB for the GR2R oracle (Table 4).

To support the first point, we compared three ways of learning hh in the log-gamma ℓ=1\ell=1 setting of Table 4 (BSDS500), with the same denoiser, recorruptor class, data, and training budget (now added to App. A.3.4):

Objective for (f,h)(f,h)PSNRSSIM
min⁡fmin⁡h\min_f\min_h GR2R loss17.440.405
min⁡fmax⁡h\min_f\max_h GR2R loss−15.610.000
min⁡fmax⁡h\min_f\max_h L2R loss (ours)30.770.871

Both naive objectives fail, as expected from the relation between the two losses (Eq. 35, App. A.1.1). Since y2=y−h(w′)/τ\mathbf y_2=\mathbf y-h(\mathbf w')/\tau and hh is zero-mean and independent of y\mathbf y,

E∥f(y1)−y2∥22=E[∥f(y1)−y∥22+2τf(y1)⊤h(w′)]+1τ2E∥h(w′)∥22,\mathbb E\Vert f(\mathbf y_1)-\mathbf y_2\Vert_2^2=\mathbb E\Big[\Vert f(\mathbf y_1)-\mathbf y\Vert_2^2+\tfrac{2}{\tau}f(\mathbf y_1)^\top h(\mathbf w')\Big]+\frac{1}{\tau^2}\mathbb E\Vert h(\mathbf w')\Vert_2^2,

where the first term is the L2R loss, i.e., the GR2R loss equals the L2R loss plus an amplitude term. Maximizing the GR2R loss over hh is dominated by this term: the learned noise scale keeps growing during training, from 0.1 to 3.48 (about 35 times the noise level). Minimizing it instead drives hh toward a degenerate distribution, and the result falls below the noisy input (20.00 dB). L2R removes exactly this amplitude term, so that hh learns the noise statistics rather than its scale.

W1.2 "This is useful, but I am not fully convinced that the resulting methodological advance is large enough for ICLR."

Beyond this new principle, L2R learns the recorruption from noisy data alone, enforces a richer family of finite-τ\tau constraints than UNSURE's single divergence (recovering UNSURE as the linear case, App. A.1.2), needs no divergence estimation (one denoiser evaluation per sample, Algorithm 1), and covers independent, correlated, and signal-dependent noise with one loss.

Empirically, L2R improves over UNSURE by up to 5.7 dB on heavy-tailed noise (Table 1). L2R also improves by +3.15 dB over a finite-τ\tau UNSURE (learned Gaussian scale, Table 4). It is also more efficient: on the Cryo-EM data, L2R trains 1.55× faster than PG-UNSURE under matched conditions.

AuthorsOpenReview ↗

Response to Reviewer 3p15 (2/2)

W2 "The claim of noise distribution agnostic denoising appears stronger than the actual assumptions of the method. Although L2R does not require the exact noise distribution or its parameters, the recorruptor architecture still needs to reflect the broad structure of the corruption process. Different designs are used for independent, spatially correlated, and signal dependent noise. The appendix also states that the recorruptor architecture acts as a user specified structural prior. Therefore, the method is not fully agnostic to the noise structure, and the current wording may overstate its generality."

We agree, and we corrected this claim in the revised paper: the abstract now states that L2R does not require knowledge of the noise distribution or its parameters, only of its broad structure. This structure is typically known from the acquisition pipeline, e.g., whether preprocessing is applied and the sensor type.

Q2 "The method uses different recorruptor structures for independent, correlated, and signal dependent noise. How should a practitioner select the appropriate recorruptor when the noise structure itself is unknown?"

In practice, the structure is typically known from the acquisition: whether an ISP or other preprocessing is applied indicates spatial correlation, and the sensor and acquisition regime (e.g., photon-limited CMOS imaging, as in SIDD, or SPAD sensors) indicates signal dependence. Otherwise, it can be diagnosed from noisy data alone. Spatial autocorrelation of the residual y−f(y)\mathbf y-f(\mathbf y) indicates the need for the spatial kernel. Dependence of the residual variance on intensity indicates the need for s(y)s(\mathbf y). We added this guidance to App. A.2.2.

Q3.1 "Could the authors clarify the practical failure cases when the chosen recorruptor class does not contain a compatible noise transformation?"

Table 7 (appendix) quantifies this; the bold rows are the recorruptor class matching each noise structure:

NoiseKernels(y)s(\mathbf y)PSNRSSIM
Log-gamma ℓ=1\ell=11×1no30.770.871
3×3no29.930.837
1×1yes26.980.721
3×3yes24.020.657
Correlated σ=0.2\sigma=0.21×1no24.210.660
3×3no29.850.863
1×1yes24.020.657
3×3yes27.050.754
Poisson–Gaussian γ=σ=0.05\gamma=\sigma=0.051×1no24.990.628
3×3no23.770.581
1×1yes27.800.791
3×3yes26.030.713

The main failure mode is a recorruptor class that lacks the structure of the noise, e.g., no spatial kernel for correlated noise (−5.64 dB) or no signal-dependent scale for Poisson–Gaussian noise (−2.81 dB).

Q3.2 "How sensitive is the method to such model mismatch?"

An unnecessary spatial kernel costs only 0.84 dB (Table 7). Within the correct class, L2R is robust to τ\tau, to the learning-rate ratio, and to the size of hh (Tables 6, 8, 9).

Q4 "Could the authors provide comparisons with more recent self supervised denoising methods to better position L2R with respect to the current state of the field?"

We thank the reviewer for this suggestion. Our comparison already includes the latest and most competitive self-supervised methods that are relevant to our setting. These include UNSURE (ICLR 2025) and GR2R (CVPR 2025), the current state of the art among SURE- and recorruption-based methods. In particular, within our RAW SIDD protocol, Blind2Sound (ICCV 2025) is the most recent and strongest competitor, and it is included in Tables 1, 2, and 3.

We hope our response and the revised paper (changes in blue) have addressed your concerns, in particular the new comparison with learning h directly within the GR2R loss (App. A.3.4). Please let us know if you have any questions regarding our response.

Reviewer_qhvHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method that jointly learns a denoiser and an auxiliary corruption generator through a minmax objective. The generator replaces manually specified recorruption statistics, while the denoiser does not require blind-spot constraints. The paper connects the objective to GR2R and UNSURE and evaluates it on synthetic non-Gaussian noise, real SIDD data, and Cryo-EM

Critical strengths
  • Learning the auxiliary corruption is a useful extension of distribution-specific recorruption methods

  • The evaluation is reasonably broad. Fixed noisy training realizations, cross-dataset testing, real SIDD evaluation, and the architecture and optimization ablations provide evidence beyond a single synthetic setting

Critical weaknesses
  • Distribution-agnostic still requires substantial structural knowledge. Tab. 7 shows performance drops when the corruption architecture is mismatched. The signal-dependent version also explicitly uses an affine variance model. In other words, the contribution is more like to avoid exact noise distributions and parameters, rather than avoiding noise priors
  • The signal-dependent extension needs a clearer justification. The derivation starts with noise independent of the clean image, whereas Eq.20 conditions the generator on the noisy observation. Conditioning on the clean image alone does not automatically justify the same moment identities for this observation-dependent generator
  • The real-data protocol and efficiency comparison need more clarification. Table 2 should explicitly identify the raw setting and state which baselines were retrained or taken from prior work.
  • Some evaluation results do not verify the claimed mechanism. In Table 4, “No learning” is simply the noisy input, not a trained denoiser with a fixed Gaussian recorruptor. A matched linear or fixed-map baseline would better isolate the benefit of learning nonlinear corruption statistics
Questions for authors
  • Which theoretical statements apply to the observation-dependent generator and the stop-gradient implementation?
  • Please confirm the raw protocol for Tab 2, the source of the baseline results, and whether the ×1.4 training cost includes the complete global recorruptor
5 comments
AuthorsOpenReview ↗

Response to Reviewer qhvH (1/2)

We thank the reviewer for the positive assessment and for the precise suggestions, which helped us sharpen the scope of our claims and isolate the mechanism of L2R.

W1 "Distribution-agnostic still requires substantial structural knowledge. Tab. 7 shows performance drops when the corruption architecture is mismatched. The signal-dependent version also explicitly uses an affine variance model. In other words, the contribution is more like to avoid exact noise distributions and parameters, rather than avoiding noise priors."

We agree and adopted this framing in the abstract and App. A.2.2. L2R avoids the exact distribution and its parameters, given a coarse structural prior: element-wise, correlated, or signal-dependent. This prior is mild in practice: spatial correlation is known from whether an ISP or other preprocessing is applied, and signal dependence from the sensor and acquisition regime (e.g., photon-limited CMOS imaging, as in SIDD, or SPAD sensors). Table 7 in the appendix quantifies the effect of a mismatched structure. The affine variance is the structural prior for signal-dependent sensors; its parameters γ\gamma and σ\sigma are not given but learned with hh (App. A.2.2).

W2.1 "The signal-dependent extension needs a clearer justification. The derivation starts with noise independent of the clean image, whereas Eq.20 conditions the generator on the noisy observation."

The additive derivation (App. A.1) establishes the correspondence with supervised training for noise independent of x\mathbf x. The observation-dependent generator of Eq. 20,

h(w′,y)=s(y)⊙h~y(w′),s(y)=γy+σ2,h(\mathbf w',\mathbf y)=s(\mathbf y)\odot\tilde h_{\mathbf y}(\mathbf w'),\qquad s(\mathbf y)=\sqrt{\gamma\mathbf y+\sigma^2},

is a practical extension motivated by the affine-variance structure of Poisson–Gaussian noise, Var[ε∣x]=γx+σ2\mathrm{Var}[\varepsilon\mid\mathbf x]=\gamma\mathbf x+\sigma^2.

W2.2 "Conditioning on the clean image alone does not automatically justify the same moment identities for this observation-dependent generator."

We agree that the identities do not transfer automatically. Eq. 20 nevertheless preserves the key property of the additive case conditionally on y\mathbf y. Its randomness comes only from w′\mathbf w', drawn independently of y\mathbf y, and h~y\tilde h_{\mathbf y} is normalized by N\mathcal N, so

E[h∣y]=s(y)⊙Ew′[h~y(w′)]=0.\mathbb E[h\mid\mathbf y]=s(\mathbf y)\odot\mathbb E_{\mathbf w'}[\tilde h_{\mathbf y}(\mathbf w')]=\mathbf 0.

Hence, given y\mathbf y, hh is a zero-mean recorruption with a local scale s(y)s(\mathbf y), so the moment identities of Eq. 22 can be stated conditionally on y\mathbf y, in the same form as in the additive case. We empirically evaluated this extension on Poisson–Gaussian noise (Table 3), SIDD, and Cryo-EM, where it performs consistently well.

Q1 "Which theoretical statements apply to the observation-dependent generator and the stop-gradient implementation?"

The main theoretical statement (App. A.1) is the correspondence with supervised training: if H\mathcal H contains the GR2R oracle recorruption hch_c, then for every denoiser satisfying the constraint of Eq. 16,

Ey,w′[f(y+τh(w′))⊤h(w′)]=0∀h∈H,\mathbb E_{\mathbf y,\mathbf w'}\big[f(\mathbf y+\tau h(\mathbf w'))^\top h(\mathbf w')\big]=0\qquad\forall h\in\mathcal H,

the self-supervised loss equals the supervised one up to a constant,

E∥f(y1)−y∥22=E∥f(y1)−x∥22+const.\mathbb E\Vert f(\mathbf y_1)-\mathbf y\Vert_2^2=\mathbb E\Vert f(\mathbf y_1)-\mathbf x\Vert_2^2+\text{const}.

This statement applies to the additive generator; for the observation-dependent generator, it serves as motivation and is validated empirically (see W2). The stop-gradient update (Algorithm 1) ascends exactly the constraint above at the current denoiser, similar to the first-order simplifications used in other min–max methods such as SAM [Foret et al., 2021]. App. A.3.3 shows it reaches similar accuracy to the full-gradient update while training 1.23× faster.

Reference:
[Foret et al., 2021] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR, 2021.

AuthorsOpenReview ↗

Response to Reviewer qhvH (2/2)

W3 "The real-data protocol and efficiency comparison need more clarification. Table 2 should explicitly identify the raw setting and state which baselines were retrained or taken from prior work."

Q2 "Please confirm the raw protocol for Tab 2, the source of the baseline results, and whether the ×1.4 training cost includes the complete global recorruptor."

We confirm that all results in Table 2 use SIDD RAW, evaluated on the validation blocks and on the benchmark through its official Kaggle evaluation. The RAW setting is now stated in the caption of Table 2. The baseline results are taken from the DCD-Net and Blind2Sound papers, which use the same SIDD RAW protocol; none were retrained. We now state the sources in Sec. 4. The ×1.4 training cost includes the complete global recorruptor and its update; with the additive recorruptor, it reduces to ×1.11. We now report both costs in App. A.2.2.

W4.1 "Some evaluation results do not verify the claimed mechanism. In Table 4, ‘No learning’ is simply the noisy input, not a trained denoiser with a fixed Gaussian recorruptor."

We agree that the "No learning" row did not isolate the mechanism; it is now renamed "Noisy input".

W4.2 "A matched linear or fixed-map baseline would better isolate the benefit of learning nonlinear corruption statistics."

We thank the reviewer for this suggestion. We added the two requested baselines to Table 4: a fixed Gaussian recorruptor, which is equivalent to R2R, and a linear recorruptor with a learned Gaussian scale, which acts as a finite-τ\tau UNSURE. We also added a random MLP whose initial weights are kept fixed, to measure the influence of optimizing hh. All use the same denoiser, data, and training budget as L2R (log-gamma ℓ=1\ell=1, BSDS500):

hhPSNRSSIM
Random MLP (fixed weights)25.300.592
Linear, learned scale (finite-τ\tau UNSURE)27.620.763
Fixed Gaussian (R2R)28.630.809
L2R (nonlinear)30.770.871

Learning the nonlinear corruption statistics improves on the random, linear, and fixed baselines by +5.47 dB, +3.15 dB, and +2.14 dB, respectively, which isolates the benefit of the learned map. Notably, on Laplace noise with b=0.1b=0.1, L2R even outperforms GR2R with the known noise distribution (28.40 vs. 28.10 dB on BSDS500, Table 1).

We hope our response and the revised paper (changes in blue) have addressed your concerns, in particular the guidance for choosing the recorruptor (App. A.2.2) and the recorruptor ablations (Table 4). Please let us know if you have any questions regarding our response.

Reviewer_nAy3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the limitations of self-supervised denoising methods based on re-noising—which typically require precise knowledge of the noise distribution—this paper proposes parameterizing the re-noising mapping as a learnable network, hh. This network consists of a monotonic mMLP, normalization layers, and convolutional kernels, with four structural variants designed to accommodate various noise types, including independent and identically distributed (i.i.d.) noise, spatially correlated noise, and signal-dependent noise. By jointly optimizing this network alongside a denoiser, ff, through a "min–max" saddle-point objective function, the approach eliminates the need for explicit reliance on noise likelihood functions or moment matching. The effectiveness of this method is demonstrated across several noise scenarios, including log-gamma noise, Laplace noise, spatially correlated Gaussian noise, and real-world, signal-dependent noise from the SIDD dataset. Furthermore, the study includes experiments conducted on real-world cryo-electron microscopy (Cryo-EM) data.

Critical strengths
  1. The GR2R method relies on precise noise statistics, which poses a significant obstacle in practical applications. The idea of ​​shifting statistical constraints from the denoiser architecture—such as "blind-spot" networks—to a learnable "recorruptor" is highly ingenious; it also preserves the generality of the architecture (i.e., the denoising function ff can be any network structure).
  2. Starting from the correlation constraints in Equations (15)–(16) and incorporating a "conic-set" assumption, the authors derive a saddle-point formulation (Equation 18) and establish its connection to UNSURE (Equations 6–7) and GR2R (Equation 10).
  3. In addition to synthetic non-Gaussian noise, this paper utilizes the real-world SIDD dataset as well as ground-truth-free data from cryo-electron microscopy (Cryo-EM).
Critical weaknesses
  1. The use of "stop-gradient" causes the actual update direction of the "recorruptor" hh to deviate from the theoretical derivation presented in the paper. In the main text, hh is updated via f(sg[y1])f(\text{sg}[y_1]) (Section 4: "a stop-gradient is applied to the input of the denoiser"); Appendix A.3.3 further clarifies that "the update of hh is driven directly by the correlation term." This implies that the fidelity term ∥f(y1)−y∥2\|f(y_1) - y\|^2 contributes no gradient to hh whatsoever; the gradient for hh is derived solely from (2/τ)⋅f(sg[y1])⊤h(w′)(2/\tau) \cdot f(\text{sg}[y_1])^\top h(w'). Because the gradient of f(y1)f(y_1) is blocked, hh is effectively pushed toward alignment with the denoiser's output—that is, the estimate of the clean image—rather than aligning with the true noise g(w)g(w) through moment matching. Given the zero-mean constraint imposed by N(⋅)N(\cdot), hh is more likely to learn the spatial structure of the images (or their dataset-wide mean) rather than the statistical properties of the noise. Mechanistically, this contradicts the claim in Equation (21) that the "learned auxiliary corruption tracks the true noise contribution." Furthermore, Equation (21) is validated only numerically in Appendix A.5, using an "≈\approx" symbol without providing any quantitative error metrics.

Please provide clarification on the following: (a) Across which dimensions is the normalization N(⋅)N(\cdot) performed? How does hh avoid collapsing into the trivial solution of simply reproducing the image structure? (b) Please provide a quantitative comparison between the marginal distribution of h∗(w′)h^*(w') and the true distribution pεp_\varepsilon (e.g., using the Kolmogorov-Smirnov distance, the first four moments, or histograms), rather than relying solely on the visualizations provided in Appendix A.4. (c) In particular, please explain why the “sg” variant performs best (Table 5: 29.85, compared to 29.78 for the combined non-“sg” variant and 29.83 for the non-combined variant). If cutting off the gradient of the “fidelity term” actually improves performance, it suggests that this term acts as noise rather than a useful signal for learning hh. Furthermore, this implies that the “min–max” interpretation of Equation (18) warrants re-examination. 2. The paper does not isolate the contribution of the “learnable recorruptor.” Although this component is a core contribution of the work, there is no control group that employs a fixed (non-learnable) recorruptor. The “No learning” row in Table 4 refers to evaluating metrics directly on noisy observations (i.e., without denoising), rather than using a fixed h(w′)=w′h(w') = w' (standard Gaussian distribution) or a GR2R-style moment-matching approach while training ff as usual. Given the concerns raised in W1 regarding the reliability of the update signal for hh, such a control experiment is crucial for clarifying the source of the performance gains shown in Table 1. Without it, one cannot determine whether the improvement stems from “learning the statistical properties of the noise” or simply from “introducing additional noise and regularization terms based on correlations.” Please include this baseline experiment. 3. The SSIM scores are generally low; under the Laplace noise setting on the DIV2K dataset, the proposed method performs even worse than a baseline model that makes no distributional assumptions—a fact that is sidestepped in the main text.

Questions for authors

See Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer nAy3 (1/4)

We thank the reviewer for the careful and detailed analysis. Before answering each point, we briefly restate the goal of L2R: a single loss and a single blind-spot-free denoiser, without knowledge of the noise distribution, versatile across a wide range of noise distributions, and linked to UNSURE and GR2R. The stop-gradient is an implementation choice within this method; below we show that it is consistent with our formulation and does not affect performance.

W1.1 "The use of ‘stop-gradient’ causes the actual update direction of the ‘recorruptor’ hh to deviate from the theoretical derivation presented in the paper."

We agree that stop-gradient differs from the full gradient of Eq. 18, but not from the constrained optimization in Eq. 16, which only requires E[f(y1)⊤h(w′)]=0\mathbb E[f(\mathbf y_1)^\top h(\mathbf w')]=0 for a given hh. Stop-gradient does not change this value and ascends exactly this quantity, as in dual ascent. We now state the exact update in Algorithm 1 (App. A.2.1).

W1.2 "In the main text, hh is updated via f(sg[y1])f(\mathrm{sg}[\mathbf y_1]) (Section 4: ‘a stop-gradient is applied to the input of the denoiser’); Appendix A.3.3 further clarifies that ‘the update of hh is driven directly by the correlation term.’"

This is the update used in all experiments, and we now state it more explicitly in Sec. 3 and Algorithm 1 (App. A.2.1).

W1.3 "This implies that the fidelity term ∥f(y1)−y∥2\Vert f(\mathbf y_1)-\mathbf y\Vert^2 contributes no gradient to hh whatsoever; the gradient for hh is derived solely from (2/τ)f(sg[y1])⊤h(w′)(2/\tau) f(\mathrm{sg}[\mathbf y_1])^\top h(\mathbf w'). Because the gradient of f(y1)f(\mathbf y_1) is blocked, hh is effectively pushed toward alignment with the denoiser's output—that is, the estimate of the clean image—rather than aligning with the true noise g(w)g(\mathbf w) through moment matching."

We agree that the fidelity term trains only ff and that hh is updated only through the correlation 2τf(sg[y1])⊤h(w′)\tfrac2\tau f(\mathrm{sg}[\mathbf y_1])^\top h(\mathbf w'). This dot product is, however, exactly the quantity that the GR2R oracle moments control. Writing ε=g(w)\boldsymbol\varepsilon=g(\mathbf w) for the noise, the GR2R oracle recorruption ω=hc(w′)\boldsymbol\omega=h_c(\mathbf w') satisfies the moment conditions of Eq. 9 (Eq. 22 in App. A.1),

E[(εi−ωiτ)(εi+τωi)k]=0,i=1,…,n,k=0,…,K,\mathbb E\Big[\Big(\varepsilon_i-\frac{\omega_i}{\tau}\Big)\big(\varepsilon_i+\tau\omega_i\big)^k\Big]=0,\qquad i=1,\ldots,n,\quad k=0,\ldots,K,

where KK is the maximum moment order, which, for the considered denoiser class, imply the correlation identity of Eq. 29 (App. A.1):

1τE[f(y1)⊤ω]=E[f(y1)⊤ε],y1=x+ε+τω.\frac1\tau\mathbb E\big[f(\mathbf y_1)^\top\boldsymbol\omega\big]=\mathbb E\big[f(\mathbf y_1)^\top\boldsymbol\varepsilon\big],\qquad \mathbf y_1=\mathbf x+\boldsymbol\varepsilon+\tau\boldsymbol\omega.

Briefly, for i.i.d. noise and a denoiser that is a polynomial of degree at most KK in each input pixel, fi(y1)f_i(\mathbf y_1) expands in powers of (εi+τωi)(\varepsilon_i+\tau\omega_i) with coefficients ckc_k independent of (εi,ωi)(\varepsilon_i,\omega_i), so

E[(εi−ωiτ)fi(y1)]=∑k≤KE[ck]E[(εi−ωiτ)(εi+τωi)k]=0\mathbb E\Big[\Big(\varepsilon_i-\frac{\omega_i}{\tau}\Big)f_i(\mathbf y_1)\Big]=\sum_{k\le K}\mathbb E[c_k]\mathbb E\Big[\Big(\varepsilon_i-\frac{\omega_i}{\tau}\Big)(\varepsilon_i+\tau\omega_i)^k\Big]=0

by Eq. 22; summing over ii gives Eq. 29. Eq. 21 is this identity evaluated at the learned h∗h^\ast. The dot product that drives hh is therefore the one fixed by the GR2R oracle moments, so at the saddle point it aligns hh with these moments rather than with the image, as confirmed by the learned statistics in W1(b).

AuthorsOpenReview ↗

Response to Reviewer nAy3 (2/4)

W1.4 "Given the zero-mean constraint imposed by N(⋅)\mathcal N(\cdot), hh is more likely to learn the spatial structure of the images (or their dataset-wide mean) rather than the statistical properties of the noise."

We thank the reviewer for raising this point. N\mathcal N only normalizes the noise map generated from w′\mathbf w', which is drawn independently of the data, so it cannot introduce image content. In the additive variants, hh is a function of w′\mathbf w' alone, hence

I(h(w′);y)≤I(w′;y)=0,I\big(h(\mathbf w');\mathbf y\big)\le I(\mathbf w';\mathbf y)=0,

i.e., hh carries no information about the image. Since hh acts pixelwise (or through a convolution) on i.i.d. w′\mathbf w', it also has no fixed spatial pattern, which rules out a dataset-wide mean. In the signal-dependent variants, y\mathbf y only modulates its amplitude (W1(a).2). Empirically, h∗h^\ast learns noise statistics: its moments follow those of the GR2R oracle recorruption (W1(b)).

W1.5 "Mechanistically, this contradicts the claim in Equation (21) that the ‘learned auxiliary corruption tracks the true noise contribution.’"

We agree that the phrase ‘tracks the true noise contribution’ was misleading: Eq. 21 refers to the GR2R oracle recorruption, which matches the noise through the denoiser-weighted correlation rather than in distribution. We have revised the sentence after Eq. 21 accordingly.

W1.6 "Furthermore, Equation (21) is validated only numerically in Appendix A.5, using an ‘≈’ symbol without providing any quantitative error metrics."

We now provide a quantitative evaluation in App. A.4 (Table 10): the variance, skewness, excess kurtosis, and KS distances of h∗h^\ast, compared with both the GR2R oracle recorruption and the noise measurement (see W1(b)).

W1(a).1 "Across which dimensions is the normalization N(⋅)\mathcal N(\cdot) performed?"

N(⋅)\mathcal N(\cdot) normalizes the generated noise map to zero mean and unit variance along its spatial dimensions.

W1(a).2 "How does hh avoid collapsing into the trivial solution of simply reproducing the image structure?"

  • Additive variants. h=kh∗N(mMLP(w′))h=k_h\ast\mathcal N(\mathrm{mMLP}(\mathbf w')) takes only w′\mathbf w' as input, so it has no access to the image. Its output is a zero-mean, stationary field that is independent of x\mathbf x. In other words, hh is a highly constrained map that applies the same scalar function to every pixel, not a generator of high-dimensional images.
  • Signal-dependent variants. Here, h(w′,y)=s(y)⊙h~y(w′)h(\mathbf w',\mathbf y)=s(\mathbf y)\odot\tilde h_{\mathbf y}(\mathbf w') (Eq. 20) does see y\mathbf y, but only in two restricted ways: s(y)=γy+σ2s(\mathbf y)=\sqrt{\gamma\mathbf y+\sigma^2} sets a nonnegative pixelwise scale, and in variant (2.b) y\mathbf y selects the weights of the monotonic map. The randomness still comes only from w′\mathbf w', drawn independently of y\mathbf y, and h~y\tilde h_{\mathbf y} is normalized by N\mathcal N. Hence, conditionally on y\mathbf y, the recorruption is zero-mean: E[h∣y]=s(y)⊙Ew′[h~y(w′)]=0\mathbb E[h\mid\mathbf y]=s(\mathbf y)\odot\mathbb E_{\mathbf w'}[\tilde h_{\mathbf y}(\mathbf w')]=\mathbf 0. The image can therefore only modulate the local amplitude of hh, as it does for the signal-dependent noise itself (variance γx+σ2\gamma\mathbf x+\sigma^2), and it cannot be reproduced as the mean of hh.

These constraints on hh reflect the robustness–optimality trade-off of self-supervised learning: a more flexible hh is robust to larger families of noise distributions, at the expense of denoising performance (App. A.2.2).

AuthorsOpenReview ↗

Response to Reviewer nAy3 (3/4)

W1(b) "Please provide a quantitative comparison between the marginal distribution of h∗(w′)h^\ast(\mathbf w') and the true distribution pεp_\varepsilon (e.g., using the Kolmogorov-Smirnov distance, the first four moments, or histograms), rather than relying solely on the visualizations provided in Appendix A.4."

We provide a more detailed statistical analysis in App. A.4 (Table 10). By construction, L2R aims to learn the auxiliary distribution required by GR2R (Eq. 9), which in general differs from pεp_\varepsilon itself. We therefore compare h∗h^\ast with both references. The cumulants of the GR2R oracle recorruption satisfy κr(ω)=τ2−rκr(ε)\kappa_r(\omega)=\tau^{2-r}\kappa_r(\varepsilon), so at τ=0.5\tau=0.5 the variance matches the noise, the skewness doubles, and the excess kurtosis is multiplied by four. App. A.4 now reports (KS computed with 10610^6 draws against the Oracle GR2R law and the noise measurement):

NoiseVar. (×10−2\times10^{-2})Skew.Ex. kurt.KS oracleKS noise
Laplace b=0.1b=0.1Learned1.9940.0412.600.1480.024
Oracle GR2R2.000012.00
Noise Measurement2.00003.00
Laplace b=0.05b=0.05Learned0.5930.097.810.1220.052
Oracle GR2R0.500012.00
Noise Measurement0.50003.00
Log-gamma ℓ=1\ell=1Learned1.052−1.907.530.0410.028
Oracle GR2R1.000−2.289.60
Noise Measurement1.000−1.142.40
Log-gamma ℓ=0.1\ell=0.1Learned1.030−3.4718.750.1650.133
Oracle GR2R1.000−3.9223.33
Noise Measurement1.000−1.965.83

The variance of h∗h^\ast stays at the noise level, while its skewness and kurtosis move well beyond those of the noise, toward those of the GR2R oracle. L2R therefore learns the statistics GR2R requires rather than copying pεp_\varepsilon. Since Eq. 9 constrains moments (tails) rather than the bulk measured by KS, we use the moments as the primary measure of agreement with the GR2R oracle.

W1(c).1 "In particular, please explain why the ‘sg’ variant performs best (Table 5: 29.85, compared to 29.78 for the combined non-‘sg’ variant and 29.83 for the non-combined variant). If cutting off the gradient of the ‘fidelity term’ actually improves performance, it suggests that this term acts as noise rather than a useful signal for learning hh."

We thank the reviewer for this careful observation. The differences in Table 5 are small (within 0.07 dB). To verify this, we evaluated the early training dynamics (first 400 epochs) over 3 seeds in the setting of Table 5: stop-gradient reaches 29.250 ± 0.048 dB and the full gradient 29.262 ± 0.058 dB, on par within one standard deviation (App. A.3.3). We therefore see stop-gradient not as improving accuracy, but as preserving it while training 1.23× faster, which is why we adopt it. We agree with the reviewer that the implicit path of the fidelity term is not needed to learn hh, since the noise statistics enter through the correlation term; the fidelity term itself remains the main training signal for ff.

W1(c).2 "Furthermore, this implies that the ‘min-max’ interpretation of Equation (18) warrants re-examination."

The min–max structure is kept: ff descends on the full loss, and hh ascends on the term of Eq. 18 in which it appears explicitly, 2τf(y1)⊤h(w′)\tfrac2\tau f(\mathbf y_1)^\top h(\mathbf w') (Algorithm 1). The full-gradient variant, which follows Eq. 18 exactly, reaches the same accuracy (W1(c).1). The specific form of Eq. 18 matters: maximizing the GR2R loss itself over hh makes the noise scale diverge (App. A.3.4). Simplifying the gradients of min–max objectives for efficiency is common practice in deep learning: SAM drops the second-order terms of its inner maximization and found that including them did not improve accuracy [Foret et al., 2021], and free adversarial training updates both players from a single backward pass [Shafahi et al., 2019].

References:
[Foret et al., 2021] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR, 2021.
[Shafahi et al., 2019] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial Training for Free! NeurIPS, 2019.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their constructive feedback. We are encouraged that the reviewers found our approach "highly ingenious" [nAy3], "principled" [3p15], and "a useful extension of distribution-specific recorruption methods" [qhvH], with an evaluation that is "reasonably broad" [qhvH] and "relatively thorough" [3p15]. The main additions to the revised paper are:

  1. Recorruptor ablations (Table 4) [nAy3, qhvH]. We added a random MLP with fixed weights, a learned Gaussian scale (a finite-τ\tau UNSURE), and a frozen Gaussian recorruptor (equivalent to R2R). In the log-gamma ℓ=1\ell=1 setting (BSDS500), L2R improves on them by +5.47 dB, +3.15 dB, and +2.14 dB respectively, showing the benefit of optimizing hh and learning the nonlinear map.
  2. Quantitative learned noise statistics (App. A.4) [nAy3]. We now report the variance, skewness, excess kurtosis, and KS distances of h∗h^\ast against both the noise measurement and the GR2R oracle recorruption. The learned higher-order statistics follow the GR2R oracle rather than the noise; for example, Laplace b=0.1b=0.1 excess kurtosis is 12.6, against 12 for the GR2R oracle and 3 for the noise.
  3. Clarified scope [qhvH, 3p15]. The abstract now states that L2R removes the need for the noise distribution and its parameters, given only its broad structure, which is encoded by the recorruptor class and discussed in App. A.2.2 (Table 7).
  4. Novelty of learning hh (App. A.3.4) [3p15]. Learning hh directly within the GR2R loss fails (17.44 dB when minimized, divergence when maximized), while L2R reaches 30.77 dB, as explained by the relation between the two losses (Eq. 35).

All changes in the revised paper are highlighted in blue. We hope these additions address the reviewers' concerns, and we would be glad to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.