acceptodds
Under review as a conference paper at ICLR 2027

Blind Spatially-Varying Denoising in Image Diffusion Models

Abstract

Image diffusion models are usually trained with a single noise level for the whole image, so denoising can only proceed uniformly over all pixels at once. The few prior works that support spatially-varying denoising require an input noise map that specifies the noise level at every pixel location. In this paper, we propose blind spatially-varying denoising, which removes this requirement. We remove the timestep input of a pixel diffusion transformer and train it on spatially-varying noise. The model predicts the clean image, so subtracting its prediction from the input leaves the noise, and the size of this remainder gives the noise level of each patch. Any pixel-space model can measure these levels. Only one trained on spatially-varying noise can also denoise patch by patch. We show that we can add this ability to pretrained models by finetuning, including a text-to-image model. The result is still a generator with almost no loss in quality. Painting noise onto an image then becomes a single way to restore and edit it. The model removes strong spatially-varying noise, fills noise-covered regions, extends borders, and edits a photograph wherever noise is applied, without requiring a mask or noise map.

Primary area:
generative models
Keywords:
image diffusion model, blind diffusion model, patch-wise denoising, pixel diffusion model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GAr5OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes blind spatially varying denoising for pixel-space diffusion models, allowing a single model to estimate and remove non-uniform noise without an externally supplied noise map. The approach removes timestep conditioning and estimates each patch’s noise level from the residual between the noisy input and the predicted clean image. An interesting idea is that blind models without timestep conditioning can estimate local noise levels even when trained only on uniformly noisy images. However, they require spatially varying noise during training to denoise each region appropriately. The estimated map indicates each patch’s starting point along the denoising trajectory. The proposed method provides a unified interface for restoration, inpainting, outpainting, and local editing, where painted noise specifies the location and strength of changes without a separate mask or map input. The authors demonstrate both training from scratch and fine-tuning pretrained generators.

Critical strengths
  1. The paper introduces blind spatially varying denoising as a new capability for image diffusion models and presents an elegant approach that unifies restoration and editing. The method achieves quite good overall results across restoration and editing tasks.
  2. The authors discuss the method’s limitations transparently and provide insightful analysis of its failure modes.
  3. I find it interesting and I feel I have learned something.
Critical weaknesses
  1. Poor performance at low noise levels as a result of the model’s reconstruction errors.
  2. Training-range mismatch in the baseline comparisons, competing methods are evaluated outside their training noise ranges, without being retrained or fine-tuned on the full evaluation range. Although the authors acknowledge this mismatch, it remains unclear whether the reported high-noise advantage would persist under matched training conditions.
Questions for authors
  1. What is the definition of sigma_255?
  2. What are the labels in figure 7?
  3. Unclear influence of training strategy and budget. In Table 1, the spatial model trained from scratch has worse FID (tuned cfg) than the fine-tuned model. It would be interesting to understand whether this reflects undertraining or a deeper advantage of learning generation first and adding spatial denoising after.

In addition, I believe that extending the original JiT baseline by 50k steps would provide a better-matched training-budget comparison. This would help determine whether the converted, uniform-trained blind model’s slight improvement over the original comes from removing timestep conditioning or simply from additional training.

3 comments

We thank the reviewer for the positive assessment and for the suggestions on matched comparisons, which we added.


W1: Poor performance at low noise levels as a result of the model's reconstruction errors.

Light noise is indeed the hardest case for the method, and we have added an analysis of it (Appendix D.2, Table A4). The noise estimate measures the residual between the input and the prediction, and this residual also contains the model's own reconstruction error. On clean images, this error sets a floor on the estimate, from 0.011 on smooth patches to 0.071 on the most textured ones (in units of the noise level 1−t1 - t). From a noise level of 0.1 on, the error is below 0.02 for all textures, so the floor matters only for nearly clean images.

Two simple measures can be used to reduce the effect. First, iterating the estimate until it settles, instead of the paper's 3 passes, adds 1.0–1.7 dB in the low-noise band (σ255\sigma_{255} from 5 to 50). The table gives restoration PSNR (dB) on 200 randomly drawn ImageNet validation images (Appendix E, Table A6):

Low-noise band (σ255\sigma_{255} 5–50)UniformSmoothPiecewiseBrightness
3 passes (paper)29.1829.0528.6328.33
Until settled30.3130.0630.3229.50

Second, patches that the model reads as clean can be copied from the input, which keeps nearly clean regions intact, as we do for editing. Denoisers trained specifically on light noise remain stronger in this range, as we state in the limitations (Appendix A).


W2: Training-range mismatch in the baseline comparisons, competing methods are evaluated outside their training noise ranges, without being retrained or fine-tuned on the full evaluation range. Although the authors acknowledge this mismatch, it remains unclear whether the reported high-noise advantage would persist under matched training conditions.

We agree this needed testing. We fine-tuned DRUNet, the strongest map-based baseline, on our full evaluation range (σ255\sigma_{255} up to 153) and reran it on our benchmark (5000 images, protocol of Table 2):

ModelUniformSmoothPiecewiseBrightnessHoles
DRUNet, original, true map22.3021.5621.9626.2419.73
DRUNet, fine-tuned, true map26.6226.4727.15–27.35
DRUNet, fine-tuned, our measured map26.4125.3426.7027.0026.93
Ours (no map)26.4926.3426.5927.2225.10

(–: the true map of brightness-dependent noise reveals the clean image's brightness, so this cell is not comparable.)

Fine-tuning removes most of the high-noise gap. Given the true map, the fine-tuned DRUNet is on par with our model on noise and better on holes. Our model reaches the same level without being given any map. On uniform noise at the strongest level (σ255=153\sigma_{255} = 153), it still leads slightly (24.6 vs. 24.1 dB; original DRUNet 17.0). Our measured map also drives the fine-tuned DRUNet to within 1.1 dB of its true-map results. The advantage over the published checkpoints at high noise is therefore mostly a matter of training range, as we now state in Section 5, and Table 2 includes the fine-tuned rows. Our contribution is a single model that measures the noise and restores without a map, at the level of a matched map-based denoiser.

Q1: What is the definition of sigma_255?

We use the flow-matching convention of JiT, z=t x+(1−t) εz = t\,x + (1-t)\,\varepsilon, with pixels in [−1,1][-1, 1]. A clean image has t=1t = 1 and pure noise has t=0t = 0. A noisy image y=x+σεy = x + \sigma\varepsilon corresponds to t=1/(1+σ)t = 1/(1+\sigma), that is, σ=(1−t)/t\sigma = (1-t)/t. On the usual 0–255 scale, the noise level is σ255=127.5 σ\sigma_{255} = 127.5\,\sigma, because the range [−1,1][-1, 1] spans 255 grey levels. For example, σ255=75\sigma_{255} = 75 corresponds to t≈0.63t \approx 0.63, and the noise-filled holes (σ=20\sigma = 20, i.e. σ255=2550\sigma_{255} = 2550) to t≈0.05t \approx 0.05. The estimated map holds one value of tt per patch. In the figures, brighter patches are noisier. We have added this conversion to Section 3 and fixed the reference in the introduction.


Q2: What are the labels in figure 7?

The first column shows the strength of the painted noise in each 16×16 patch, from black (no noise) to white (pure noise). The second column shows the resulting input. Each remaining column is one method, and its label says what the method receives besides the input. The edit map is the painted strength. The true map is the actual noise level of every pixel. The two differ only in row 3, where the photograph was already noisy before we painted on it. For SDEdit + paste-back, we run SDEdit on the whole image and then paste the original back outside the painted region. Blended Diffusion and RePaint receive the painted region as a binary mask, DiffPIR receives the true noise map, and our model receives only the input. We have rewritten the caption to define each label. Please note that this figure is now Figure 8.


Q3: Unclear influence of training strategy and budget. In Table 1, the spatial model trained from scratch has worse FID (tuned cfg) than the fine-tuned model. It would be interesting to understand whether this reflects undertraining or a deeper advantage of learning generation first and adding spatial denoising after.

To separate the two, we continued training the from-scratch spatial model for 200k more steps, with its own optimizer state and learning rate, and compared it with the converted model (Appendix H, Table A10). The table gives FID on 50k samples, at the best guidance scale (3.5 for every model) and without guidance. "Images seen" counts all training images; for the converted model it includes the pretraining of its source model.

ModelImages seenFID, tunedFID, no cfg
From scratch, 600 epochs769M4.7737.5
+ 50k steps820M4.6436.6
+ 100k steps871M4.5636.1
+ 150k steps922M4.5835.9
+ 200k steps974M4.4934.9
Converted820M3.6536.2

With guidance, more training helps only slowly, and the from-scratch model stays well behind the converted one, even with more training (4.49 vs. 3.65). At the budgets we tested, learning to generate first and adding spatially varying noise afterwards gives the better guided generator. Without guidance, the from-scratch model was undertrained. It passes the converted model after 150k more steps (35.9 vs. 36.2).


Q4: In addition, I believe that extending the original JiT baseline by 50k steps would provide a better-matched training-budget comparison. This would help determine whether the converted, uniform-trained blind model's slight improvement over the original comes from removing timestep conditioning or simply from additional training.

Thank you for this suggestion. We trained the original JiT-B/16 for the same 50k steps as the blind uniform model, with the same recipe, but kept its timestep input (Appendix H, Table A9). FID uses 50k samples, each model at its own best guidance scale (in brackets), and without guidance:

Model (JiT-B/16)FID, tuned (cfg)FID, no cfg
Original3.60 (3.0)26.0
+50k steps, timestep kept3.49 (3.0)26.7
+50k steps, blind, uniform noise3.51 (3.5)30.9

The blind model indeed improves on the original only because of the extra training. With the same training, the model that keeps the timestep reaches the same FID (3.49 vs. 3.51). Removing the timestep therefore costs nothing with guidance, but it costs 4.2 FID without guidance (30.9 vs. 26.7). The released JiT-B/16 shows the same pattern (3.63 originally, 3.42 with 50k more steps and the timestep kept).


We hope these additions answer the reviewer's questions and are happy to clarify any further questions if they come up.

Reviewer_GAr5OpenReview ↗

I keep my accept recommendation

Thank you for the careful responses and additional experiments, which address my main concerns. I maintain my acceptance recommendation, as I believe this is an interesting contribution to the community.

Reviewer_6Ys2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a blind spatially varying denoising method that enables diffusion models to process images with varying noise levels across different regions without requiring a time step or an external noise map. The model directly predicts the clean image and uses the residual between the input and the prediction to estimate local noise levels, thereby determining the estimated time step. Through training on spatially varying noise, the model acquires local denoising capabilities, allowing it to restore noise-filled regions and perform noise-painting edits.

Critical strengths
  1. A clear distinction is made between local noise estimation and local denoising capability. The authors conducted extensive experiments to analyze the difference between local noise estimation and local denoising capability, finding that models trained only on uniform noise can estimate local corruption levels yet fail to restore differently corrupted regions. This finding provides valuable insights into the underlying mechanisms.
  2. The method is straightforward and makes full use of the existing model outputs. Performing flow matching in pixel space allows for direct use of the model’s inputs and outputs to estimate noise, eliminating the need to train a separate estimation network.
  3. The method offers practical value by reducing the burden of user-provided controls. Without supplying explicit noise maps or separate masks, users can specify where and how strongly to modify an image by painting noise onto it, enabling denoising, inpainting, outpainting, and local editing through a unified interface.
Critical weaknesses
  1. Residuals cannot completely distinguish between true noise and model prediction errors. Residuals also contain reconstruction errors and their correlation terms with noise. Complex textures may be incorrectly classified as noise. Experiments with low-noise data have shown that this significantly limits high-fidelity reconstruction performance.
  2. Patch-level estimation limits spatial control accuracy. If a single patch contains both clean regions and regions with strong noise, a single time value cannot accurately describe the internal differences within it. Fine-grained noise training can improve adaptability but cannot eliminate initialization mismatches.
  3. The reliability of fixed-point estimation is unclear. A small number of iterations proved effective in experiments, but it remains unclear whether oscillations or incorrectly fixed points occur when dealing with complex textures, large holes, or out-of-distribution inputs.
  4. The scope of applicability is limited. The current algorithm can only operate on prediction models for clean images in pixel space and is difficult to transfer to latent space models.
Questions for authors
  1. Fixed-point stability: How does the reconstruction error vary under different initializations and numbers of iterations? Is there any evidence of non-convergence or convergence to an incorrect state?
  2. Influence of patch size: The paper presents model results for different patch sizes but lacks ablation studies on patch sizes under fixed image resolutions and other training conditions. We recommend further analysis of the impact of patch size on the stability of noise estimation, the accuracy of boundary reconstruction, and computational cost.
  3. Global vs. Local Reading Selection: Can the method automatically determine whether the input noise is approximately uniform, and thereby select a global, local, or fused estimate?
  4. Sampler Dependency: Aside from Euler and Heun, does changing the sampler affect the error in the noise map?
  5. Generalization to latent models: Can the underlying idea of this algorithm be generalized to diffusion models in latent spaces?
4 comments

We thank the reviewer for the thoughtful technical questions. We have added a step-by-step derivation of the noise estimate with an error bound (Appendices C and D), along with analyses of fixed-point convergence (Appendix E), noise boundaries inside a patch (Appendix J) and the impact of choosing different patch sizes (Appendix K). We answer each point below.


W1: Residuals cannot completely distinguish between true noise and model prediction errors. Residuals also contain reconstruction errors and their correlation terms with noise. Complex textures may be incorrectly classified as noise. Experiments with low-noise data have shown that this significantly limits high-fidelity reconstruction performance.

We agree that the residual contains the model's reconstruction error e=x−x^e = x - \hat{x} as well as the noise. In the revision we bound its effect (Appendix D). By the triangle inequality, the noise estimate is off by at most about tfloor=∥e∥/d+∥x^∥2t_{\text{floor}} = \lVert e \rVert / \sqrt{d + \lVert \hat{x} \rVert^2}. The bound holds for any ee, including error correlated with the noise.

We measured it on the 5000 held-out images of Table 2, with patches sorted by texture (Table A4). From noise level 1−t=0.11 - t = 0.1 on, the error is below 0.02 for every texture group. On clean patches the reconstruction error shows: textured patches read a noise level of 0.071, smooth ones 0.011. So textures can indeed be read as weak noise, and at low noise this limits restoration. Below σ255≈40\sigma_{255} \approx 40, denoisers trained on that range are stronger, as we state in the limitations (Appendices A and T).


W2: Patch-level estimation limits spatial control accuracy. If a single patch contains both clean regions and regions with strong noise, a single time value cannot accurately describe the internal differences within it. Fine-grained noise training can improve adaptability but cannot eliminate initialization mismatches.

One noise level indeed cannot describe a patch that is half clean and half noisy. We measured how large this effect is (Appendix J). We placed a noise boundary at controlled offsets from the patch grid on 200 ImageNet validation images. In the table below, the columns give the noise level 1−t1 - t left / right of the boundary (0 clean, 1 pure noise), and the cells give PSNR (dB) in a 16 px band on the cleaner side. When the boundary cuts a patch, the estimate for that patch is dominated by its noisy part. It is off by up to 0.48, compared to at most 0.02 elsewhere. With the default pipeline, the clean part of the patch is then treated as noisy and redrawn ("default" columns). Training on noise maps whose boundaries cut patches does not remove this. Smaller patches shrink the affected area but do not remove it either (Appendix K).

The problem lies in using one noise level per patch. This can be solved without retraining by using a patch-level sampler. It first measures a noise level per pixel, from the residual smoothed over 3×3 neighborhoods. Each patch then starts at the noise level of its noisiest pixels, so the model sees a patch with a single level, as in training. After every step, we copy the cleaner pixels back from the input and add noise to them at the patch's current level, as RePaint [A] does for masked pixels. The clean part of a cut patch is thus kept instead of redrawn. This recovers 30.7 instead of 15.5 dB in the worst case ("patch-level" columns), at a cost of about 1 dB where the boundary already lies on the patch grid.

Boundary position0 / 1, default0 / 1, patch-level0 / 0.5, default0 / 0.5, patch-level0.2 / 0.8, default0.2 / 0.8, patch-level
On a patch edge33.131.933.231.829.027.9
4 px into a patch27.230.530.531.426.527.6
Halfway through a patch15.530.723.631.417.627.5

In our editing experiments, strengths are painted on the patch grid, where the default pipeline works as is. For edits that do not follow the grid, the patch-level sampler can be the better choice.

[A] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. CVPR 2022.

W3: The reliability of fixed-point estimation is unclear. A small number of iterations proved effective in experiments, but it remains unclear whether oscillations or incorrectly fixed points occur when dealing with complex textures, large holes, or out-of-distribution inputs.

Q1: Fixed-point stability: How does the reconstruction error vary under different initializations and numbers of iterations? Is there any evidence of non-convergence or convergence to an incorrect state?

We tested the fixed point on the three cases mentioned above and added those results in Appendix E. For that, we used the 100 most textured images of our ImageNet validation set, 100 images with large holes (25–47% of the image), and 100 out-of-distribution images (synthetic grids of grey squares). Each image was started from six initial maps: all clean, all pure noise, one global estimate, and three random maps. We then ran 20 passes of the fixed-point loop (scale by the map, predict, re-read).

No non-convergence or oscillation. Every run converged. Apart from numerical jitter below 10⁻⁴, no estimate moved back and forth.

One fixed point. All six starts end at the same map and at the same restored image.

No wrong fixed point. In one hole image, the settled map reads clean, strongly textured patches at a noise level of about 0.2. Every start reaches this same map, so it is not a second fixed point. It is the reconstruction-error floor analyzed in Appendix D.2.

Number of passes. From any start, the map is within 0.01 of its final value after 4–5 passes. From the paper's start, 3 passes are enough on textured and out-of-distribution inputs. Large holes need about 5: after 3 passes, PSNR outside the hole is 29.7 dB, against 32.0 dB once settled. On the restoration benchmark, we compared 3 passes with iterating until no patch changes by more than 10⁻³ (6.6 passes on average), on the 200 ImageNet validation images of Table A19 with the same noise draws. We use two noise ranges: the default band, with noise standard deviation σ₂₅₅ ∈ [13, 153] on a 0–255 scale plus holes, and the low band, σ₂₅₅ ∈ [5, 50]. More passes never lower the result (PSNR in dB below, Table A6). They help most on holes and at low noise, and little in the default band. We keep 3 passes as a cheap default and report the rest as an ablation.

BandPassesUniformSmoothPiecewiseBrightnessHoles
Default3 passes26.9326.7126.8827.7025.22
DefaultSettled26.9826.8027.0627.9127.38
Low3 passes29.1829.0528.6328.33–
LowSettled30.3130.0630.3229.50–

W4. The scope of applicability is limited. The current algorithm can only operate on prediction models for clean images in pixel space and is difficult to transfer to latent space models.

Q5: Generalization to latent models: Can the underlying idea of this algorithm be generalized to diffusion models in latent spaces?

A blind pixel-space model trained on spatially varying noise reads the noise level of every patch and can do restoration and editing without a noise map. This works because the white noise added to an image in pixel space is the same kind of noise the model was trained on.

The recipe would work for a latent model if the noise were added directly to the latent, since that is the noise a latent model is trained on. But in a noisy image, the noise is in the pixels and passes through the encoder first. We tested this with the SDXL encoder on 200 ImageNet images. After encoding, the noise depends on the image content, and strong and weak noise look similar. A region with four times the noise standard deviation (σ255=100\sigma_{255} = 100 vs. 25) carries 16 times the noise energy in pixels, but only 2.2 times in the latent, while the energy at a single noise level varies by 40–60% across regions (5% in pixels). The residual reading therefore no longer works directly in latent space.

Q2: Influence of patch size: The paper presents model results for different patch sizes but lacks ablation studies on patch sizes under fixed image resolutions and other training conditions. We recommend further analysis of the impact of patch size on the stability of noise estimation, the accuracy of boundary reconstruction, and computational cost.

We have added this ablation (Appendix K). We trained the region model with P=8P = 8, 16 and 32 at 256 px for 200 epochs, changing nothing else. The P=16P = 16 model reaches FID 6.99.

P=8P = 8P=16P = 16P=32P = 32
Spread of the noise estimate (theory 1/2d1/\sqrt{2d})4.6% (5.1%)2.2% (2.6%)1.2% (1.3%)
Noise level read on clean images0.0100.0380.075
Clean-side PSNR at a boundary halfway through a patch (dB)28.817.112.1
Restoration, smooth noise, PSNR (dB)25.9025.4024.12
FID-50k, tuned4.976.9919.45
GFLOPs per forward pass2185015

Stability of noise estimation: Larger patches give steadier estimates, close to the theory, but reconstruct less accurately.

Accuracy of boundary reconstruction: A patch cut by a noise boundary gets one reading, set by its noisy part, with the same error at every PP. Smaller patches lose fewer clean pixels at such a boundary. At P=16P = 16, the patch-level sampler of Appendix J mitigates this.

Computational cost: Smaller patches restore and generate better, at about four times the compute per halving of PP. Our model uses P=16P = 16, as JiT-B/16 does. A repeat at 128 px with P=4P = 4, 8 and 16 shows the same pattern.


Q3: Global vs. Local Reading Selection: Can the method automatically determine whether the input noise is approximately uniform, and thereby select a global, local, or fused estimate?

Under uniform noise, the per-patch estimates differ only slightly, so a large spread between them signals non-uniform noise. The method can therefore check this spread and use the global estimate when it is small and the local one otherwise.


Q4: Sampler Dependency: Aside from Euler and Heun, does changing the sampler affect the error in the noise map?

The sampler does not affect the error in the noise map in any meaningful way. We compared Euler and Heun with two samplers of a different kind, DPM-Solver++(2M) [B] and a stochastic DDIM sampler [C] that adds fresh noise at every step (η=0.5\eta = 0.5 and 1), at matched numbers of network evaluations (10, 25 and 50). We used 500 ImageNet validation images, 125 for each of four corruption types of Table 2 (uniform, smooth and piecewise noise, and holes). The noise map is estimated once before sampling, so its starting error is the same for every sampler. During sampling, the model re-reads the map at every step, and the read level stays close to the scheduled level for every sampler: within 0.0005–0.0016 for the deterministic samplers and up to 0.004 for the stochastic one.

The sampler does change restoration. The three deterministic samplers stay within 0.3 dB of each other (within 0.8 dB on holes), while the stochastic DDIM sampler loses up to 8 dB on average at 50 evaluations, because the fresh noise discards part of the noisy input at every step.

[B] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li and Jun Zhu. DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models. arXiv:2211.01095, 2022.

[C] Jiaming Song, Chenlin Meng and Stefano Ermon. Denoising Diffusion Implicit Models. ICLR 2021.


We hope these additions answer the reviewer’s questions and are happy to clarify any remaining points.

Dear Reviewer 6Ys2,

We wanted to check whether our responses have addressed your concerns and if there are any remaining questions. We are happy to clarify anything before the deadline which is approaching very soon!

Thank you!

Reviewer_R5b1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes blind spatially varying denoising using a timestep-free pixel-space diffusion model to estimate local noise levels. Without an additional noise map or mask, the model supports spatially varying denoising, noise-filled hole completion, outpainting, and local image editing.

Critical strengths
  1. The proposed blind spatially varying denoising method handles different noise levels across different regions of a single image.
  2. Without an additional noise map, the proposed model supports spatially varying denoising, noise-filled hole completion, outpainting, and local image editing.
Critical weaknesses
  1. The derivation of Equation (2) in the Method section is not sufficiently detailed. Please provide a step-by-step derivation in the appendix to clarify the underlying assumptions and mathematical reasoning.
  2. Although the proposed method does not require an explicit mask or noise map, it still relies on selectively adding noise to predefined regions. Can the existing model perform inpainting by adding uniform noise to the entire image, without knowing the locations of missing regions?
Questions for authors
  1. Can the proposed spatially varying denoising method effectively handle real-world noise?
  2. Does spatially varying noise provide any benefits for standard image generation (i.e., generating images from pure noise)? In particular, does it improve training convergence speed or generation quality metrics (e.g., FID)?
  3. In lines 122-123, the authors state that the method supports local image edits. Please provide additional visual examples to demonstrate this capability.
  4. The protocol in Table A1 yields 13,500 votes (3×60×3×25), which differs from the 13,506 votes reported in the main text in line of 680; the authors should explain this discrepancy.
  5. The captions of Figures A7 and A8 do not match the figures: (1) Please correct the captions: the captions list five corruptions (smooth noise, piecewise noise, brightness-dependent noise, a noise-filled hole, and a noise stroke), but the figures show six (uniform noise, smooth noise, piecewise noise, brightness-dependent noise, a noise-filled hole, and a noise stroke). (2) the captions list "our restoration" as the second row and "the blind model with its global clock" as the third, while the figures show "blind" second and "ours" third. Please use consistent labels and row ordering throughout. (3) "Holes and strokes are sampled; the rest is one x ̂." is unclear; please explain the sentence x ̂.
  6. The phrase "this ability" in the second contribution is ambiguous; please specify the ability to which it refers.
3 comments

We are thankful for the reviewer's valuable feedback and respond to their comments individually below.


W1: "The derivation of Equation (2) in the Method section is not sufficiently detailed. Please provide a step-by-step derivation in the appendix to clarify the underlying assumptions and mathematical reasoning."

We thank the reviewer for this suggestion. We have added a step-by-step derivation of Eq. (2) to Appendix C. We summarize it here.

Eq. (2) recovers tt from the size of the residual between the noisy patch zz and the network's prediction x^\hat{x} of the clean patch:

t^=1−∥z−x^∥d+∥x^∥2,(2)\hat{t} = 1 - \frac{\lVert z - \hat{x} \rVert}{\sqrt{d + \lVert \hat{x} \rVert^2}}, \qquad \text{(2)}

where dd is the number of entries in a patch.

Step 1: Relate the residual to tt. From Eq. (1), the noisy patch is z=tx+(1−t)εz = tx + (1-t)\varepsilon, where xx is the clean patch. Subtracting xx from both sides gives

z−x=(1−t)(ε−x).z - x = (1-t)(\varepsilon - x).

Taking squared norms:

∥z−x∥2=(1−t)2∥ε−x∥2.\lVert z - x \rVert^2 = (1-t)^2 \lVert \varepsilon - x \rVert^2.

Step 2: Compute the expected size of ε−x\varepsilon - x. Expanding the norm gives

∥ε−x∥2=∥ε∥2−2⟨ε,x⟩+∥x∥2.\lVert \varepsilon - x \rVert^2 = \lVert \varepsilon \rVert^2 - 2 \langle \varepsilon, x \rangle + \lVert x \rVert^2.

The noise ε∈Rd\varepsilon \in \mathbb{R}^d has i.i.d. N(0,1)\mathcal{N}(0,1) entries and is independent of xx. Its expected energy is therefore dd, and its expected inner product with xx is zero. Taking expectations:

E∥ε−x∥2=d+∥x∥2.\mathbb{E}\lVert \varepsilon - x \rVert^2 = d + \lVert x \rVert^2.

Step 3: Replace the realized value by its expectation. For large dd, both the noise energy and the noise–image cross term concentrate around their means, so

∥z−x∥2≈(1−t)2(d+∥x∥2).\lVert z - x \rVert^2 \approx (1-t)^2 \left( d + \lVert x \rVert^2 \right).

Solving for 1−t1 - t:

1−t≈∥z−x∥d+∥x∥2.1 - t \approx \frac{\lVert z - x \rVert}{\sqrt{d + \lVert x \rVert^2}}.

Step 4: Substitute the prediction. The clean patch xx is unknown at inference time, so we use the network's prediction x^\hat{x} in its place. We write the result as t^\hat{t}, an estimate of tt. This gives Eq. (2).

Assumptions. The derivation relies on:

  1. ε\varepsilon has i.i.d. N(0,1)\mathcal{N}(0,1) entries and is independent of xx.
  2. For large dd, ∥ε−x∥2\lVert \varepsilon - x \rVert^2 is close to its expectation, so we can replace it by its expectation. For our 1616 px patches (d=768d = 768) and a perfect prediction, the standard deviation of 1−t^1-\hat{t} across noise draws is at most 1/2d≈2.55%1/\sqrt{2d} \approx 2.55\% of its value.

Appendix C also derives the corresponding form of the estimate that restoration uses on a noisy image, y=x+σεy = x + \sigma \varepsilon. It explains how the two forms relate, and where the method uses each.

 


 

W2: "Although the proposed method does not require an explicit mask or noise map, it still relies on selectively adding noise to predefined regions. Can the existing model perform inpainting by adding uniform noise to the entire image, without knowing the locations of missing regions?"

The model can only find a missing region if that region is noisier than the rest of the image. Under the same noise level everywhere, the estimated noise map is uniform and the missing region is treated as image content. We have added this to the limitations (Appendix A).

For inpainting, the painted noise therefore acts as the mask. Since it is part of the image, the model does not need a separate mask input, and the amount of noise sets how strongly a region is changed. For images that are already noisy, no locations are needed. The model finds the noisy regions by itself.

Q1: Can the proposed spatially varying denoising method effectively handle real-world noise?

Our model handles real-world noise to a useful degree without any training on it, but it does not compete with models trained on SIDD (Smartphone Image Denoising Dataset). Appendix R evaluates this zero-shot on SIDD. Reading one global noise level, our uniform-trained blind model reaches 27.69 dB from a noisy input at 23.66 dB, ahead of published denoisers trained on Gaussian noise (CDnCNN-B 26.20 dB, Restormer 23.99 dB). Our measured map also lets DRUNet, a denoiser that takes a noise map as input, reach 27.59 dB. Models trained on SIDD itself reach about 40 dB (e.g. Restormer, 40.02 dB), so real camera noise is clearly better served by training on it. Our result shows that the method transfers to real noise without such training.


Q2: Does spatially varying noise provide any benefits for standard image generation (i.e., generating images from pure noise)? In particular, does it improve training convergence speed or generation quality metrics (e.g., FID)?

Our model is blind and reads the noise level per patch. The point of this design is that the same model still generates images from pure noise, and can also restore and edit images without a noise map. Generation quality with CFG stays comparable to the conditioned model (Appendix H, Table A9). We did not observe faster convergence or better FID than the conditioned model. Spatially varying training adds the ability to restore and edit, which the conditioned model does not have.


Q3: In lines 122-123, the authors state that the method supports local image edits. Please provide additional visual examples to demonstrate this capability.

We added Figure 6 with visual examples of local edits that change part of the object/scene (e.g. the centre of a daisy and the coast below a lighthouse). Only the painted region changes, and each noise draw gives a different fill. Furthermore, Figure A6 shows object inpainting with nine fills per image, on ImageNet with our JiT model and on COCO with MiniT2I, and Figure A5 shows edits with MiniT2I. These add to the stroke edit in Figure 7 and the comparison with baselines in Figure 8.


Q4: The protocol in Table A1 yields 13,500 votes (3×60×3×25), which differs from the 13,506 votes reported in the main text in line of 680; the authors should explain this discrepancy.

The main study has 540 pairs (60 images, 3 settings, 3 baselines), and we requested 25 votes for each, 13,500 in total. The platform delivered 13,506, as six pairs received a 26th vote from one extra rater. We kept these votes. The preference for each image is the share of its votes, so an extra vote does not give a pair more weight. The four rows of the figure comparison were rated separately, with 50 votes per pair, 600 in total. We have corrected the description in Appendix B.


Q5: Captions of Figures A7 and A8

Thanks for spotting this. We fixed the captions for the figures accordingly in our revision.


Q6: The phrase "this ability" in the second contribution is ambiguous; please specify the ability to which it refers.

"This ability" referred to blind spatially-varying denoising, i.e. reading the noise level of each patch and restoring it at that level, without a noise map as input. The second contribution, abstract, and conclusion now says so explicitly.


We hope that we were able to resolve the reviewer's concerns, and we are happy to clarify any further questions. We would be grateful if the reviewer takes them into account in their final assessment.

Dear Reviewer R5b1,

As the discussion period is ending soon, we wanted to check whether our responses have addressed your concerns and if there are any remaining questions. We are happy to clarify anything before the deadline!

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.