- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes blind spatially varying denoising for pixel-space diffusion models, allowing a single model to estimate and remove non-uniform noise without an externally supplied noise map. The approach removes timestep conditioning and estimates each patch’s noise level from the residual between the noisy input and the predicted clean image. An interesting idea is that blind models without timestep conditioning can estimate local noise levels even when trained only on uniformly noisy images. However, they require spatially varying noise during training to denoise each region appropriately. The estimated map indicates each patch’s starting point along the denoising trajectory. The proposed method provides a unified interface for restoration, inpainting, outpainting, and local editing, where painted noise specifies the location and strength of changes without a separate mask or map input. The authors demonstrate both training from scratch and fine-tuning pretrained generators.
Critical strengths
- The paper introduces blind spatially varying denoising as a new capability for image diffusion models and presents an elegant approach that unifies restoration and editing. The method achieves quite good overall results across restoration and editing tasks.
- The authors discuss the method’s limitations transparently and provide insightful analysis of its failure modes.
- I find it interesting and I feel I have learned something.
Critical weaknesses
- Poor performance at low noise levels as a result of the model’s reconstruction errors.
- Training-range mismatch in the baseline comparisons, competing methods are evaluated outside their training noise ranges, without being retrained or fine-tuned on the full evaluation range. Although the authors acknowledge this mismatch, it remains unclear whether the reported high-noise advantage would persist under matched training conditions.
Questions for authors
- What is the definition of sigma_255?
- What are the labels in figure 7?
- Unclear influence of training strategy and budget. In Table 1, the spatial model trained from scratch has worse FID (tuned cfg) than the fine-tuned model. It would be interesting to understand whether this reflects undertraining or a deeper advantage of learning generation first and adding spatial denoising after.
In addition, I believe that extending the original JiT baseline by 50k steps would provide a better-matched training-budget comparison. This would help determine whether the converted, uniform-trained blind model’s slight improvement over the original comes from removing timestep conditioning or simply from additional training.
3 comments
Q1: What is the definition of sigma_255?
We use the flow-matching convention of JiT, , with pixels in . A clean image has and pure noise has . A noisy image corresponds to , that is, . On the usual 0–255 scale, the noise level is , because the range spans 255 grey levels. For example, corresponds to , and the noise-filled holes (, i.e. ) to . The estimated map holds one value of per patch. In the figures, brighter patches are noisier. We have added this conversion to Section 3 and fixed the reference in the introduction.
Q2: What are the labels in figure 7?
The first column shows the strength of the painted noise in each 16×16 patch, from black (no noise) to white (pure noise). The second column shows the resulting input. Each remaining column is one method, and its label says what the method receives besides the input. The edit map is the painted strength. The true map is the actual noise level of every pixel. The two differ only in row 3, where the photograph was already noisy before we painted on it. For SDEdit + paste-back, we run SDEdit on the whole image and then paste the original back outside the painted region. Blended Diffusion and RePaint receive the painted region as a binary mask, DiffPIR receives the true noise map, and our model receives only the input. We have rewritten the caption to define each label. Please note that this figure is now Figure 8.
Q3: Unclear influence of training strategy and budget. In Table 1, the spatial model trained from scratch has worse FID (tuned cfg) than the fine-tuned model. It would be interesting to understand whether this reflects undertraining or a deeper advantage of learning generation first and adding spatial denoising after.
To separate the two, we continued training the from-scratch spatial model for 200k more steps, with its own optimizer state and learning rate, and compared it with the converted model (Appendix H, Table A10). The table gives FID on 50k samples, at the best guidance scale (3.5 for every model) and without guidance. "Images seen" counts all training images; for the converted model it includes the pretraining of its source model.
| Model | Images seen | FID, tuned | FID, no cfg |
|---|---|---|---|
| From scratch, 600 epochs | 769M | 4.77 | 37.5 |
| + 50k steps | 820M | 4.64 | 36.6 |
| + 100k steps | 871M | 4.56 | 36.1 |
| + 150k steps | 922M | 4.58 | 35.9 |
| + 200k steps | 974M | 4.49 | 34.9 |
| Converted | 820M | 3.65 | 36.2 |
With guidance, more training helps only slowly, and the from-scratch model stays well behind the converted one, even with more training (4.49 vs. 3.65). At the budgets we tested, learning to generate first and adding spatially varying noise afterwards gives the better guided generator. Without guidance, the from-scratch model was undertrained. It passes the converted model after 150k more steps (35.9 vs. 36.2).
Q4: In addition, I believe that extending the original JiT baseline by 50k steps would provide a better-matched training-budget comparison. This would help determine whether the converted, uniform-trained blind model's slight improvement over the original comes from removing timestep conditioning or simply from additional training.
Thank you for this suggestion. We trained the original JiT-B/16 for the same 50k steps as the blind uniform model, with the same recipe, but kept its timestep input (Appendix H, Table A9). FID uses 50k samples, each model at its own best guidance scale (in brackets), and without guidance:
| Model (JiT-B/16) | FID, tuned (cfg) | FID, no cfg |
|---|---|---|
| Original | 3.60 (3.0) | 26.0 |
| +50k steps, timestep kept | 3.49 (3.0) | 26.7 |
| +50k steps, blind, uniform noise | 3.51 (3.5) | 30.9 |
The blind model indeed improves on the original only because of the extra training. With the same training, the model that keeps the timestep reaches the same FID (3.49 vs. 3.51). Removing the timestep therefore costs nothing with guidance, but it costs 4.2 FID without guidance (30.9 vs. 26.7). The released JiT-B/16 shows the same pattern (3.63 originally, 3.42 with 50k more steps and the timestep kept).
We hope these additions answer the reviewer's questions and are happy to clarify any further questions if they come up.
I keep my accept recommendation
Thank you for the careful responses and additional experiments, which address my main concerns. I maintain my acceptance recommendation, as I believe this is an interesting contribution to the community.
We thank the reviewer for the positive assessment and for the suggestions on matched comparisons, which we added.
W1: Poor performance at low noise levels as a result of the model's reconstruction errors.
Light noise is indeed the hardest case for the method, and we have added an analysis of it (Appendix D.2, Table A4). The noise estimate measures the residual between the input and the prediction, and this residual also contains the model's own reconstruction error. On clean images, this error sets a floor on the estimate, from 0.011 on smooth patches to 0.071 on the most textured ones (in units of the noise level 1−t). From a noise level of 0.1 on, the error is below 0.02 for all textures, so the floor matters only for nearly clean images.
Two simple measures can be used to reduce the effect. First, iterating the estimate until it settles, instead of the paper's 3 passes, adds 1.0–1.7 dB in the low-noise band (σ255 from 5 to 50). The table gives restoration PSNR (dB) on 200 randomly drawn ImageNet validation images (Appendix E, Table A6):
Second, patches that the model reads as clean can be copied from the input, which keeps nearly clean regions intact, as we do for editing. Denoisers trained specifically on light noise remain stronger in this range, as we state in the limitations (Appendix A).
W2: Training-range mismatch in the baseline comparisons, competing methods are evaluated outside their training noise ranges, without being retrained or fine-tuned on the full evaluation range. Although the authors acknowledge this mismatch, it remains unclear whether the reported high-noise advantage would persist under matched training conditions.
We agree this needed testing. We fine-tuned DRUNet, the strongest map-based baseline, on our full evaluation range (σ255 up to 153) and reran it on our benchmark (5000 images, protocol of Table 2):
(–: the true map of brightness-dependent noise reveals the clean image's brightness, so this cell is not comparable.)
Fine-tuning removes most of the high-noise gap. Given the true map, the fine-tuned DRUNet is on par with our model on noise and better on holes. Our model reaches the same level without being given any map. On uniform noise at the strongest level (σ255=153), it still leads slightly (24.6 vs. 24.1 dB; original DRUNet 17.0). Our measured map also drives the fine-tuned DRUNet to within 1.1 dB of its true-map results. The advantage over the published checkpoints at high noise is therefore mostly a matter of training range, as we now state in Section 5, and Table 2 includes the fine-tuned rows. Our contribution is a single model that measures the noise and restores without a map, at the level of a matched map-based denoiser.