- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method to select automatically the best-performing denoiser among a set of off-the-shelf denoisers, given a noisy image with unknown noise type and noise strength. No training is necessary. The method is based on computing basic statistics of the noise directly on the noisy image itself, such as estimations of the variance of the noise, skewness, kurtosis... and then makes a prediction via nearest neighbor selection on a calibration set. The authors claim that their method is better than simply considering the best overall denoiser or the one chosen based on a blind estimation of the PSNR.
Critical strengths
A lot of off-the-shelf denoisers are available, from classical ones to neural networks. Each have their strengths and weaknesses, and some are dedicated to specific noise types. From the practitioner's point of view, it may be difficult to know in advance which denoiser will be the most efficient on their input noisy image. The paper tackles this issue, which is little studied in the literature, to the best of my knowledge. One of the main advantage of the method is that no additional training is required: once chosen, the denoiser can be used directly, without fine-tuning.
Critical weaknesses
Overall, I am not enthusiastic about this paper, which I find rather weak in terms of contributions and novelty. I do not think the paper meets the acceptance bar for ICLR, despite the numerous experiments. Here are the main weaknesses in my opinion:
Major:
- The main idea is not very original and not very sophisticated: the authors basically propose to compute some basic statistics of the noise from the input noisy image itself (the descriptor) and then use the denoiser which was identified to perform best for this descriptor (routing procedure), identified via a simple nearest neighbor technique on a calibration set.
- For the routing procedure, a calibration set consisting of pairs of noisy/clean images is necessary. Denoising must be run, for each noisy image of the calibration set, for each denoiser in the pool of denoiser and for each condition (i.e. noise type)! If the practitioner wants to denoise a single noisy image, it is not worth it and it would be much faster to test them all on the input noisy image and choose visually which output is best.
- The comparisons are not always convincing. For example, it is clear that the best overall denoiser is a poor baseline if the noise types considered are very different. In Figure 2 and 9, the best overall denoiser is TV for example, which is one of the simplest (and least efficient) denoiser, its only quality being that it is versatile. By the way, I am surprised that a denoiser such as SCUNet is not the best overall denoiser. Do you have explanations? I suspect that vertical banding noise and other very specific noises, which may not be present in the training set of SCUNet (but coud be added), weights on the overall performance. Moreover, comparisons could have be made with self-supervised techniques such as blind-spot networks (noise2self, noise2void), neighbor2neighbor, self2self, noise2vst, etc, when the noise is not spatially correlated, and AP-BSN, PUCA, etc, when it is.
Minor:
- Figure 2 is not very clear and coud be improved.
- Typo line 228: ImageNette -> ImageNet
Questions for authors
- The descriptor maps the input noisy image to a vector of size by computing many arbitrary statistics, such as the global variance of the horizontal/vertical differences, or the spatial correlation. Have you thought about learning this descriptor?
- I don't understand the difference between specialist and general-purpose denoisers in Table 3. Why is S_SwinIR not considered to be a specialist model since it was trained on Gaussian noise with . Same remark holds for DRUNet where the value of needs to be provided. Moreover, M_gaussian_25, S DRUNet_s25 and S_SwinIR_s25 are placed in three different pools, it doesn't make sense to me as they are all supposed to be dedicated to handle Gaussian noise with . By the way, what is the point in distinguishing between Classical/Specialist/General-purpose and Hybrid candidates? A denoiser is a denoiser in the end, isn't it?
- What if the noise type of the single-noisy image is different from the ones that are expected, i.e. the ones described in Table 1, and that no clean/noisy pairs are available so that you cannot add references of the new noise? What about Poisson-Gaussian noise for example or compression noise (for example JPEG artifacts)?
3 comments
Response part 2
Q1: learning the descriptor. Yes, and we have now tried it. We trained a CNN on the noisy image pixels, with the same calibration set and labels, to predict the calibration condition, from which the best denoiser is read as in our method. It did about as well as D13 but not better (Table A below and Appendix L). It also has to be retrained whenever a new noise type is added, while D13 only needs some clean/noisy pairs of the new noise.
Table A. Learned alternatives trained on the same data and labels. Routing gain (dB) over the best overall denoiser on the 26 withheld conditions. For each learned alternative, the table gives its best variant in each pool, chosen after the test results were seen, which favors the alternatives. An asterisk marks a significant difference from D13 with nearest-neighbor routing (95% paired interval excluding zero).
| Method | Classical | Specialist | Pretrained | Hybrid | All 36 |
|---|---|---|---|---|---|
| D13, nearest neighbor (ours) | +1.36 | +1.58 | +0.14 | +0.99 | +1.20 |
| Learned router on D13 (best of 8 variants) | +1.36 | +1.56 | +0.05 | +0.85 | +1.10 |
| CNN on image pixels | +1.29 | +1.53 | −0.11* | +0.97 | +1.04* |
Q2: the distinction between pools. We agree that a denoiser is a denoiser, and the revision now reports results for all 36 as a single set (Section 4.2 and Appendix E). We kept the separate pools because they show that selection helps most when the denoisers have different strengths. “General-purpose” was also a poor name for networks like DRUNet and SwinIR, which in our pool are run at a single noise level, and we now call them “pretrained.”
Q3: an unexpected noise type, with no clean/noisy pairs. If there are no clean/noisy pairs for a new noise type, we cannot expect the method to choose well for it, but it can often detect the situation before any denoiser is run. On real camera images from SIDD, the descriptor flagged 90% of them as unlike anything in the calibration set, and those images can simply be sent to the best overall denoiser. A noise-type classifier gives no such warning. The one we reimplemented assigned images of unfamiliar noise types to one of its four classes with a mean confidence of 0.93. For the revision, we tried the two cases you mention (Appendix O and Figure 10). (i) Poisson–Gaussian noise needed no new pairs. The descriptor placed these images next to the Poisson conditions they combine, so only 0.5% of them were flagged. With our specialist networks, routing sent them to the Poisson specialists and gained 3.93 dB, which is 91% of what a perfect per-image choice would achieve. When SCUNet was among the candidates, there was almost nothing to gain over it, and routing lost 0.2 to 0.3 dB. (ii) JPEG is closer to the situation you described. The descriptor flagged 45% of the JPEG images, against 7% of the withheld synthetic images, and without JPEG pairs routing fell 2.04 dB below the best overall denoiser. The flag alone is therefore not enough for JPEG, and labeled examples are needed. We then added labeled JPEG examples at each quality level and one denoiser designed for compression artifacts (FBCNN), without retraining anything. Routing then gained 0.52 dB over the best overall denoiser, 67% of what a perfect per-image choice would achieve, and 0.46 dB at a quality level that had no examples of its own (Appendix O).
Minor points. We have moved Figure 2 to Appendix J and are redrawing it with all 36 denoisers, for which the best overall denoiser is SCUNet. ImageNette is the ten-class subset of ImageNet (Howard, 2019), and we will describe it where it first appears.
In summary. You were concerned that the method is too simple, that calibration is too costly to be worth it, and that the best overall denoiser is a weak baseline. On simplicity, a few statistics of the noisy image and a single denoiser run chose better than blind quality criteria that test all the denoisers and compare the outputs, which is the automated version of choosing visually. Calibration is done once and shared with the code, and anyone can extend it with a new noise type or denoiser for all users, so a practitioner pays only for the descriptor and one denoiser per image.
Your questions also led to new results. A learned descriptor did not do better than D13, all 36 denoisers are now also routed as one set with SCUNet as the best overall denoiser, and we tested the JPEG and Poisson-Gaussian cases you raised. We have also rewritten the abstract and introduction so that the main result is stated plainly, and we would be grateful if you could take another look with these changes in mind.
I sincerely thank the authors for their response and appreciate the substantial effort they have made to address my concerns. Nevertheless, despite these efforts, I still do not think that the paper meets the acceptance bar for ICLR. As I noted in my initial review, I am not enthusiastic about the core idea, which I find limited in terms of contribution and novelty. For this reason, I will maintain my original rating.
Response part 1
Thank you for your detailed review. We have uploaded a revised PDF in which new text is marked in blue. On why we split the denoisers into separate pools when a denoiser is a denoiser in the end, we fully agree, and the revision now reports results for all 36 routed as a single set (Section 4.2 and Appendix E). With all of them combined, the best overall denoiser is SCUNet (as you expected), and our method still gains 1.20 dB over it with one denoising call per image. On whether the descriptor could be learned, we trained a CNN on the image pixels with the same labels, and it performed about as well as D13 without improving on it, while D13 requires no training (Appendix L).
We also understand why the paper read as a set of basic statistics combined with a nearest-neighbor lookup, since the results were spread across four pools and many sub-cases. However, the paper shows that this simple procedure chooses better than seven methods that run every denoiser and score the outputs, and the new experiments show that a learned router does not improve on it. It works because each calibration image is labeled with the denoiser that serves it best rather than with its corruption, so the denoiser can be chosen without identifying the noise. Since the learned and handcrafted descriptors perform comparably, the handcrafted one has a practical advantage: each statistic measures a known property of the noise, so when selection fails on a new type of noise one can see which property is missing and add a statistic for it, without retraining any denoiser. We also clarify below that calibration is done once for a given set of denoisers rather than once per image. We think the paper is clearer and stronger as a result of your review and, in light of the new results below, we hope you will give it another look.
W1: originality and sophistication. We kept the method simple on purpose, and we think the main result is more notable for it. With only a few basic statistics of the noise, computed on the noisy image, and a single denoiser run, it picked better denoisers than seven blind quality criteria (NIQE, BRISQUE, MUSIQ, CLIP-IQA+, TOPIQ, ARNIQA, and MC-SURE). Each of those has to run all the denoisers on the image and compare the outputs. We also tried learned alternatives trained on the same calibration set, and they did not do better. At no point is any denoiser retrained or fine-tuned.
W2: cost of calibration. You are right that calibration means running every denoiser on the noisy/clean pairs. This happens once for a given set of denoisers, and the result is shared with the code, so a practitioner with a single noisy image only computes its descriptor, which takes about 23 ms on a CPU, and runs one denoiser. The calibration set can also grow over time. Anyone who adds clean/noisy pairs of a new noise type, or a new denoiser, needs to run only the new combinations, and once the addition is in the shared repository, every user benefits from it without retraining anything. On real camera noise, adding such pairs recovered most of the loss without affecting results on the noise types already covered. Testing all the denoisers and choosing visually, by contrast, means running every one of them on every image. The blind quality criteria above are essentially automated versions of that choice, and our method did better than all of them while running only one denoiser.
W3: the best overall denoiser, SCUNet, and self-supervised methods. We agree that the best overall denoiser is a weak baseline when the noise types are very different. We report it because it is what a practitioner would run without knowing the noise type, and the blind quality criteria above are the stronger comparison. TV comes out as the best overall denoiser only among the classical filters, which is the set shown in Figures 2 and 9 (8 and 9 in the revised version). With all 36 denoisers combined, it is SCUNet (as you suspected), and your explanation holds for impulse noise. SCUNet is about 16 dB behind the best denoiser on impulse noise, and that family alone accounts for half of our 1.20 dB gain over it, with mixtures contributing another 30%. Banding contributes only about 6%, although SCUNet is 8 dB behind there as well. On the calibration images our banding network beats the notch filter, so banding images are sent to it, but it was trained on horizontal stripes at a single period and does poorly on the vertical and longer-period stripes in the test set, where the notch filter does well. A denoiser that is best on the calibration images but fragile on noise that differs from them can therefore reduce the gain, and we will discuss this in the revision.
Self-supervised denoisers would fit in as further candidates, and we will add pretrained blind-spot nets (e.g. AP-BSN) in the final version. Methods that train on the noisy image itself, such as Self2Self, would need a training run for every image, so they fall outside the setting we consider.