- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GaborCanvas, an image-adaptive Gabor representation for self-supervised image restoration. A shared network takes local image content and predicts spatially varying Gabor parameters, including window orientation, scales, frequencies, and phases. When the local atom matrix has full row rank, the corresponding dual synthesis operator is constructed explicitly. The local reconstructions are then combined by overlap averaging. Under the stated conditions, this reconstructs the unmodified coefficients exactly.
For denoising, the method learns coefficient thresholds using the R2R loss. For SR and inpainting, it uses internal self-supervision and differentiable reconstruction with a Gabor analysis-sparsity penalty. In the implementation, SR updates the frame over several reconstruction rounds, while inpainting uses a quadratic term around a biharmonic initialization. The paper includes continuous and discrete frame results, restoration benchmarks, and additional experiments on spatial adaptation, sparse approximation, atom types, the inpainting initialization, and numerical reconstruction accuracy.
Critical strengths
-
The main methodological value is the combination of input-conditioned local Gabor parameters, spatially varying analysis systems, and corresponding local duals. The network predicts interpretable spatial-frequency structure, rather than directly producing restored pixels. In denoising, it is possible to see separately what is done by analysis, coefficient modification, and synthesis.
-
Appendix B.2 compares with learned Gaussian atoms and reports gains of 2.77 and 2.99 dB on Kodak24. This is useful evidence for the Gabor family against that alternative. It is not exactly a "remove only cosine modulation" experiment, since the Gaussian counterpart also learns its centers and scales.
-
Appendix B.4 includes the biharmonic initialization baseline. Adding the Gabor regularizer improves inpainting PSNR by 0.84 and 0.49 dB at the two missing-pixel ratios. So there is already evidence that the method adds something beyond the classical initialization. These experiments are a main reason for my positive assessment.
Critical weaknesses
-
The novelty needs to be discussed against closer prior work. An important missing work is GDLNet (Janjušević et al., 2022). It learns real 2D Gabor analysis/synthesis dictionaries in a sparse-coding-inspired denoising network. The broad combination of learned Gabor representations, sparsity, and self-supervised denoising already has precedent. This method is not the same as GaborCanvas. In particular, the input-conditioned per-location predictor and explicit local dual construction seem to be important differences. At present, the reader has to work out this positioning.
-
The AWGN R2R construction is explicit, but the noise generation and parameter estimation for SIDD and FMD are not described at the same level. These details matter for the validity and interpretation of the self-supervised training.
-
The baseline training conditions also need some clarification. For example, standard TTT-MIM uses supervised pretraining and then self-supervised test-time adaptation. It should not be grouped as entirely clean-target-free unless the authors used a different training protocol. This is a question about the comparison conditions, not evidence that GaborCanvas itself used clean targets.
-
ZSSR is a relevant missing SR baseline because it also learns from pairs constructed inside the input image. This is especially important for the broad claim about GT-free SR: at x4, the PSNR improvement over DIP is only 0.010 dB, although SSIM and LPIPS improve. A matched comparison would help. One established learned-frame baseline would also be useful in addition to the Gaussian-atom comparison. The supervision conditions should be kept clear when making these comparisons.
-
Finally, a small parameter count does not tell the actual cost here. There is image-specific fitting, local matrix computation, and iterative image reconstruction. End-to-end runtime and peak memory are needed. I am not claiming the method is necessarily slow, but the current tables do not allow a practical efficiency assessment.
Questions for authors
-
Could the authors explain the precise differences from GDLNet's self-supervised Gabor denoising, nonstationary Gabor-frame constructions, and region-adaptive learned frames? In particular, which part is new in the input-conditioned local parameterization, and which part follows from existing sparse representation and dual-frame results? This could change my novelty assessment.
-
For SIDD and FMD, what noise distribution, parameter estimates, and image domain are used for R2R? What data are used to choose hyperparameters and stopping rules, including any clean validation data? Please also clarify whether TTT-MIM uses its standard supervised pretraining, and which method was actually implemented for the PIP/Fourier INR row.
-
How does the method compare with ZSSR under the same downsampling and evaluation protocol? A comparison with one established learned-frame method under comparable information conditions would also help. I am asking for these because they are close to the current claims, not asking the authors to run every related method.
-
Please report runtime and peak memory at representative image resolutions, with the hardware specified. It would be useful to separate image-specific fitting from final reconstruction and include local matrix computations and iterative updates. Parameter counts alone are not enough for this part.
6 comments
Response to Reviewer JedJ (2/4)
(b) Hyperparameters and checkpoint selection. For SIDD, we train for four epochs (16,531 steps). The re-corruption procedure and the four-epoch budget are inherited from the official R2R code, the atom count and patch size (96 atoms, 8×8) and the learning rates were set empirically without validation data. As in the official R2R code, the model is evaluated on the SIDD validation set during training, here on a fixed subset of 32 of the 1,280 validation blocks every 250 steps, and we report the checkpoint with the highest PSNR on this subset, at step 15,500, near the end of training. Clean references enter only this checkpoint selection; they are not used in the training loss or for the hyperparameters. The same checkpoint and settings are used for SIDD Benchmark without further selection; its score is obtained through the benchmark's online evaluation server. For FMD, we optimize a separate model for each noisy image using a fixed number of 500 updates, without early stopping. The multiplier in its noise level, the atom count, and the patch size are chosen by mean PSNR against the clean references of validation images from different modality (5 Confocal, 4 Two-Photon, and 3 Wide-Field), which are among all 48 evaluation images. In both cases, the training loss uses no clean targets; clean references enter only model selection. We acknowledge that GT-free is accurate only for the training stage, and we will use the term in this narrower sense. To our knowledge, such validation-based model selection is the default protocol on these benchmarks, so the compared self-supervised methods are under the same condition.
We also evaluated sensitivity to the main representation hyperparameters by varying them individually in the Gaussian-denoising model, using all 24 Kodak24 images at with the noise realizations of Table 1. With patches, PSNR varies by only 0.16 dB across 88–144 atoms; performance decreases at 72 and 80 atoms.
Table R4-1. Number of atoms, with patches; PSNR / SSIM:
| 72 | 80 | 88 | 96 | 104 | 112 | 128 | 144 |
|---|---|---|---|---|---|---|---|
| 29.41 / .815 | 29.80 / .829 | 29.98 / .835 | 30.03 / .837 | 29.91 / .831 | 30.07 / .836 | 29.98 / .836 | 29.96 / .835 |
When varying the patch size while maintaining approximately 1.5 atoms per pixel, PSNR varies by 0.08 dB across – patches.
Table R4-2. Patch size / number of atoms; PSNR / SSIM:
| 6×6 / 54 | 7×7 / 74 | 8×8 / 96 | 9×9 / 122 | 10×10 / 150 |
|---|---|---|---|---|
| 29.80 / .827 | 29.95 / .834 | 30.03 / .837 | 30.01 / .835 | 29.86 / .830 |
These results support robustness around the adopted configuration.
(c) TTT-MIM. Yes, TTT-MIM is first trained on 38,400 ImageNet images corrupted with synthetic Gaussian noise (variance 0.005), with a supervised denoising loss against the clean images and a masked-image-modeling loss; it is then adapted to each noisy test image with 8 self-supervised gradient steps (Mansour et al., 2024). It therefore uses external clean training data and should not be grouped as entirely clean-target-free. We did not re-run it; its SIDD Validation, SIDD Benchmark, and FMD results are quoted from Table 1 of Guo et al. (NeurIPS 2025). In the revision, Table 2 will list it separately under “supervised pretraining + self-supervised test-time adaptation.”
(d) The PIP/Fourier INR row. It is our implementation of a Fourier-feature MLP (Tancik et al., 2020): 64 Gaussian random features (σ=10) and a 4×256 ReLU MLP predicting a bounded residual over bicubic upsampling. It is fitted for 1,800 Adam steps on low-resolution (LR) consistency through the bicubic downsampler (MSE plus total variation). The label "PIP" was inaccurate; in the revision, Table 3 will rename the row "Fourier-feature INR (residual)".
Response to Reviewer JedJ (3/4)
W4 / Q3. Comparisons with ZSSR and a learned-frame method; 0.010 dB gain over DIP at ×4 SR.
Response. We thank the reviewer for suggesting these comparisons.
(a) ZSSR. We re-implemented ZSSR in PyTorch following Shocher et al. (2018): an 8-layer, 64-channel residual network trained on internal low-resolution pairs, followed by back-projection. We use the same LR inputs, bicubic downsampler, and evaluation protocol as Table 3.
Table R4-3. Set14; PSNR-Y / SSIM-Y / LPIPS, with the same evaluation border crop:
| Method | ×2 | ×4 | ×8 |
|---|---|---|---|
| ZSSR | 32.05 / .911 / .151 | 26.69 / .753 / .339 | 23.32 / .559 / .510 |
| GaborCanvas | 31.37 / .906 / .167 | 26.65 / .751 / .341 | 23.63 / .578 / .505 |
We agree that the 0.01 dB margin over DIP at ×4 is not meaningful. We will add ZSSR to Table 3 and revise Section 4.2 to describe GaborCanvas as a competitive GT-free SR method. On Set14, its performance is close to ZSSR at ×4, and it outperforms ZSSR on all three metrics at the larger upscaling factor of ×8.
(b) Learned-frame and Gabor-dictionary baselines. We added DDTF (Cai et al., 2014), which learns a tight frame from a single noisy image without external training data and supports exact reconstruction when coefficients are unchanged. We implemented Algorithm 1 with its published denoising settings, applied separately to each RGB channel.
We also fit the authors’ GDLNet implementation per image using our R2R loss, noise realizations, and known noise level. We use MoG-3 with 30 stages, 169 subbands, and 11×11 filters, adapted to RGB (557,700 parameters). Starting from its own initialization, it is trained on 48×48 random crops with Adam for 400 updates. These results evaluate GDLNet adapted to our single-noisy-image setting.
Table R4-4. Same images, noise realizations, and metrics as Table 1, with additional CBSD68 evaluation on 256×256 center crops. Kodak24 and McMaster18: PSNR / SSIM; CBSD68: PSNR.
| Method | Kodak24 σ=10 | Kodak24 σ=25 | McMaster18 σ=10 | McMaster18 σ=25 | CBSD68 σ=15 / 25 / 50 |
|---|---|---|---|---|---|
| DDTF | 33.45 / .907 | 28.59 / .778 | 34.37 / .925 | 29.41 / .817 | 30.31 / 27.63 / 24.45 |
| GDLNet (R2R, per image) | 33.84 / .923 | 29.16 / .821 | 33.96 / .917 | 29.41 / .820 | 30.92 / 28.60 / 25.05 |
| GaborCanvas | 35.31 / .937 | 30.03 / .837 | 34.95 / .929 | 29.93 / .835 | 32.50 / 29.45 / 25.69 |
GaborCanvas exceeds DDTF by 0.52–2.19 dB and GDLNet by 0.52–1.58 dB across these settings.
The 400-update budget was selected among 200, 400, 800, and 1,600 updates by PSNR on Kodak24 at σ=25 (28.75, 29.16, 28.20, and 28.80 dB). GDLNet takes 57.4 s per image with 0.8 GiB peak memory, compared with 23.5 s and 2.9 GiB for GaborCanvas (W5).
These comparisons assess the methods as a whole; Table 5 isolates the contribution of spatial adaptation within our pipeline. We will add DDTF and GDLNet to Table 1 and the CBSD68 results to Appendix F.
Response to Reviewer JedJ (1/4)
We thank the reviewer for careful reading and instructive feedback.
W1 / Q1. Positioning against related work; new versus established components.
Response. We thank the reviewer for identifying these related works. Our contribution has two complementary aspects.
(1) Learning image-conditioned, spatially varying Gabor frames. GDLNet learns Gabor filters shared across spatial locations within each layer. DDTF learns one filter bank shared across an image, while region-adaptive extensions, such as piecewise DDTF (Du et al., 2026), learn separate frames for segmented regions. Our shared network predicts atom-specific orientations, frequencies, and phases from local image content at every location, without requiring a regional partition. Thus, the representation adapts both to the input image and to different locations within it, while retaining an explicit Gabor parameterization.
Table 5 (main paper) isolates the benefit of this spatial adaptation: under the same fixed-threshold protocol, the spatially varying frame improves PSNR by 0.50–0.55 dB over a learned stationary Gabor frame across all four settings. Additional comparisons with GDLNet under our self-supervised protocol and with DDTF are provided in our response to W4. GDLNet itself is trained supervised with MSE on BSD432 clean/noisy pairs; our per-image R2R experiment below adapts its architecture to our noisy-only training setting.
(2) Frame analysis with nonuniform local frequencies. In the time-adaptive nonstationary Gabor construction of Balazs et al. (2011), windows and frequency spacings may vary across locations, but frequencies remain uniformly spaced within each local atom set. Their painless frame proof uses this structure and suitable support conditions to obtain a diagonal frame operator. Our atom-dependent nonuniform frequencies break the Fourier orthogonality used in that argument, so their proof cannot be applied directly. We establish explicit, verifiable sufficient conditions for frame preservation; our continuum proof combines Gaussian fiber-matrix estimates, nonharmonic Fourier bounds, and phase-error control. Please also see our response to Reviewer Yg8B, W1, for further discussion.
What follows established methods. The local pseudo-inverse, coefficient shrinkage, and R2R loss follow established methods. Our contribution lies in the image-conditioned, spatially varying Gabor frame and the accompanying analysis of its frame property.
References.
W2 / W3 / Q2. SIDD/FMD training protocols; TTT-MIM pretraining; the PIP/Fourier INR row.
Response. We thank the reviewer for these questions. We clarify the training protocols and baseline conditions below and will include these details in the revised manuscript, submitted before the discussion deadline.
(a) SIDD and FMD: noise models and image domains. For SIDD, we use the RAW-domain re-corruption procedure of R2R (Pang et al., 2021). For each noisy RAW image y in SIDD-Medium, we generate the input–target pair
(y+2z,y−z/2),z∼N(0,β1y+β2),
using the supplied camera noise-level parameters (β1,β2) for the Bayer channels. The re-corrupted images are rendered to sRGB through the camera ISP, and training uses 128×128 crops from the resulting pairs. Separately, a per-channel median absolute deviation (MAD) estimate from the sRGB network input, σ^=MAD(d)/0.6745 with d a 2×2 diagonal-difference high-pass response, sets the scale of the normalized thresholds.
For FMD, we use Gaussian re-corruption with its noise level estimated from the noisy image by MAD. The image intensities are the 8-bit grayscale values divided by 255, with no variance-stabilizing transform, and the noise is treated as homoscedastic Gaussian in this domain with level σ=cσ^, where σ^=MAD(d)/0.6745 uses the same diagonal difference d as for SIDD and c is a per-modality multiplier (4 for Confocal, 2 for Two-Photon, 8 for Wide-Field). The R2R input–target pair is built at each update from a random 48×48 crop y of the noisy image,
(y+ασε,y−σε/α),ε∼N(0,I), α=0.5,
so the input and target perturbations have variances σ2/4 and 4σ2, and the noise level given to the model is σ1+α2.