- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes UAI-Net, an uncertainty-aware framework for image inpainting. The method predicts a spatial uncertainty map supervised by the rank of reconstruction errors, which is then used to guide feature fusion and adaptive refinement depth. The authors also introduce a learned halting mechanism with an integer depth target to address training instability. Experiments on CelebA-HQ, Places2, and FFHQ evaluate reconstruction quality, uncertainty estimation, and adaptive refinement.
Critical strengths
The paper is generally well-written, with clear explanations that make it easy to follow.
Critical weaknesses
-
Lack of a clear underlying insight. The paper proposes using uncertainty estimated from reconstruction errors to determine the refinement depth for image inpainting. However, the authors do not adequately explain why regions with larger reconstruction errors should receive more refinement steps. The method seems to be driven mainly by architectural design choices, without a convincing insight that justifies them. For a paper targeting ICLR, I would expect a clearer motivation for the core design.
-
Limited empirical improvements. The ablation results do not convincingly demonstrate the effectiveness of the proposed design. Despite introducing multiple components, the improvements are marginal. For example, in Table 3, learned halting improves PSNR over the simple scheduled strategy by only 0.01 dB for 10–20% masks and 0.02 dB for 50–60% masks. Such small differences make it difficult to determine whether learned halting provides a meaningful benefit over a simple predefined rule.
Questions for authors
-
What is the key insight behind the proposed method? In particular, why should reconstruction error or the uncertainty derived from it determine refinement depth? The authors should provide a more convincing explanation of this design choice.
-
Please improve the presentation of the figures. Some figures, particularly the architecture and module diagrams (e.g., Figures 1 and 2), are overly complicated and difficult to read. Simplifying these diagrams and increasing the readability of the labels would significantly improve the presentation.
-
Consider adopting a stronger backbone. The current model falls behind several existing methods in reconstruction quality and does not achieve state-of-the-art performance. While SOTA performance is not strictly necessary, adopting a more competitive backbone could improve the overall results and make the proposed method more competitive for a conference like ICLR.
8 comments
Response 2 to Reviewer fMRg – Submission 29718
Addendum to W1: why the learned signal rather than a geometric heuristic?
Our W1 response shows that the perceptual response to additional refinement is stronger in high-U regions. A natural follow-up is whether U provides information beyond a simpler, training-free geometric signal. We therefore repeated the forced-depth analysis using terciles defined by distance to the nearest known pixel instead of U, keeping the model, images, masks, and evaluation procedure unchanged.
| Mask | LPIPS change using U: low / mid / high | LPIPS change using distance: low / mid / high |
|---|---|---|
| 10-20% | -7.3% / -9.3% / -12.6% | -8.5% / -10.1% / -12.0% |
| 20-30% | -6.4% / -9.6% / -13.3% | -8.3% / -10.0% / -12.6% |
| 30-40% | -5.7% / -9.3% / -13.3% | -7.6% / -9.7% / -12.5% |
| 40-50% | -4.2% / -8.2% / -12.9% | -6.5% / -8.6% / -11.9% |
| 50-60% | -4.1% / -8.0% / -12.5% | -6.1% / -8.4% / -11.5% |
Distance-to-boundary is not an uninformative control: regions farther from known content also exhibit larger perceptual improvements from additional refinement. However, the high-to-low tercile spread is 5.3-8.7 percentage points under U, compared with 3.5-5.4 under distance. Thus, the separation of perceptual responses is 1.51-1.61 times larger under U across all five buckets.
These results show that U stratifies the perceptual response to additional refinement more sharply than this geometric heuristic across all five mask buckets. It is also consistent with our error-ranking evaluation, which is a separate measurement on 500 images per bucket using 1,000 sampled hole pixels per image: there, trained U reaches an AUROC of 0.8293 against 0.6693 for the distance heuristic in the 10-20% bucket, and 0.7989 against 0.6742 in the 50-60% bucket, so the margin narrows as holes grow.
We acknowledge that this comparison covers one training-free geometric alternative, one model, and one dataset. It does not establish that U is the best possible allocation signal; alternatives such as feature magnitude and predictive entropy remain untested. We therefore present this result as evidence supporting the choice of U over a simple geometric heuristic, rather than claiming that it is uniquely optimal.
Response 3 to Reviewer fMRg – Submission 29718
*W2. Limited empirical improvements, and whether learned halting beats a predefined rule
Review: "The ablation results do not convincingly demonstrate the effectiveness of the proposed design. Despite introducing multiple components, the improvements are marginal. For example, in Table 3, learned halting improves PSNR over the simple scheduled strategy by only 0.01 dB for 10-20% masks and 0.02 dB for 50-60% masks."
The reviewer raises two points: whether the components justify their complexity, and whether learned halting beats a predefined schedule. We address both and state where the evidence does not support an advantage.
What each component measurably provides. The PSNR differences cited are accurate and we do not dispute their magnitude. PSNR alone, however, does not capture the measured benefits of the uncertainty and allocation components.
| Component | Measurable benefit | Evaluation |
|---|---|---|
| Variance head | Retaining the 70% lowest-uncertainty images increases hole PSNR by 1.06-1.59 dB; selection uses predicted variance, no ground truth | Selective prediction |
| Variance head | Normalised AURC of 0.783-0.822 with one forward pass, compared with 0.664-0.694 for two-pass flip-TTA | Error ranking |
| Uncertainty map U | Stratifies the perceptual response to refinement 1.51-1.61 times more sharply than distance-to-boundary across five buckets | Tercile LPIPS |
| Learned halting | Matches the hand-designed U-based schedule with 0.161-0.170 fewer per-pixel refinement updates | Per-pixel depth |
| Adaptive depth | No fidelity or wall-clock advantage demonstrated | PSNR and runtime |
We acknowledge the limitation in the last row. In a matched-batch second-backbone experiment, frozen and trainable configurations differed by 4.06 steps in depth spread but only 0.0115 dB in PSNR. Raising the budget from K=4 to K=8 changed PSNR by +0.0009 dB overall, with two buckets worsening. We therefore withdraw any fidelity or computational-efficiency claim for adaptive depth.
The remaining ablation results also show small PSNR differences. The full model exceeds the no-uncertainty variant by 0.06-0.09 dB across the five mask buckets, while retrained K=1 remains within 0.07 dB. We therefore do not use PSNR alone to establish each component's contribution; we report separately measured benefits using the metrics relevant to each. Paired Wilcoxon tests on identical images and masks, in our appendix, give additional evidence that the differences are consistent: under a Bonferroni threshold of 0.05/30, 22 of 30 tests remain significant, 9 of 10 for full versus no-uncertainty, 8 of 10 for full versus K=1, 5 of 10 for K=1 versus no-uncertainty. Their small magnitudes should not be interpreted as large practical gains.
Learned halting against the predefined schedule. As discussed in W1, additional refinement improves perceptual quality while worsening pixelwise distortion. We therefore complement PSNR with spatially resolved LPIPS and per-pixel budgets. At 30-40% masks, the results are:
| Policy | Mean per-pixel depth | Tercile LPIPS: low / mid / high U |
|---|---|---|
| Uniform k=3 | 3.000 | 0.0760 / 0.1170 / 0.1616 |
| Uniform k=4 | 4.000 | 0.0772 / 0.1174 / 0.1620 |
| Hand-designed U-based schedule, round(1+3U) | 2.702 | 0.0759 / 0.1169 / 0.1615 |
| Learned halting | 2.532 | 0.0759 / 0.1169 / 0.1615 |
First, learned halting achieves the same displayed tercile-wise LPIPS as uniform three-step refinement at a lower mean per-pixel depth. Second, uniform four-step refinement does not improve on uniform three-step: across the five buckets it is worse in 14 of the 15 tercile-bucket comparisons and better in one by 0.0001, so additional refinement is not uniformly beneficial. Third, compared with the hand-designed U-based schedule, learned halting uses 2.489-2.594 mean per-pixel steps versus 2.650-2.764, a reduction of 0.161-0.170 updates per pixel. Across the 15 bucket-tercile comparisons, learned halting achieves lower spatial LPIPS in nine cells and equal LPIPS in the remaining six (to four decimal places).
The claim we defend. Learned halting matches the measured perceptual quality of the hand-designed U-based schedule with fewer per-pixel updates. It does not establish a meaningful perceptual-quality advantage over it.
Implementation limitation. These reductions represent per-pixel refinement updates, not measured computational or wall-clock savings. Under our hard-halting implementation, the loop still executes all four batch-level passes, because early termination requires every active cell to have halted. Realizing computational savings requires sparse execution, which we have not implemented.
We will revise the manuscript to distinguish per-pixel update reduction from computational efficiency, report the allocation comparison using perceptual metrics alongside PSNR, and present the variance head and allocation policy as components with separately measured benefits.
Response 1 to Reviewer fMRg – Submission 29718
W1 and Q1. Motivation and underlying principle
We thank the reviewer for raising these concerns. The experiments below were conducted during the discussion period to examine the underlying motivation of our method.
Review, W1: "The authors do not adequately explain why regions with larger reconstruction errors should receive more refinement steps. The method seems to be driven mainly by architectural design choices, without a convincing insight that justifies them."
Review, Q1: "What is the key insight behind the proposed method? In particular, why should reconstruction error or the uncertainty derived from it determine refinement depth?"
The key insight. The perceptual response to iterative refinement is spatially non-uniform: regions with higher predicted reconstruction-error rankings exhibit greater perceptual improvement from additional refinement. However, this can come at the cost of pixelwise fidelity.
We agree that the original manuscript did not sufficiently explain this principle. It established that the predicted uncertainty map correlates with reconstruction difficulty. Still, a correlation between U and error does not justify why regions with higher U should benefit from additional refinement. We have therefore revisited the motivation and conducted the following experiment.
The insight implies two distinct predictions: (i) additional refinement should produce larger perceptual improvements in high-uncertainty regions, and (ii) these improvements may coexist with increased pixelwise distortion. Together, these predictions distinguish a perceptual-refinement explanation from the simpler assumption that additional refinement necessarily reduces reconstruction error.
To test both predictions, we fixed the uncertainty map before refinement and divided the hole pixels into per-image low-, medium-, and high-uncertainty terciles. We then forced all hole cells to undergo exactly k = 1, 2, 3, and 4 refinement steps, with the same model, images, masks, and procedure. U is computed before the refinement loop, so it remains identical at every depth, and the same pixels belong to each tercile throughout. Terciles are computed within each image, preventing a global offset in U from explaining the trend. We measured spatially resolved LPIPS, averaged over the pixels belonging to each tercile, rather than using whole-image LPIPS.
Across all five CelebA-HQ mask buckets, increasing depth from one to three steps reduced spatially resolved LPIPS more strongly in high-uncertainty regions (12.5-13.3%) than in low-uncertainty regions (4.1-7.3%), with the magnitude of LPIPS improvement increasing consistently from the low- to the high-uncertainty tercile in every bucket. Over the same depth change, high-uncertainty regions showed a 7.0-15.3% increase in MSE.
Both predictions are supported by this experiment across the five evaluated buckets, revealing a perceptual–distortion trade-off. Thus, the relevance of U to refinement is not simply that it identifies regions with larger reconstruction errors, but that it identifies regions exhibiting a stronger perceptual response to additional refinement. This explains why our PSNR-based evidence showed marginal gains: the proposed mechanism exhibits a perceptual effect that is not adequately captured by pixelwise fidelity. We therefore withdraw any implication that additional refinement necessarily improves reconstruction fidelity.
We acknowledge the scope of this finding. The experiment measures the response of a trained model to forced-depth interventions, using 120 images per bucket, a single dataset, checkpoint, and perceptual metric. It does not establish that the relationship generalizes to independently trained models, datasets, or architectures. Forcing uniform depth also takes the model away from the halting policy it was trained under. The experiment does not show that uncertainty causes perceptual improvement or that learned halting is the optimal way to exploit this relationship. Furthermore, U is trained to rank the model's own reconstruction errors and should not be interpreted as a calibrated estimate of the true conditional distribution.
The experiment provides empirical motivation for using the predicted error-ranking signal to guide refinement: regions with higher U exhibit a stronger perceptual response to additional refinement, even though their pixelwise distortion increases. We will revise the manuscript to articulate this motivation, distinguish perceptual improvement from pixelwise fidelity, and clarify that the finding supports the proposed design without establishing a universally optimal refinement policy.