acceptodds
Under review as a conference paper at ICLR 2027

UAI-Net: Uncertainty-Aware Image Inpainting with Learned Adaptive Refinement Depth

Abstract

Most inpainting models fill in missing regions without indicating which parts of the fill are likely to be wrong, and those that do estimate a confidence map use it only to order pixels within a fixed number of refinement passes. This paper makes the model estimate its own difficulty and act on it. UAI-Net predicts a coarse spatial uncertainty map, trained to reproduce the rank ordering of the model's reconstruction error, and uses it twice: to weight the fusion of local, global-attention and boundary features, and to decide, through a straight-through halting head, how many refinement steps (one to four) each location of the refinement grid receives. Training the halting head is the main difficulty: with a continuous depth target its logits collapse to zero and the executed depth is decided by floating-point rounding; an integer target with a small confidence penalty removes the problem. On CelebA-HQ, Places2 and FFHQ (256x256, free-form masks over 10-60% of the image, 1,500 test images each) the map correlates with the true per-pixel error at a within-image Spearman rho of 0.60-0.67, and the model's predicted variance detects badly filled pixels better than the flip-disagreement or snapshot-ensemble uncertainty of stronger inpainters, in one forward pass. On CelebA-HQ the least-uncertain tercile receives 1.7 refinement steps and the most-uncertain 3.2 (2.54 on average), matching the quality of uniform four-step refinement; the block is still evaluated densely, so wall-clock time is unchanged. Against a PSNR-optimised variant at equal PSNR, UAI-Net gives 8-15% lower LPIPS and 10-17% lower FID.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Inpainting, Uncertainty Estimation, Adaptive Computation, Computer Vision, Image Restoration, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fMRgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes UAI-Net, an uncertainty-aware framework for image inpainting. The method predicts a spatial uncertainty map supervised by the rank of reconstruction errors, which is then used to guide feature fusion and adaptive refinement depth. The authors also introduce a learned halting mechanism with an integer depth target to address training instability. Experiments on CelebA-HQ, Places2, and FFHQ evaluate reconstruction quality, uncertainty estimation, and adaptive refinement.

Critical strengths

The paper is generally well-written, with clear explanations that make it easy to follow.

Critical weaknesses
  1. Lack of a clear underlying insight. The paper proposes using uncertainty estimated from reconstruction errors to determine the refinement depth for image inpainting. However, the authors do not adequately explain why regions with larger reconstruction errors should receive more refinement steps. The method seems to be driven mainly by architectural design choices, without a convincing insight that justifies them. For a paper targeting ICLR, I would expect a clearer motivation for the core design.

  2. Limited empirical improvements. The ablation results do not convincingly demonstrate the effectiveness of the proposed design. Despite introducing multiple components, the improvements are marginal. For example, in Table 3, learned halting improves PSNR over the simple scheduled strategy by only 0.01 dB for 10–20% masks and 0.02 dB for 50–60% masks. Such small differences make it difficult to determine whether learned halting provides a meaningful benefit over a simple predefined rule.

Questions for authors
  1. What is the key insight behind the proposed method? In particular, why should reconstruction error or the uncertainty derived from it determine refinement depth? The authors should provide a more convincing explanation of this design choice.

  2. Please improve the presentation of the figures. Some figures, particularly the architecture and module diagrams (e.g., Figures 1 and 2), are overly complicated and difficult to read. Simplifying these diagrams and increasing the readability of the labels would significantly improve the presentation.

  3. Consider adopting a stronger backbone. The current model falls behind several existing methods in reconstruction quality and does not achieve state-of-the-art performance. While SOTA performance is not strictly necessary, adopting a more competitive backbone could improve the overall results and make the proposed method more competitive for a conference like ICLR.

8 comments
AuthorsOpenReview ↗

Response 1 to Reviewer fMRg – Submission 29718

W1 and Q1. Motivation and underlying principle

We thank the reviewer for raising these concerns. The experiments below were conducted during the discussion period to examine the underlying motivation of our method.

Review, W1: "The authors do not adequately explain why regions with larger reconstruction errors should receive more refinement steps. The method seems to be driven mainly by architectural design choices, without a convincing insight that justifies them."

Review, Q1: "What is the key insight behind the proposed method? In particular, why should reconstruction error or the uncertainty derived from it determine refinement depth?"

The key insight. The perceptual response to iterative refinement is spatially non-uniform: regions with higher predicted reconstruction-error rankings exhibit greater perceptual improvement from additional refinement. However, this can come at the cost of pixelwise fidelity.

We agree that the original manuscript did not sufficiently explain this principle. It established that the predicted uncertainty map correlates with reconstruction difficulty. Still, a correlation between U and error does not justify why regions with higher U should benefit from additional refinement. We have therefore revisited the motivation and conducted the following experiment.

The insight implies two distinct predictions: (i) additional refinement should produce larger perceptual improvements in high-uncertainty regions, and (ii) these improvements may coexist with increased pixelwise distortion. Together, these predictions distinguish a perceptual-refinement explanation from the simpler assumption that additional refinement necessarily reduces reconstruction error.

To test both predictions, we fixed the uncertainty map before refinement and divided the hole pixels into per-image low-, medium-, and high-uncertainty terciles. We then forced all hole cells to undergo exactly k = 1, 2, 3, and 4 refinement steps, with the same model, images, masks, and procedure. U is computed before the refinement loop, so it remains identical at every depth, and the same pixels belong to each tercile throughout. Terciles are computed within each image, preventing a global offset in U from explaining the trend. We measured spatially resolved LPIPS, averaged over the pixels belonging to each tercile, rather than using whole-image LPIPS.

MaskLPIPS change, k=1 to k=3: low / mid / high UMSE change, high U
10-20%-7.3% / -9.3% / -12.6%+7.0%
20-30%-6.4% / -9.6% / -13.3%+8.9%
30-40%-5.7% / -9.3% / -13.3%+11.8%
40-50%-4.2% / -8.2% / -12.9%+13.5%
50-60%-4.1% / -8.0% / -12.5%+15.3%

Across all five CelebA-HQ mask buckets, increasing depth from one to three steps reduced spatially resolved LPIPS more strongly in high-uncertainty regions (12.5-13.3%) than in low-uncertainty regions (4.1-7.3%), with the magnitude of LPIPS improvement increasing consistently from the low- to the high-uncertainty tercile in every bucket. Over the same depth change, high-uncertainty regions showed a 7.0-15.3% increase in MSE.

Both predictions are supported by this experiment across the five evaluated buckets, revealing a perceptual–distortion trade-off. Thus, the relevance of U to refinement is not simply that it identifies regions with larger reconstruction errors, but that it identifies regions exhibiting a stronger perceptual response to additional refinement. This explains why our PSNR-based evidence showed marginal gains: the proposed mechanism exhibits a perceptual effect that is not adequately captured by pixelwise fidelity. We therefore withdraw any implication that additional refinement necessarily improves reconstruction fidelity.

We acknowledge the scope of this finding. The experiment measures the response of a trained model to forced-depth interventions, using 120 images per bucket, a single dataset, checkpoint, and perceptual metric. It does not establish that the relationship generalizes to independently trained models, datasets, or architectures. Forcing uniform depth also takes the model away from the halting policy it was trained under. The experiment does not show that uncertainty causes perceptual improvement or that learned halting is the optimal way to exploit this relationship. Furthermore, U is trained to rank the model's own reconstruction errors and should not be interpreted as a calibrated estimate of the true conditional distribution.

The experiment provides empirical motivation for using the predicted error-ranking signal to guide refinement: regions with higher U exhibit a stronger perceptual response to additional refinement, even though their pixelwise distortion increases. We will revise the manuscript to articulate this motivation, distinguish perceptual improvement from pixelwise fidelity, and clarify that the finding supports the proposed design without establishing a universally optimal refinement policy.

AuthorsOpenReview ↗

Response 2 to Reviewer fMRg – Submission 29718

Addendum to W1: why the learned signal rather than a geometric heuristic?

Our W1 response shows that the perceptual response to additional refinement is stronger in high-U regions. A natural follow-up is whether U provides information beyond a simpler, training-free geometric signal. We therefore repeated the forced-depth analysis using terciles defined by distance to the nearest known pixel instead of U, keeping the model, images, masks, and evaluation procedure unchanged.

MaskLPIPS change using U: low / mid / highLPIPS change using distance: low / mid / high
10-20%-7.3% / -9.3% / -12.6%-8.5% / -10.1% / -12.0%
20-30%-6.4% / -9.6% / -13.3%-8.3% / -10.0% / -12.6%
30-40%-5.7% / -9.3% / -13.3%-7.6% / -9.7% / -12.5%
40-50%-4.2% / -8.2% / -12.9%-6.5% / -8.6% / -11.9%
50-60%-4.1% / -8.0% / -12.5%-6.1% / -8.4% / -11.5%

Distance-to-boundary is not an uninformative control: regions farther from known content also exhibit larger perceptual improvements from additional refinement. However, the high-to-low tercile spread is 5.3-8.7 percentage points under U, compared with 3.5-5.4 under distance. Thus, the separation of perceptual responses is 1.51-1.61 times larger under U across all five buckets.

These results show that U stratifies the perceptual response to additional refinement more sharply than this geometric heuristic across all five mask buckets. It is also consistent with our error-ranking evaluation, which is a separate measurement on 500 images per bucket using 1,000 sampled hole pixels per image: there, trained U reaches an AUROC of 0.8293 against 0.6693 for the distance heuristic in the 10-20% bucket, and 0.7989 against 0.6742 in the 50-60% bucket, so the margin narrows as holes grow.

We acknowledge that this comparison covers one training-free geometric alternative, one model, and one dataset. It does not establish that U is the best possible allocation signal; alternatives such as feature magnitude and predictive entropy remain untested. We therefore present this result as evidence supporting the choice of U over a simple geometric heuristic, rather than claiming that it is uniquely optimal.

AuthorsOpenReview ↗

Response 3 to Reviewer fMRg – Submission 29718

*W2. Limited empirical improvements, and whether learned halting beats a predefined rule

Review: "The ablation results do not convincingly demonstrate the effectiveness of the proposed design. Despite introducing multiple components, the improvements are marginal. For example, in Table 3, learned halting improves PSNR over the simple scheduled strategy by only 0.01 dB for 10-20% masks and 0.02 dB for 50-60% masks."

The reviewer raises two points: whether the components justify their complexity, and whether learned halting beats a predefined schedule. We address both and state where the evidence does not support an advantage.

What each component measurably provides. The PSNR differences cited are accurate and we do not dispute their magnitude. PSNR alone, however, does not capture the measured benefits of the uncertainty and allocation components.

ComponentMeasurable benefitEvaluation
Variance headRetaining the 70% lowest-uncertainty images increases hole PSNR by 1.06-1.59 dB; selection uses predicted variance, no ground truthSelective prediction
Variance headNormalised AURC of 0.783-0.822 with one forward pass, compared with 0.664-0.694 for two-pass flip-TTAError ranking
Uncertainty map UStratifies the perceptual response to refinement 1.51-1.61 times more sharply than distance-to-boundary across five bucketsTercile LPIPS
Learned haltingMatches the hand-designed U-based schedule with 0.161-0.170 fewer per-pixel refinement updatesPer-pixel depth
Adaptive depthNo fidelity or wall-clock advantage demonstratedPSNR and runtime

We acknowledge the limitation in the last row. In a matched-batch second-backbone experiment, frozen and trainable configurations differed by 4.06 steps in depth spread but only 0.0115 dB in PSNR. Raising the budget from K=4 to K=8 changed PSNR by +0.0009 dB overall, with two buckets worsening. We therefore withdraw any fidelity or computational-efficiency claim for adaptive depth.

The remaining ablation results also show small PSNR differences. The full model exceeds the no-uncertainty variant by 0.06-0.09 dB across the five mask buckets, while retrained K=1 remains within 0.07 dB. We therefore do not use PSNR alone to establish each component's contribution; we report separately measured benefits using the metrics relevant to each. Paired Wilcoxon tests on identical images and masks, in our appendix, give additional evidence that the differences are consistent: under a Bonferroni threshold of 0.05/30, 22 of 30 tests remain significant, 9 of 10 for full versus no-uncertainty, 8 of 10 for full versus K=1, 5 of 10 for K=1 versus no-uncertainty. Their small magnitudes should not be interpreted as large practical gains.

Learned halting against the predefined schedule. As discussed in W1, additional refinement improves perceptual quality while worsening pixelwise distortion. We therefore complement PSNR with spatially resolved LPIPS and per-pixel budgets. At 30-40% masks, the results are:

PolicyMean per-pixel depthTercile LPIPS: low / mid / high U
Uniform k=33.0000.0760 / 0.1170 / 0.1616
Uniform k=44.0000.0772 / 0.1174 / 0.1620
Hand-designed U-based schedule, round(1+3U)2.7020.0759 / 0.1169 / 0.1615
Learned halting2.5320.0759 / 0.1169 / 0.1615

First, learned halting achieves the same displayed tercile-wise LPIPS as uniform three-step refinement at a lower mean per-pixel depth. Second, uniform four-step refinement does not improve on uniform three-step: across the five buckets it is worse in 14 of the 15 tercile-bucket comparisons and better in one by 0.0001, so additional refinement is not uniformly beneficial. Third, compared with the hand-designed U-based schedule, learned halting uses 2.489-2.594 mean per-pixel steps versus 2.650-2.764, a reduction of 0.161-0.170 updates per pixel. Across the 15 bucket-tercile comparisons, learned halting achieves lower spatial LPIPS in nine cells and equal LPIPS in the remaining six (to four decimal places).

The claim we defend. Learned halting matches the measured perceptual quality of the hand-designed U-based schedule with fewer per-pixel updates. It does not establish a meaningful perceptual-quality advantage over it.

Implementation limitation. These reductions represent per-pixel refinement updates, not measured computational or wall-clock savings. Under our hard-halting implementation, the loop still executes all four batch-level passes, because early termination requires every active cell to have halted. Realizing computational savings requires sparse execution, which we have not implemented.

We will revise the manuscript to distinguish per-pixel update reduction from computational efficiency, report the allocation comparison using perceptual metrics alongside PSNR, and present the variance head and allocation policy as components with separately measured benefits.

Reviewer_79iHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UAI-Net, an image-inpainting model that predicts a coarse spatial map supervised on the within-image ranking of its reconstruction error. This map controls feature fusion and conditions a learned halting head that assigns one to four refinement updates to each location of a grid. Experiments cover CelebA-HQ, Places2 and FFHQ with five free-form-mask buckets.

Critical strengths
  • Uncertainty guides the predictions and makes some interpretibility of predictions. The same predicted map controls feature fusion and spatial refinement depth. Learned halting assigns more updates to uncertain regions while preserving quality comparable to uniform refinement.
  • Quantitative evaluation of error prediction. Across three datasets, the variance head achieves AUROC of 0.85+ and outperforms the evaluated disagreement-based estimates.
Critical weaknesses
  • Limited reconstruction quality. UAI-Net significantly behind strong baselines: it achieves 1+ dB loose in PSNR. The uncertainty benefits have not been shown to compensate for this gap.
  • Unclear benefit of the proposed mechanisms. A retrained one-step model achieves almost identical quality, while adaptive refinement provides no runtime savings.
Questions for authors
  • Are the “48 test images per bucket” used for checkpoint selection (Tab. 12) disjoint from both evaluation sets? If not, can you report results with separate validation and test data?
  • Are the small gains over retrained K=1 consistent across training seeds? What practical benefit justifies adaptive refinement given the similar quality and lack of runtime savings (Sec. 4.4)?
  • Can you report the final hole-PSNR after the 10% correction in Tab. 2 for UAI-Net and the baselines? This would clarify whether better error detection compensates for lower reconstruction quality.
  • Can you report the Tab. 2 uncertainty metrics for the K=1 and no-U ablations, keeping the same variance head and NLL supervision? This would isolate the contribution of routing and halting.
  • It concerns me that the most recent baseline was taken from 2025; many powerful new models have been released since then, and they should be incorporated into the final ICLR 2027 paper.
8 comments
AuthorsOpenReview ↗

Response 1 to Reviewer 79iH – Submission 29718

Response 1 to Reviewer 79iH – Submission 29718

We thank the reviewer for five questions that are all answerable with measurements, which is what we have done. All figures below are new work from the discussion period. We hope our revisions and responses merit reconsideration of a higher evaluation.

Q1. Checkpoint selection

Review: "Are the '48 test images per bucket' used for checkpoint selection (Tab. 12) disjoint from both evaluation sets? If not, can you report results with separate validation and test data?"

Response: They are not disjoint. They are the first 48 test images per bucket, and the submission should have stated this explicitly. On all three datasets, the rule selected the final epoch (CelebA-HQ 50/50, Places2 (30/30), FFHQ (30/30), so it never discarded a candidate. Re-evaluating those weights from scratch on the full 1,500-image protocol reproduces the submitted numbers, with a maximum |deltaPSNR| of 0.0000 dB on every bucket of all three datasets. We do not claim the procedure was sound; it was a protocol error. We can establish only that, in these runs, it did not change the reported results. For the camera-ready, selection moves to a held-out split drawn from the training set.

Q2. Seed consistency, and what justifies adaptive refinement

Review: "Are the small gains over retrained K=1 consistent across training seeds? What practical benefit justifies adaptive refinement given the similar quality and lack of runtime savings?"

Response: Within the limits of three seeds, yes. The gain over retrained K=1 averages +0.049 dB with an across-seed standard deviation of 0.013 dB and the same sign in all five buckets. That is about four times the observed seed-level standard deviation. We read that as repeatable rather than statistically significant, since three seeds are a limited estimate of training variability rather than a formal test. The learned policy is equally stable: mean depth moves from 2.58 to 2.49 across mask sizes (sd <= 0.017), and the uncertainty-error rank correlation has sd <= 0.0013.

On what the refinement buys, the answer is calibration rather than fidelity. Collapsing the loop to a single step drops error-detection AUROC from 0.885 to 0.784 and degrades calibration error roughly 20-fold, from 1.9e-4 to 3.7e-3, while keeping the variance head unchanged. Q4 explains why we describe this as an association rather than a demonstrated mechanism.

Q3. Hole-PSNR after the 10% correction

Review: "Can you report the final hole-PSNR after the 10% correction in Tab. 2 for UAI-Net and the baselines? This would clarify whether better error detection compensates for lower reconstruction quality."

Response: One clarification should have been made in the submission: this metric replaces the selected pixels with ground truth, so it is an oracle analysis of how well each method ranks its own errors, and not a deployable restoration procedure. We will relabel it accordingly. Table 1 reports 500 images per bucket, with both passes measured together.

Table 1: hole-PSNR before and after the 10% oracle correction

MaskUAI-Net before/afterHINT before/afterSPA-Former before/afterLaMa after
10-20%23.39 / 26.7024.36 / 26.4824.70 / 26.8024.59
20-30%22.33 / 25.5423.30 / 25.4023.76 / 25.7923.49
30-40%21.63 / 24.7522.70 / 24.7523.15 / 25.1522.83
40-50%20.96 / 23.9722.13 / 24.0922.49 / 24.4222.08
50-60%20.33 / 23.2421.56 / 23.4621.90 / 23.7721.46

UAI-Net begins about 1 dB behind HINT in every bucket. After correction, it leads in the two smallest buckets, ties at 30-40% and still trails in the two largest, while SPA-Former remains ahead throughout. The answer is therefore a qualified yes: better detection closes much of the gap without making our method the strongest. The narrower claim holds everywhere and can be read from the table: Our correction gain exceeds HINT's and SPA-Former's by 1.01-1.21 dB across all buckets.

The remaining points are in the next comment.

AuthorsOpenReview ↗

Response 2 to Reviewer 79iH – Submission 29718

Q4. Table 2 metrics for the K=1 and no-U ablations

Review: "Can you report the Tab. 2 uncertainty metrics for the K=1 and no-U ablations, keeping the same variance head and NLL supervision? This would isolate the contribution of routing and halting."

Response: Table 2 reports them, with the same variance head and NLL supervision in every row.

Table 2: uncertainty metrics under the two ablations, CelebA-HQ, 10-20% bucket, 500 images

Modelsigma^2 AUROCcalibration errorU AUROCU rho
Full0.8851.9e-40.8290.668
no-U0.8841.6e-40.5230.002
K=1 retrained0.7843.7e-30.7560.450

These separate the behavior of the two outputs without isolating their causes. Removing the routing and its supervision leaves variance-based detection almost unchanged, from 0.885 to 0.884, while U largely loses its error-ranking ability, from 0.829 to 0.523 and from a rank correlation of 0.668 to 0.002. The K=1 ablation is associated with substantially worse calibration, from 1.9e-4 to 3.7e-3, but it removes the refinement iterations and their supervision together, so we read it as an association rather than proof that the loop alone causes the better calibration.

Q5. The age of the baselines

Review: "It concerns me that the most recent baseline was taken from 2025; many powerful new models have been released since then, and they should be incorporated into the final ICLR 2027 paper."

Response: We should answer this precisely rather than appear to answer it. Three additions are now evaluated under our protocol and included in the paper, at another reviewer's request: FcF (WACV 2023), MI-GAN (ICCV 2023) and MAT (CVPR 2022). All three predate our existing baselines, so they broaden the comparison but do not address your point, which is about models newer than those we already compare against. We have not added a post-2025 method, and we state that as a limitation rather than imply coverage we do not have. Two practical obstacles stand behind it: the available Places2 checkpoints for the three additions are 512-pixel, making them resolution-mismatched references rather than controlled comparisons, and the MAT CelebA-HQ-256 download returns HTTP 404 as of 14 November 2026, with no working mirror and only a retrained replacement; its authors note may differ from the published numbers. For the camera-ready, we will investigate and include the strongest suitable post-2025 method with a publicly available checkpoint that can be evaluated under our protocol.

W1. Limited reconstruction quality

Review: "UAI-Net significantly behind strong baselines: it achieves 1+ dB lower in PSNR. The uncertainty benefits have not been shown to compensate for this gap."

Response: We tested whether the contribution depends on our backbone by grafting our heads onto HINT.

Table 3: our heads on a stronger trunk, CelebA-HQ, mean of five buckets

ModelPSNRHole PSNRSSIMLPIPSFID
HINT, frozen27.74422.7620.89840.04377.52
HINT + heads, last stage free28.29123.3090.90800.05388.20
UAI-Net, own backbone26.64821.6650.88480.05899.61

Our heads add 0.55 dB on top of HINT and stand 1.64 dB above our own backbone, after six epochs rather than fifty. The depth policy also transfers, but only when HINT's final decoder stage is allowed to co-adapt with our heads: the tercile spread is then +1.999 at K=4, widening to +3.118 at K=8 with the hardest tercile at 6.39-6.55 steps, clear of the ceiling. On a wholly frozen trunk, the allocation inverts instead, which we report. Importantly, the additional budget does not produce a measurable improvement in fidelity, and we report that alongside the positive result. These results indicate that the choice of backbone contributes substantially to the observed gap. One caveat we state rather than leave to be found: the run with the final stage trainable used batch size 4, whereas the frozen run used batch size 8, for memory reasons, so that particular comparison is not fully controlled. The K=4 and K=8 runs are both at batch size 4.

Thank you for the additional experiments and clarifications, the new results strengthen some evidence, but issues with the evaluation protocol remain unresolved, and the practical benefits of the proposed modules remain insufficiently demonstrated. Therefore, I want to maintain my rating of "Weak rejection".

I would recommend focusing on this points for a resubmission:

  1. Isolate the contribution of each proposed module;
  2. Correct the evaluation protocol and add modern methods into the evaluation;
  3. Proof a clear benefit over simpler baselines.
Reviewer_rz5uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UAI-Net, a gated-convolution inpainting network built around a 32×32 uncertainty map U. U is trained to reproduce the rank order of the model's own reconstruction error and is used in two places: to weight the fusion of local, global and boundary features, and to guide a straight-through halting head that gives each refinement cell one to four steps. The authors also identify a "halting collapse" in which continuous depth targets drive the halting logits to zero, and they resolve it with integer targets and a confidence penalty. A separate heteroscedastic variance head detects poorly filled pixels better than flip disagreement.

Critical strengths

1.The ablations are transparent and the paired significance tests are useful.

Critical weaknesses
  1. The core contributions could be stronger. The single-step model is within 0.07 dB of the full model, and no wall-clock time is saved. The strongest detector is the standard NLL variance head (Kendall & Gal, 2017) rather than U. Comparing the rounding fix with PonderNet's KL prior or ACT's ponder cost would show more clearly whether it is needed. 2.Checkpoint selection may leak test information. Table 12 says checkpoints are chosen by LPIPS on 48 test images per bucket. Reselecting on a held-out validation split would make every reported number easier to trust. 3.Some baselines are missing. MAT (Li et al., 2022a) is cited and has 256×256 CelebA-HQ weights. FcF [1] and MI-GAN [2] are other natural comparisons. For uncertainty, MC dropout [3], distribution-free intervals [4], and sample variance from stochastic inpainters would be more informative than flip disagreement, which is a weak proxy for deterministic networks. The pluralistic latent-code model in Table 1 could serve the last role directly.
  2. The detection protocol could be clarified. Each detector is scored on its own model's worst 20% of pixels, so the targets differ in difficulty. A shared target would make the comparison cleaner. Several point estimates also sit at the edge of their bootstrap intervals (e.g., 0.878 [0.870, 0.879]), and 100 resamples is a small number.
  3. Fidelity and statistical robustness are limited. HINT and SPA-Former lead by 0.9 to 1.8 dB, and Places2 uses only 50k images. With one run per configuration and effect sizes around 0.05 dB, results over several seeds would strengthen the conclusions.
  4. Some numbers are inconsistent. The ρ ranges differ between the abstract (0.60–0.67), the contributions (0.65–0.67) and Table 2 (0.607–0.662).
  5. Positioning and citations need a check. The claim that no prior work evaluates its map against error could be softened, since PSM analyses how its variance evolves. Please also verify "SeGuE-Net (Liao et al., 2021)". It may refer to SGE-Net (ECCV 2020).
  6. Sparse execution would help the efficiency claim. Spatially sparse execution such as DynConv [5] would turn the 36% reduction in per-pixel updates into a measurable speedup.

[1] Jain et al., Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand, WACV 2023. [2] Sargsyan et al., MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices, ICCV 2023. [3] Gal & Ghahramani, Dropout as a Bayesian Approximation, ICML 2016. [4] Angelopoulos et al., Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging, ICML 2022. [5] Verelst & Tuytelaars, Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference, CVPR 2020.

Questions for authors

See weaknesses.

10 comments
AuthorsOpenReview ↗

Response 1 to Reviewer rz5u – Submission 29718

We thank the reviewer for a precise and constructive review. Almost every point raised led to a new experiment. Where an experiment was not feasible, we say so and describe the substitute. All results below are newly reported, and the sample size for each is stated in its table caption. We hope our revisions and responses merit reconsideration of a higher evaluation.

W1.1. Checkpoint selection may leak test information

Review: "Checkpoint selection may leak test information. Table 12 says checkpoints are chosen by LPIPS on 48 test images per bucket. Reselecting on a held-out validation split would make every reported number easier to trust."

Response: The reviewer is correct, and we do not defend the rule. We examined what it did in practice: on all three datasets it selected the final epoch (CelebA-HQ 50/50, Places2 30/30, FFHQ 30/30), and therefore never discarded a candidate. Re-evaluating those weights from scratch reproduces the submitted PSNR values, with a maximum |deltaPSNR| of 0.0000 dB across all five buckets of all three datasets. The reported numbers are therefore unaffected, although the procedure itself was unsound, and we state that rather than defend it. For the camera-ready, we will move selection to a held-out split drawn from the training set and state the protocol explicitly in the appendix.

W1.2. The core contribution could be stronger

Review: "The core contributions could be stronger. The single-step model is within 0.07 dB of the full model, and no wall-clock time is saved. The strongest detector is the standard NLL variance head (Kendall & Gal, 2017) rather than U."

Response: Both observations are correct and are stated in the paper. What the refinement loop provides is calibration rather than PSNR. Every row of Table 1 retains the same variance head and the same NLL supervision, so the comparison isolates the loop and the routing rather than the detector.

Table 1: uncertainty ablations, CelebA-HQ, 10-20% bucket, 500 images

Modelsigma^2 AUROCcalibration errorU AUROC
Full0.8851.9e-40.829
no-U0.8841.6e-40.523
K=1 retrained0.7843.7e-30.756

Removing the routing leaves error detection essentially unchanged, whereas collapsing the loop costs 0.10 AUROC and degrades calibration 20-fold. We regard this as an association rather than a demonstrated mechanism, since these ablations remove the refinement iterations and their supervision together, and we do not claim otherwise. Across three seeds, the gain over retrained K=1 averages +0.049 dB (sd 0.013), with the same sign in all five buckets.

W1.3. Comparison with PonderNet and ACT

Review: "Comparing the rounding fix with PonderNet's KL prior or ACT's ponder cost would show more clearly whether it is needed."

Response: We ran both comparisons with matched gradients. Mean PSNR spans only 0.05 dB across the four variants, so the question is one of allocation rather than fidelity. The depth profiles in Table 2 use a 200-image subset per bucket rather than the full 1,500.

Table 2: depth allocation under three regularisers, CelebA-HQ

Variantmean deptherror-tercile gap
ours2.575+0.164 (positive CI in 5/5 buckets)
PonderNet1.691+0.011 (positive CI in 2/5)
ACT1.004-0.008 (CI < 0 in 5/5; 88% halt at step 1)

PonderNet largely reproduces its input-independent prior, while ACT collapses to a single step and so reaches our K=1 ablation by another route. Our own gap is modest, and we present it as such.

AuthorsOpenReview ↗

Response 2 to Reviewer rz5u – Submission 29718

W2. The Bootstrap Intervals

We thank the reviewer for highlighting the limitations of our original bootstrap analysis. We agree that 100 resamples were insufficient for reliable interval estimation and that the reported point estimates and confidence intervals required closer examination. We therefore re-examined the bootstrap implementation, identified an inconsistency in the original procedure, and conducted a corrected analysis.

The quoted interval, reproduced

Reproducing the original calculation on the 20–30% CelebA-HQ bucket gives the reported 0.878 [0.870, 0.879]. We subsequently found that the point estimate and the bootstrap interval were computed using different pixel samples, meaning that the interval was not valid for the estimate it accompanied.

The cause

The point estimate used the 1,000-pixel per-image subsample, whereas the bootstrap used only the first 200 of those pixels. Consequently, the interval described a different estimator from the one it bracketed. We measured the resulting discrepancy rather than inferring its effect. Both bootstrap distributions below are compared against the same point estimate, computed using 1,000 pixels per image; only the resampling procedure differs.

Resampling used for the intervalPoint estimateResample meanDifferenceSkewnessPercentile of the point estimate
Submitted bootstrap: 200 px per image, B = 1000.8775770.875408-0.002169-0.6784.0
Corrected bootstrap: 1,000 px per image, B = 10,0000.8775770.877602+0.000025-0.0449.6

The submitted bootstrap distribution had a mean of 0.8754, approximately 0.0022 below the point estimate of 0.8776, and a skewness of -0.67. Consequently, the point estimate lay at the 84th percentile of that distribution, near its upper bound. This explains the discrepancy highlighted by the reviewer. We report these values as a measured discrepancy between the point estimate and the resampling distribution with which it was paired, rather than as a formal bias result, which would require a defined estimand and a repeated-sampling argument.

The correction

We now use the same 1,000-pixel per-image sample for both the point estimate and the bootstrap. We resample images rather than individual pixels to preserve within-image dependence. Images are sampled with replacement, and an image selected more than once contributes its 1,000 sampled pixels only once per selection. All 10,000 replicates yield a defined AUROC, and the confidence interval is calculated using the 2.5th and 97.5th percentiles of those replicates.

For the quoted cell, the corrected result is:

0.8776 [0.8735, 0.8816]

The resample mean now differs from the point estimate by 2.5e-5 rather than 2.2e-3. The distribution is close to symmetric (skewness -0.04), and the point estimate lies at the 49.6th percentile of its bootstrap distribution.

Numerical validation

For efficiency, the resampling procedure evaluates AUROC using per-image score histograms rather than direct ranking. We compared this approach against an exact mid-rank AUROC computation for all 50 cells in the detection table, comprising five mask buckets and ten detector configurations: our variance head, U, flip-TTA, and distance-to-boundary, together with flip-TTA and distance-to-boundary for each of LaMa, HINT, and SPA-Former. Every cell was evaluated using the worst-20% target. The largest absolute disagreement was 7.0e-7. Thus, the histogram-based computation is a close numerical approximation to the exact statistic.

We also recomputed the two narrowest intervals using a separate implementation that sorts the pooled scores once and evaluates a weighted rank statistic instead of binning. This independent implementation used 40,000 replicates. The two implementations generated their bootstrap samples using different random streams. Consequently, the comparison reflects Monte Carlo variation as well as any implementation differences and does not isolate the latter. Across the four endpoints compared—the ordinary and adjusted lower limits for both intervals—the largest absolute difference was 0.00011.

We report 95% confidence intervals throughout. For comparisons between detectors evaluated on the same images, we use paired differences rather than separate confidence intervals because their estimates are correlated.

Scope

This correction concerns the statistical reporting of the error-detection analysis. It does not change any fidelity result in the paper, nor does it establish an inference-time speedup or any fidelity gain from a larger refinement budget. These remain limitations of the work and are addressed separately.

The analysis is implemented in a new script that leaves the submitted evaluation code unmodified. It caches the per-image score and error arrays once, allowing different target definitions, error thresholds, and resampling counts to be recomputed and independently checked.

AuthorsOpenReview ↗

Response 3 to Reviewer rz5u – Submission 29718

W2 continued. A shared target, and paired comparisons

The reviewer is right: each detector was scored against its own model's worst 20% of pixels, so the targets differed in difficulty. We report three analyses addressing different questions.

1. Within-model detection. Each signal predicts its own output's errors, the deployment-relevant question. Within each model, all four signals share one error map.

2. Pooled-threshold sensitivity. One threshold—the 80th percentile of pooled hole errors across all four models—is applied to each model's own error map. This equalises the threshold, not the labels, so it is a sensitivity analysis, not a shared-target experiment. No AUROC changes by more than 0.006.

3. Identical pixel labels. We fix one reference error map, define its worst 20% of pixels as positive, and score every detector on those same labels.

The mean reference is the pixel-wise mean of UAI-Net, HINT, SPA-Former, and LaMa error maps on the same images, masks, hole region, and pixel coordinates. All use the same channel-mean absolute-error definition and [0,1] scale, justifying averaging without per-model normalization; correlations with the composite are 0.876–0.911. This composite is derived from the four models' own errors; it supplies common labels, not an independent measure of difficulty. We also use HINT's map alone as an alternative reference.

A pixel is positive only when its reference error is strictly greater than the threshold; ties are negative. The mean reference has no threshold ties, giving exact base rates of 0.100000, 0.200000, and 0.300000. HINT's map is derived from 8-bit reconstructions and has ~1,750 distinct values; threshold ties therefore reduce base rates to 0.1982–0.1990 (worst-20%) and 0.2976–0.2988 (worst-30%). This label rule is distinct from mid-rank handling of tied detector scores in AUROC. As a check, the distance control gives identical AUROC for all four models under shared labels.

For identical labels (mean reference), the 20–30% bucket, worst-20% threshold and 500 images, AUROCs are: σ² 0.8945 [0.8906, 0.8985]; U 0.8373 [0.8326, 0.8420]; flip-TTA: HINT 0.8137, ours 0.8100, SPA-Former 0.7957, LaMa 0.7807; distance 0.6839. The revision reports every interval.

Paired comparisons

Separate AUROC intervals do not establish an ordering because detectors share images and estimates are correlated. We therefore bootstrap paired AUROC differences, not individual AUROCs. Images are the resampling unit: one draw of indices per cell is reused for both detectors, all pixels of each sampled image remain together, and the difference is computed within each replicate (B=40,000).

Each entry is the minimum across 15 cells (5 buckets × 3 thresholds) of the ordinary 95% lower endpoint | Bonferroni-adjusted lower endpoint. These are interval endpoints, not point estimates or minimum observed differences. The adjusted endpoint uses two-sided tails 0.05/(2×15), a 99.667% interval per cell; its lower limit interpolates between the 67th and 68th smallest of the 40,000 replicates.

Paired differenceOwn errorsIdentical, mean ref.Identical, HINT ref.
σ² − U+0.0467 | +0.0452+0.0464 | +0.0447+0.0387 | +0.0371
σ² − flip-TTA (ours)+0.0588 | +0.0576+0.0739 | +0.0726+0.0655 | +0.0642
σ² − flip-TTA (HINT)+0.0686 | +0.0670+0.0750 | +0.0734+0.0465 | +0.0451
σ² − flip-TTA (SPA-Former)+0.0829 | +0.0815+0.0894 | +0.0880+0.0711 | +0.0697
σ² − flip-TTA (LaMa)+0.0919 | +0.0901+0.1046 | +0.1026+0.0990 | +0.0972
U − flip-TTA (ours)−0.0106 | −0.0129+0.0036 | +0.0011+0.0128 | +0.0102

Scope. Each 15-cell family is one comparison under one target definition, corrected across its 15 bucket-threshold cells. We make no joint family-wise claim across comparisons or target definitions. Every variance-head comparison has an adjusted interval above zero in all 15 cells of its family. The weakest adjusted endpoint among these five comparisons is +0.0371 (σ² − U, HINT reference, 20–30% bucket, worst-30% threshold). The revision lists all 270 endpoints.

Thus, the variance head is a more effective error detector than the evaluated flip-TTA and U signals under these CelebA-HQ protocols. Cross-model identical-label results are limited to CelebA-HQ, the only dataset for which baseline reconstructions are retained. U's relative performance is dataset- and target-dependent, as discussed in the next comment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.