acceptodds
Withdrawn from ICLR 2027

To Inpaint, or Not to Inpaint: Auxiliary-Free Shadow Removal via Inpainting and Post Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Shadows degrade downstream vision tasks, yet most shadow removal methods depend on auxiliary inputs such as shadow masks, surface normals, or depth maps, which are often unavailable or inaccurate at deployment. We propose Inpaint2Free (I2F), an auxiliary-free shadow removal framework that exploits the mask conditioning of a pretrained inpainting diffusion model. I2F is fine-tuned in two stages: first with ground-truth shadow masks to learn localized shadow removal, then with Full-one masks so that inference requires only the shadow image. We further propose Post Fusion, which estimates a high-recall pseudo mask from the difference between the removal result and the input, and uses it to restore non-shadow details from the original image. On ADSP, the two-stage fine-tuning substantially improves shadow restoration, while Post Fusion markedly improves non-shadow and full-image fidelity. When compared with mask-dependent baselines using FDRNet-predicted masks on ADSP, auxiliary-free I2F achieves a shadow-region PSNR of 41.178 dB, outperforming all baselines. The framework thus removes the dependence on both auxiliary inputs and upstream prediction models, providing a practical standalone solution for real-world shadow removal.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image restoration, Shadow removal, Diffusion model, Computer Vision

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_tdvWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the shadow removal task, especially trying to address the problem of unavailable or imperfect auxiliary inputs at inference. The authors propose a two-stage framework with Post Fusion to achieve auxiliary-free shadow removal. Stage 1 trains a shadow removal model formulated as an inpainting task, while Stage 2 adapts the model to accept an all-one mask, eliminating the need for an externally supplied shadow mask. Both stages use the shadow image as conditioning. Post Fusion estimates a pseudo shadow mask, and the final result is a composite of the prediction and the original image. The authors evaluate their method on ADSP, ISTD+, and SRD, using masks predicted by BDRAR and FDRNet for the compared mask-based methods.

Critical strengths
  1. The authors show a clear auxiliary dependency in current mask-based shadow removal methods, and they propose a straightforward two-stage model to address the problem of imperfect masks. The authors formulate shadow removal as an inpainting task in Stage 1 and remove the dependency on externally supplied masks through a second training stage.

  2. The authors further propose to improve their auxiliary-free framework by estimating a pseudo mask followed by compositing. This reduces the errors caused by I2F by constraining where reconstructed pixels are retained and preserving the original image details elsewhere.

Critical weaknesses
  1. The comparisons do not sufficiently establish the advantage of the proposed inpainting-based framework for removing mask dependence. Can the authors apply a similar strategy to the compared methods? For example, replace their mask inputs with all-one masks and conduct fine-tuning comparable to Stage 2. The relative drop in performance can help clarify the contributions of the proposed model and the two-stage training strategy.

  2. The practical usefulness of the complete two-stage framework across different data conditions needs more evidence. The reconstruction-fidelity advantage on ADSP in the main paper does not consistently extend to ISTD+ and SRD, as shown in supplementary Tables 3 and 5. Furthermore, the authors skip Stage 1 on SRD because reliable annotated masks are unavailable, leaving the benefit of the complete strategy unclear for such datasets. To show the applicability of the framework, can the authors conduct cross-domain evaluation on scenes from WSRD, or qualitative evaluation on more diverse images from SBU? Alternatively, can the authors demonstrate whether a Stage-1 model trained on a source dataset improves Stage-2 adaptation to new data without masks, compared with direct Stage-2 training from scratch? Either would strengthen the justification for using the proposed framework.

  3. The use of predicted masks from detection models raises my concern about the consistency of the scope definition. ADSP targets cast-shadow removal while preserving the occluder and its attached shadow. ISTD+/SRD, on the other hand, primarily contain cast shadows. However, some regions predicted by BDRAR/FDRNet may correspond to genuine attached shadows rather than incorrect detections, potentially creating a mismatch between the detection and removal scopes. The suitability of these predictions for a cast-shadow removal task therefore needs to be verified. Can the authors first report the detection models’ checkpoints and training datasets, and then report how much of the predicted-mask error comes from the object region? Can the authors then conduct a controlled evaluation excluding the occluder region from the predicted masks? This could help determine how much the comparison is affected by this mismatch. This may also contribute to the different performance trends between ADSP and ISTD+/SRD, but further analysis is needed to establish that explanation.

Questions for authors
  1. Please see my question in Weakness 1.

  2. Please see my question in Weakness 2.

  3. Please see my question in Weakness 3.

  4. Can the authors compare with Stable Shadow Removal (Detail-Preserving Latent Diffusion for Stable Shadow Removal, CVPR 2025)? It is also a diffusion-based, mask-free shadow removal method closely related to the proposed method. Its version without DINOv2 features could serve as a good auxiliary-free baseline.

  5. I suggest that the authors clarify the scope of ‘auxiliary-free.’ Their methodology and experiments primarily address shadow mask dependence, while the effects of geometric or semantic auxiliary cues are not examined to the same extent.

  6. I suggest adding error maps to Figs. 1, 5, and 7 to make the improvements from I2F to Fused easier to assess. I also suggest standardizing figure and table caption formatting, increasing the spacing, and using consistent panel sizes and column alignment in Fig. 8.

Reviewer_bL3jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a shadow removal method called Inpaint2Free (I2F), which requires only a shadow-containing image as input during the inference stage. Based on the Stable Diffusion 2 Inpainting model and integrated with ControlNet and LoRA, the method is adapted through a two-stage training process: the first stage utilizes annotated shadow masks, while the second stage employs "all-one" masks (masks consisting entirely of ones). Subsequently, a "Post Fusion" module estimates the shadow regions by analyzing the differences between the input image and the generated image—incorporating Gaussian Mixture Model (GMM)-based thresholding and morphological refinement—and restores the areas outside these regions using the content from the original input image. Evaluation was conducted on the ADSP, ISTD+, and SRD datasets, including an assessment of the predicted masks and the retraining of baseline models that rely on such masks. Experimental results demonstrate that this method achieves state-of-the-art performance on the ADSP dataset. For the SRD dataset, which lacks reliable annotated masks, the training process consists solely of the second stage.

Critical strengths
  1. I find the inference setup proposed in the paper to be highly practical. By requiring only an input image to operate, this shadow-removal system eliminates any dependency on externally provided "shadow masks." The authors have also conducted a rigorous evaluation of this setup: they utilized predicted masks from two different sources and retrained the mask-dependent baseline models using these predictions.

  2. The "Post Fusion" mechanism addresses specific flaws in the generated results, with its effectiveness clearly reflected across various metrics. On the ADSP dataset, it improves the full-image PSNR from 31.510 dB to 32.690 dB and reduces the VGG-LPIPS score from 0.2006 to 0.0586.

Critical weaknesses
  1. A primary concern is the lack of a comparison with a highly relevant "image-only" diffusion-based approach. The paper cites the work of Xu et al., "Detail-Preserving Latent Diffusion for Stable Shadow Removal" (CVPR 2025), which also leverages a pre-trained latent diffusion model and explicitly focuses on detail preservation. Given that this method also relies solely on the input image for inference, such a comparison is essential. While the supplementary material discusses differences in the use of DINO features and overall architecture, it fails to clearly articulate the specific, practical advantages of I2F over this approach.

  2. On the ISTD+ dataset, the "Fused" method achieves a full-image PSNR of 32.521 dB, whereas HomoFormer—when retrained using FDRNet masks—reaches 34.411 dB. On the SRD dataset, the corresponding figures are 29.170 dB and 30.712 dB, respectively. Although the results presented for ADSP are impressive, these performance gaps make the precise scope of the proposed improvements somewhat ambiguous. A direct, like-for-like comparison with the aforementioned diffusion-based model would help clarify whether the core strengths of the proposed method lie in restoration quality, detail preservation, inference cost, or performance within specific data environments.

  3. In the ablation studies, two key components of the proposed method lack sufficient explanation. Regarding the training pipeline, the full ADSP model undergoes 1.122 million updates in Stage-1, followed by an additional 1.973 million updates in Stage-2. Table 4 does not specify the total number of updates or the loss function scheduling scheme used for the "no-GT-tuning" configuration. Without this information, it is difficult to determine whether the performance gains stem from the introduction of shadow masks early in the training process, or simply from differences in the optimization budget (i.e., the total amount of training). If the optimization budget was kept consistent across configurations, providing the relevant details would largely resolve this concern.

  4. Regarding GMM thresholding, Table 5 presents only statistics for the masks, without showing the quality of the final fused images resulting from different thresholding methods. The argument in the paper that "failing to detect shadow pixels poses a significant risk" is sound. However, compared to MET, GMM yields a lower IoU and produces larger masks. This highlights a necessary trade-off: expanding the mask area preserves more content from the diffusion model’s output, but it simultaneously reduces the fidelity to the original image. The current table does not explicitly indicate which threshold setting achieves the best final reconstruction quality.

Questions for authors
  1. How does I2F perform compared to the "latent-diffusion with detail preservation" method (cited from CVPR 2025 in the paper) when evaluated under identical data, resolution, and assessment protocols? I am particularly interested in the inpainting metrics and inference costs, as such a comparison would help clarify the practical contribution of the proposed pipeline.

  2. For the "no-GT-tuning" experiments in Table 4, is the total number of training updates the same as that used for the full model? Please specify the number of updates, the loss values, and how the optimizer states were handled for these experiments. If the training budget (i.e., the number of updates) differs, would a model trained using an "all-one mask"—given the same 3.095 million ADSP updates—still demonstrate the advantages attributed to "mask curriculum learning"?

  3. If the GMM approach were replaced by Otsu’s method or the MET method—while keeping all subsequent morphological operations unchanged—how would the PSNR for the shadowed/non-shadowed regions and the LPIPS for the entire image be affected? These results would help clarify whether higher mask coverage truly improves the quality of the output that the "Post Fusion" stage aims to preserve.

Reviewer_Ge7tOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Inpaint2Free, a standalone shadow removal framework that eliminates the dependency on external auxiliary inputs such as shadow masks, depth maps or surface normals. Built upon a pretrained inpainting diffusion model with ControlNet and LoRA adaptation, the method adopts a two-stage fine-tuning strategy. To compensate for detail degradation in non-shadow regions caused by global diffusion-based restoration, the authors further design a Post Fusion module. Experiments on the ADSP benchmark show that the proposed auxiliary-free method outperforms mask-dependent state-of-the-art methods working with predicted shadow masks on shadow-region restoration, and Post Fusion substantially improves non-shadow and full-image fidelity.

Critical strengths
  1. The paper clearly articulates the deployment gap between ideal laboratory evaluation and real-world scenarios, and quantitatively demonstrates that inaccurate predicted auxiliary inputs cause consistent performance degradation across existing methods.

  2. The progressive transition from mask-guided learning to auxiliary-free inference cleverly exploits the mask conditioning mechanism of inpainting models. It retains the benefit of explicit spatial supervision during training while removing the requirement for external masks at test time, providing a reasonable curriculum learning scheme for adapting general inpainting models to the shadow removal task.

  3. The recall-oriented pseudo mask estimation via GMM thresholding is technically appropriate for the fusion task, as false negatives are more harmful than false positives. The module achieves a favorable trade-off between shadow removal completeness and non-shadow detail preservation with low computational overhead.

  4. The paper provides thorough quantitative and qualitative evaluation, including cross-method comparison under both ideal and realistic deployment settings, component-level ablation studies, and detailed statistical analysis of the estimated pseudo masks. The results consistently support the effectiveness of each proposed component and the overall framework.

Critical weaknesses
  1. As a diffusion-based method, the framework inevitably has higher inference latency and computational cost than lightweight CNN or Transformer baselines. Although computational cost is mentioned as a limitation, no quantitative comparison of inference speed, parameter count or FLOPs is provided, leaving the practical deployment efficiency unclear.

  2. The paper validates the improvement brought by Stage-1 GT mask tuning, but does not include a baseline that trains directly with full-one masks from scratch. Without this comparison, it is difficult to fully disentangle the contribution of the two-stage curriculum from the inherent capacity of the inpainting backbone.

  3. The quality of Post Fusion is strongly coupled with the performance of the initial I2F removal result. For faint shadows, highly textured scenes or other failure cases of I2F, the difference map will be unreliable and produce low-quality pseudo masks, which may in turn degrade the final output. The paper does not analyze such failure modes or test robustness under low-quality initial results.

  4. Core quantitative results in the main text are only presented on the ADSP dataset, while results on standard benchmarks SRD and ISTD+ are placed in the supplementary material. This weakens the demonstration of cross-dataset generalization in the main paper.

  5. The pseudo mask is binarized, which may produce unnatural transitions in penumbra regions. The paper does not specifically evaluate performance on soft shadow boundaries or discuss how the method handles gradual illumination changes at shadow edges.

Questions for authors

see above.

Reviewer_5qmFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes I2F, a shadow removal method that uses an inpainting model and does not need shadow masks during testing. The model is trained in two stages, first using ground-truth shadow masks and then full-one masks. The authors also propose Post Fusion to preserve details in non-shadow regions. Experiments on three datasets show good shadow removal results, and the ablation studies show that both the two-stage training and Post Fusion help improve the results.

Critical strengths

** The method removes the need for external shadow masks during inference, which makes it more practical to use. ** The two-stage training strategy improves shadow removal, as shown by the results in Table 4. ** Post Fusion helps preserve details in non-shadow regions and improves overall image quality.

Critical weaknesses

** The paper compares I2F with methods that need extra inputs, such as shadow masks, normals, or depth maps. These comparisons are useful. However, the paper does not compare I2F with other mask-free methods, such as DeS3, ShadowRefiner, and Xu et al. (CVPR 2025). Since these methods work under the same setting as I2F, it is hard to tell how much I2F improves over existing mask-free approaches.

** In Stage 2, the model uses an all-one mask, so the mask and masked image are constant, leaving the shadow image through ControlNet as the only image-dependent input. This raises the question of whether the inpainting backbone is really needed. The paper does not compare it with standard Stable Diffusion using the same ControlNet and LoRA setup. Also, Table 4 does not make clear whether the model without Stage 1 receives the same total training budget. This makes it hard to tell whether the improvement comes from mask-guided training or simply from training longer.

Questions for authors

Please check the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.