acceptodds
Withdrawn from ICLR 2027

Physical-Anchor Diffusion: A Coarse-to-Fine Framework for Shadow-Aware Image Harmonization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Realistic image compositing requires jointly resolving foreground appearance, illumination, and cast-shadow consistency. These factors are mutually constrained: appearance adjustment alone cannot correct lighting mismatch, while unconstrained shadow generation may improve local realism but violate scene geometry. We therefore propose Physical-Anchor Diffusion (PAD), a unified coarse-to-fine framework for shadow-aware image compositing. PAD first harmonizes the foreground in intrinsic albedo space and estimates background illumination using an environment-aware spherical-harmonic representation to relight the foreground consistently with the scene. It then combines the reconstructed scene geometry with the estimated illumination direction to construct a physically grounded coarse shadow prior that encodes shadow direction, contact region, and global layout. Conditioned on this prior, a diffusion model refines the shadow by recovering soft boundaries and realistic local appearance. By constraining global shadow structure with geometry while delegating fine-scale appearance synthesis to diffusion, PAD balances physical plausibility and visual realism. Experiments on the DESOBAv2 benchmark and real-world compositing cases demonstrate that PAD achieves stronger shadow structural fidelity while improving the perceived coherence of foreground appearance, illumination, and cast shadows in the final composites.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Harmonization, Shadow Generation, Controllable Relighting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6yEpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Physical-Anchor Diffusion (PAD), a framework for foreground harmonization and cast-shadow generation in image compositing. The first stage adapts foreground appearance and illumination through intrinsic decomposition, albedo harmonization, and spherical-harmonic relighting. Then the second stage uses estimated geometry and lighting to compute a coarse shadow mask, which guides a ControlNet-based diffusion model in generating the final composite. The proposed framework connects foreground relighting with geometry-guided shadow generation. Experiments on DESOBAv2, real-world examples, and a user study evaluate shadow quality and overall visual consistency, reporting improvements over the compared methods.

Critical strengths

The paper proposes a unified pipeline that first harmonizes and relights the inserted foreground, then constructs a coarse shadow prior from estimated geometry and illumination before diffusion refinement. The authors aim to use geometric projection to anchor shadow direction, contact relationships, and global layout, while assigning soft boundaries and local appearance to diffusion refinement, with the goal of improving consistency between foreground lighting and cast shadows. The controlled comparison in Table 3 supports the usefulness of the geometry-based prior, showing improvements in local reconstruction and structural similarity over the tested alternatives.

Critical weaknesses
  1. Insufficient justification and validation of the foreground adaptation stage.

    Relationship to prior work. The first stage follows the intrinsic decomposition, albedo harmonization, and foreground relighting pipeline of Careaga et al. [1]. PAD replaces several components, but does not clearly explain what problems these changes address or demonstrate their benefits over the original pipeline.

    Input to intrinsic decomposition. Careaga et al. [1] formulate the decomposition separately for the foreground source image and the background image. PAD applies it directly to the unharmonized composite. The inserted foreground still contains shading from its source scene, and decomposing the composite may incorrectly encode some of this shading in the foreground albedo. Since subsequent relighting replaces the foreground shading while retaining the harmonized albedo, such errors would carry into the relit result. The paper does not explain how this issue is addressed.

    Normal estimation. PAD uses MoGe-2 for foreground geometry and Omnidata for background normals. However, the paper does not explain why different models are used for the two regions or specify how their outputs are aligned and combined into the normal map used for relighting.

    Compatibility between albedo, normals, and illumination. Careaga et al. [1] fit lighting parameters to the estimated background shading and normals, explicitly connecting these estimates. PAD instead combines independently estimated albedo, geometry, and EMLight illumination. The paper does not explain how the normal and lighting coordinate conventions are aligned, or how the brightness scales and color representations of the albedo and rendered shading are matched before multiplication.

    Validation of the intermediate result. Figure 3 provides a qualitative component ablation, but does not establish whether these modifications improve foreground relighting. A direct comparison of the stage-one output against Careaga et al. [1] is missing. Since subsequent geometry estimation and shadow generation use this intermediate image, its quality is consequential to the reliability of the full pipeline.

  2. The benefit of diffusion refinement is not established. Given estimated geometry and lighting, conventional shadow rendering is a natural alternative for fixed-view compositing. The paper provides no quantitative ablation removing diffusion refinement, and Figure 3 does not demonstrate a clear improvement over the corresponding variant. Table 3 keeps the diffusion model fixed and therefore cannot establish its contribution. This leaves the benefit of a central component of the proposed coarse-to-fine framework insufficiently supported.

[1] Chris Careaga, S. Mahdi H. Miangoleh, and Yağız Aksoy. Intrinsic Harmonization for Illumination-Aware Compositing. SIGGRAPH Asia, 2023.

Questions for authors
  1. Clarification of normal estimation: How are foreground normals obtained from MoGe-2, and how are they aligned and combined with the Omnidata estimates? Why are both models needed?

  2. Evidence for the changes to intrinsic decomposition: Why is intrinsic decomposition applied directly to the unharmonized composite? Careaga et al. [1] formulate this step separately for the foreground source image and the background image. How do the authors verify that the lighting mismatch in the composite does not bias the estimated foreground albedo? Can the authors provide a direct comparison of the stage-one relighting results against Careaga et al. [1] under matched input conditions?

  3. Evidence for diffusion refinement: What quantitative improvement does diffusion refinement provide over both the variant without refinement and conventional shadow rendering using the same estimated geometry and lighting? Can the authors provide clearer visual comparisons demonstrating the improvement that is difficult to discern in Figure 3?

Reviewer_1ikCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Physical-Anchor Diffusion, a two-stage framework for shadow-aware image harmonization. The first stage adapts foreground appearance and illumination using pretrained intrinsic decomposition, harmonization, and lighting estimation methods. The second stage constructs a coarse shadow prior from estimated scene geometry and illumination direction, followed by conditional diffusion refinement. Experiments on DESOBAv2 show improvements in local reconstruction and structural similarity, although shadow-mask accuracy remains less competitive.

Critical strengths

(1) The paper addresses the joint consistency of foreground appearance, illumination, and cast shadows in image compositing. The progressive framework provides a reasonable approach to combining physical constraints with generative refinement. (2) PAD achieves competitive results on DESOBAv2, particularly in LRMSE, GSSIM, and LSSIM. The controlled coarse-prior ablation and user study provide additional evidence of its practical value.

Critical weaknesses

(1) Limited methodological novelty and insufficient differentiation from closely related work. Stage 1 integrates existing pretrained methods for intrinsic decomposition, albedo harmonization, illumination estimation, and reshading, with limited technical modifications. In Stage 2, the main contribution lies in geometry-guided shadow generation followed by diffusion-based refinement. However, similar coarse-to-fine strategies have already been explored in prior work. For example, Foreground Harmonization and Shadow Generation for Composite Image (Zhou et al., 2024) has investigated joint foreground harmonization and shadow generation through a coarse-shadow-guided diffusion framework. More closely related, Physics-Grounded Shadow Generation from Monocular 3D Geometry Priors and Approximate Light Direction (Hu et al., arXiv:2512.06174, 2025) employs monocular point-map geometry, a dominant illumination direction, and a coarse shadow estimate to guide diffusion-based shadow generation. Although PAD adopts a different projection and prior-construction strategy, its distinct methodological contributions and advantages over these existing approaches remain insufficiently established.

(2) Concerns about experimental results and insufficient validation of key components. Although PAD achieves promising improvements in several reconstruction and structural similarity metrics, the considerably worse shadow-mask accuracy raises questions about the effectiveness of its physical anchoring. Moreover, the qualitative results and existing ablations do not fully clarify the contribution of Diffusion Refinement or its potential impact on image fidelity. The robustness of the physical prior to illumination and geometry estimation errors, as well as the effects of several heuristic design choices, requires further investigation. Specific concerns are detailed in the Questions For Authors section.

Questions for authors

Q1. A clearer discussion of the differences between PAD and Zhou et al., 2024 and Hu et al., 2025 is needed to clarify PAD's distinct motivation and technical contributions.

Q2. In Table 2, PAD achieves substantial improvements in LRMSE and LSSIM but performs considerably worse than GPSDiffusion and VSDiffusion in GBER and LBER. Although the authors attribute this discrepancy to soft-shadow generation, it remains unclear whether the predicted shadow regions are accurately localized. Moreover, PAD does not achieve the best GRMSE on the BOS subset. A more detailed analysis of these differences, particularly the potential trade-off between shadow-mask accuracy and overall reconstruction quality, would strengthen the experimental conclusions.

Q3. In Figure 3, the result without Diffusion Refinement (c) appears to preserve sharper textures than the full model (b), particularly in the bag folds, ropes, and background structures. This raises concerns about potential appearance degradation introduced by refinement. Furthermore, the shadow masks in Figures 3(b) and 3(c) appear similar, while the mask in (b) contains a seemingly isolated region without a clearly corresponding shadow in the RGB result. Clarifying whether diffusion refinement modifies the shadow mask or affects non-shadow regions would be helpful. In addition, although Table 3 provides quantitative comparisons of coarse shadow priors, an additional quantitative ablation with and without Diffusion Refinement would better establish its contribution.

Q4. Figure 2 lacks comparisons with VSDiffusion, a competitive baseline in Table 2, as well as the corresponding ground-truth shadow results. The first example also exhibits a foreground color shift, which is difficult to assess without the corresponding ground-truth image.

Q5. Table 4 shows relatively limited degradation under small illumination-direction perturbations but a noticeable decline at 20°. Relating this sensitivity to the actual estimation errors of EMLight would provide a more meaningful assessment of robustness. Moreover, although Appendix A acknowledges that intermediate geometry and illumination estimation errors may propagate to the final results, their respective contributions to the observed failures are not quantitatively analyzed. A more detailed analysis of these error sources, together with sensitivity evaluations of key hyperparameters in Appendix B, would help clarify the limitations and robustness of the framework.

Q6. The construction of the coarse shadow prior relies on partitioning nearby background points into lower and upper/side receiver regions. The rationale for this partition and its applicability to different receiver geometries and illumination conditions could be further justified. Moreover, the projected contours undergo contact correction and RANSAC-based affine deformation, whose effects on the physical consistency of the resulting shadow prior remain unclear. Additional ablations of these operations would help validate the proposed design.

Reviewer_SAZYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a shadow-aware image harmonization framework that combines pretrained models for foreground relighting and geometry estimation with physics-guided shadow projection and diffusion refinement. It achieves promising results, but its novelty and physical accuracy are limited by its reliance on existing methods and simplified physical assumptions.

Critical strengths
  1. The paper incorporates physical constraints into shadow generation and achieves promising results.

  2. The paper is well-written and easy to follow.

Critical weaknesses
  1. All the components in the proposed framework are based on existing methods, which makes the overall methodological novelty somewhat limited. Moreover, these methods have their own limitations, especially since tasks such as intrinsic decomposition, illumination estimation, and geometry reconstruction are inherently ill-posed. Simply combining these methods may lead to error accumulation and propagation throughout the pipeline, potentially affecting the reliability of the final results.

  2. Although the method incorporates physical constraints, I have concerns about the extent to which it can ensure physical consistency, as several approximations are introduced. For example, selecting the direction with the highest non-ambient illumination energy as the dominant light direction assumes a single dominant light source rather than fully modeling the environment illumination. Similarly, approximating shadow-receiving surfaces as planes and applying a 2D affine transformation may not accurately represent 3D shadow projection, especially for non-planar surfaces or complex scene geometry. These approximations seem likely to limit the physical accuracy of the generated shadows.

  3. The intrinsic decomposition results themselves do not seem sufficiently accurate. For example, even in Figure 1, the estimated albedo contains noticeable errors, as the shadows on the ground are not completely removed. This raises concerns about how well the overall framework would perform when the intermediate intrinsic decomposition is inaccurate.

  4. It is unclear whether the proposed method can handle more complex shadow interactions, such as when a newly inserted object casts a shadow onto another object in the scene. Could the authors provide additional results on more challenging scenarios involving complex geometry and shadow interactions?

  5. The paper lacks a direct evaluation of the proposed coarse physical shadow prior. Although Table 3 demonstrates that the physical prior improves the final diffusion results, it does not necessarily indicate that the prior itself is physically accurate. Could the authors provide quantitative evaluations of the coarse shadow prior, such as shadow mask IoU, shadow direction error, or contact alignment error, to better validate the effectiveness of the proposed physical modeling?

  6. Recent multimodal image-editing models, such as GPT-based image-editing systems, may also be capable of performing this task. It would be interesting to include comparisons with such methods to better demonstrate the advantages of the proposed framework.

  7. Since existing methods can estimate environment illumination from an image and perform foreground relighting, a modular pipeline combining these methods could potentially achieve similar foreground harmonization results. It would be helpful if the authors could clarify the additional benefits of the proposed framework compared with such an alternative, particularly in terms of shadow generation and physical consistency.

  8. Minor Suggestion: Figure 1 could be clearer about which components are trainable and which are frozen. For example, it is not immediately clear from the figure whether the geometry reconstruction module requires training.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.