acceptodds
Under review as a conference paper at ICLR 2027

Ratio-RFE: From Appearance Fitting to Source-Relative Relighting Field Estimation for Image Harmonization

Abstract

Image harmonization aims to adjust a composite foreground to match the background while preserving its content and structure. Existing methods typically learn the desired target appearance directly, requiring appearance adjustment and content preservation to be handled within the same prediction process. This work instead formulates image harmonization as source-relative relighting field estimation. Motivated by the approximately shared foreground reflectance between the composite and harmonized images, the required source-to-target change is represented as a source-relative log-ratio relighting field, by canceling the shared reflectance. The harmonized foreground can then be obtained from the observed source through analytical reconstruction based on relighting field, without explicitly recovering reflectance or absolute lighting. Based on this formulation, a source-relative Relighting Field Estimation (in the log-Ratio form) based image harmonization framework, termed as Ratio-RFE, is proposed. It is developed with Phong-inspired Ambient-Diffuse-Specular (ADS) representations, where the Ambient, Diffuse and Specular components are estimated separately based on their different characteristics. An ADS-guided dynamic aggregation is further developed, where these representations guide spatially adaptive aggregation for unified relighting field estimation. Moreover, transformation-aligned learning is used to directly constrain the predicted log-ratio relighting field together with the reconstructed appearance. Experiments demonstrate state-of-the-art performance on iHarmony4 and strong cross-dataset generalization to ccHarmony without fine-tuning. Moreover, although trained only at , Ratio-RFE generalizes directly to and inputs without retraining, demonstrating strong cross-resolution generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image harmonization, image compositing, intrinsic decomposition, appearance transformation, high-resolution image harmonization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BLiVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Ratio‑RFE, a novel image harmonization framework reformulating the task as source‑relative log‑ratio relighting‑field estimation. Phong‑inspired Ambient‑Diffuse‑Specular (ADS) pathways separately model global low‑frequency shifts, spatially‑varying shading, and local high‑frequency specular highlights. An ADS‑MK dynamic aggregation module fuses these complementary representations to produce a unified relighting field. Experiments achieve state‑of‑the‑art quantitative and qualitative results on iHarmony4. Without fine‑tuning, Ratio‑RFE generalizes well to the out‑of‑distribution ccHarmony dataset. Comprehensive ablations validate the log‑ratio formulation, ADS multi‑path design, dynamic aggregation, and ratio‑domain supervision.

Critical strengths
  1. Novel physically‑motivated log‑ratio relighting‑field formulation for image harmonization. Most existing harmonization approaches either directly predict final RGB outputs or perform explicit intrinsic decomposition into reflectance and illumination. This work avoids the ill‑posed full intrinsic decomposition while still embedding image‑formation physics into network design.

  2. Impressive practical generalization performance across dataset domain and resolution. The model trained purely on iHarmony4 obtains SOTA results on ccHarmony. Moreover, it generalizes to high-resolution inference without retraining, while other methods degrade substantially on higher resolutions.

Critical weaknesses
  1. Phong‑inspired ADS pathways are learned feature representations rather than physically‑grounded decomposition. Although the architecture is motivated by Phong ADS lighting model, the three branches do not perform true physical decomposition of ambient/diffuse/specular components. There is no ground‑truth supervision for individual ADS terms, and the paper provides limited evidence that each pathway strictly corresponds to its intended physical lighting component.

  2. Limited analysis of failure cases and real‑world challenging cases. Experimental evaluation mainly uses standard benchmark datasets iHarmony4 and ccHarmony. The paper lacks systematic study of tough real-world scenarios: e.g., foreground shadow cast by new background, highly‑transparent foreground objects, or large‑scale color gamut mismatch. It is unclear how well the physical assumption holds under these real‑world conditions, which might be the practical limits of the proposed approach.

Questions for authors
  1. Could you provide further analysis or experiments to verify to what extent each pathway actually captures its target lighting phenomenon, rather than just learning complementary generic image features?
  2. Why does the log‑ratio prediction formulation bring substantial cross‑resolution generalization benefit compared with direct RGB regression? Can you provide an intuitive or mathematical explanation for this observation?
  3. In transformation‑aligned training, you apply auxiliary L1 loss on the clipped ground‑truth log‑ratio field RgtR^{gt}. Does clipping RgtR^{gt} to [−rmax,rmax][-r_{max}, r_{max}] discard useful supervision for large lighting corrections?
Reviewer_ndv3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the image harmonization task. The authors formulate it as a source-relative log-ratio correction. Motivated by approximately shared foreground reflectance, the formulation avoids explicitly estimating reflectance or absolute illumination. The authors introduce three branches for global, spatial, and high-frequency features, together with dynamic aggregation, to estimate the correction field.

Critical strengths
  1. The authors propose a straightforward log-ratio formulation for image harmonization. The experimental results show the benefit of predicting the correction field and using direct ratio supervision.

  2. The authors demonstrate the effectiveness of the three proposed branches and dynamic aggregation and achieve the best performance among the compared methods on iHarmony4. The cross-dataset evaluation further supports generalization to a different appearance-transformation distribution.

Critical weaknesses
  1. The physical interpretation of the three branches needs more evidence. The experiments show that global, spatial, and high-frequency features are useful, but not that they correspond to ambient, diffuse, and specular effects. If there is no evidence that the three branches match these effects, I would like the authors to simply call them global, spatial, and high-frequency branches and clarify each component’s role based on what it actually learns. Inspiration from the lighting model alone is insufficient to justify these names without evidence that the branches capture the corresponding lighting effects.

  2. The comparison is incomplete. DiffHarmony++ (Enhancing Image Harmonization with Harmony-VAE and Inverse Harmonization Model, ACM MM 2024) is missing from the compared methods. Its reported performance is competitive with that of the proposed method. Therefore, the authors need to clarify the advantages of Ratio-RFE.

Questions for authors
  1. Can the authors provide representative qualitative results on Hday2night examples with large lighting changes and conduct experiments on IllumHarmony (SIDNet: Learning Shading-Aware Illumination Descriptor for Image Harmonization, IEEE TETCI 2024) or IndoorHarmony (Spatially-Varying Illumination-Aware Indoor Harmonization, IJCV 2024)? Most examples in iHarmony4 and ccHarmony primarily involve color and tone transformations. Evaluation on datasets with genuine shading changes would provide more convincing evidence for the proposed relighting formulation and lighting-inspired design.

  2. Please see my concern in Weakness 2.

  3. Can the authors clarify the small-change assumption used for Eq. 4? The normalized diffuse and specular changes are not necessarily small after separating the ambient term, particularly when lighting direction or highlights change.

Reviewer_qAhTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ratio-RFE, an image harmonization framework that reframes the task from direct appearance fitting to source-relative relighting field estimation. It adopts a U-Net with RAIN normalization and employs three Phong-inspired representations, namely Ambient, Diffuse, and Specular (ADS), to guide the spatially adaptive aggregation of dynamically filtered candidate responses. The harmonized image is then analytically reconstructed using the predicted relighting field. The model, trained only at 256×256, generalizes to 1024×1024 and 2048×2048 without retraining, outperforming prior methods at higher resolutions.

Critical strengths
  1. Reformulating image harmonization as source-relative log-ratio relighting-field estimation, followed by analytical reconstruction, provides a new modeling perspective for image harmonization.

  2. The method introduces three complementary pathways inspired by the Phong model to capture global low-frequency appearance variations, spatially varying shading patterns, and localized high-frequency variations, respectively. These representations guide the spatially adaptive aggregation of dynamically filtered foreground responses. Experimental results demonstrate the effectiveness of the proposed module.

Critical weaknesses
  1. The connection between the physical motivation provided by the Phong model and the learned representations of the ADS pathways remains unclear. Although the three pathways are designed to model global low-frequency appearance variations, spatially varying shading patterns, and localized high-frequency variations, there is insufficient evidence that these learned representations correspond to the respective physical lighting components. Further analysis of the pathway features and the three mixing weights in ADS-MK under different lighting conditions would help validate the connection between the proposed module and its physical motivation.

  2. The ablation study in Table 5 does not sufficiently distinguish the respective contributions of the ADS pathways and the ADS-MK aggregation mechanism. According to Eq. (9), the dynamic filtering operator generates three foreground candidate responses, Yl1,Yl2,Yl3Y_l^1, Y_l^2, Y_l^3, while Eq. (10) uses the ADS features FlA,FlD,FlSF_l^A, F_l^D, F_l^S, among other inputs, to predict the corresponding spatial mixing weights gl1,gl2,gl3g_l^1, g_l^2, g_l^3. However, the paper does not clarify whether the numbers of candidate responses generated in Eq. (9) and mixing weights predicted in Eq. (10) remain fixed at three in the single-pathway ablations. Additional ablation experiments are recommended to investigate how the relationship between the number of candidate responses and the number of pathways affects model performance.

  3. The three-pathway ablation study in Table 5(a) lacks the pairwise combinations A+D, A+S, and D+S, making it difficult to fully assess the contribution of each pathway. In addition, Table 1 does not report the fMSE metric, and the fMSE value of 152.25 in Table 2 differs from the value of 137.58 reported in Table 5. The authors must clarify the reason for this discrepancy and specify the training and evaluation datasets used in the ablation experiments in Table 5.

  4. The paper does not report the model's parameters and computational cost, so it is unclear whether the performance gains come from the additional model capacity introduced by the new multi-branch structure.

Questions for authors

My main concerns are placed in the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.