acceptodds
Withdrawn from ICLR 2027

GS-PI++: Light-Aware Generative PBR Decomposition of Objects in Gaussian Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recovering physically based surface attributes from Gaussian scenes is an ill-posed inverse problem because observed appearance entangles intrinsic reflectance, surface orientation, and illumination. Object-level inference provides a convenient unit for reusable learned priors, but isolating an object removes contextual observations that can help interpret its appearance, including information about the source illumination and the surrounding scene. Uncertainty in surface orientation further compounds this ambiguity, since errors in normals can be partially compensated by changes in predicted material parameters. We introduce GS-PI++, a light-aware generative framework for PBR decomposition of object instances in Gaussian scenes. For each target object, we construct an object-centered scene appearance probe by removing the target instance and rendering outward from its center. The probe composites visible near-field scene content with the source far-field environment, providing a compact directional observation of the conditions under which the object was observed. Conditioned on the object observations and this probe, a point diffusion model jointly predicts base color, roughness, metallicity, and surface normals. The learned attribute predictor avoids per-scene inverse-rendering optimization. Experiments on synthetic assets and real scanned objects show substantial improvements in material recovery over the evaluated baselines, including comparisons in which source illumination is also provided to competing inverse-rendering methods. The recovered attributes further support downstream Gaussian-based relighting and material editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Material Decomposition, Conditional Diffusion Models, Inverse Rendering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JTMvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for inverse rendering in a reconstructed 3D Gaussian scene. It combines object observations with surrounding scene context and source illumination to jointly estimate material properties and surface normals, supporting downstream relighting and material editing.

The overall novelty is moderate, primarily extending GS-PI through illumination conditioning, joint material–normal prediction, and architectural changes. However, all quantitative evaluations assume access to the GT environment map, which limits the generalization of the framework. Experiments show that the proposed framework achieves substantial improvements in material decomposition and relighting.

Critical strengths
  1. The paper is clearly written and easy to follow, with a well-explained technical design.

  2. Incorporating source illumination and surrounding scene context is well motivated and helps resolve ambiguity in material decomposition.

  3. Quantitative and visual comparisons demonstrate strong material decomposition and competitive relighting performance, including when baselines receive the same GT environment maps.

Critical weaknesses
  1. Dependence on known illumination. The framework relies on the provided GT environment lighting, which limits the application scenarios of the framework.

  2. Limited evaluation of highly reflective objects. Dedicated tests on glossy and highly reflective objects would better assess the benefits of illumination and surrounding scene conditioning.

  3. Limited evaluation of cluttered scenes. Beyond the reported failure case, systematic evaluation of complex object arrangements is needed to assess robustness to occlusion and preprocessing errors.

  4. Limited novelty. The method extends GS-PI’s point-diffusion framework through illumination conditioning, joint material–normal prediction, and network modifications. These incremental contributions also bring new limitations: the need for GT environment lighting.

Questions for authors
  1. How could the framework operate with unknown illumination? Could the authors discuss a practical approach and its limitations?

  2. Could the authors provide experiments on glossy and highly reflective objects to better validate the benefits of illumination conditioning?

  3. Could the authors provide evaluations in scenes with more complex object layouts, including at least high-resolution visual comparisons?

Reviewer_mp25OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • object-based PBR decopmosition of 3DGS objects.
  • scene decomposition first into individual objects
  • convert 3DGS into its point-cloud for each object -> feed into point cloud DiT that predicts albedo/roughness/metallic/normal;
  • further PBR gaussian refinement on top
  • re-integration into overall scene
Critical strengths
  • important problem and motivation
  • we need pbr estimates for gaussian-reconstructed scenes.
Critical weaknesses
  • the datasets are pretty contrived and limited to object-level result; the paper claims scenes but why not actually run it on actual 3dgs reconstruction?
  • this makes the scene decomposition part is kind of irrelevant given that the results are done basically on objects.
  • I don't find the results overly convincing; e.g., the video is a nice addition but w
  • the text reads overly AI-generated which makes it bloated and hard to follow; I would hope for a top-tier paper to make it more concise.
  • would've loved to see some downstream applications; e.g., a video for relighting etc.
Questions for authors

Can you show something that is less contrived 3dgs settings? E.g., Tanks and Temples or ScanNet++ scenes? Alternatively if not possible tone down the pitch and show things on object-based reconstructions such as uco3d (uncommon objects 3d).

Overall, I'm on the edge and would love to be convinced that this would work in real-world settings.

Reviewer_wmkhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GS-PI++, a method for estimating PBR material properties and surface normals of objects within a scene. It uses a point-based diffusion model conditioned on a reconstructed point cloud, image observations, and a scene lighting probe. The probe is generated by removing the target object and rendering the surrounding scene outward from its center.

Critical strengths
  • Incorporating scene context into object-level material estimation is well motivated.
  • Jointly predicting materials and normals is well motivated, as errors in surface orientation can otherwise be absorbed into material estimates.
  • The authors evaluate the method on multiple datasets covering a variety of objects and shapes.
Critical weaknesses
  • The paper targets material recovery for objects within scenes, but the experiments primarily evaluate isolated objects under HDR environment lighting. The scene example in Fig. 4(c) is presented as a failure case, and DTC appears to be the only dataset with tabletop surroundings. Consequently, the full pipeline, including object removal, inpainting, and scene-probe generation, is insufficiently evaluated. Additional synthetic scenes with quantitative evaluation would help validate the complete pipeline and its intended use case. -- The qualitative results predominantly feature objects with weak specular reflections; only one chair in the supplementary video contains a small metallic area. More examples of highly metallic and strongly specular objects should be included to demonstrate the method’s performance across diverse materials.

  • Table 3 suggests that much of the relighting improvement may come from access to source illumination. Providing the illumination improves IRGS from 19.15 to 23.76 dB, compared with 24.92 dB for GS-PI++. IRGS also achieves slightly better SSIM and LPIPS. These results raise questions about the benefit of the proposed method itself.

  • Some of the training and experimental details are not very clear.

    • The method requires a reconstructed Gaussian representation as input, but its construction is unclear. Are the Gaussians optimized from multi-view renderings? How is the initial point cloud obtained?
    • The evaluation set sizes for DTC and GSO are not specified.
    • The training configuration is incomplete. Please specify the number of training iterations, runtime, batch size, optimizer, and learning-rate schedule.
Questions for authors
  • Please add more experiments to address the critical weaknesses. And answer the missing details mentioned.
  • Another question regarding the method: The scene probe is generated with the target object removed, so it cannot capture self-interreflections between different parts of the object. Can the network implicitly account for these effects? If not, please discuss this in limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.