acceptodds
Withdrawn from ICLR 2027

Mat2GS: Feed-Forward Relighting with Explicit Materials on 2D Gaussians

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Relighting is essential for seamlessly integrating 3D assets into diverse environments, enabling applications such as augmented reality, simulation, and digital content creation. However, existing relighting methods often rely on implicit appearance modeling or image-space material prediction coupled with deferred shading, and struggle to faithfully reproduce specular highlights and shadows under novel illumination. To mitigate this issue, we propose a feed-forward relighting framework based on 2D Gaussian Splatting (2DGS) that directly attaches explicit material attributes to 2D Gaussians, establishing a unified representation of geometry and materials. Through accurate geometry and material prediction, our method produces realistic specular highlights and shadows under novel environment lighting. We further construct a large-scale synthetic dataset by rendering a nearly 110K object subset from Objaverse, providing images with various viewpoints and illumination under corresponding material and geometry. Experiments demonstrate that our method outperforms existing open-source state-of-the-art approaches in relighting quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
inverse rendering, relighting, feedforward reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z7YSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mat2GS is a feedforward framework that jointly predicts geometry and PBR materials from posed images to construct relightable 2D Gaussian assets without per-scene refinement. Its renderer combines metallic–roughness shading, environment lighting, and ray tracing for visibility and single-bounce indirect illumination. Experiments show improved context- and novel-view relighting over evaluated baselines, strong material recovery, and competitive normal accuracy.

Critical strengths

1.Strong relighting performance and cross-view consistency. Mat2GS achieves better relighting quality than the evaluated open-source baselines while preserving geometric and material consistency across viewpoints.

Critical weaknesses
  1. Unclear motivation for the relighting dataset. The authors present the Objaverse-based relighting dataset as a core contribution, yet training relies on direct supervision of material attributes without a relighting loss. The additional benefit of the relighting-specific dataset construction to model training is therefore not clearly demonstrated. 2.Limited methodological novelty. The proposed method primarily combines modules adapted from existing frameworks, without clearly demonstrating substantive innovations in either the individual components or their integration. Its methodological contribution therefore appears largely incremental.
Questions for authors

1.Dataset contribution. Given that training relies on direct material supervision without a relighting loss, what additional benefit does the multi-illumination dataset provide? Could the authors quantify this benefit through a controlled ablation comparing single- and multi-illumination training with matched objects, views, and training budgets?

2.Methodological novelty. What are the key technical innovations beyond combining modules from existing frameworks? Could the authors identify these novel design choices and provide controlled ablations that isolate their contributions while keeping the training data, supervision, and backbone fixed?

Reviewer_7vSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper shows Mat2GS, a feedforward framework for reconstructing relightable 3D objects from posed multi-view RGB images. The method includes a pretrained geometric backbone with Gaussian and material prediction heads to produce surface-aligned 2D Gaussians carrying explicit material attributes. The rendering pipeline combines split-sum physically based shading with Gaussian ray tracing to account for visibility and single-bounce indirect illumination. The authors construct a synthetic dataset from approximately 110,000 Objaverse objects, providing multi-view images under different lighting conditions alongside geometry and material annotations. Experiments evaluate material recovery, normal estimation, and relighting quality against released baseline models, while ablation studies examine the effects of normal supervision and explicit rendering.

Critical strengths
  1. The authors construct a synthetic dataset covering approximately 110,000 objects, with multiple viewpoints and lighting conditions and corresponding geometry and material annotations, providing a controlled setting for learning to distinguish intrinsic properties from illumination-dependent appearance.

  2. The rendering pipeline combines environment lighting with Gaussian ray tracing for visibility and single-bounce indirect illumination. This integration provides a geometric basis for computing self-shadowing and interreflection, while the metallic–roughness material model supports different surface reflectance characteristics.

Critical weaknesses
  1. Mat2GS is trained on 70,000 objects from the newly constructed dataset, whereas MVInverse and DiffusionRenderer are evaluated using released models, with no reported retraining or fine-tuning on the same training split Consequently, the performance differences may reflect differences in training distributions and supervision, in addition to the proposed method. Adapting representative baselines to the same training data would help establish the contribution of the methodological changes.

  2. In Table 2, explicit rendering improves PSNR over the implicit variant by 0.19 dB on context views and 0.33 dB on novel views, while the implicit variant achieves better LPIPS in both settings and better SSIM on context views. These results do not establish a consistent image-quality advantage for explicit rendering. Furthermore, the paper lacks controlled comparisons of 2D Gaussians against alternative representations and quantitative ablations isolating metallic modeling, visibility, and indirect illumination.

Questions for authors

Were MVInverse and DiffusionRenderer retrained or fine-tuned on the newly constructed training dataset? If not, could you provide results for at least one representative baseline adapted to the same training split, to help distinguish the contribution of the proposed method from that of the dataset and its supervision?

What are the principal technical differences between Mat2GS and closely related feedforward Gaussian inverse-rendering methods such as InvSplat? Is there controlled evidence that the use of 2D Gaussians or the proposed geometry–material coupling provides an advantage over alternative representations? Please also clarify why InvSplat is not included in the experimental comparisons.

Reviewer_gR9NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to reconstruct relightable 2D Gaussian asset from posed multiview RGB images. A shared multiview backbone (AnySplat + VGGT) predicts geometry and Gaussians with albedo, metallic, and roughness. Relighting combines analytic environment shading with Gaussian ray tracing for visibility and single-bounce indirect illumination. The predictor uses direct geometry/material supervision without an image-space relighting loss. The paper constructs an approximately 110K object synthetic dataset and reports improved relighting metrics on synthetic objects, views, and illumination.

Critical strengths
  • The method produces a reusable asset with explicit geometry and materials, supporting novel viewpoints and illumination without per-object optimization. Integrating this output with visibility and indirect-light computation provides a useful reconstruction-to-relighting pipeline.

  • The large synthetic dataset supplies aligned geometry and material supervision. The reported relighting PSNR gains and qualitative examples are promising evidence of the system's effectiveness on the evaluated distribution.

Critical weaknesses
  • The incremental contribution is insufficiently isolated. InvSplat already predicts materialized Gaussians feedforward, while IRGS provides materialized 2D Gaussians and ray-traced transport. Mat2GS's contribution therefore centers on its particular integration, representation choice, and hybrid renderer. The reported cross-method gains show promising system-level performance, but do not clearly establish how much improvement comes from the proposed framework compared with the backbone and task-specific training data. The experiments do not clearly show that the proposed renderer improves relighting: the comparison with the IRGS renderer shows only qualitative results, without quantitative evaluation. Additionally, the quantitative evaluation does not isolate the benefit of using 2D rather than 3D Gaussians within the proposed framework.

  • The main quantitative comparisons needs fairness. Metrics are computed at different resolutions across methods, and IRGS uses an unspecified smaller subset without matched-subset results for the other methods. Is it possible to retain ReLi3D's native 512x512 conditioning input while rendering its reconstructed asset at a common evaluation resolution, such as 256x256, and compute all methods' metrics at that same resolution for a more consistent comparison?

  • Lack of real-world examples. The evaluation is limited to the Objaverse-derived synthetic dataset, with no quantitative or qualitative results on real captures. Although the synthetic tests demonstrate generalization within this synthetic setting, this limits the evidence for the method's practical use in reconstructing and relighting captured assets. I suggest to add a few examples with real-world captures and their relighting comparisons with the real captures.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.