acceptodds
Under review as a conference paper at ICLR 2027

Inverse Rendering without Training from Scratch: Unsupervised Post-Training of Feedforward 3D Gaussian Models

Abstract

Despite the great success of feedforward 3D Gaussian models in scene reconstruction, their learned representations cannot provide editable physical information. While optimization-based inverse rendering methods are able to produce these attributes, they are not compatible to feedforward paradigm. To close this gap, we introduce an unsupervised post-training framework that adapts feedforward 3D Gaussian models to inverse rendering without training a new network from scratch. We first analyze the propagation of geometry errors from off-the-shelf Gaussian scaffold produced by feedforward models to PBR maps, and are motivated to develop a post-splat decoding paradigm. Specifically, the features produced by feedforward 3D Gaussian models are aggregated to construct scene memory. Afterwards, a learnable adapter is introduced to generate view-independent material latents conditioned on the scene memory, which are then decoded into PBR parameters after splatting. Experiments across seven backbones demonstrate that our framework achieves state-of-the-art performance against previous approaches with only 1.5%–7.0% additional inference parameters. Moreover, the ablation experiments validate the effectiveness of our post-splat decoding paradigm and scene memory, while qualitative results on real-world scenes illustrate promising relighting and illumination editing results.

Primary area:
generative models
Keywords:
Relighting, Inverse rendering, 3D Gaussian splatting, Feed-Forward Scene Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sJm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an unsupervised post-training framework that adapts pretrained, feedforward 3D Gaussian Splatting models to inverse rendering without training a new network from scratch. By freezing the original reconstruction backbone and analyzing how residual geometry errors propagate into nonlinear PBR shading, the authors replace traditional pre-splat parameter estimation with a post-splat decoding paradigm. In this approach, view-independent material latents are splatted onto the image plane and decoded into PBR parameters conditioned on an aggregated multi-view scene memory. Evaluated across seven feedforward backbones on RealEstate10K, the method adds only a small number of additional parameters while matching or exceeding native novel-view reconstruction fidelity and enabling real-world relighting.

Critical strengths
  • The paper resolves the geometry-to-PBR error propagation bottleneck by rasterizing 64-D material latents rather than blending explicit 8-D PBR parameters before splatting, preserving higher-order distribution moments essential for nonlinear shading, preventing geometry misalignment from degrading material estimation.

  • The framework adapts frozen feedforward 3DGS models to inverse rendering without requiring ground-truth PBR supervision or retraining architectures from scratch. It demonstrates plug-and-play compatibility across seven distinct backbones while introducing only limited parameter overhead.

Critical weaknesses
  • Absence of Quantitative Intrinsic and Relighting Evaluation: The empirical evaluation relies entirely on source-light novel-view synthesis metrics (PSNR, SSIM, LPIPS) on RealEstate10K alongside qualitative crops, offering zero numerical validation on intrinsic material accuracy (e.g., albedo, normal angular, roughness) or ground-truth HDR relighting benchmarks. Because inverse rendering is a heavily ill-posed factorization problem subject to fundamental material-illumination ambiguities, high RGB reconstruction fidelity under the original capture lighting can easily be achieved while baking illumination into albedo or learning unphysical specular shortcuts. Without quantitative verification against ground-truth material datasets or comparisons against prior feedforward inverse rendering pipelines, it is impossible to substantiate whether the estimated decomposition is physically meaningful or merely an over-parameterized appearance fit.

  • Architectural Violations of View-Invariance: Key components of the post-splat and illumination architectures introduce screen-space dependencies that conflict with the view-independent nature of physical scene properties. Applying 8-group Group Normalization across the spatial dimensions of the target-view rasterized feature map forces the decoded albedo, roughness, and normals at any given 3D point to vary based on camera framing, image boundaries, and visible scene context. Similarly, conditioning the global HDR environment cubemap prediction on the rendered target-view feature buffer risks making the estimated scene lighting viewpoint-dependent. These choices compromise the multi-view and temporal consistency inherent to explicit 3D Gaussian representations.

Questions for authors
  • Does evaluating the post-splat decoder along continuous camera trajectories reveal temporal flickering or material drift caused by screen-space GroupNorm and view-dependent feature rasterization?

  • Why does the illumination head ingest target-view rendered feature maps to predict a global scene environment cubemap? Does the predicted cubemap vary when querying different target camera poses within the same scene, and how is cross-view lighting consistency enforced?

  • How does the framework perform quantitatively on established synthetic/real inverse rendering datasets with ground-truth intrinsics and known HDR relighting?

  • Why were external feedforward inverse rendering pipelines (such as MVInverse or MAIR++) excluded from the empirical comparisons, and how does your unsupervised adapter compare to them in terms of inference speed and material accuracy?

  • Because the adapter is trained solely on re10k, how well does the post-trained model generalize zero-shot to completely different scene distributions without retraining or fine-tuning? Have you evaluated cross-dataset transfer to confirm that training on residential interiors does not degrade the zero-shot capabilities of backbones

5 comments

Comment:

Thank you for recognizing our post-splat design for mitigating geometry-to-PBR error propagation and our lightweight adaptation across frozen backbones without PBR supervision. We address the quantitative evaluations, illumination-head description, feed-forward comparisons, and cross-dataset transfer below.

R3.W1 — Quantitative intrinsic recovery and relighting

To address the quantitative evaluation concern, we reproduce below the TensoIR and Synthetic4Relight results reported in our response to Reviewer 1, Weakness 1, including comparisons with MVInverse, MAIR++, GI-GS, and GS-IR.

(1) TensoIR.

Repr.InferencePBR sup.Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Normal cos. ↑Relight PSNR ↑
MVInverse2DFFSupervised18.5620.8740.1520.90218.420
MAIR++2DFFSupervised21.1520.8900.1580.85921.029
GI-GS3DOptimizationUnsupervised24.2510.8820.1500.89826.972
GS-IR3DOptimizationUnsupervised24.1810.8800.1550.91226.549
Ours (AnySplat)3DFFUnsupervised24.9140.8870.1540.86226.911

(2) Synthetic4Relight.

Repr.InferencePBR sup.Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Relight PSNR ↑
MVInverse2DFFSupervised18.4050.8760.14120.099
MAIR++2DFFSupervised20.4920.8860.13122.103
GI-GS3DOptimizationUnsupervised22.9960.9120.11228.185
GS-IR3DOptimizationUnsupervised21.9780.8960.12826.893
Ours (AnySplat)3DFFUnsupervised21.3220.8950.14926.405

We use ground-truth intrinsic properties from TensoIR to quantitatively assess material recovery. Compared with feed-forward methods MVInverse and MAIR++, our albedo and relighting PSNR are higher on TensoIR and Synthetic4Relight.

TensoIR is absent from the reported training data of MVInverse, so domain differences may affect its performance [2].

Compared with per-scene optimization methods GI-GS and GS-IR, we achieve higher albedo PSNR and comparable relighting on TensoIR.

Our method uses deferred rendering based on GI-GS. Without intrinsic ground-truth supervision, some illumination may be baked into the estimated intrinsic parameters, an issue also present in GI-GS [1].

Nevertheless, quantitative evaluation against intrinsic ground truth supports competitive material accuracy.

R3.W2 / Q1 / Q2 — Cross-view consistency and illumination prediction

Since Weakness 2 and Questions 1–2 concern the same architectural issues, we address them jointly here, we first evaluate cross-view material consistency through flow-reprojection experiments, then clarify the actual inputs to the illumination head and how the scene-level environment map is shared across target views.

We conducted flow-reprojection experiments following MVInverse [2] on RealEstate10K, DL3DV, Synthetic4Relight, and TensoIR to evaluate cross-view material consistency by comparing material properties at corresponding surface locations across views.

The table reports Albedo / Metallic / Roughness RMSE ↓.

DatasetMVInverseMAIR++GI-GSGS-IROurs (AnySplat)
RealEstate10K0.039 / 0.030 / 0.0150.050 / N/A / 0.1080.057 / 0.086 / 0.0460.047 / 0.074 / 0.0520.024 / 0.040 / 0.020
DL3DV0.045 / 0.030 / 0.0110.054 / N/A / 0.1020.058 / 0.081 / 0.0610.067 / 0.089 / 0.0690.035 / 0.065 / 0.028
Synthetic4Relight0.031 / 0.014 / 0.0050.051 / N/A / 0.0810.018 / 0.017 / 0.0300.020 / 0.014 / 0.0280.013 / 0.039 / 0.017
TensoIR0.041 / 0.027 / 0.0070.060 / N/A / 0.1230.021 / 0.022 / 0.0280.023 / 0.022 / 0.0260.018 / 0.054 / 0.022

Despite using no intrinsic ground-truth supervision, our method maintains competitive cross-view material consistency on the evaluated datasets.

Illumination-head description and reproducibility. We apologize for the incorrect description of the illumination-head inputs in the manuscript. The text mistakenly identified the rendered target-view feature buffer as its input.

Figure 2 correctly shows the actual inputs: the output tokens of the frozen feed-forward 3DGS backbone and the global scene latent, matching our implementation.

For a fixed set of input images, the scene-level HDR cubemap is shared across target views, and changing the target camera does not change the illumination-head inputs.

We will correct Section 4.2 and Appendix G.5 and subsequently release the implementation code and test and evaluation scripts.

R3.Q3 — Quantitative evaluation on synthetic and real scenes

We report complementary evaluations on TensoIR for ground-truth intrinsic recovery and relighting under known HDR illumination, and on NeRF-OSR for real outdoor scene relighting.

(1) TensoIR.

Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Normal cos. ↑Relight PSNR ↑
MVInverse18.5620.8740.1520.90218.420
MAIR++21.1520.8900.1580.85921.029
GI-GS24.2510.8820.1500.89826.972
GS-IR24.1810.8800.1550.91226.549
Ours (AnySplat)24.9140.8870.1540.86226.911

(2) NeRF-OSR.

Relight PSNR (dB) ↑
MVInverse + adapter19.857
MAIR++ + adapter17.652
GI-GS17.378
R3GW21.728
LumiGauss20.978
Ours (Full)19.421

The limited training views in our NeRF-OSR evaluation may constrain per-scene optimization methods such as GI-GS and contribute to its weaker relighting performance.

Across synthetic and real-scene evaluations, our method achieves competitive relighting and inverse rendering without intrinsic ground-truth supervision, supporting its applicability across scene domains.

R3.Q4 — Feed-forward baselines and inference efficiency

Following the reviewer's suggestion, we have added quantitative comparisons with MVInverse and MAIR++.

Material accuracy and inference speed. We assess material accuracy against ground-truth intrinsic properties on TensoIR (Question 3) and report inference speed on DL3DV below.

Time
Ours (AnySplat)0.0925
MVInverse0.0497
MAIR++0.1025

Our method offers competitive material accuracy and inference speed. Its inference cost includes material rasterization and decoding in the deferred rendering pipeline for PBR.

R3.Q5 — Cross-dataset zero-shot transfer

We directly test the AnySplat-based model post-trained on RealEstate10K on DL3DV, without retraining or fine-tuning.

Transfer results for the other backbones are also reported below.

BackboneDense PSNR: Native → PBR (dB)Dense ΔPSNR (dB)Sparse PSNR: Native → PBR (dB)Sparse ΔPSNR (dB)
AnySplat25.714 → 28.024+2.31021.716 → 23.851+2.135
DA322.165 → 24.650+2.48511.509 → 11.482−0.027
C3G21.398 → 21.365−0.03320.939 → 20.976+0.037
NoPoSplat21.599 → 24.355+2.75621.326 → 23.327+2.001
SPFSplat25.151 → 26.437+1.28721.518 → 22.926+1.408
YoNoSplat24.402 → 25.673+1.27122.989 → 24.213+1.224

Our method remains effective on most backbones in these evaluated cases. Entries average five scenes selected after evaluation; DA3-dense evaluates input-view reconstruction.

References

[1] Hongze Chen, Zehong Lin, and Jun Zhang. GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering. ICLR 2025. Appendices H and I.2; authors' response to Reviewer zyyD, Q1.

[2] Xiangzuo Wu, Chengwei Ren, Jun Zhou, Xiu Li, and Yuan Liu. MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds. arXiv:2512.21003, 2025, v2 §§3.2, 4.2, and 7 (Table 5).

[3] JunYong Choi, SeokYeong Lee, Haesol Park, Seung-Won Jung, Ig-Jae Kim, and Junghyun Cho. MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation. arXiv:2408.06707, v2, 2025, §§IV-D, IV-E, and V-A.

Reviewer_JwDxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an unsupervised post-training framework that adapts pretrained feedforward Gaussian models to inverse rendering. The framework uses estimated Gaussian geometry from the feed-forward Gaussian model and learn latents for material and lighting, and the scene before decoding into multi-view PBR maps to preserve mixtures of shading responses. Learned latents are splatted by the given camera and then decoded to PBR maps. The paper analyzes information loss in pre-splat PBR aggregation and evaluates the framework across seven backbones on RealEstate10K benchmark, reporting RGB reconstruction results, component ablations, and qualitative relighting comparisons.

Critical strengths
  1. Interesting post-splat material decoding formulation. The paper aggregates PBR intrinsic features before decoding int PBR maps, rather than directly blending explicit PBR parameters, providing a plausible way to retain information while exploiting rich features from fixed feedforward Gaussian representations for inverse rendering.

  2. Broad applicability across feedforward backbones. The framework is evaluated on seven pretrained Gaussian backbones with separately trained adapters improving RGB reconstruction while adding reletively few inference parameters (1.5%~7.0%).

Critical weaknesses
  1. Insufficient validation of intrinsic reconstruction and relighting Tables 1–2 primarily evaluate RGB reconstruction, which does not support whether the recovered albedo, normals, and illumination are physically plausible. The intrinsic and relighting metrics in Fig. 5 and Table 3 measure prediction drift under geometry perturbations rather than errors against ground truth. It is therefore unclear whether the improved PBR rendering reflects more accurate decomposition or compensation for geometry and illumination errors. Similarly, Fig. 4 demonstrates responses to different illumination maps, but without target-light ground truth, it is difficult to identify the advantage of SPFSplat PBR over the other methods. Although Sec. 5.1 acknowledges the lack of relighting ground truth in RE10K, quantitative evaluation on a benchmark such as NeRF-OSR, used by LumiGauss and R3GW, would provide more direct evidence and further support the claim on enhancing inverse rendering.

  2. Insufficient analysis of the sources of improvement. Appendix E motivates post-splat decoding through preservation of additional mixture information, but the paper does not provide empirical evidence that the learned material latents retain and use this information besides RGB reconstruction ablation in Table 2. A comparable decoder applied to the aggregated 8-D PBR parameters, or a controlled experiment with identical mean PBR parameters but different roughness variances, would help test this explanation. Fig. 5/Table 3 also evaluates only on proposed method, thus whether the proposed method reduces drift under the same perturbations remains unclear. Moreover, adding scene conditioning raises PSNR from 24.62 to 24.82 dB while introducing approximately 11.5M additional parameters, compared with the 0.054M added by the post-splat variant without scene conditioning. A parameter-matched control is needed to distinguish the benefit of scene information from increased capacity. Lastly, since quantitative evaluation is restricted to RE10K which is the dataset used for adapter training, whether this benefit persists on other datasets remains unknown. The connection between the theoretical motivation, the roles of these modules, and the reported improvements needs clearer explanation and empirical support.

Questions for authors
  1. Could the authors provide reconstruction results for the predicted environment maps alongside albedo, normals, roughness, and metallic? If the ground truth is available, quantitative comparisons and corresponding visualizations would help assess whether the enhanced RGB reconstruction is achieved through correct decomposition.

  2. What inference-time overhead does the proposed adaptation introduce beyond the frozen backbone?

4 comments

Comment:

Thank you for recognizing the value of post-splat material decoding and the broad applicability of our lightweight framework across feed-forward backbones.

We address the quantitative evaluations, representation and scene-conditioning ablations, visualization plans, and inference overhead below.

R2.W1 & Q1 — Intrinsic recovery, relighting, and environment-map visualization

We have added quantitative evaluations on TensoIR and NeRF-OSR to assess intrinsic recovery and relighting, alongside the complementary Synthetic4Relight evaluation.

Ground-truth intrinsic and relighting evaluation. TensoIR provides reference albedo and normal maps together with images under known environment illumination [1].

We evaluate recovered albedo and normals against these references and compare renderings under held-out HDR environment maps with the corresponding target images. Synthetic4Relight provides an additional evaluation of albedo and relighting.

(1) TensoIR.

Repr.InferencePBR sup.Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Normal cos. ↑Relight PSNR ↑
MVInverse2DFFSupervised18.5620.8740.1520.90218.420
MAIR++2DFFSupervised21.1520.8900.1580.85921.029
GI-GS3DOptimizationUnsupervised24.2510.8820.1500.89826.972
GS-IR3DOptimizationUnsupervised24.1810.8800.1550.91226.549
Ours (AnySplat)3DFFUnsupervised24.9140.8870.1540.86226.911

(2) Synthetic4Relight.

Repr.InferencePBR sup.Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Relight PSNR ↑
MVInverse2DFFSupervised18.4050.8760.14120.099
MAIR++2DFFSupervised20.4920.8860.13122.103
GI-GS3DOptimizationUnsupervised22.9960.9120.11228.185
GS-IR3DOptimizationUnsupervised21.9780.8960.12826.893
Ours (AnySplat)3DFFUnsupervised21.3220.8950.14926.405

NeRF-OSR. We also quantitatively evaluate relighting on the real outdoor scene lk2 using 18 context views and five test views.

Relight PSNR (dB) ↑
MVInverse + adapter19.857
MAIR++ + adapter17.652
GI-GS17.378
R3GW21.728
LumiGauss20.978
Ours (Full)19.421

Our albedo and relighting PSNR exceed those of the feed-forward MVInverse on TensoIR and Synthetic4Relight.

TensoIR is absent from its reported training data [2], so domain shift may affect its performance.

A smaller domain gap may help explain its slight relighting advantage on NeRF-OSR. Our unsupervised feed-forward results approach the per-scene optimization methods GI-GS and GS-IR on TensoIR and outperform GI-GS on NeRF-OSR, although a gap remains on Synthetic4Relight.

Overall, our method offers competitive intrinsic recovery and relighting, while not being the best across all datasets and metrics.

Predicted environment maps. In the final manuscript, we will show predicted environment maps alongside albedo, normal, roughness, and metallic maps, input images, reconstructed RGB images, and available ground-truth references for direct visual comparison.

We will subsequently release the datasets used in our experiments, the training pipeline, trained model weights, and the testing and evaluation pipeline.

R2.W2 — Representation, decoder, and scene-conditioning controls

We report additional representation ablations, a three-method material-drift diagnostic, and evaluations on DL3DV below.

1. Learned features versus a decoder applied to aggregated PBR parameters. Following your suggestion, we repeated the ablation experiments on SPFSplat.

We evaluated PBR8-PostDecoder, which splats explicit 8-D PBR parameters and applies a learned material decoder afterward.

We compare it with Post-Latent-8, which splats unconstrained 8-D features before the same decoder architecture.

The two variants use the same training and evaluation settings.

SPFSplat variantAdded inference params (M)PSNR ↑SSIM ↑LPIPS ↓
PBR8-PostDecoderref.24.3650.8150.118
Post-Latent-8+0.00024.5760.8160.117
Ours w/o Scene Latent, 64-D+0.04924.6280.8190.114
Ours (Full), 64-D+13.45125.2990.8280.106

The matched 8-D comparison improves PSNR by 0.211 dB, with identical dimensionality, decoder architecture, and parameter count.

Thus, adding a decoder after explicit PBR aggregation does not reproduce the performance obtained by learning the representation that is aggregated.

Increasing the learned feature width to 64 provides a further 0.052 dB. These results support preserving task-adapted material features through splatting.

2. Comparative material drift under perturbed training geometry. Given the time constraints, we conducted three paired experiments for Direct-PBR, Ours w/o Scene Latent, and Ours (Full).

Each pair compares training on clean geometry with training on ε = 0.1 perturbed geometry, using the same training and evaluation settings across the three methods.

Scale-aligned L1 ↓Direct-PBROurs w/o Scene LatentOurs (Full)
Albedo0.0160.0160.014
Metallic0.0410.0350.033
Roughness0.0510.0250.026

Both post-splat variants have lower mean drift than Direct-PBR for all three materials.

Ours (Full) further reduces albedo and metallic drift relative to the variant without scene conditioning, while the latter has lower roughness drift.

These results support the robustness of our post-splat formulation to geometry errors inherited from frozen feed-forward models in the evaluated cases.

3. Scene conditioning and parameter overhead.

Scene conditioning adds 13.402M parameters on SPFSplat and 11.512M on YoNoSplat, accounting for just 2.02% and 1.14% of their complete inference networks, respectively.

Thus, scene information is introduced with a small parameter overhead, and the models with scene conditioning achieve better reconstruction results.

4. Evaluation beyond RE10K. We additionally evaluate the PBR-adapted models on DL3DV.

BackboneDense ΔPSNR (dB)Sparse ΔPSNR (dB)
AnySplat+2.310+2.135
DA3+2.485−0.027
C3G−0.033+0.037
NoPoSplat+2.756+2.001
SPFSplat+1.287+1.408
YoNoSplat+1.271+1.224

These DL3DV cases show improved reconstruction for AnySplat, NoPoSplat, SPFSplat, and YoNoSplat under both view settings, while C3G and DA3-sparse remain close to their native reconstruction scores.

Our post-training framework therefore retains competitive reconstruction performance on the evaluated DL3DV cases beyond RE10K.

R2.Q2 — Inference overhead

The matched inference-time and peak-memory measurements are reported in the table below.

BackboneForward + one-view rendering (ms), Base → Ours-PBRPeak allocated memory (GiB), Base → Ours-PBR
AnySplat87.080 → 128.5102.970 → 3.470
DA370.660 → 119.3306.580 → 7.080
SPFSplat43.610 → 89.2102.570 → 3.150
C3G64.700 → 98.5704.050 → 4.200
FLARE105.330 → 142.8305.590 → 6.120
NoPoSplat46.130 → 95.1902.550 → 3.110
YoNoSplat52.620 → 98.8303.770 → 4.370

The additional inference time comes from the modules introduced during post-training and the PBR renderer, which perform material decoding, illumination prediction, and physically based shading. A forward pass plus one rendered view takes 89.2–142.8 ms.

We consider the additional 33.9–49.1 ms an acceptable trade-off for enabling material decomposition and relighting in a feed-forward pipeline.

References

[1] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. TensoIR: Tensorial Inverse Rendering. CVPR, 2023, pp. 165–174. DOI: 10.1109/CVPR52729.2023.00024.

[2] Xiangzuo Wu, Chengwei Ren, Jun Zhou, Xiu Li, and Yuan Liu. MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds. arXiv:2512.21003, 2025, v2 §3.2 (Training).

[3] JunYong Choi, SeokYeong Lee, Haesol Park, Seung-Won Jung, Ig-Jae Kim, and Junghyun Cho. MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation. arXiv:2408.06707, v2, 2025, §V-A (Training strategy and losses).

Thank you for the detailed responses and additional experiments, addressing my concerns regarding inference overhead, evaluation beyond RE10K, geometry perturbation, the benefit of learned material latents, and scene conditioning.

Overall, I find the framework practically useful. However, the additional intrinsic recovery and relighting results still is not sufficiently convincing that the method achieves accurate intrinsic decomposition. As this remains a central concern for the paper’s inverse-rendering contribution, I maintain my original rating.

I again thank the authors for their efforts in addressing my concerns during the rebuttal.

Reviewer_FQPsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an unsupervised post-training framework that adapts pretrained feedforward 3D Gaussian Splatting models to inverse rendering. Instead of predicting PBR attributes per Gaussian, the method decodes materials (albedo, normals, roughness, metallicity) after splatting, and adds scene-conditioned feature aggregation via a scene latent; illumination is estimated jointly. Training uses only multi-view RGB reconstruction with smoothness and diffuse–specular regularization. The adapter adds 1.5%–7.0% parameters and is applied to several backbones (e.g., SPFSplat, C3G) trained on RealEstate10K. Validation consists mainly of target-view RGB metrics (PSNR/SSIM/LPIPS) under sparse- and dense-view protocols, qualitative relighting and comparisons with existing inverse-rendering methods, a geometry-perturbation study of a Direct-PBR baseline, and a theoretical argument that richer feature representations cannot increase optimal prediction risk.

Critical strengths

1.Practical, backbone-agnostic formulation. Adapting existing pretrained feedforward 3DGS models to inverse rendering without supervision on materials or lighting is a relevant problem, and the post-splat adapter is lightweight (1.5%–7.0% extra parameters) and can be attached to multiple backbones. This matters because it would let inverse rendering benefit from progress in feedforward reconstruction without retraining. 2.Promising cross-backbone evidence. The method is evaluated on several different backbones under two protocols, and in a number of settings it maintains or improves RGB reconstruction quality while producing material and lighting decompositions. This supports the generality of the idea, although (see Weaknesses) the evidence is not yet sufficient to establish the central claims.

Critical weaknesses
  1. The evaluation primarily reports target-view RGB reconstruction metrics, which do not establish the accuracy of recovered albedo, normals, roughness, metallicity, or illumination: different combinations of material properties and lighting can produce similar rendered images. The authors acknowledge that RealEstate10K lacks ground-truth illumination and controlled relighting observations, so relighting is evaluated only qualitatively. Comparisons with existing inverse-rendering methods are likewise largely qualitative and cover limited scenes, and the main quantitative tables compare PBR-adapted outputs mainly with the native renderings of the same backbones. This is consequential because inverse rendering is the paper’s core claim; without quantitative evaluation on synthetic scenes with ground-truth PBR parameters (or real scenes captured under controlled lighting) and matched comparisons with existing inverse-rendering methods, it is unclear whether the framework produces physically meaningful decompositions or improves on prior work.

  2. The geometry-perturbation experiment in Figure 5 evaluates Direct-PBR but does not provide a corresponding comparison with the proposed post-splat method under identical perturbations. It therefore demonstrates baseline sensitivity without establishing the claimed advantage of the proposed approach. Moreover, the theoretical result that a richer feature representation cannot increase optimal prediction risk does not guarantee that a finite-capacity network will learn more robust material predictions. This is consequential because robustness to geometry errors is the main motivation for the post-splat design. A controlled comparison of Direct-PBR, post-splat decoding without the scene latent, and the full method under identical geometry perturbations is needed.

Questions for authors

Clarifications: 1.Cross-backbone regressions and protocol differences. Table 1 shows regressions after adaptation in some settings: SPFSplat is worse in sparse-view PSNR, SSIM, and LPIPS, and C3G has worse LPIPS under both protocols. The manuscript also notes different training context counts for SPFSplat, and Appendix H indicates that C3G uses a different training objective with several regularization losses disabled. Could you clarify these protocol differences, explain the observed regressions, and state whether the results under a unified protocol still support the claimed generality? 2.Train/test overlap. The adapter is trained on 6,128 RealEstate10K scenes. What datasets were the pretrained backbones originally trained on, and can you confirm that none of the evaluation scenes were seen during backbone pretraining? 3.Efficiency. The 1.5%–7.0% parameter overhead alone does not establish computational efficiency. Please report adapter training cost, inference latency, and memory requirements relative to the base backbones. 4.Material identifiability and cross-view consistency. Materials and illumination are estimated from multi-view RGB reconstruction with smoothness and diffuse–specular regularization, which may not resolve the ambiguity between reflectance and illumination. Also, PBR maps are decoded from splatted features whose aggregation weights depend on viewpoint. Can you evaluate material consistency across corresponding surface regions in different views, and stability under different viewpoints and lighting conditions? 5.If quantitative comparisons with existing inverse-rendering methods cannot be provided, would you consider moderating the claims of superior material decomposition and relighting performance?

4 comments

Comment:

Thank you for recognizing the practical value of our lightweight, backbone-agnostic approach to unsupervised inverse rendering and its promising results across backbones. We address the added experiments, protocol corrections, training-data separation, and claim scope below.

R1.W1 — Quantitative intrinsic recovery and relighting

We added evaluations on TensoIR and Synthetic4Relight. Recovery depends on the frozen backbone's geometry and features, motivating our choice of AnySplat, pretrained on nine diverse datasets [2]. In the tables, FF denotes feed-forward inference.

(1) TensoIR.

Repr.InferencePBR sup.Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Normal cos. ↑Relight PSNR ↑
MVInverse2DFFSupervised18.5620.8740.1520.90218.420
MAIR++2DFFSupervised21.1520.8900.1580.85921.029
GI-GS3DOpt.Unsupervised24.2510.8820.1500.89826.972
GS-IR3DOpt.Unsupervised24.1810.8800.1550.91226.549
Ours (AnySplat)3DFFUnsupervised24.9140.8870.1540.86226.911

(2) Synthetic4Relight.

Repr.InferencePBR sup.Albedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Relight PSNR ↑
MVInverse2DFFSupervised18.4050.8760.14120.099
MAIR++2DFFSupervised20.4920.8860.13122.103
GI-GS3DOpt.Unsupervised22.9960.9120.11228.185
GS-IR3DOpt.Unsupervised21.9780.8960.12826.893
Ours (AnySplat)3DFFUnsupervised21.3220.8950.14926.405

Across both datasets, our method achieves the highest albedo and relighting PSNR among the evaluated feed-forward methods. As an unsupervised approach, it also approaches the performance of per-scene optimization methods, demonstrating competitive results without test-time optimization.

R1.W2 — Robustness to geometry perturbations

Given the time constraints, we compared Direct-PBR, post-splat decoding without the scene latent, and the full method at geometry-perturbation levels of ε = 0 and ε = 0.1.

Albedo aligned L1 ↓Metallic aligned L1 ↓Roughness aligned L1 ↓
Direct-PBR0.0160.0410.051
Post-splat w/o scene latent0.0160.0350.025
Ours (Full)0.0140.0330.026

In this evaluation, both post-splat variants, including the full method, reduce albedo, metallic, and roughness drift relative to Direct-PBR. These results support the robustness of post-splat decoding to geometry errors. We will include this three-way comparison in the revised manuscript.

R1.Q1 — Cross-backbone regressions and evaluation protocols

We apologize for the misleading wording in the main text and will revise it, together with the inaccurate description in Appendix H.

SPFSplat training and evaluation. Except for FLARE, all adapters, including SPFSplat, are trained with 10–20 context views and three target views. Sparse-view evaluation uses two context views, with matched target frames and evaluation settings for Vanilla and Ours-PBR. Evaluating the SPFSplat adapter under sparse-view conditions after multi-view training may explain its reduced sparse-view performance. Due to time constraints, we will add a sparse-view training protocol for SPFSplat and the corresponding results to the final version of the paper.

C3G objective and regularization. We apologize for the incorrect description in Appendix H, which was included by mistake. We added regularization losses to the open-source C3G training code and, after carefully rechecking the implementation, confirmed that its training objective is aligned with those of the other methods. We will revise Appendix H and subsequently release the code, trained weights, and test and evaluation scripts. C3G's compact 3D representation [4] may limit fine-detail modeling and thus its PBR decomposition performance.

R1.Q2 — Training-data provenance and test-set separation

Our adapter follows the NoPoSplat protocol on RealEstate10K, using the predefined training and test splits distributed on Hugging Face. We retain this separation when training each adapter. The reconstruction-training datasets for the backbone variants used in our experiments are listed below. RealEstate10K refers to its training split.

BackboneTraining datasets
AnySplatHypersim, ARKitScenes, BlendedMVS, ScanNet++, CO3D-v2, Objaverse, Unreal4K, WildRGBD, DL3DV
Depth Anything 3 (DA3)AriaDigitalTwin, AriaSyntheticENV, ARKitScenes, BlendedMVS, CO3D-v2, DL3DV, Hypersim, MapFree, MegaDepth, MegaSynth, MVSSynth, Objaverse, OmniObject, OmniWorld, PointOdyssey, ReplicaVMAP, ScanNet++, SceneNetRGBD, TartanAir, Trellis, vKITTI2, WildRGBD
SPFSplatRealEstate10K
C3GRealEstate10K
FLAREMegaDepth, ARKitScenes, BlendedMVS, ScanNet++, CO3D-v2, Waymo, WildRGBD, DL3DV, RealEstate10K
NoPoSplatRealEstate10K
YoNoSplatRealEstate10K

The published protocols for RealEstate10K-trained backbones preserve the training/test split.

We can therefore confirm that our adapter training excludes the evaluation scenes, and the published backbone protocols support separation from the RealEstate10K test split.

R1.Q3 — Training cost, inference latency, and memory

We agree that the 1.5%–7.0% parameter increase measures parameter efficiency, while computational cost requires separate evaluation. We have now compiled adapter training costs and measured inference latency and peak memory against each base backbone.

Training cost. Table 1 reports the actual adapter training time recorded in our training logs.

Table 1. Original backbone training and additional adapter training.

BackboneOriginal training reported in its paperEstimated A100-equivalent GPU timePost training costs (h)
AnySplat15k steps; 16 × A800; ≈2 daysApprox. 768.0 hours30.5
DA3Giant stage: 200k steps; 128 × H100; ≈10 daysApprox. 76,800.0 hours45.0
SPFSplat1 × A100; batch size 12; duration not reportedApprox. 315.0 hours35.5
C3G450k steps; 8 × H100; duration not reportedApprox. 9,800.0 hours26.5
FLAREMain training: 200 epochs; 64 × A800; ≈14 daysApprox. 21,504.0 hours12.2
NoPoSplatTwo-view training: 2.4M image pairs; 8 × GH200; ≈6 hApprox. 120.0 hours27.0
YoNoSplat224 × 224 stage: 150k steps; 16 × GH200; duration not reportedApprox. 1,500.0 hours20.0

Inference latency and memory. We compare Base and Ours-PBR for each backbone under a unified single-scene evaluation protocol.

Table 2. Matched inference measurements: Base → Ours-PBR.

BackboneForward median (s)Rendering median (s/view)Sum (s)Peak allocated memory (GiB)
AnySplat0.086 → 0.1020.001 → 0.0270.087 → 0.1292.970 → 3.470
DA30.069 → 0.0880.002 → 0.0310.071 → 0.1196.580 → 7.080
SPFSplat0.041 → 0.0590.003 → 0.0300.044 → 0.0892.570 → 3.150
C3G0.063 → 0.0700.002 → 0.0290.065 → 0.0994.050 → 4.200
FLARE0.104 → 0.1120.002 → 0.0310.105 → 0.1435.590 → 6.120
NoPoSplat0.044 → 0.0630.002 → 0.0320.046 → 0.0952.550 → 3.110
YoNoSplat0.050 → 0.0720.003 → 0.0270.053 → 0.0993.770 → 4.370

The additional inference cost and latency mainly come from the post-trained modules and the deferred-shading computation used for PBR rendering.

We consider this overhead acceptable for the added material decomposition and relighting capabilities.

We will include these tables and measurement details in the revision and distinguish parameter efficiency from runtime and memory overhead.

R1.Q4 — Material recovery and cross-view consistency

We agree that recovering intrinsic materials and maintaining consistency across views are central objectives of inverse rendering.

We have added complementary quantitative evaluations of material recovery on TensoIR and cross-view material consistency on four datasets, using our AnySplat-based variant. These provide empirical evidence of recovery and consistency.

Material recovery and relighting. We quantitatively evaluate albedo recovery and relighting on TensoIR under a common rendering setup.

Table 1. TensoIR material recovery and common-renderer relighting.

MethodInference settingAlbedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Relighting PSNR ↑
MVInverseFF18.5620.8740.15218.420
MAIR++FF21.1520.8900.15821.029
GI-GSPer-scene optimization24.2510.8820.15026.972
GS-IRPer-scene optimization24.1810.8800.15526.549
Ours (AnySplat)FF24.9140.8870.15426.911

Our method achieves the highest albedo and relighting PSNR among the evaluated feed-forward methods and approaches the performance of per-scene optimization methods, demonstrating competitive material recovery and relighting without test-time optimization.

Cross-view material consistency. Following MVInverse's flow-reprojection evaluation approach, we assess consistency on four different datasets using RMSE between aligned material maps. Since MAIR++ does not predict metallicity, its metallicity error is reported as N/A.

Table 2. Cross-view material consistency: Albedo / Metallic / Roughness RMSE ↓.

DatasetMVInverseMAIR++GI-GSGS-IROurs (AnySplat)
RealEstate10K0.039 / 0.030 / 0.0150.050 / N/A / 0.1080.057 / 0.086 / 0.0460.047 / 0.074 / 0.0520.024 / 0.040 / 0.020
DL3DV0.045 / 0.030 / 0.0110.054 / N/A / 0.1020.058 / 0.081 / 0.0610.067 / 0.089 / 0.0690.035 / 0.065 / 0.028
Synthetic4Relight0.031 / 0.014 / 0.0050.051 / N/A / 0.0810.018 / 0.017 / 0.0300.020 / 0.014 / 0.0280.013 / 0.039 / 0.017
TensoIR0.041 / 0.027 / 0.0070.060 / N/A / 0.1230.021 / 0.022 / 0.0280.023 / 0.022 / 0.0260.018 / 0.054 / 0.022

Table 2 shows that, compared with the feed-forward methods MVInverse and MAIR++, our method achieves lower albedo reprojection RMSE on all four datasets and lower roughness RMSE than MAIR++, while MVInverse retains lower metallicity and roughness errors.

Compared with the unsupervised per-scene optimization methods GI-GS and GS-IR, our method achieves lower albedo and roughness RMSE on all four datasets.

Together with the TensoIR ground-truth evaluation, these results support competitive material recovery and cross-view consistency through feed-forward inference without target-scene optimization. We will include these results in the revised manuscript.

R1.Q5 — Aligning claims with the evaluation scope

In addition to providing the quantitative comparisons in our response to Question 4, we will revise the abstract, contribution statements, and related discussion to replace broad claims of superior material decomposition and relighting with conclusions matched to the evaluated settings.

References

[1] Wu et al. MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds. arXiv:2512.21003, §3.2 and Training Details.

[2] Jiang et al. AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views. arXiv:2505.23716, §4.1 and Appendix A.

[3] Chen et al. GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering. ICLR 2025, Appendices H and I.2.

[4] Honggyu An et al. C3G: Learning Compact 3D Representations with 2K Gaussians. arXiv:2512.04021, 2025 (v2, 2026), §1 and Appendix D.2. Paper.

Comments on the paper

1 comment

General Response to All Reviewers

We sincerely thank all three reviewers for their constructive feedback and for recognizing the practical value and applicability across backbones of our unsupervised post-training framework , as well as the post-splat material decoding design . Your comments have helped us strengthen the experimental evidence and clarify the contributions and scope of our method.

Our revision focuses on concerns shared by multiple reviewers, including quantitative validation of material recovery and relighting, the roles of post-splat decoding and scene conditioning, cross-view consistency and cross-dataset transfer, and computational overhead. Our main additions are as follows:

  1. Quantitative material recovery and relighting evaluation (Appendix J).

    We added evaluations on TensoIR and Synthetic4Relight, including comparisons with MVInverse, MAIR++, GI-GS, and GS-IR. Using available intrinsic ground truth and target-light images, these experiments provide direct evidence of material recovery and relighting beyond source-light RGB reconstruction.

  2. Controlled representation and geometry-perturbation comparisons (Sec. 5.4 and Appendix L).

    We compared aggregating explicit PBR parameters with aggregating learned features under matched 8-D dimensionality, decoder architecture, and parameter count. We also compared Direct-PBR, post-splat decoding without the scene latent, and the full method under matched geometry perturbations. These experiments further examine the role of the post-splat representation and its sensitivity to geometry errors.

  3. Cross-view material consistency evaluation (Appendix K).

    We added material-reprojection evaluations on RealEstate10K, DL3DV, Synthetic4Relight, and TensoIR, comparing albedo, metallicity, and roughness at corresponding surface locations across views. These measurements complement the ground-truth material evaluations by assessing cross-view consistency alongside recovery accuracy. Together, the results show that our method achieves competitive material recovery and cross-view consistency within an unsupervised, feed-forward framework.

  4. Cross-dataset transfer evaluation (Appendix M).

    We directly evaluated the RE10K post-trained adapters on selected DL3DV scenes without retraining or fine-tuning. Comparisons of native and adapted reconstruction across multiple backbones under dense and sparse input settings provide additional evidence of the adapters’ applicability beyond RE10K.

  5. Training cost, inference latency, and memory measurements (Appendix O).

    We added adapter training budgets, matched comparisons of inference latency and peak GPU memory between the base backbones and their adapted models, and runtime comparisons with other feed-forward inverse-rendering methods. These measurements distinguish parameter overhead from computational overhead.

We have also revised the abstract and contribution statements to better reflect the evaluated settings and the scope of the evidence. Revisions and additional results are highlighted in blue in the revised manuscript.

We welcome further discussion with the reviewers and would appreciate your feedback on whether these additions address the shared concerns. We would be happy to clarify any remaining questions or points that require further explanation. Thank you again for your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.