acceptodds
Under review as a conference paper at ICLR 2027

InvSplat: Inverse Feed-Forward Scene Splatting

Abstract

Inverse rendering recovers geometry and material properties from images and is a prerequisite for relighting and material editing. Recent learning-based methods predict materials from images, but despite exploiting multi-view information, their image-space predictions can remain inconsistent across viewpoints. Meanwhile, per-scene optimization methods are computationally expensive and, in our experiments, achieve lower material accuracy than learned approaches despite using many views and can bake illumination into material maps. We therefore predict materials directly in 3D, as part of a feed-forward Gaussian splatting reconstruction. We observe that a direct combination of a 2D material estimator with a feed-forward reconstruction model is insufficient: the two networks must share features, and surface normals derived from the reconstructed geometry absorb texture, which degrades relighting. Based on these observations, we present InvSplat, which reconstructs 3D Gaussians with albedo, metallic, roughness and normals from posed images in 1.5 seconds. InvSplat predicts normals from appearance features and uses a point-light relighting loss that jointly constrains materials and normals without introducing errors from reconstructed geometry into the shading supervision. From two input views, InvSplat achieves higher relighting PSNR than the evaluated 2D baselines and produces more temporally consistent materials than methods processing all 32 frames of a video sequence, while maintaining comparable per-view material accuracy. Its explicit 3D representation further enables novel-view rendering of materials, relighting, and material editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3d scene reconstruction, inverse rendering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oa5xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InvSplat, a feed-forward 3D Gaussian Splatting framework that jointly reconstructs scene geometry and intrinsic material properties (albedo, roughness, and metallic), enabling applications such as relighting and material editing. InvSplat combines the geometric reconstruction capabilities and cross-view consistency of feed-forward 3DGS with the material priors of 2D intrinsic prediction models. The geometry and material branches exchange features during decoding to improve joint prediction. Experiments demonstrate improved multi-view material consistency and relighting quality.

Critical strengths
  1. Novel and well-motivated architecture. Combining geometry reconstruction and material estimation in a unified feed-forward framework is an interesting design. The dual-branch feature fusion is well motivated, and the ablation experiments demonstrate its effectiveness. The proposed Point-Light Relighting Loss is also a useful addition, with experimental evidence supporting its contribution.

  2. Strong results under sparse-view settings. With only two input views, InvSplat achieves impressive multi-view material consistency and relighting performance.

Critical weaknesses
  1. Confusing architecture illustration. Figure 2 does not clearly reflect the feature exchange described in Section 3.2: "The two branches share features during decoding: intrinsic features contribute to depth estimation, while the high-resolution image features of the geometry branch sharpen albedo prediction." However, the figure only illustrates the information flow from the intrinsic branch to the geometry branch, without showing the geometry-to-albedo connection. Moreover, the arrow from the intrinsic branch is ambiguous and may suggest that intrinsic features are fed into both the Depth Head and GS Head, rather than specifically contributing to depth prediction. The figure should be revised to accurately illustrate both feature-sharing paths.

  2. Limited evaluation of input view counts. The main quantitative evaluation focuses exclusively on the two-view setting. Although the supplementary material includes qualitative results with four input views, quantitative performance with different numbers of input views is missing. It remains unclear whether the model generalizes to more input views and how increasing the number of views affects reconstruction quality and multi-view material consistency.

  3. Incomplete dual-branch fusion ablation. Table 3 demonstrates the overall benefit of feature fusion, but does not isolate the individual contributions of Intrinsic-to-Depth and Geometry-to-Albedo feature sharing. Disabling these two connections separately would help clarify their respective effects on depth estimation and albedo prediction.

  4. Lack of quantitative normal prediction ablation. Appendix E provides only qualitative comparisons of different normal prediction strategies. It is difficult to determine the actual improvement without quantitative results. Moreover, Figure 10 suggests that the dedicated Normal Head produces smoother normals, but may also suppress some high-frequency geometric details, such as thin lamp cords and pillow folds. Quantitative comparisons of normal accuracy and relighting quality would help clarify this trade-off.

Questions for authors
  1. Relighting with estimated depth for 2D baselines. In Table 1, could the authors evaluate MVInverse with depth estimated by an off-the-shelf model, such as Depth Anything V2, and report the corresponding Relight-Dpred score? This would provide a more complete comparison under inference-time conditions.

  2. Geometry features for normal prediction. Have the authors considered feeding geometry-branch features into the Normal Head? Since depth and normal estimation are closely related and can mutually benefit each other, incorporating geometry features may further improve normal prediction, particularly for fine geometric details.

1 comment

We thank the reviewer for your thoughtful review and positive assessment. We have clarified the architecture, expanded the quantitative evaluation, and improved InvSplat following the suggestion to incorporate geometry features into normal prediction.

W1. Method figure improvement

We agree that figure should be more detailed. We have new version in a revision.

W2. Number of input views

InvSplat inherits ReSplat's variable-view architecture. As in ReSplat, performance generally benefits from matching training and inference view counts: adapting to more views requires retraining with those view counts, without architectural changes. We use two views because InteriorVerse's scattered cameras make larger overlapping sets difficult to sample. Our sampling supports larger sets and mixed-view training, the supplement qualitatively demonstrates four-view inference.

W3/Q2. Feature fusion and geometry-aware intrinsic prediction

Our original fusion shares complementary feature pathways from ReSplat and MVInverse:

  • Intrinsic → Depth: MVInverse's decoded multi-view ViT features, used for intrinsic prediction, replace ReSplat's monocular ViT features for depth estimation.
  • Geometry → Albedo: ReSplat's high-resolution CNN features replace MVInverse's high-resolution ResNeXt features for albedo prediction.

This removes two redundant backbones while preserving the original feature pathways, testing whether separate networks can effectively share representations. We did not disable the two feature flows separately, as each reuses features that were already beneficial in the corresponding original network.

Following Q2, we extend geometry-to-intrinsic fusion: ReSplat's Point Transformer aggregates features in 3D and provides high-resolution detail. We inject these features into the final decoding stage of all four intrinsic heads (albedo, metallic, roughness, normals). This improves material and normal accuracy:

ModelAlbedo PSNR ↑Albedo SSIM ↑Albedo LPIPS ↓Metallic RMSE ↓Roughness RMSE ↓Normal cosine ↑Depth AbsRel ↓
Original fusion22.370.8800.1930.09860.12560.96300.0453
+ 3D features22.460.8840.1860.09720.12580.96510.0445

Note: All variants in this and the following ablation were retrained for 40k steps (vs. 20k in the submission) with identical base losses, without intermediate geometry and relighting supervision (ablated separately).

W4/Q2. Quantitative normal prediction ablation

We agree that quantitative comparisons are needed to assess normal accuracy and its effect on relighting and provide the following table:

Normal predictionNormal Cos ↑Relight DGTD_{GT} ↑ (GT mat.)Relight DGTD_{GT} ↑ (Rend. mat.)
From depth0.814515.8915.67
Geometry GS head0.952221.7320.86
Normal GS head0.963022.8821.86
Normal GS head + 3D features0.965122.9421.93

This shows that high-resolution geometry-aware features complement the pretrained normal prior while retaining the advantages of a dedicated normal prediction head. We have added this ablation to the main paper, together with zoomed qualitative comparisons.

Q1. Relighting with estimated depth for 2D baselines

We have extended Table 1 to evaluate MVInverse with estimated depth from Pi3, the geometry model used in MVInverse's relighting pipeline, and Depth Anything 3.

MethodRelight DGTD_{GT} ↑Relight DpredD_{pred} ↑
MVInverse + Pi3 / DA321.9121.44 / 20.90
MVInverse* + Pi3 / DA321.4421.00 / 20.48
InvSplat (revised, full model)22.5021.88

InvSplat achieves the best relighting performance even when the baselines use strong external depth estimators. Notably, Pi3 produces more accurate depth than InvSplat, yet InvSplat outperforms it in relighting.

These results suggest that accurate depth and independently predicted materials are not sufficient for optimal relighting. InvSplat instead jointly learns geometry, materials, and normals in a shared 3D representation, with relighting supervision constraining their combined shading under novel illumination.

Reviewer_z8CvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper reconstructs a 3D Gaussian scene with albedo, metallic, roughness, and normals from posed images in a feed-forward pass. It combines pretrained ReSplat geometry and MVInverse material branches, exchanges features between them, predicts shading normals with a dedicated head, and adds point-light relighting supervision. Training uses synthetic InteriorVerse material and geometry labels. The resulting scene supports novel-view material rendering, relighting, and material editing. Evaluation shows comparable per-view material accuracy to strong image-space baselines, improved cross-view and temporal consistency, and gains in controlled relighting quality.

Critical strengths

The method provides a practical integration of sparse-view geometry and material estimation into an editable 3D representation, with a reported reconstruction time of 1.5 seconds. Its consistency results are useful: on 100 RealEstate10K sequences, it reconstructs from only the first and last frames and achieves lower temporal reprojection error than baselines processing all 32 frames. For example, albedo error is 0.020 versus 0.025 for DiffusionRenderer and 0.027 for fine-tuned MVInverse (Table 2). The relighting objective also improves PSNR by 0.65 dB with ground-truth depth and 0.51 dB with predicted depth (Table 4). These results support the utility of the integrated system.

Critical weaknesses
  1. The methodological advance over existing feed-forward components is limited. ReSplat already provides feed-forward Gaussian reconstruction, and MVInverse already predicts materials in a feed-forward manner. Moreover, MVInverse demonstrates a pipeline that combines its predictions with feed-forward geometry to construct a materialized point cloud for relighting and consistent editing [1, Section 4.5]. Thus, fast reconstruction and these downstream capabilities are not sufficient by themselves to establish a significant contribution. The distinctive additions here are the joint Gaussian prediction, feature exchange, normal-head design, and relighting objective. I find these useful engineering choices, but insufficiently substantial as a methodological advance.

  2. The demonstrated incremental benefits do not compensate for the limited methodological novelty. Table 3 does include a jointly fine-tuned MVInverse + ReSplat baseline, but the material improvements are small: albedo RMSE decreases from 0.0901 to 0.0883 and roughness RMSE from 0.1283 to 0.1254. Against fine-tuned MVInverse, per-view accuracy is mixed: albedo PSNR is 22.23 versus 22.92 dB on InteriorVerse and 19.57 versus 20.48 dB on Structured3D (Tables 1 and 6). Although the temporal gains are meaningful, Table 2 compares against image-space predictions, leaving unclear how much the proposed coupling adds beyond assigning those predictions to an existing 3D reconstruction. Real-data consistency and qualitative demonstrations also leave physical material and relighting accuracy on real scenes unquantified. My concern is the significance of the demonstrated advance, rather than an identified technical error.

Questions for authors
  1. Relative to the jointly fine-tuned MVInverse + ReSplat baseline in Table 3, how much does the proposed integration improve temporal consistency and relighting quality? A matched comparison using the existing sequence and relighting protocols would help distinguish benefits of the proposed coupling from benefits of using any shared 3D representation. Please specify which training losses are shared in this comparison.

  2. Is there quantitative evidence that the recovered materials improve relighting accuracy on real scenes, beyond temporal consistency? The real-data examples are useful, but the reported relighting metrics use a controlled shadow-free renderer. Existing paired real captures under changed lighting, if available, would help assess whether the practical benefit extends to physically accurate relighting. This concerns the scope of the demonstrated benefit; I am not claiming that the method cannot cast shadows.

1 comment

Thank you for your thoughtful review and for recognizing our efficient scene-level reconstruction, temporal consistency from two views and the benefit of relighting supervision.

W2/Q1. Contributions and model variants

Our approach reconstructs scenes in one pass as 3D Gaussians with explicit materials and shading normals, enabling novel-view material rendering, relighting, and editing. Rendering and relighting supervision yield 1.3–1.8 dB higher relighting PSNR than attaching MVInverse* predictions to Pi3 depth or RGB Gaussians on our Infinigen benchmark with cast shadows (Q2). We compare two variants sharing representation and training: jointly trained MVInverse* + ReSplat* (Dual-branch), and InvSplat, which fuses them via feature sharing. Following R3 Q2, we inject Point Transformer features to intrinsic heads (+3D features). All variants use the same base losses (L1+LPIPS for materials, normal cosine, rendered depth), without intermediate geometry or relighting losses, ablated separately to isolate architectural effects. All models are retrained for 40k steps to reduce the gap with 2d baselines; the revised ablation includes consistency and relighting metrics:

ModelFlow RMSE Albedo/Metallic/Roughness ↓Albedo PSNR/SSIM/LPIPSMetallic RMSE ↓Roughness RMSE ↓Normal cos. ↑Relight DGTD_{GT} /DpredD_{pred} ↑
Dual-branch (MVInverse* + ReSplat*)0.0654/0.0554/0.048022.50/0.880/0.1930.09790.12600.962921.93/21.29
InvSplat, original fusion0.0638/0.0550/0.047922.37/0.880/0.1930.09860.12560.963021.86/21.23
InvSplat, original fusion + 3D features0.0637/0.0544/0.047022.46/0.884/0.1860.09720.12580.965121.93/21.29

InvSplat + 3D features matches Dual-branch relighting, with sharper albedo (LPIPS 0.186 vs. 0.193), better normals (0.9651 vs. 0.9629), and slightly better cross-view consistency, using 166M fewer parameters (14%) and 1.3× faster inference. On RealEstate10K, all three models achieve similar temporal consistency (flow RMSE within 0.001). We revised the accuracy–efficiency discussion in the paper.

We reassess intermediate geometry and relighting supervision:

Training objectivesRelight DGTD_{GT}/DpredD_{pred} ↑Depth AbsRel ↓
Base losses21.93/21.290.0445
Base + intermediate geometry losses21.89/21.350.0434
Base + intermediate geometry + relighting (full model)22.50/21.880.0436

The relighting loss adds 0.61/0.52 dB and carries over to full relighting with shadows (19.99→20.22 dB on Infinigen).

W1. Value beyond existing feed-forward predictors

We follow MVInverse's own relighting pipeline (Sec. 4.5) with Pi3 or Depth Anything 3 depth and added results to Table 1:

MethodRelight DGTD_{GT} ↑Relight DpredD_{pred} ↑
MVInverse + Pi3/DA321.9121.44/20.90
MVInverse* + Pi3/DA321.4421.00/20.48
InvSplat, submitted22.3621.63
InvSplat, revised22.5021.88

InvSplat relights best, already in the submitted version. Accurate geometry and accurate per-view materials are not sufficient on their own: relighting requires materials, normals and geometry that are consistent with each other, which InvSplat obtains by training them jointly in 3D with a relighting loss that shades them together. The same combinations are evaluated with full relighting and cast shadows below.

Q2. Full relighting with cast shadows

Real captures lack GT materials and most models lack RGB forward renderers. We therefore construct a benchmark of 29 Infinigen indoor scenes with five point-light configurations each on input and held-out views using predicted geometry, materials, normals, and ray-traced shadows.

We compare our model with two naive feed-forward pipelines:

  • Geometry model: MVInverse* on Pi3 depth, with shadows marched through the depth map
  • RGB reconstruction model: MVInverse* materials assigned to ReSplat-RGB* Gaussians, fine-tuned on InteriorVerse with RGB L1/LPIPS and rendered-depth losses, with shadows traced as for ours
MethodRelit PSNR ↑Visibility L1 ↓Visibility SSIM ↑
MVInverse* + Pi3, depth march18.950.0600.806
MVInverse* + ReSplat-RGB*18.390.0960.743
InvSplat20.220.0550.820
  • Depth maps limit shadows by design: each map is per-view, so occluders seen only in the other view cast no shadow, and the discrete samples make shadows sensitive to step size and thresholds.
  • RGB-trained Gaussians are not suited for shadow tracing: they are more opaque and thicker.

In summary, predicting materials as Gaussian attributes outperforms combinations of existing predictors (W1, Q2), fusion with 3D features matches the Dual-branch variant at lower cost, and relighting supervision provides the largest relighting gain (W2/Q1).

Reviewer_3C5wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents InvSplat, a feed-forward method that reconstructs a 3D Gaussian scene with geometry, materials, and surface normals from as few as two posed images. It combines the geometry branch of ReSplat with the material estimation branch of MVInverse and shares features between them. A separate head predicts normals, and a point-light relighting loss jointly supervises materials and normals. The resulting scene supports novel-view rendering, relighting, and material editing. Experiments show improved cross-view and temporal consistency over the tested 2D methods, with competitive material accuracy and a reconstruction time of about 1.5 seconds.

Critical strengths
  • The method reconstructs an editable indoor scene from just two views in about 1.5 seconds, making it useful for scene editing and relighting.

  • The motivation is clear. Sharing features between geometry and material estimation, using a separate normal prediction head, and adding relighting supervision are all sensible choices.

  • The results show better consistency than the tested 2D methods, while keeping material estimation accuracy close to the strongest baseline.

Critical weaknesses
  • The method combines existing components, including ReSplat for geometry, MVInverse for materials, and a standard physically based shading model. The main changes are feature sharing, normal prediction, and training losses, but their gains over strong baselines seem limited. Feature sharing only slightly improves over jointly fine-tuning the two networks, the normal-head comparison is mainly qualitative, and using five training lights instead of one improves Relight-DGT from only 22.34 to 22.36 dB.

  • The paper would be more convincing as a practical scene-level system with stronger quantitative tests of novel-view materials and full relighting. Most material metrics are measured at the input views, while the main relighting metric uses ground-truth depth and excludes cast shadows. The camera-centered light test also provides limited evidence of shadow quality under other light positions.

  • Some closely related references are missing such as "F-RNG: Feed-Forward Relightable Neural Gaussians". It also combines reconstruction and intrinsic decomposition priors to produce relightable Gaussians from sparse views. Although it focuses on objects rather than indoor scenes, it is better to elaborate the diffrences between these two works

Questions for authors

see the weaknesses

1 comment

Thank you for your thoughtful review and for recognizing our efficient scene-level reconstruction, design choices, and improved consistency over 2D methods. We strengthened architectural ablations and added quantitative normal and full relighting evaluations with shadows.

W1. Contributions and model variants

Our approach reconstructs scenes in one pass as 3D Gaussians with explicit materials and shading normals, enabling novel-view material rendering, relighting, and editing. Rendering and relighting supervision yield 1.3–1.8 dB higher relighting PSNR than attaching MVInverse* predictions to Pi3 depth or RGB Gaussians on our Infinigen benchmark with cast shadows (R2 Q2). We compare two variants with the same representation and training: jointly trained MVInverse* + ReSplat* (Dual-branch), and InvSplat, which fuses them through feature sharing. Following R3 Q2, we inject Point Transformer features to intrinsic heads (+3D features). All models are retrained for 40k steps; the revised ablation includes consistency and relighting metrics:

ModelFlow RMSE Albedo/Metallic/Roughness ↓Albedo PSNR/SSIM/LPIPSMetallic RMSE ↓Roughness RMSE ↓Normal cos. ↑Relight DGTD_{GT} /DpredD_{pred} ↑
Dual-branch (MVInverse* + ReSplat*)0.0654/0.0554/0.048022.50/0.880/0.1930.09790.12600.962921.93/21.29
InvSplat, original fusion0.0638/0.0550/0.047922.37/0.880/0.1930.09860.12560.963021.86/21.23
InvSplat, original fusion + 3D features0.0637/0.0544/0.047022.46/0.884/0.1860.09720.12580.965121.93/21.29

InvSplat + 3D features matches the Dual-branch variant in relighting and yields sharper albedo (LPIPS 0.186 vs. 0.193) and more accurate normals (0.9651 vs. 0.9629), with slightly better cross-view consistency, with 166M fewer parameters (14%) and 1.3× faster inference by removing duplicated backbones. We revised the paper to reflect accuracy–efficiency trade-off.

With the revised architecture fixed, we reassess intermediate geometry and relighting supervision:

Training objectivesRelight DGTD_{GT}/DpredD_{pred} ↑Depth AbsRel ↓
Base losses21.93/21.290.0445
Base + intermediate geometry losses21.89/21.350.0434
Base + intermediate geometry + relighting (full model)22.50/21.880.0436

Five training lights instead of one add little, and we removed this claim, but relighting loss usage is beneficial and helps to outperform strongest 2d baselines (MVInverse 21.91/21.44dB).

W1. Quantitative normal-head ablation

We compare depth-derived, GS head, and dedicated normal-head predictions, with and without 3D feature injection.

Normal predictionNormal Cos ↑Relight DGTD_{GT} ↑ (GT mat.)Relight DGTD_{GT} ↑ (Rend. mat.)
From depth0.814515.8915.67
Geometry GS head0.952221.7320.86
Normal GS head0.963022.8821.86
Normal GS head + 3D features0.965122.9421.93

Geometry-aware features complement the pretrained normal prior and dedicated normal head. We added this ablation to the main paper, together with zoomed qualitative comparisons.

W2. Full relighting and shadow evaluation

We agree that the camera-centered visibility test provides limited evidence of shadow quality. We therefore construct a benchmark of 29 Infinigen indoor scenes with five point-light configurations each, evaluating input and held-out views using predicted geometry, materials, normals, and ray-traced shadows. GT images are rendered in Cycles with direct illumination. This benchmark replaces the camera-centered test in the revised paper (Tab. 2, Fig. 5), and its relit PSNR and visibility are part of the ablation (Tab. 4).

LossesRelit PSNR ↑ (input/novel)Visibility L1 ↓ (input/novel)Visibility SSIM ↑ (input/novel)
Base19.93/19.590.054/0.0590.811/0.799
+ Intermediate geometry19.99/19.670.052/0.0540.826/0.819
+ Intermediate Geometry + Relighting20.22/19.870.055/0.0580.820/0.811

This directly evaluates full relighting under novel lights, cast shadows and held-out views.

W3. Relation to F-RNG

We thank the reviewer for highlighting F-RNG and discussed it in the revision. Both combine reconstruction and intrinsic priors, but differ in representation, training, and setting. F-RNG learns latent reflectance decoded by a neural renderer, whereas InvSplat predicts explicit albedo, metallic, roughness, and normals per Gaussian for PBR and direct editing. F-RNG freezes its pretrained large models and uses intrinsic maps for reflectance distillation, InvSplat jointly fine-tunes coupled geometry and intrinsic branches with material, geometry, and relighting supervision. Finally, F-RNG focuses on objects, while InvSplat targets indoor scenes and cast shadows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.