acceptodds
Under review as a conference paper at ICLR 2027

LUMEN-GS: Towards Light-Disentangled and Intrinsic-Aware Gaussian Splatting for Feed-forward Sparse-View Street Reconstruction

Abstract

Feed-forward reconstruction models have recently advanced spatial intelligence and simulation but mainly predict color radiance fields, treating scenes as non-interactive assets without exploring intrinsic properties and lighting representations for controllable editing. Bridging this gap requires a feed-forward paradigm that jointly represents intrinsic properties and outdoor illumination, instead of the hours of per-scene optimization required by iterative inverse rendering. Therefore, we present Lumen-GS, to our knowledge the first feed-forward reconstruction system that integrates a physically based rendering (PBR) pipeline for street scenes. In a single forward pass, it jointly predicts intrinsic primitives, sunlight and global environment radiance within a sunlight-integrated representation from sparse-view images at a single timestamp, supporting reconstruction, relighting, and harmonization. Lumen-GS aggregates multi-view features via a Hybrid Ring representation, with sunlight direction from GPS/IMU metadata. It further introduces a Physically-Aligned Decoder (PAD), whose intrinsic prediction heads are jointly supervised by teacher pseudo-labels and Physically-Based Rendering to improve disentanglement. Results on nuScenes show that Lumen-GS remains competitive with the strongest feed-forward baselines in novel-view synthesis and with iterative inverse-rendering methods in PBR quality, while supporting object insertion, harmonization, and data augmentation for downstream perception. These results demonstrate the potential of Lumen-GS for lighting-controllable street-scene simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed-forward Reconstruction, Relighting, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_DwYjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents LUMEN-GS, a feed-forward Gaussian Splatting framework for sparse-view street-scene reconstruction with intrinsic material decomposition and illumination modeling. Unlike existing feed-forward reconstruction methods that mainly focus on predicting geometry and appearance, LUMEN-GS aims to recover both Gaussian primitives and their physical attributes in a single forward pass, enabling relighting and scene editing without expensive per-scene optimization.

Critical strengths
  1. This paper explores an interesting and practically valuable direction by extending feed-forward Gaussian reconstruction to jointly predict scene geometry and intrinsic physical properties. Feed-Forward Gassuian prediction with physical attributes is very meaningful direaction.

  2. The proposed method achieves competitive rendering quality compared with existing sparse-view feed-forward reconstruction methods(DepthSplat, UniSplat, EvolSplat...). The paper also provides comparisons with optimization-based inverse rendering methods, such as UrbanIR and InvRGB+L, demonstrating the potential of feed-forward reconstruction for physically based scene modeling. Overall, the experimental is extensive and the resultas are good.

  3. The authors combine pixel-volume Gaussian reconstruction with PBR-based intrinsic learning. Appendix Table 9 shows that PBR-related supervision also benefits NVS performance. Although the approach is relatively straightforward and relies on teacher-generated pseudo-labels, pioneering feed-forward prediction of physically meaningful Gaussians for street scenes would be a valuable contribution if the novelty claim holds.

  4. Ablation studies and qualitative results are sufficient.

Critical weaknesses
  1. Training protocols and fairness of the NVS baseline comparisons: For the sparse-view feed-forward reconstruction baselines in Table 1, particularly MVSplat, DepthSplat, and EVolSplat, were these models retrained or fine-tuned on nuScenes, or were publicly available pretrained checkpoints directly used for evaluation?
  2. The quantitative PBR evaluation mainly measures reconstruction under the original lighting conditions. Despite promising qualitative results, it remains unclear whether the predicted intrinsic properties enable physically consistent relighting under novel illumination.
Questions for authors

No

1 comment
AuthorsOpenReview ↗

Response to Reviewer DwYj - 1

(Weakness-1) Training protocols and fairness of the NVS baseline comparisons: For the sparse-view feed-forward reconstruction baselines in Table 1, particularly MVSplat, DepthSplat, and EVolSplat, were these models retrained or fine-tuned on nuScenes, or were publicly available pretrained checkpoints directly used for evaluation?

We apologize for not clearly stating how the baselines were run. In Table 1, all baselines other than XYZCylinder (a baseline newly added in our revision) and UniSplat are fully trained, since most of them differ from us in training and evaluation protocols as well as in datasets. XYZCylinder has not released its code, and UniSplat has not released a checkpoint on the nuScenes dataset. We therefore adopt the results reported in the original papers, since their evaluation protocols match ours.

(Weakness-2) The quantitative PBR evaluation mainly measures reconstruction under the original lighting conditions. Despite promising qualitative results, it remains unclear whether the predicted intrinsic properties enable physically consistent relighting under novel illumination.

We agree that paired captures of the same scene and poses under different illumination would provide the most direct quantitative evaluation of relighting. Such paired data are not available in nuScenes or, to our knowledge, in existing public driving datasets. As a complement to our quantitative physical-consistency results, we measure the temporal stability of the predicted scene-level sunlight and ambient illumination within each nuScenes bin. We run inference on 2,842 timestamped six-view samples from 200 validation bins and compute the coefficient of variation (CV = std/mean) of each predicted illumination across the samples in each bin. The median intra-bin CV is 0.89% for sunlight and 0.54% for ambient light. Ambient-light CV is at most 1.99%, while sunlight CV is below 3.41% in 95% of bins. These results support the temporal stability of the predicted scene-level illumination under the assumption that illumination remains approximately constant within each short bin. They do not directly measure relighting fidelity against ground truth under a changed target illumination, which remains a limitation.

QuantityMetricMedianp95MaxBins within criterion
Sun intensityCV (%)0.893.4111.4498.0% (< 5%)
Ambient intensityCV (%)0.541.371.99100% (< 2%)
Reviewer_V371OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Lumen-GS, a feed-forward model that reconstructs a static street scene from six calibrated surround-view images at a single timestamp. It directly predicts 3D Gaussian geometry, material properties, visibility, and illumination, using a GPS/IMU-derived sun-direction prior in its rendering pipeline. The model combines a hybrid pixel-volume representation with a novel Physically-Aligned Decoder (PAD), whose latent interactions follow the structure of BRDF and light transport. Training combines teacher-generated pseudo-labels for intrinsics properties, RGB and PBR reconstruction losses, and geometry-derived visibility supervision. The resulting representation supports relighting and insertion of preprocessed assets. Experiments report competitive reconstruction, favorable insertion preferences, and improved vehicle detection after augmentation with Lumen-GS generated nighttime and adverse weather images.

Critical strengths
  • The paper addresses a practically important limitation of feed-forward street reconstruction: conventional radiance representations offer limited control over materials and illumination. Predicting an editable representation in a single feed-forward manner is a useful advance over per-scene inverse-rendering optimization.

  • The paper is well-organized and technically solid. The experiments examine several important design choices: PAD, teacher supervision, sunlight modeling, etc. Noteworthy experiments include gains on two downstream applications: objection insertion with harmonization and data augmentation to vehicle detection tasks.

  • Lumen-GS retains competitive reconstruction quality while additionally predicting intrinsic properties and illumination. Table 2 reports strong PBR reconstruction performance under the sparse-input setting, supporting the usefulness of the combined representation.

Critical weaknesses
  • My main concerns are mainly about the evaluation of Lumen-GS.

    • XYZCylinder is missing from Table 1, despite its close architectural relationship to Lumen-GS.

    • Quantitative reconstruction and intrinsic evaluation remain concentrated on nuScenes. A quantitative transfer experiment under a matched protocol would strengthen the generalization evidence. For instance, Carla-Centric (as evaluated by XYZCylinder) could additionally test more substantial viewpoint changes.

Questions for authors
  • Please address the issues raised in Weaknesses.

  • For the MVInverse + DepthAnything V3 baseline in Table 2, what motivates the use of Depth Anything V3 over the used MoGe-2 in Lumen-GS?

  • Do the PBR metrics in Table 2 evaluate reconstruction under the observed illumination or relighting under a different target illumination? What exactly are the reference images and lighting protocols?

5 comments
AuthorsOpenReview ↗

Response to Reviewer V371 - 1

We thank the reviewer for highlighting the importance of editable, relightable feed-forward reconstruction and for recognizing the strengths of our ablations, downstream experiments, and sparse-input PBR results. We address the remaining concerns about baseline coverage, cross-dataset evaluation, depth-estimator choice, and PBR evaluation protocol.

(Weakness-1 & Question-1) XYZCylinder is missing from Table 1, despite its close architectural relationship to Lumen-GS.

We thank the reviewer for pointing out this oversight on our part. We have added the quantitative results for XYZCylinder to Table 1 in the paper:

ModelsLight DisentangledPSNR↑LPIPS↓SSIM↑PCC↑D-RMSE↓Infer FPS↑Infer Mem.(GB)↓
XYZCylinder❌24.970.2310.7500.887–2.19.81
Lumen-GS (Ours)✅25.510.2210.7540.8885.411.799.15

Compared with XYZCylinder, Lumen-GS improves PSNR by 0.54 dB, reduces LPIPS by 0.010, and improves SSIM and PCC by 0.004 and 0.001, respectively. It uses 0.66 GB less inference memory, while running at 1.79 FPS compared with 2.10 FPS for XYZCylinder. D-RMSE is not reported for XYZCylinder.

(Weakness-2 & Question-1) Quantitative reconstruction and intrinsic evaluation remain concentrated on nuScenes. A quantitative transfer experiment under a matched protocol would strengthen the generalization evidence. For instance, Carla-Centric (as evaluated by XYZCylinder) could additionally test more substantial viewpoint changes.

We thank the reviewer for this suggestion. We agree that a quantitative transfer experiment under a matched protocol would strengthen the generalization evidence. Following XYZCylinder, we first built the Carla-Centric dataset, and subsequently added the main baseline comparison experiments, following the evaluation protocol of XYZCylinder. Table 2 has been revised accordingly.

ModelsPSNR↑LPIPS↓SSIM↑PCC↑
PixelSplat14.670.5650.4120.554
MVSplat15.320.5070.4570.566
DrivingRecon13.440.7400.4630.604
EVoiSplat14.340.5390.4610.754
DepthSplat16.270.5030.5080.581
DrivingForward15.380.5410.4450.696
STORM16.870.4980.5140.783
Omni-Scene15.540.5580.4620.551
XYZCylinder18.400.3590.6220.817
Lumen-GS (Ours)18.590.3620.6310.871

On Carla-Centric, Lumen-GS achieves the best PSNR, SSIM, and PCC among the listed methods, while XYZCylinder has slightly lower LPIPS (0.359 vs. 0.362).

For the quantitative transfer experiment, since the training and evaluation protocols of nuScenes and Carla-Centric differ substantially (in XYZCylinder, these are treated as two aspects of evaluating a reconstruction model), directly transferring a model trained on nuScenes to Carla-Centric leads to a noticeable performance drop, as shown in the table below. We therefore consider fully fine-tuning the nuScenes-trained weights on Carla-Centric and then evaluating the resulting model.

ModelsPSNR↑LPIPS↓SSIM↑PCC↑
Lumen-GS (zero-shot transfer from nuScenes)14.720.5120.4940.436
Lumen-GS (nuScenes pretrained, fine-tuned on Carla-Centric)18.660.3610.6370.870
Lumen-GS (trained from scratch on Carla-Centric)18.590.3620.6310.871

The results show that the model carrying nuScenes knowledge, after full fine-tuning, can outperform the LUMEN-GS trained from scratch. However, for a fair comparison, the main table still reports the from-scratch version.

AuthorsOpenReview ↗

Response to Reviewer V371 - 2

(Question-2) For the MVInverse + DepthAnything V3 baseline in Table 2, what motivates the use of Depth Anything V3 over the used MoGe-2 in Lumen-GS?

Depth estimatorPSNR↑SSIM↑LPIPS↓PBR-PSNR↑PBR-SSIM↑PBR-LPIPS↓D-RMSE↓CB-RMSE↓
Metric3Dv225.520.7520.22322.540.7130.2715.720.088
DA325.490.7530.22022.550.7150.2725.450.088
MoGe-325.500.7530.22222.540.7120.2695.360.087
MoGe-2 (Ours)25.510.7540.22122.550.7150.2695.410.088

We use DA3 in the MVInverse composition to construct a feed-forward 2D-to-3D reconstruction baseline. MVInverse predicts per-view intrinsic maps in 2D, while DA3 provides pixel-aligned geometry. We back-project DA3's per-pixel depth predictions through the camera intrinsics to obtain one 3D Gaussian per input pixel, without per-scene optimization (Wu et al., 2025; Lin et al., 2025b). This choice is motivated by DA3's role as a feed-forward geometry model, not by a claim that it is a better depth estimator than MoGe-2. Quantitatively, DA3 reports a 23.6% average gain in geometric accuracy over VGGT on its visual-geometry benchmark. On its feed-forward 3DGS/NVS benchmark, the DA3 variant achieves 21.33 dB PSNR on DL3DV, compared with 20.96 dB for VGGT (Wang et al., 2025). These results support DA3's suitability for the geometry component of our baseline, although they are not a direct comparison with MoGe-2 on our data. In our depth-estimator ablation, DA3 and MoGe-2 give comparable Lumen-GS results: both reach 22.55 dB PBR-PSNR and 0.088 C_B-RMSE, while MoGe-2 gives a slightly higher RGB PSNR (25.51 vs. 25.49 dB). We therefore use MoGe-2 within Lumen-GS and DA3 to instantiate the external feed-forward 2D-to-3D baseline.

Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth Anything 3: Recovering the visual space from any views. In International Conference on Learning Representations (ICLR), 2026.

Xiangzuo Wu, Chengwei Ren, Jun Zhou, Xiu Li, and Yuan Liu. MVInverse: Feed-forward multiview inverse rendering in seconds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5294–5306, 2025.

(Question-3) Do the PBR metrics in Table 2 evaluate reconstruction under the observed illumination or relighting under a different target illumination? What exactly are the reference images and lighting protocols?

The PBR metrics in Table 2 evaluate novel-view renderings under the observed illumination, not relighting under a different target illumination. For each test novel view, the reference is the original captured RGB image under the scene's observed illumination. The dataset does not provide paired reference images of the same scene under alternative illuminations, so quantitative evaluation under a changed target illumination is not possible. PBR evaluation follows the same test-view protocol as the RGB evaluation, with PBR-renderer outputs compared against the same reference images.

In the Dense setting, the iterative methods use all frames in the nuScenes scene bin for per-scene optimization. In the Sparse setting, UrbanIR, InvRGB+L, and MVInv.+DA3 use the same six views from a single timestamp as Lumen-GS. All methods are evaluated on the same test novel views. This separates a like-for-like sparse-input comparison from the dense-input results, which show how the iterative methods perform when given all frames in the scene bin. Using only sparse input, LUMEN-GS achieves results competitive with those of iterative methods trained densely over the entire bin.

Thank you for adding additional insights and clarifications.

Could the authors describe or hypothesize why zero-shot transfer to Carla-Centric results in poor results? Nevertheless, I think fine-tuning and training from scratch are perfectly acceptable.

Reviewer_Dh7sOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a feed-forward reconstructor of a relightable Gaussian representation for street scenes (typical e.g. in autonomous driving applications). Existing fast reconstruction models can render new camera views, but often bake lighting into color. Methods that separate materials from light usually fit each scene for hours using direct optimization, and need many more views. This paper reconstructs a plausible scene from a single omni-directional capture, combined with geolocation data to help estimate the lighting.

Critical strengths

Relightable feed-forward reconstruction is a highly valuable direction. This paper is a nice step towards such reconstruction for outdoor scenes, with many downstream applications.

The use of a decoder instead of classical re-rendering is a welcome improvement, instead of attempting to compute shadows and global illumination from approximate Gaussian geometry for every edit with all of its artifacts. Shadow visibility and a local illumination residual are predicted and stored on the Gaussians.

The results shown are quite convincing: albedo estimation, relighting, viewpoint change, and new weather conditions.

Critical weaknesses

Perhaps not critical, but I have the following criticisms.

First, why only street scenes? The method seems to be extensible to any outdoor scene reconstruction from a single viewpoint, for applications in architecture, agriculture, entertainment, etc.

Second, why only a single viewpoint as input? Two viewpoints would already allow a much longer street to be simulated. The authors claim an out of memory issue, but I'm a bit skeptical; this could be demonstrated at the cost of lower resolution, for example.

Finally, it would really help to show higher resolution result videos in the supplementary, to truly judge the result quality.

Questions for authors

Could the method work for outdoor scenes beyond typical streets? Could higher resolution videos be provided in the final version?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Dh7s - 1

We thank the reviewer for recognizing the value of relightable feed-forward reconstruction and for finding our albedo, relighting, viewpoint-change, and weather results convincing. We also appreciate the positive assessment of our decoder's per-Gaussian shadow-visibility and illumination-residual predictions. We address the remaining questions about scope, multi-frame input, and supplementary-video quality below.

(Weakness-1 & Question-1) Why only street scenes? The method seems extensible to outdoor scene reconstruction for applications in architecture, agriculture, entertainment, etc.

Our method takes six synchronized surround-view images captured at a single timestamp. The lighting representation itself is not tied to street scenes, but our feature-aggregation module is designed for this vehicle-mounted camera layout, and we evaluate the method on street scenes. We focus on driving as a representative application with synchronized surround-view data and GPS/IMU metadata available at scale, and as a key component of our closed-loop simulation pipeline. Extending the method to architecture, agriculture, or entertainment would require suitable capture configurations and may require a different feature-aggregation module. Hybrid Ring is designed for vehicle surround-view capture. BEVFormer- or TPVFormer-style aggregation is one possible direction for other camera layouts, but we leave this adaptation for future work.

No encoder feature or intrinsic-prediction head takes the sun direction as input, so intrinsic prediction does not require precise geographic coordinates at inference stage. This is consistent with our zero-shot qualitative examples in Fig. 27 (Appendix A.8), which use inputs without solar-direction annotations. However, training still uses the sun direction as an auxiliary prior, and relighting still requires a specified or estimated direction. Tab. 12 and Fig. 16 quantify the effect of noisy directions: across the tested noise levels of σ = 5° and 10°, the PBR-PSNR drop is at most 0.13 dB; it is 1.84 dB at σ = 25° and 1.22 dB with the estimated direction.

Our current focus on street scenes also reflects data availability. The training setup benefits from large-scale synchronized surround-view captures paired with GPS/IMU-derived sun directions. Other outdoor domains may require different feature aggregation and may not provide equally precise geolocation. For datasets such as Waymo, which removes precise geolocation for privacy but provides city-level metadata, manual lookup or LLM-assisted geocoding could provide approximate sun-prior labels. We estimate that city-level coordinates may introduce about 3% error in solar-position estimation. The noise results above suggest that small perturbations are tolerable, but we have not evaluated this annotation pipeline directly. Thus, approximate geolocation annotation is one possible route to extending training beyond datasets with exact GPS/IMU.

AuthorsOpenReview ↗

Response to Reviewer Dh7s - 2

(Weakness-2) Why only a single viewpoint as input? Two viewpoints would already allow a much longer street to be simulated. The authors claim an out-of-memory issue, but I'm a bit skeptical; this could be demonstrated at the cost of lower resolution, for example.

Our model already takes six synchronized camera views at one timestamp. We interpret the suggested second viewpoint as an additional timestamp from another vehicle pose, which would increase longitudinal scene coverage. We profiled a two-timestamp variant with each image resized to 112 × 200. Its peak memory was OOM, compared with 71 GB per GPU for the single-timestamp setting at 224 × 400. Thus, halving both image dimensions did not reduce peak memory in this setting. We attribute the increase to cross-frame attention and the larger scene representation, which introduces more volumetric Gaussians.

Under the 80-GB/GPU memory budget, the two-timestamp configuration supports a voxel split of at most 2, so we report split = 2 rather than the default split = 3. The table below reports this two-timestamp variant at 112 × 200 alongside the single-timestamp split = 2 reference from Tab. 13, evaluated at the native 224 × 400 resolution. The voxel split is matched, but this is still not a fully matched comparison because both timestamp count and per-image resolution differ. The metric differences therefore cannot be attributed to the additional timestamp alone; we report this as a feasibility and quality point for the tested two-timestamp configuration.

SettingTimestampsResolutionVoxel splitPBR-PSNR ↑PBR-SSIM ↑PBR-LPIPS ↓C_B-RMSE ↓
Single-timestamp reference1224 × 400221.050.6980.2920.102
Two-timestamp variant2112 × 200219.7250.66750.26330.111

The two-timestamp setting has lower PSNR and SSIM and higher C_B-RMSE, while LPIPS is lower. Because the memory limit prevents testing voxel splits above 2 for this configuration, these results do not establish the performance of a two-timestamp model at the default split = 3. Together with the memory profile above, they show that this lower-resolution multi-timestamp setting is feasible but - at the cost of fewer volumetric Gaussians.

Reducing the number of volumetric Gaussians is also not a free way to preserve visibility quality. In a separate ablation, using one Gaussian per voxel, one-third of our default split count, lowers the sampled-ray hit rate to 6.52%. Most sampled rays then miss the volumetric geometry, and PBR-PSNR falls by 2.32 dB compared with the default setting. Tab. 13 in paper reports the corresponding results:

SplitAvg. Gaussians (M)Ray hit ratePBR-PSNR ↑PBR-SSIM ↑PBR-LPIPS ↓C_B-RMSE ↓
00.464.27%18.820.6010.3470.149
11.026.52%20.230.6760.3110.134
21.7519.49%21.050.6980.2920.102
3 (default)2.3940.92%22.550.7150.2690.088

A complete multi-timestamp model would also require robust temporal alignment and handling of moving objects. We leave this extension to future work.

(Weakness-3 & Question-2) It would really help to show higher-resolution result videos in the supplementary, to truly judge the result quality.

The current model is trained at 224 × 400, and the supplementary clips use this native resolution. We agree that compression reduced their visual clarity. We have replaced the highly compressed videos in the “static/videos/nvs” folder with lower-compression versions and added low-loss versions in the “low-loss” folder. These updates reduce compression artifacts but do not increase the resolution of the model outputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.