- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SpectralSplat studies how to adjust the weather and lighting appearance of driving scenes after reconstructing them with feed-forward 3D Gaussian Splatting. Built on UniSplat, the method introduces a global appearance code into color prediction: a zero code produces base colors, while the actual code produces colors under the corresponding appearance. To obtain training pairs of the same scene under different conditions, the authors combine physical rendering priors with diffusion relighting. Reconstruction, appearance-swap, base-color consistency, and other losses encourage the separation of scene content and appearance. At inference, a prebuilt appearance library allows different appearances to be applied while keeping the reconstructed geometry fixed. The method also caches features instead of fixed colors, allowing accumulated Gaussians to be recolored using the current code. Experiments evaluate reconstruction on Waymo and nuScenes and demonstrate appearance swapping, library-based control, and code interpolation on Waymo. The method primarily adjusts overall scene appearance rather than providing a complete physical simulation of weather.
Critical strengths
-
The problem has practical value, and its integration with feed-forward reconstruction is natural. For driving scenes, being able to switch appearances after reconstruction without optimizing each scene again is a useful capability. Beyond demonstrating relighting for an individual scene, this paper focuses on incorporating this capability into a reusable feed-forward reconstruction system. I think this is a valuable direction.
-
The design of the historical feature cache makes sense. If the cache stores fixed RGB values, changing the appearance code may create color inconsistencies between historical Gaussians and the current frame. Caching features and then predicting colors consistently using the selected code is an intuitive solution that fits the needs of temporal reconstruction.
Critical weaknesses
-
The contribution beyond existing appearance-conditioned GS methods needs further clarification. Using appearance embeddings to adjust Gaussian colors and using MLPs to predict conditioned colors have appeared in prior work. What stands out here is the integration of these designs with feed-forward driving-scene reconstruction, an appearance library, and temporal caching. However, I am not yet fully convinced that the proposed code extraction and library construction are necessary compared with simpler alternatives. For example, with the same synthetic data and training budget, could directly learning a condition embedding for each weather condition or time of day achieve similar results? The paper also acknowledges that an advantage over this alternative has not been established. I think this would be a useful baseline to help identify which parts of the design account for the gains.
-
The central capability of reusing the appearance library across scenes still lacks sufficiently direct quantitative validation. The appendix states that the current library is constructed from the augmented Waymo validation split. The existing visualizations therefore do not establish whether appearance can still transfer reliably when the library construction scenes and test scenes are completely disjoint. In addition, Table 3 uses appearance codes encoded from each image pair rather than the averaged codes in the library, so it does not directly evaluate the library as used in practice. Given that the authors also show unreliable transfer between real images, I think the current evidence more strongly supports control within the synthetic appearance distribution. Broader conclusions about content–appearance disentanglement should be made cautiously.
Questions for authors
-
Regarding Weakness 1: Could you add a baseline that directly learns a condition embedding for each weather condition or time of day, using the same backbone, synthetic data, and training budget? I would like to see what clear benefits extracting codes from images and averaging them into a library provides over this simpler alternative, to better establish the contribution of this design.
-
Regarding Weakness 2: Could you clarify whether the library construction scenes overlap with the test scenes, and quantitatively evaluate the averaged library codes when the two sets are completely disjoint? Beyond the appearance swapping in Table 3 using codes from the paired images themselves, I would like to see whether fixed library codes can also reliably control appearance in new scenes.
3 comments
W2/Q2 (library overlap, centroids).
"Could you clarify whether the library construction scenes overlap with the test scenes, and quantitatively evaluate the averaged library codes when the two sets are completely disjoint?"
Yes, the submitted library overlapped with the test scenes. Following your suggestion:
- we rebuilt the library from 48 training-split scenes only;
- we evaluated its fixed centroids on all 202 validation scenes (2,020 frames), with relit targets balanced over the nine conditions.
CIs are 95%, from a bootstrap over scenes.
Table R1b (main rows; the full table is in our reply to Reviewer 49n4):
| Code used at render time | PSNR↑ | SSIM↑ | LPIPS↓ |
|---|---|---|---|
| Per-image code from the target (upper bound) | 22.59 | 0.724 | 0.450 |
| Library as submitted (24 validation scenes) | 20.65 | 0.695 | 0.501 |
| Library from 48 training scenes (disjoint) | 20.70 | 0.698 | 0.500 |
| Library from the other half of validation (disjoint, 2-fold) | 20.70 | 0.697 | 0.498 |
| Mean of all codes (no condition information) | 19.80 | 0.683 | 0.504 |
| Centroid of a wrong condition | 19.07 | 0.669 | 0.508 |
- The library generalizes to unseen scenes: disjoint libraries score the same as the submitted one.
- The centroids carry condition information: +0.90 dB [0.82, 0.99] over the mean code and +1.64 dB [1.51, 1.77] over a wrong condition. The gain is largest at night (+1.8 to +2.5 dB).
- They control the appearance of new scenes:
- A probe recognizes the intended condition from renders of unseen scenes as often as for per-image codes (9-way: 25.7% vs. 21.3%; chance 11%; Table R1a, in our reply to Reviewer 49n4).
- Mean luminance tracks the target conditions with r = 0.97.
Control within the synthetic distribution. We agree, and we will phrase the claim this way. To quantify how far the edits reach real appearance, we classified library renders of 325 unseen daytime frames. The classifier was trained only on real Waymo photos and reaches 97% accuracy on held-out real scenes. Table R6:
| Library code | Real-photo classifier | No edit | CLIP zero-shot | No edit |
|---|---|---|---|---|
| rain (day) | 57.8% | 0.6% | 69.5% | 4.0% |
| rain (golden hour) | 72.0% | 0.6% | 60.3% | 4.0% |
| clear night | 0.6% | 0.0% | 56.0% | 13.2% |
| rainy night | 0.9% | 0.0% | 56.6% | 13.2% |
- Rain: codes make unseen scenes read as real rain, for a classifier that never saw a synthetic image.
- Night: CLIP reads 56% of the edited scenes as night, against 13% without an edit. The real-photo classifier rarely does, because real Waymo night scenes are dominated by street lamps and headlights (local light sources).
- The model reproduces its training distribution faithfully.
- Our renders are 2–3× closer to the synthetic targets than unedited renders are.
- Their remaining distance to real photos (KID×10³ 0.61–0.69) matches that of the generator's own images (0.46–0.67), so a stronger generator carries over directly.
We hope these results address your concerns. If they do, we would be grateful if you would consider updating your score. We are happy to answer any further questions.
Thank you for the additional experiments and clarifications. Your rebuttal addresses some of my concerns. After considering your response, I have decided to maintain my original score.
We thank the reviewer for the thoughtful review. We are glad the reviewer sees reusable feed-forward appearance control as a valuable direction and found the feature-cache design sensible.
We ran both experiments you asked for. In short:
W1 (contribution over appearance-conditioned GS).
We agree that appearance embeddings and conditioned color MLPs are established.
Q1 (learned per-condition table).
On seen conditions, yes, about the same. On a new condition, no: it is 2.4 dB worse. Thank you for suggesting this baseline.
clear_nightremoved from fine-tuning. The starting model has seen night-like lighting, so this tests adding a condition without retraining.Table R4:
Optimized table, frozen decoder. We also optimized nine codes directly for reconstruction through the frozen submitted decoder. They match the library centroids in PSNR (−0.06 dB [−0.13, 0.01]) and improve LPIPS by 0.030.