acceptodds
Withdrawn from ICLR 2027

PAM-ToD: Plug-and-Play Appearance Modeling for Cross-Time-of-Day 3D Gaussian Splatting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting a pre-trained 3D Gaussian Splatting (3DGS) road scene to a new time of day requires learning appearance changes from a few anchor images while preserving consistent, real-time rendering. We propose PAM-ToD, a lightweight plug-in that learns color corrections while keeping the pre-trained 3DGS parameters fixed. PAM-ToD scales each Gaussian's existing color to model illumination changes and uses an additive term for additional brightness, such as when street lamps turn on at night. Under a simplified image formation model, unchanged surface albedo can be eliminated from the relation between source and target appearances, allowing us to learn these corrections without separately estimating albedo and illumination. The model corrects colors across the scene while allowing the corrections to vary by location and by Gaussian. To guide learning from a few anchor images, it discourages abrupt spatial changes in these corrections. We also introduce CARLA-ToD, a benchmark with matching geometry, camera poses, and moving-object trajectories across three times of day. A few target-time anchor images are used to train each plug-in, while separate views are used for evaluation. Across the static and dynamic settings, PAM-ToD achieves higher PSNR and lower LPIPS than the baselines, even when the anchor images come from a single synchronized capture across multiple cameras.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Autonomous Driving, Relight

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DfzEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAM-ToD formulates cross-time scene transfer as adapting a frozen road-scene 3D Gaussian Splatting model to a target time of day using a small set of posed target-time anchor images. This paper applies a multiplicative log-gain and nonnegative additive emission correction to Gaussian colors, using a global term, multiscale spatial grids, and per-Gaussian residuals with several regularizers. The paper also introduces CARLA-ToD, a controlled static and dynamic benchmark with matched trajectories and camera poses across morning, noon, and night.

Critical strengths
  • Freezing geometry and base colors while attaching a relatively small appearance module is a sensible design for repeated simulation use.
  • CARLA-ToD makes per-pixel cross-time evaluation much more meaningful than evaluation over independently captured drives with changing objects and poses.
  • Visualization shows PAM-ToD better retains broad road and façade structure under large morning/noon/night shifts than the displayed competing methods.
Critical weaknesses
  • The appearance transport prior matches source-Gaussian and target-anchor RGB distributions by within-class luminance quantiles. It does not use correspondence, and therefore assumes that rank order within a semantic class is preserved across potentially different spatial regions, materials, occlusion patterns, and shadow configurations.
  • The comparison protocol between different models is not consistent. See Question 1
Questions for authors
  • PAM-ToD uses sparse-anchor adaptation setting. While GS-W is trained with eight frames from different times of day, inverse-rendering methods are trained separately for each time and evaluated through environment-map swapping and the diffusion methods are applied to StreetGS renderings. Would each diffusion baseline receive target-time image, anchor, reference, or conditioning information? The evaluation would be more convincing with at least one strong frozen-3DGS adaptation baseline trained on exactly the same target anchors, such as a global affine RGB transform, a global-plus-grid color field without semantic priors, and a regularized per-Gaussian correction baseline.
  • The real-world validation is limited. Could the authors to evaluate cross-time appearance transfer using real captures under different illumination conditions, for example using repeated traversals from Oxford RobotCar [1].

[1] https://robotcar-dataset.robots.ox.ac.uk/datasets/

3 comments
AuthorsOpenReview ↗

Authors rebuttal

[Part 1/3]

We thank the reviewer for recognizing our frozen-geometry design and the CARLA-ToD benchmark, as well as for the constructive suggestions. We address each point below.

W1. The appearance transport prior assumes that luminance ranks are preserved within a semantic class

We agree that when materials, shadows, or occlusions differ within a semantic class, matching luminance quantiles does not always yield the exact appearance change.

However, the appearance transport prior does not strictly assume the preservation of the luminance rank of each individual Gaussian. Rather, it provides a statistical prior on appearance changes, derived from per-class luminance distributions.

The final appearance is determined by optimization against the target anchor images. In regions observed by the anchors, this optimization explicitly corrects the mismatch between the prior and the actual illumination change.

Our ablation study supports this. In Table 4, removing the appearance transport prior lowers PSNR from 24.72 to 24.54 dB, drops SSIM from 0.808 to 0.803, and worsens LPIPS from 0.285 to 0.289. Although the scenes contain regions where the rank assumption does not hold, the prior does not degrade accuracy and serves as an effective overall prior.

In unobserved regions, however, errors from the prior may remain. We have stated this as a limitation in the Conclusion of the revised PDF.

[Part 2/3]

W2 / Q1. Fairness of the baseline comparison: do the diffusion baselines receive target-time information?

We agree that the paper did not state the inputs to each baseline clearly enough. The full setup is as follows:

MethodTarget-time dataBase 3DGS
GS-IR / GI-GS / LumiGaussFull sequenceOwn reconstruction
GS-W{Ak}\{A_k\} (K=8K{=}8)Own reconstruction
LumiNet / DiffusionRenderer / UniRelight{Ak}\{A_k\} (K=8K{=}8)StreetGS (frozen)
PAM-ToD{Ak}\{A_k\} (K∈{1,8}K\in\{1,8\})StreetGS (frozen)

Every {Ak}\{A_k\} in the table represents exactly the same set of anchor images used by PAM-ToD. The diffusion baselines are conditioned on these anchor images with K=8K{=}8. LumiNet uses them directly as reference images. DiffusionRenderer and UniRelight take environment lighting as input, so they are fed an equirectangular HDR map built from these images. All three methods are applied to the same StreetGS renderings as PAM-ToD. GS-W is trained on the source sequence, with the frames at the anchor timesteps replaced by {Ak}\{A_k\}. The inverse rendering baselines train a separate model on the full sequence for each time of day and swap the environment maps at test time; thus, they utilize significantly more target-time information than PAM-ToD. Under the Novel View Synthesis protocol, the held-out views are strictly unseen by all methods.

We fully agree that a frozen-3DGS adaptation baseline trained with exactly the same anchors is the most informative comparison. Following your suggestion, we have added four simple baselines:

  1. A global affine RGB transform shared by all Gaussians (3×33\times3 matrix and bias).
  2. A per-Gaussian affine transform.
  3. A per-Gaussian affine transform with regularized
  4. A global gain + spatial field correction without semantic priors.

All baselines utilize the same frozen StreetGS and exactly the same anchor images {Ak}\{A_k\} as PAM-ToD, for both K=1K{=}1 and K=8K{=}8.

Table R1. Appearance corrections trained on the same frozen StreetGS with the same anchors (same setting as Tables 3–5)

MethodSSIM ↑ (K=1K{=}1)PSNR ↑ (K=1K{=}1)LPIPS ↓ (K=1K{=}1)SSIM ↑ (K=8K{=}8)PSNR ↑ (K=8K{=}8)LPIPS ↓ (K=8K{=}8)
Base (no adaptation)0.47110.170.4870.47110.170.487
Global affine0.69118.510.3560.69719.090.350
Per-Gaussian affine0.59514.490.4150.76923.160.341
Regularized per-Gaussian affine0.60514.910.4110.77323.610.341
Global gain + spatial field (w/o priors)0.64717.800.4040.76722.060.341
PAM-ToD0.72021.110.3270.80824.720.285

PAM-ToD outperforms all baselines across every metric at both K=1K{=}1 and K=8K{=}8.

  • Global affine cannot represent spatially varying illumination, meaning more anchors barely improve performance (18.51 → 19.09 dB).
  • Per-Gaussian affine reaches 23.16 dB at K=8K{=}8 but drops sharply to 14.49 dB at K=1K{=}1, falling below the performance of the global transform. With few anchors, its numerous free parameters overfit the anchor images, and Gaussians outside the anchor views are never updated. Adding regularized a marginal improvement of about 0.4 dB (14.49 → 14.91 dB at K=1K{=}1).
  • Global gain + spatial field (w/o priors) reaches 22.06 dB at K=8K{=}8 but only 17.80 dB at K=1K{=}1. Even with the same correction structure, corrections for unobserved regions cannot be stably estimated from a few anchors without the semantic priors.

By contrast, PAM-ToD achieves an impressive 21.11 dB even at K=1K{=}1.

[Part 3/3]

Q2. Validation on real-world data

Thank you for this excellent suggestion. We agree that validation with real target-time images was the most significant gap in our original experiments. We therefore ran a new real-world experiment, and the results are very encouraging: PAM-ToD outperforms all baselines by a large margin. We believe this experiment makes the paper substantially stronger.

The Oxford RobotCar dataset was designed for visual odometry, which may introduce localization errors into novel view evaluation. Therefore, we used nuPlan multi-traversal data, where the exact same route is driven repeatedly under varying lighting conditions. We reconstruct StreetGS from a source traversal, adapt it using anchor images {Ak}\{A_k\} taken from a target traversal, and evaluate on held-out views from the target traversal across four scenes. Because dynamic elements like parked vehicles and pedestrians differ between traversals, metrics are computed strictly on static regions observed in both.

Table R2. Quantitative results on nuPlan (Novel View Traversal)

MethodSSIM ↑ (K=1K{=}1)PSNR ↑ (K=1K{=}1)LPIPS ↓ (K=1K{=}1)SSIM ↑ (K=8K{=}8)PSNR ↑ (K=8K{=}8)LPIPS ↓ (K=8K{=}8)
Base (StreetGS, no adaptation)0.54915.210.2790.54915.210.279
MTGS0.57915.790.3590.57615.920.353
WildGaussians0.51314.280.4010.51914.490.401
GS-W0.50313.730.3830.50913.970.381
PAM-ToD0.60517.810.2740.75922.420.195

PAM-ToD proves highly effective on real target-time images as well. At K=8K{=}8, it improves PSNR by 7.2 dB over the base StreetGS (15.21 → 22.42 dB) and outperforms the best baseline, MTGS, by a margin of 6.5 dB. Even at K=1K{=}1, where anchors are sourced from a single timestep, it consistently outperforms all baselines across all metrics while maintaining real-time rendering above 70 FPS.

We have added these real-world results to the main paper (Table 6 and Figure 4 of the revised PDF). We have also clarified in the revised PDF that the Waymo experiment, which uses synthesized anchors, validates the adaptation of scenes reconstructed from real data rather than accuracy against real target-time images.

Reviewer_ME1kOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAM-ToD, a plug-in method for adapting a pretrained 3DGS scene to different times of day using a few target-time anchor images. It keeps the original 3DGS fixed and learns appearance corrections through a multiplicative illumination term and an additive emission term, together with hierarchical regularization. The paper also introduces CARLA-ToD dataset, with matched geometry and camera poses across different times of day. Experiments on CARLA-ToD and the Waymo Open Dataset show improvements in rendering quality while maintaining real-time performance.

Critical strengths

S1. The paper provides a practical way to adapte an existing 3DGS autonomous driving scene without retraining, where reconstructing the same scene under different lighting conditions can be expensive. The combination of multiplicative illumination and additive emission is also intuitive, and the results show that it works well with only a few anchor images.

S2. The paper introduces CARLA-ToD with consistent geometry and dynamic object trajectories across different times of day, which makes the evaluation more controlled.

Critical weaknesses

W1. My main concern is the limited validation on real-world time-of-day changes. CARLA-ToD provides exactly matched geometry, camera poses, and dynamic object trajectories, which is useful for evaluation but also makes the setting more controlled than real-world driving. The Waymo experiments are helpful, but the target-time anchor images are synthesized using an image editing model rather than captured under real lighting conditions. I am not sure how well the method would generalize to real-world scenes with more complex lighting changes, shadows, and imperfectly aligned observations. More validation with real target-time images would make the results more convincing. I think it would be very meaningful to include a small real-world dataset with scenes captured at different times of day. If no suitable dataset is available, even collecting a few scenes could provide valuable evidence. While this may require some additional effort, I believe it would significantly strengthen the paper's claims.

W2. The paper compares against several inverse rendering and diffusion-based methods, but these methods use quite different representations and training settings. Table 5 provides useful ablations, but I would also like to see a simple appearance correction baseline trained on the same frozen StreetGS with the same anchor images. This would help clarify how much improvement is from the proposed formulation and regularization, rather than from directly optimizing the appearance of an already reconstructed scene.

Questions for authors

Q1. Have you tested the method using real target-time images instead of synthesized anchors?

Q2. How does PAM-ToD compare with a simpler appearance correction method trained on the same frozen StreetGS and anchor images?

Q3. The experiments mainly use StreetGS as the base model. Does PAM-ToD also work with other pre-trained 3DGS architectures?

Q4. Appendix A.7 mentions 8,000 optimization iterations for each plug-in. How long does the adaptation take in practice, and how does this compare with the training cost of the baselines?

Q5. Some existing works, such as SpectralSplat as mentioned also in the paper, have also addressed similar problems in appearance modeling for driving scenes. Could the authors provide a more direct comparison to better clarify the advantages of PAM-ToD over these methods?

6 comments
AuthorsOpenReview ↗

Authors rebuttal

[Part 1/3]

We thank the reviewer for the detailed review and for recognizing the practical value of reusing existing road-scene 3DGS alongside the CARLA-ToD benchmark. We address each of your points below.

W1 / Q1. Limited real-world validation: was the method evaluated on real target-time images?

The submitted manuscript did not include evaluations on real target-time images, as the Waymo anchors were synthesized. We agree that this was the most notable gap in our evaluation. Thank you for this valuable suggestion. Following it, we ran a new real-world experiment, and PAM-ToD performs strongly on real target-time images.

We utilized nuPlan multi-traversal data, where the same route is repeatedly driven under different lighting conditions. A StreetGS reconstructed from a source traversal is adapted using anchor images {Ak}\{A_k\} from a target traversal. We then evaluate the model on held-out views of the target traversal, focusing on static regions consistently observed in both traversals.

Table R2. Quantitative results on nuPlan (Novel View Traversal)

MethodSSIM ↑ (K=1K{=}1)PSNR ↑ (K=1K{=}1)LPIPS ↓ (K=1K{=}1)SSIM ↑ (K=8K{=}8)PSNR ↑ (K=8K{=}8)LPIPS ↓ (K=8K{=}8)
Base (StreetGS, no adaptation)0.54915.210.2790.54915.210.279
MTGS0.57915.790.3590.57615.920.353
WildGaussians0.51314.280.4010.51914.490.401
GS-W0.50313.730.3830.50913.970.381
PAM-ToD0.60517.810.2740.75922.420.195

At K=8K{=}8, PSNR increases from 15.21 dB (base StreetGS) to 22.42 dB, outperforming the best baseline (MTGS) by 6.5 dB. Even under the extreme sparsity of K=1K{=}1, PAM-ToD surpasses all baselines across every metric. These results show that PAM-ToD works well not only on synthetic data but also on real scenes. We believe this experiment substantially strengthens the paper, and we have added it to the main paper (Table 6 and Figure 4 of the revised PDF).

[Part 2/3]

W2 / Q2. Comparison with simple appearance-correction baselines trained on the same frozen StreetGS with the same anchors

We agree that this comparison is essential for isolating the source of our method's improvements. We added four simple baselines lacking the priors introduced in our method:

  1. A global affine RGB transform shared by all Gaussians.
  2. A per-Gaussian affine transform.
  3. A per-Gaussian affine transform with regularized .
  4. A global gain + spatial field correction without semantic priors.

All these baselines use the exact same frozen StreetGS and the same anchor images {Ak}\{A_k\} as PAM-ToD, for both K=1K{=}1 and K=8K{=}8, and are trained using only the anchor reconstruction loss.

Table R1. Appearance corrections trained on the same frozen StreetGS with the same anchors (same setting as Tables 3–5)

MethodSSIM ↑ (K=1K{=}1)PSNR ↑ (K=1K{=}1)LPIPS ↓ (K=1K{=}1)SSIM ↑ (K=8K{=}8)PSNR ↑ (K=8K{=}8)LPIPS ↓ (K=8K{=}8)
Base (no adaptation)0.47110.170.4870.47110.170.487
Global affine0.69118.510.3560.69719.090.350
Per-Gaussian affine0.59514.490.4150.76923.160.341
Regularized per-Gaussian affine0.60514.910.4110.77323.610.341
Global gain + spatial field (w/o priors)0.64717.800.4040.76722.060.341
PAM-ToD0.72021.110.3270.80824.720.285

PAM-ToD outperforms all baselines across every metric at both K=1K{=}1 and K=8K{=}8. The performance gap is most pronounced at K=1K{=}1, where anchors originate from a single timestep. Under this setting, the per-Gaussian affine transform only reaches 14.91 dB even with regularization, whereas PAM-ToD achieves 21.11 dB. Furthermore, the global gain + spatial field correction (which shares our architectural correction structure) stalls at 17.80 dB at K=1K{=}1 when deprived of our priors.

These results clearly demonstrate that the improvement does not stem merely from added expressiveness, but rather from the carefully designed structure and priors that enable stable correction estimation from just a few anchors.

[Part 3/3]

Q3. Does the method work with other pre-trained 3DGS architectures?

Yes. Because PAM-ToD operates strictly on Gaussian centers and pre-rasterization RGB colors, it is by design independent of any specific 3DGS architecture. It applies color corrections before rasterization without modifying the underlying geometry or opacity, making it seamlessly applicable to vanilla 3DGS, PVG(IJCV2026), OmniRe(ICLR2025), and other Gaussian variants.

To verify this, we chose PVG (IJCV 2026) as a second base model. OmniRe shares its backbone with StreetGS, whereas PVG has a different Gaussian architecture and is also used by recent methods such as DeSiRE-GS (CVPR 2025). We replaced the frozen StreetGS with a frozen PVG and applied PAM-ToD with the same recipe (CARLA-ToD Static, Reconstruction). Table R3 shows the results, which we have added to the Appendix of the revised PDF.

Table R3: Results with a different base model

Base modelMethodK=1K{=}1 SSIM ↑PSNR ↑LPIPS ↓K=8K{=}8 SSIM ↑PSNR ↑LPIPS ↓FPS ↑
StreetGSBase0.47110.170.4870.47110.170.48766.7
StreetGS+ PAM-ToD0.74022.050.2980.82625.660.24953.4
PVGBase0.45010.060.5730.45010.060.57348.1
PVG+ PAM-ToD0.69020.160.3990.80224.570.32844.7

Both bases start at about 10 dB without adaptation. With PVG, PAM-ToD reaches 20.16 dB at K=1K{=}1 and 24.57 dB at K=8K{=}8, comparable to 22.05 dB and 25.66 dB with StreetGS. PAM-ToD therefore transfers to another base model without changing its recipe. The plug-in also adds little rendering cost: 1.6 ms per frame on PVG (48.1 → 44.7 FPS on an RTX A6000), comparable to the 3.7 ms on StreetGS (66.7 → 53.4 FPS, Table 1).

Q4. Adaptation time and computational cost compared with the baselines

On a single RTX A6000 GPU, optimizing our plug-in for 8,000 iterations takes approximately 11 minutes. Because each iteration renders one anchor image, the adaptation time does not scale with KK and remains identical for both K=1K{=}1 and K=8K{=}8. By comparison, the initial StreetGS reconstruction (which our plug-in keeps frozen) takes 46 minutes.

MethodIterationsTraining timeCost of adding one time of day
GS-IR30k148 min148 min
GI-GS30k307 min307 min
GS-W30k60 min60 min
StreetGS30k46 min46 min
+ PAM-ToD8k46 + 11 min11 min

Q5. Comparison with SpectralSplat

Thank you for pointing us to this work. Because the code and pre-trained weights for SpectralSplat are not publicly available, a direct numerical comparison was not possible. However, we can highlight the structural differences between our approaches.

SpectralSplat relies on a neural network pre-trained on large-scale driving data to predict new Gaussian representations from input images. Consequently, its ability to generalize to unseen lighting conditions may be inherently tied to the distribution of its pre-training dataset.

In contrast, PAM-ToD adapts the appearance of a pre-reconstructed 3DGS representation without modifying its geometry, relying solely on target-time anchor images {Ak}\{A_k\}. It requires no pre-training, and optimizing a new scene takes only about 11 minutes. A key practical advantage of PAM-ToD is that existing 3DGS assets already integrated into simulation environments can be quickly repurposed for different times of day without the need for full reconstruction.

Reviewer_iZJFOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents PAM-ToD, an appearance adapter for frozen, pretrained road-scene 3D Gaussian Splatting models. The paper also introduces CARLA-ToD, a benchmark with matched geometry, camera poses, and dynamic trajectories across lighting conditions.

Critical strengths

The method is sort of meaningful. Adapting existing 3DGS assets can reduce repeated reconstruction and storage costs. The reported improvements are substantial, while rendering remains approximately 50 fps after adaptation.

Critical weaknesses
  1. Several important prior methods are missing from the experimental comparisons, particularly WildGaussians (NeurIPS 2024) and DAVIGS (AAAI 2025). WildGaussians introduces per-Gaussian affine color transformations, while DAVIGS combines global appearance embeddings with spatially consistent local features to model appearance variations. Both are closely related to the proposed appearance correction framework and are cited in the Related Work section, yet neither is evaluated as a baseline. Comparisons under the same sparse-anchor adaptation setting would be valuable.

  2. The experimental evaluation is limited to CARLA-ToD and Waymo, with different categories of baselines evaluated under different settings. For example, GS-W, GS-IR, GI-GS, and LumiGauss are evaluated only on CARLA-ToD Static (Table 1), whereas diffusion-based methods are compared on CARLA-ToD Dynamic (Table 2). This separation makes it difficult to assess the relative advantages of different approaches under a unified evaluation protocol.

Questions for authors

How does the proposed appearance correction framework fundamentally differ from the per-Gaussian affine color transformations?

Could the authors clarify the input, supervision, and adaptation settings for each baseline to ensure fair comparisons?

3 comments
AuthorsOpenReview ↗

Authors rebuttal

[Part 1/3]

We sincerely thank the reviewer for recognizing the practical value of adapting existing 3DGS assets and the significant performance gains our method achieves at real-time rendering speeds.

W1. Missing comparisons with WildGaussians and DAVIGS

Difference from DAVIGS

While a direct empirical comparison would be ideal, the source code for DAVIGS is not publicly available, preventing us from reproducing their results. Instead, we clarify the methodological differences and outline our concerns. DAVIGS jointly trains the 3DGS and its appearance model, predicting appearance based on global embeddings and local features. PAM-ToD, conversely, freezes an existing 3DGS and directly optimizes an appearance correction layer using only a few target anchor images.

If DAVIGS were trained from scratch using only a limited set of anchors, its appearance embeddings and decoder would suffer from sparse supervision, likely leading to severe generalization issues in unobserved regions. PAM-ToD naturally mitigates this by propagating appearance changes to unobserved regions via hierarchical corrections and robust semantic priors, reliably adapting to a new time of day in approximately 11 minutes.

Comparison with WildGaussians

We successfully compared PAM-ToD against WildGaussians in our real-world nuPlan experiment. Because WildGaussians couples appearance embedding learning with geometry reconstruction, it must be trained from scratch on the source traversal alongside the target anchor images. Notably, both methods utilize the exact same set of anchor images.

MethodSSIM ↑ (K=1K{=}1)PSNR ↑ (K=1K{=}1)LPIPS ↓ (K=1K{=}1)SSIM ↑ (K=8K{=}8)PSNR ↑ (K=8K{=}8)LPIPS ↓ (K=8K{=}8)
Base (StreetGS, no adaptation)0.54915.210.2790.54915.210.279
MTGS0.57915.790.3590.57615.920.353
WildGaussians0.51314.280.4010.51914.490.401
GS-W0.50313.730.3830.50913.970.381
PAM-ToD0.60517.810.2740.75922.420.195

PAM-ToD comfortably outperforms WildGaussians across every metric. At K=1K{=}1, it achieves a 3.53 dB higher PSNR (17.81 vs. 14.28 dB), and at K=8K{=}8, it achieves a massive 7.93 dB improvement (22.42 vs. 14.49 dB). Furthermore, PAM-ToD maintains rendering speeds above 70 FPS, compared to 50.3 FPS for WildGaussians.

We also evaluated WildGaussians on the CARLA-ToD Static dataset. Following the GS-W protocol, it was trained from scratch on the source sequence, with the frames at the anchor timesteps replaced by {Ak}\{A_k\} (K=8K{=}8).

MethodSSIM ↑ (Recon.)PSNR ↑ (Recon.)LPIPS ↓ (Recon.)SSIM ↑ (NVS)PSNR ↑ (NVS)LPIPS ↓ (NVS)FPS ↑
WildGaussians0.76022.280.3710.76122.660.36622.6
PAM-ToD (K=1K{=}1)0.74022.050.2980.72921.890.29753.4
PAM-ToD (K=8K{=}8)0.82625.660.2490.81425.450.24753.4

Given the same K=8K{=}8 anchors, PAM-ToD achieves 3.38 dB higher PSNR in Reconstruction and 2.79 dB higher PSNR in Novel View Synthesis. Even at K=1K{=}1, PAM-ToD remains highly competitive and demonstrates a clear LPIPS improvement over WildGaussians. It also renders more than twice as fast (53.4 vs. 22.6 FPS).

[Part 2/3]

W2. No unified evaluation protocol

We initially evaluated baselines separately because methods such as GS-W, GS-IR, GI-GS, and LumiGauss assume inherently static scenes and cannot accurately model the dynamic objects present in CARLA-ToD Dynamic. Thus, we evaluated them against PAM-ToD on CARLA-ToD Static. Conversely, diffusion baselines operate directly on StreetGS renderings and can process dynamic objects. We therefore compared them on CARLA-ToD Dynamic, which provides a scenario closer to real-world driving.

However, we acknowledge that splitting these protocols makes cross-method comparisons difficult. To rectify this, we ran the diffusion baselines for both Reconstruction and Novel View Synthesis on CARLA-ToD Static.

MethodSSIM ↑ (Recon.)PSNR ↑ (Recon.)LPIPS ↓ (Recon.)SSIM ↑ (NVS)PSNR ↑ (NVS)LPIPS ↓ (NVS)FPS ↑
LumiNet0.59614.590.4170.58614.600.4180.084
DiffusionRenderer0.55914.300.4860.53613.830.5690.083
UniRelight0.60515.730.4730.56515.240.5630.068
PAM-ToD (K=1K{=}1)0.74022.050.2980.72921.890.29753.4
PAM-ToD (K=8K{=}8)0.82625.660.2490.81425.450.24753.4

The three diffusion baselines struggle. In stark contrast, PAM-ToD achieves 22.05 dB (Reconstruction) and 21.89 dB (Novel View Synthesis) even at K=1K{=}1, with fewer anchors than the K=8K{=}8 used to condition the diffusion baselines—improvements of 6.32 dB and 6.65 dB over the best baseline, UniRelight. PAM-ToD also solidly outperforms all diffusion baselines in both SSIM and LPIPS. Crucially, PAM-ToD renders at 53.4 FPS, whereas the diffusion baselines operate at less than 0.1 FPS, rendering them entirely unsuitable for real-time simulation. This mirrors our findings on CARLA-ToD Dynamic, confirming that our conclusions hold firm under a unified protocol.

[Part 3/3]

Q1. What is the essential difference from a per-Gaussian affine color transformation?

While a simple per-Gaussian affine color transformation and PAM-ToD share a foundational approach of multiplicative and additive color correction, they differ crucially in how parameters are shared and constrained when estimating appearance changes from sparse observations.

A standard per-Gaussian affine transformation assigns independent parameters to every Gaussian. As a result, corrections fail to propagate to unobserved Gaussians, and the model rapidly overfits when anchor images are sparse. PAM-ToD, conversely, decouples appearance changes into three hierarchical levels: a scene-wide global gain, a regional spatial field, and a local per-Gaussian residual. This structured parameter sharing and regularization enable robust adaptation even from highly limited views.

Empirically, under identical settings, a per-Gaussian affine transformation achieves only 14.49 dB at K=1K{=}1, whereas PAM-ToD reaches 21.11 dB. Therefore, the core contribution of PAM-ToD is not the affine transformation itself, but the structured correction architecture and optimization design that make few-shot appearance adaptation well-posed and effective.

Q2. Inputs, supervision, and adaptation setting of each baseline

MethodTarget-time dataBase 3DGSCost of adding one time of day
GS-IR / GI-GS / LumiGaussFull sequenceOwn reconstructionOne full reconstruction
GS-W{Ak}\{A_k\} (K=8K{=}8)Own reconstructionOne retraining
LumiNet / DiffusionRenderer / UniRelight{Ak}\{A_k\} (K=8K{=}8)StreetGS (frozen)None (per-frame inference)
PAM-ToD{Ak}\{A_k\} (K∈{1,8}K\in\{1,8\})StreetGS (frozen)One plug-in optimization (~11 min)

Every {Ak}\{A_k\} in this table represents the exact same set of anchor images utilized by PAM-ToD. The diffusion baselines are conditioned on these anchor images with K=8K{=}8, and they are applied to the same frozen StreetGS renderings as PAM-ToD. The inverse rendering baselines are trained on the full target-time sequence, thereby leveraging significantly more target-time information than our method. Finally, under the Novel View Synthesis protocol, the held-out test views remain strictly unseen by all evaluated methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.