acceptodds
Withdrawn from ICLR 2027

SyncGS: Self-Aligning Dynamic RGB and Thermal Gaussian Splatting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dynamic reconstruction from RGB and Thermal streams benefits from their complementary appearance and thermal cues. In practice, however, independently recorded streams often exhibit residual temporal offsets and viewpoint discrepancies, which complicate reliable cross-modal reconstruction. We present SyncGS, a self-aligning framework for dynamic Gaussian reconstruction from RGB and Thermal streams. SyncGS uses cross-modal motion cues to build a motion cost volume with local spatial tolerance and continuously queries this volume to estimate a sequence-level temporal offset. During optimization, Thermal image supervision uses the current temporal estimate to learn an effective spatial correction shared across the sequence under a shared virtual-camera model. To limit mutual compensation between calibration and scene variables, we introduce Observable-Owned Optimization, which assigns motion, image-alignment, and reconstruction objectives to the temporal, spatial, and scene variables, respectively, while retaining a shared forward state. Experiments on DynamicRGBT-Scenes show improved reconstruction quality in both RGB and Thermal modalities, recovery of imposed temporal offsets, and resilience to controlled pose-initialization perturbations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic Gaussian Splatting, RGB-T Reconstruction, Multimodal Reconstruction, Temporal Alignment, Spatial Alignment, Thermal Imaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NaKvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a system for estimating a dynamic 3D Gaussian splatting representation of a scene from a combination of RGB and thermal cameras. It focuses in particular on the problem of aligning spatially and temporally these different types of cameras that, due to hardware constraints, cannot overlap spatially or be perfectly synchronised temporally. It builds on "Dynamic Thermal Gaussians: Multimodal 4D Gaussian Splatting", taking their setup and basic formulation, and investigating the problem of improving spatial and temporal calibration in the setup. To do this, a loss based on cross-correlation is considered in addition to the usual reconstruction losses in 3D Gaussian Splatting (3DGS), and an estimation algorithm called "Observable-Owned Optimisation" is introduced to optimise the model parameters, including the calibration ones, in a more robust manner, less prone to overfitting. These ideas are ablated, and the overall reconstruction method works better than the original "Dynamic Thermal Gaussians" on their dataset.

Critical strengths
  • Generalising from the specific setting presented in the paper, the ideas may be interesting to somebody that has to estimate 3D reconstruction from a heterogeneous collection of cameras, and in particular cameras that record different modalities.

  • The paper makes an effort to develop intuition about the specific setup of the problem and to translate these in novel aspects of the formulation.

  • The paper demonstrates a quantitative improvement on "Dynamic Thermal Gaussians: Multimodal 4D Gaussian Splatting" (DTG), on which it is based (see Table 1).

  • The paper contains an ablation (see Table 3) that shows that the proposed "Observable-Owned Optimisation" (OOO; see Section 3.4) is significantly less prone to overfitting, estimating correctly the temporal shift between RGB and thermal cameras, whereas vanilla optimisation of the same loss function does not.

Critical weaknesses
  • OOO is introduced as a major contribution of the paper. While the idea is justified empirically by one ablation experiment in Table 3, and while it makes intuitive sense to try to address overfitting by reducing the number of parameter with respect to which a function is optimised over, I found the mathematical justification and framing quite lacking. OOO essentially reduces to optimising different terms of a loss function with respect to only selected subsets of variables that are deemed do be more "directly" associated to those terms (this is, effectively, the effect of blocking gradients for other terms). This is done in the hope of avoiding "one variable form absorbing the error of another" (line 269). However, this also does not necessarily result in a descent direction for the overall loss function. The overall loss function is computed in the forward pass, but not optimised in the backward pass (the individual terms are, separately, and in a coordinate fashion). The paper needs to justify this better.

  • The paper is very heavily based on the cited, and compared, DTG. Figure 1 in this paper is obviously derived from Figure 1 in the "Dynamic Thermal Gaussians" (DTG) one. The paper focuses on a subproblem that was not necessarily directly addressed by DTG, namely of joint station-temporal snychrosnisatio between cameras, but the increment is not a major one.

  • DTG itself appears to be a pre-print which does not appear to be peer-reviewed. The setup introduced in DTG (dataset and evaluation protocol), and on which this paper is based, is, in my opinion, problematic. First, the dataset is somewhat small, consisting of just 12 sequences. Second, each sequence is captured by just two camera rigs, each consisting of one RGB camera and one thermal camera. Presumably, although this is not said very clearly, this means that one rig is used for reconstruction, and the other for evaluation. While one rig can capture hundreds of frames of a scene (typically enquired by 3DGS reconstruction without introducing learned priors), here the scenes are dynamic and still observed by just two semi-simultaneous viewpoints (RGB and thermal). In fact, the scene is first reconstructed using only the RGB modality. This should not be enough to reconstruct well dynamic objects using Dynamic Gaussian Splatting, and it may explain why RGB PSNR is somewhat low. In short, the small size of the dataset, the limited input information used to reconstruct dynamic scene for the given methodology (3DGS), and the lack of peer-review of this setup, data, and benchmark, weaken the solidity of the empirical evaluation.

  • The problem addressed by the paper, reconstruction from a single camera rig consisting of one RGB camera and one thermal camera, is fairly specialised. It may not be of broad enough interest for the ICLR community.

Questions for authors
  • Is there a way of justifying "Observable-Owned Optimisation" theoretically, and in a rigorous manner? What is the algorithm actually doing?

  • Is it enough to use a single viewpoint to reconstruct well a dynamic scene using what is effectively a variant of 4DGS without introducing strong statistical priors?

  • Can the authors detail a bit more the reconstruction and evaluation protocol? This is still not very clear to me even after reading DTG. Is a single RGB + Thermal camera pair used for reconstruction, and the other is used for evaluation? How did you calibrate the two camera rigs for benchmarking?

Reviewer_axiMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the aligning issue of RGB and Thermal cameras for reconstruction. Different sensors are often unsynchronized in time and misaligned in space. To address this, the authors leverage dynamic thermal gaussions for reconstruction. They estimate temporal correspondence from cross-modal motion, and learn a spatial correction for each rgb and thermal camera pair. To avoid the errors form different components affecting each other, they constrained the different loss items to update their corresponding parameters. The authors evaluate their method on DynamicRGBT-Scenes.

Critical strengths

The temporal misalignment and inconsistent viewpoints are practical challenges for rgb-thermal reconstruction.

The authors provide a clear design, they use motion variations to calibrate temporal shifts, correct spatial alignment with thermal image residuals, and design constrained reconstruction loss for each module.

The authors provide the ablations of temporal calibration and spatial alignment. Also, the authors evaluate the method under perturbations.

Critical weaknesses

The paper uses a shared virtual-camera model with a fixed projection viewpoint. The authors need to elaborate more on how the intrinsics, lens distortion, geometry shifts between RGB and thermal are addressed.

The observable-owed optimization needs an ablation compared with normal optimization.

Questions for authors

What is the performance of the method under both temporal and spatial perturbations simultaneously?

Reviewer_ZTUSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SyncGS extends dynamic RGB/thermal Gaussian reconstruction with temporal and spatial calibration. It estimates a temporal offset from a fixed cross-modal motion-correlation volume, learns a relative camera transformation, and assigns the calibration and reconstruction losses to separate groups of variables. The goal is to improve alignment while reducing the extent to which the scene representation absorbs calibration errors.

Critical strengths
  • The problem is well motivated. RGB and thermal streams can differ in timing and viewpoint, and a flexible scene model may achieve a low rendering loss without recovering the correct alignment.

  • The reconstruction gains are consistent. Relative to DTG, average thermal PSNR improves from 32.68 to 34.92 dB and RGB PSNR from 24.37 to 25.49 dB, with PSNR improvements in both modalities on all 12 scenes.

Critical weaknesses
  • The evidence for the proposed optimization scheme is stronger for time alignment than for spatial alignment. Table 3 shows that allowing reconstruction gradients to update the temporal offset can be harmful. However, the spatial experiment in Table 4 reports rendering PSNR after pose perturbations, rather than the error of the recovered transformation. Rendering quality alone cannot distinguish accurate calibration from compensation by the scene.

  • The calibration and comparison protocols are not described in enough detail. It is unclear which observations are used to fit the offsets and transformations, how the train/test split relates to the two rigs, and whether DTG receives the same RGB initialization and total scene-update budget. These details matter for attributing the gains.

Questions for authors
  • Can you provide the explicit spatial-alignment loss and, for the existing pose-perturbation experiment, the rotation/translation error of the recovered transformation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.