- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a system for estimating a dynamic 3D Gaussian splatting representation of a scene from a combination of RGB and thermal cameras. It focuses in particular on the problem of aligning spatially and temporally these different types of cameras that, due to hardware constraints, cannot overlap spatially or be perfectly synchronised temporally. It builds on "Dynamic Thermal Gaussians: Multimodal 4D Gaussian Splatting", taking their setup and basic formulation, and investigating the problem of improving spatial and temporal calibration in the setup. To do this, a loss based on cross-correlation is considered in addition to the usual reconstruction losses in 3D Gaussian Splatting (3DGS), and an estimation algorithm called "Observable-Owned Optimisation" is introduced to optimise the model parameters, including the calibration ones, in a more robust manner, less prone to overfitting. These ideas are ablated, and the overall reconstruction method works better than the original "Dynamic Thermal Gaussians" on their dataset.
Critical strengths
-
Generalising from the specific setting presented in the paper, the ideas may be interesting to somebody that has to estimate 3D reconstruction from a heterogeneous collection of cameras, and in particular cameras that record different modalities.
-
The paper makes an effort to develop intuition about the specific setup of the problem and to translate these in novel aspects of the formulation.
-
The paper demonstrates a quantitative improvement on "Dynamic Thermal Gaussians: Multimodal 4D Gaussian Splatting" (DTG), on which it is based (see Table 1).
-
The paper contains an ablation (see Table 3) that shows that the proposed "Observable-Owned Optimisation" (OOO; see Section 3.4) is significantly less prone to overfitting, estimating correctly the temporal shift between RGB and thermal cameras, whereas vanilla optimisation of the same loss function does not.
Critical weaknesses
-
OOO is introduced as a major contribution of the paper. While the idea is justified empirically by one ablation experiment in Table 3, and while it makes intuitive sense to try to address overfitting by reducing the number of parameter with respect to which a function is optimised over, I found the mathematical justification and framing quite lacking. OOO essentially reduces to optimising different terms of a loss function with respect to only selected subsets of variables that are deemed do be more "directly" associated to those terms (this is, effectively, the effect of blocking gradients for other terms). This is done in the hope of avoiding "one variable form absorbing the error of another" (line 269). However, this also does not necessarily result in a descent direction for the overall loss function. The overall loss function is computed in the forward pass, but not optimised in the backward pass (the individual terms are, separately, and in a coordinate fashion). The paper needs to justify this better.
-
The paper is very heavily based on the cited, and compared, DTG. Figure 1 in this paper is obviously derived from Figure 1 in the "Dynamic Thermal Gaussians" (DTG) one. The paper focuses on a subproblem that was not necessarily directly addressed by DTG, namely of joint station-temporal snychrosnisatio between cameras, but the increment is not a major one.
-
DTG itself appears to be a pre-print which does not appear to be peer-reviewed. The setup introduced in DTG (dataset and evaluation protocol), and on which this paper is based, is, in my opinion, problematic. First, the dataset is somewhat small, consisting of just 12 sequences. Second, each sequence is captured by just two camera rigs, each consisting of one RGB camera and one thermal camera. Presumably, although this is not said very clearly, this means that one rig is used for reconstruction, and the other for evaluation. While one rig can capture hundreds of frames of a scene (typically enquired by 3DGS reconstruction without introducing learned priors), here the scenes are dynamic and still observed by just two semi-simultaneous viewpoints (RGB and thermal). In fact, the scene is first reconstructed using only the RGB modality. This should not be enough to reconstruct well dynamic objects using Dynamic Gaussian Splatting, and it may explain why RGB PSNR is somewhat low. In short, the small size of the dataset, the limited input information used to reconstruct dynamic scene for the given methodology (3DGS), and the lack of peer-review of this setup, data, and benchmark, weaken the solidity of the empirical evaluation.
-
The problem addressed by the paper, reconstruction from a single camera rig consisting of one RGB camera and one thermal camera, is fairly specialised. It may not be of broad enough interest for the ICLR community.
Questions for authors
-
Is there a way of justifying "Observable-Owned Optimisation" theoretically, and in a rigorous manner? What is the algorithm actually doing?
-
Is it enough to use a single viewpoint to reconstruct well a dynamic scene using what is effectively a variant of 4DGS without introducing strong statistical priors?
-
Can the authors detail a bit more the reconstruction and evaluation protocol? This is still not very clear to me even after reading DTG. Is a single RGB + Thermal camera pair used for reconstruction, and the other is used for evaluation? How did you calibrate the two camera rigs for benchmarking?