acceptodds
Under review as a conference paper at ICLR 2027

Physics-inspired RGB-T 3D Gaussian Splatting with Multimodal Interaction

Abstract

RGB-Thermal (RGB-T) 3D reconstruction combines the detail richness of RGB with the all-weather capability of thermal imaging, but faces challenges in jointly balancing both modalities and modeling their physical correlations. Prior RGB-T methods either degrade RGB reconstruction quality or overlook cross-modal physical relationships, leading to artifacts in distant regions and near object boundaries. To address these challenges, we propose PRT-3D, a novel physics-inspired RGB-T 3D Gaussian Splatting (3DGS) reconstruction framework. Unlike previous data-driven methods, PRT-3D draws inspiration from the physical principles governing thermal imaging to design cross-modal interaction networks that enhance reconstruction. Specifically, we introduce a Light Transmission Network (LTN) that, inspired by atmospheric transmission attenuation, leverages the depth consistency of the RGB modality to improve thermal transmission distance estimation. Furthermore, we design a Thermal Conduction Network (TCN) that, inspired by heat diffusion effects, exploits the edge consistency of RGB images to mitigate thermal boundary blurring. Experiments demonstrate that our method significantly improves thermal reconstruction quality over state-of-the-art approaches (e.g., improving thermal PSNR over ThermalGaussian by 3.72 dB on average), while maintaining RGB quality comparable to or surpassing previous best results, and requiring even less GPU memory on a single RTX 3090 GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal AI, RGB-Thermal (RGB-T), 3D Reconstruction, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cCWNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PRT-3D for RGB-T dual-modal 3D Gaussian Splatting reconstruction. Its main components include the Light Transmission Network (LTN), which leverages depth information from both RGB and thermal modalities to estimate atmospheric attenuation-related parameters and improve thermal reconstruction quality, and the Thermal Conduction Network (TCN), which utilizes edge information from both modalities to predict compensation residuals for thermal images, thereby enhancing boundary quality. The two modalities share Gaussian geometric parameters while maintaining separate opacity and appearance attributes.

Critical strengths
  1. The paper addresses the challenges of large reconstruction errors in distant regions and blurred object boundaries in thermal reconstruction. The overall framework is clearly structured, and the proposed modules are well aligned with the identified problems.

  2. Compared with ThermalGaussian, the proposed method achieves an average improvement of 3.72 dB in Thermal PSNR on scenes such as RGBT-Scenes, while largely preserving RGB reconstruction quality.

Critical weaknesses
  1. The paper overlooks several relevant methods, including MMOne (ICCV 2025), ThermoSplat (2026), and PhysIR-Splat (CVPR 2026), and does not sufficiently discuss the existing physics-based modeling designs in Thermal3D-GS (ECCV 2024). These omissions weaken the paper's novelty positioning and undermine the credibility of its state-of-the-art (SOTA) performance claims.
  2. LTN and TCN are essentially learning-based networks that leverage cross-modal depth and edge information, respectively. The current experiments are insufficient to disentangle whether the performance improvements stem from the proposed physical formulations or simply from increased network capacity and cross-modal feature fusion. Furthermore, ThermalGaussian has already considered the smoothing characteristics of thermal imaging, and the paper does not provide a sufficiently comprehensive discussion of prior work incorporating physical priors.
  3. RGB texture edges do not necessarily correspond to temperature boundaries, and vice versa. Although TCN employs a learnable compensation mechanism rather than enforcing strict edge alignment, the paper lacks an analysis of modality-specific edges and registration errors. Consequently, it remains unclear whether the proposed method can effectively prevent the erroneous transfer of edge information across modalities.
Questions for authors
  1. Could the authors provide comparisons with MMOne, ThermoSplat, and PhysIR-Splat under unified experimental settings, and further clarify the key differences between the proposed method and existing physics-based modeling approaches such as Thermal3D-GS?
  2. Could the authors include a parameter-matched conventional cross-modal fusion network as a baseline and independently evaluate the contributions of the exponential attenuation formulation in LTN and the edge information in TCN? Such experiments would help establish whether the performance gains genuinely arise from the proposed physics-inspired designs.
  3. In scenarios where RGB and thermal edges are inconsistent or slight registration errors are present, could TCN introduce spurious edges into the thermal reconstruction? Could the authors provide quantitative experiments or visual analyses to evaluate the method's robustness under these conditions?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer cCWN (Part 1/4)

Thanks a lot for the careful and constructive review. We are glad that you found our paper “clear and easy to follow”, the framework “clearly structured”, the proposed modules “well aligned with the identified problems”, and that our method “largely preserving RGB reconstruction quality”. Our replies to your three questions follow.

Q1: Comparisons with more baselines and Clarification of key differences.

Q1.1: Could the authors provide comparisons with MMOne, ThermoSplat, and PhysIR-Splat under unified experimental settings?

Our reply: We appreciate these great suggestions. We ran all three methods under the same setting as PRT-3D: the same train/test split, 30K iterations, the same resolution and the same evaluation protocol. MMOne and PhysIR-Splat are run with their official code. ThermoSplat has no public code, so we re-implemented it from its paper.

Table A. RGBT-Scenes.

MethodThermal PSNR↑RGB PSNR↑
MMOne (ICCV 2025)25.3225.32
ThermoSplat (Neur2026)22.0622.03
PhysIR-Splat (CVPR 2026)25.7225.32
PRT-3D (ours)28.9124.94

Table B. Cross-dataset test on ThermoScenes (average over the 10 original ThermoNeRF scenes). All methods use the same configuration as on RGBT-Scenes, without per-dataset tuning.

MethodThermal PSNR↑RGB PSNR↑
MMOne (ICCV 2025)32.3022.33
ThermoSplat (Neur2026)28.9219.63
PhysIR-Splat (CVPR 2026)28.2122.51
PRT-3D (ours)32.8221.96

Thermal reconstruction. PRT-3D achieves the best thermal PSNR on both datasets. On RGBT-Scenes, it exceeds PhysIR-Splat, MMOne and ThermoSplat by 3.19, 3.59 and 6.85 dB. On the cross-dataset test, it exceeds them by 4.61, 0.52 and 3.90 dB. We attribute this to the cross-modal interaction in our physics-inspired module: the other methods either do not model thermal physics (MMOne, ThermoSplat) or model it from the thermal modality alone (PhysIR-Splat), whereas PRT-3D supplies RGB depth and edge evidence exactly where the texture-poor thermal branch is unreliable.

RGB reconstruction. The RGB quality is close across methods: PRT-3D is within 0.38 dB of the best on RGBT-Scenes and within 0.55 dB on ThermoScenes. This small trade-off comes from the design focus. MMOne and PhysIR-Splat decouple the two modalities, so the RGB branch is optimized with little interference from the thermal one. PRT-3D instead couples the modalities through the physics modules, which benefits the information-poor thermal modality most and can slightly constrain RGB where the two modalities disagree.

In summary, under a unified setting and against the most recent methods, PRT-3D delivers the best thermal reconstruction on both datasets while keeping RGB quality comparable. This confirms that physics-inspired cross-modal interaction is an effective and complementary route to high-quality RGB-T reconstruction. We will add these comparisons and this discussion to the paper.

AuthorsOpenReview ↗

Reply to Reviewer cCWN (Part 2/4)

Q1: Comparisons with more baselines and Clarification of key differences (continued)

Q1.2: Clarify the key differences between the proposed method and existing physics-based modeling approaches such as Thermal3D-GS?

Our reply: Thanks for your guidance. We compare PRT-3D with the two physics-based methods, Thermal3D-GS and PhysIR-Splat. The key differences are as follows.

1. Multimodal interaction. This is the central difference. Thermal3D-GS is thermal-only. PhysIR-Splat reconstructs both modalities but keeps them decoupled: they merely share geometry, and its physical model is driven by the thermal modality alone. PRT-3D is the only physics-based method built on cross-modal interaction: both physical corrections are estimated jointly from the two modalities, with the choice of inputs derived from a MAP analysis (Sec. 4.4–4.5). The interaction is bidirectional: RGB evidence corrects the thermal reconstruction, and the gradients flowing back into the shared geometry also improve RGB (from 24.44 to 24.94 dB PSNR, Table A2).

2. Attenuation. Thermal3D-GS predicts the transmission distance with a network whose inputs are the Gaussian position and the capture time; since the camera position is not an input, this distance has no geometric link to the rendered viewpoint. PhysIR-Splat computes the distance geometrically for each Gaussian primitive but assumes a single attenuation coefficient for the entire scene, which cannot adapt to spatially varying conditions. In contrast, PRT-3D estimates the attenuation per pixel from the rendered surface depth, so it follows the actual viewing geometry and adapts spatially. Moreover, it is the only one that uses the depth of both modalities.

3. Conduction. PhysIR-Splat does not model the boundary blur caused by heat diffusion, and the conduction module of Thermal3D-GS relies on the thermal image alone, whose edges are weak and blurred. Our TCN uses the RGB edges as a cross-modal reference, which provides sharp and well-localized boundaries. As a result, PRT-3D reduces the cross-modal edge discrepancy by 18.3% on high-contrast scenes (Table A3), and the TCN improves not only thermal PSNR (+0.20 dB) but also RGB PSNR (+0.14 dB) through the shared geometry, which a thermal-only module cannot do.

In summary, PRT-3D differs from existing physics-based methods in that it lets the two modalities interact inside the physical model. This brings three advantages: the physical corrections are guided by cross-modal evidence rather than by the thermal modality alone; the attenuation follows the actual viewing geometry and adapts spatially; and the conduction term is anchored to the sharp RGB boundaries. As a result, the interaction benefits both modalities, improving thermal and RGB reconstruction at the same time.

AuthorsOpenReview ↗

Reply to Reviewer cCWN (Part 3/4)

Q2: Could the authors include a parameter-matched conventional cross-modal fusion network as a baseline and independently evaluate the contributions of the exponential attenuation formulation in LTN and the edge information in TCN?

Our reply: Thank you for the inspiring suggestions. We added all three comparisons. All variants share the backbone, the training schedule and the data split.

a. Parameter-matched cross-modal fusion network (PRT-3D w/o physics). It takes the rendered images and depth maps of both modalities (IR,IT,dR,dTI_R, I_T, d_R, d_T) as inputs, has the same number of parameters as LTN and TCN combined (around 46.5 K), and adds its output to the rendered thermal image, without any physics-inspired structure.

b. Exponential attenuation formulation in the LTN (PRT-3D w/o exp). We keep the LTN inputs and capacity but replace e−βle^{-\beta l} by an unconstrained gain α\alpha predicted by the network.

c. Edge information in the TCN (PRT-3D w/o edge). We keep the TCN capacity but feed it the rendered images ITI_T and IRI_R instead of their Laplacian edge maps.

MethodThermal PSNR↑RGB PSNR↑
PRT-3D28.9124.94
PRT-3D w/o physics28.5624.58
PRT-3D w/o exp28.2824.62
PRT-3D w/o edge28.5824.50
Ref: PRT-3D w/o LTN28.6024.58
Ref: PRT-3D w/o TCN28.8324.68
Ref: PRT-3D w/o LTN and TCN28.4024.44

The reference rows are from Tables A1 and A2 of the supplementary material. The results show that:

  1. With the same inputs and number of parameters, the generic fusion network reaches 28.56 dB, which is 0.35 dB below PRT-3D. It recovers only 0.16 dB (31%) of the 0.51 dB gain brought by the two modules. Capacity and access to cross-modal information alone do not explain the main gain.

  2. Removing the exponential formulation lowers thermal PSNR by 0.63 dB, and removing the edge inputs lowers it by 0.33 dB. In both cases the result is worse than removing the module altogether. The two modules are therefore effective because of their physics-inspired design, not merely because a network is added.

  3. Each module also contributes individually (+0.43 dB for LTN, +0.20 dB for TCN), and their combination is best in both modalities.

These results indicate that the gains mainly arise from the physics-inspired designs rather than from added capacity or generic fusion.

Reviewer_rQoGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PRT-3D studies joint RGB-thermal reconstruction using 3D Gaussian Splatting. The two modalities share Gaussian geometry but have separate opacity and appearance attributes. A Light Transmission Network (LTN) takes rendered RGB and thermal depth maps to predict an attenuation coefficient and transmission distance. A Thermal Conduction Network (TCN) uses the rendered thermal image and cross-modal Laplacian edges to predict an additive correction. Together, they produce a thermal prediction of the form I_T exp(-βl) + D, optimized alongside RGB reconstruction with L1 and D-SSIM losses.

On 10 RGBT-Scenes scenes plus an additional low-light scene, the paper reports thermal PSNR of 28.91 dB versus ThermalGaussian's 25.19 dB, comparable average RGB quality, and rendering at 24.81 FPS. Further evaluation covers 22 ThermoScenes scenes and comparisons with thermal-only methods. The paper also compares network-based conditioning with explicit consistency losses, while referring detailed component ablations to supplementary material.

Critical strengths
  1. Thermal PSNR improves consistently over ThermalGaussian on the 11 RGBT-Scenes test scenes, with additional results on 22 ThermoScenes scenes.
  2. The comparison between explicit consistency losses and network-based conditioning is informative, and the full system retains a reported rendering speed of 24.81 FPS.
Critical weaknesses
  1. The contribution relative to prior work is unclear. Thermal3D-GS already models atmospheric transmission and thermal conduction but is not discussed. The paper should distinguish its RGB conditioning and image-space formulation from these existing ideas.
  2. The physical interpretation is not established. LTN's output depends on β and l only through their product, so reconstruction loss alone cannot identify them individually. The additive TCN residual can compensate for attenuation changes. The MAP priors motivate the inputs, but the final objective does not explicitly enforce them; improved reconstruction therefore does not establish recovery of the claimed physical effects.
  3. The experiments do not isolate the source of improvement. Controls with matched capacity, no RGB conditioning, and raw versus corrected thermal outputs are needed to distinguish physical modeling from generic image refinement. Detailed ablations are deferred to supplementary material absent from the supplied PDF. The thermal PSNR advantage over GSDF is only 0.33 dB, although GSDF does not provide the same joint RGB-T output.
Questions for authors
  1. Novelty and controlled comparisons. Which aspects of LTN/TCN go beyond the transmission and conduction modeling in Thermal3D-GS? Please provide results for PRT-3D without either module, without RGB conditioning, and with a generic refinement network matched in inputs and capacity. These comparisons would separate the base representation, cross-modal information, and specific module design.
  2. Physical meaning of the learned quantities. Are thermal images calibrated radiance, temperature measurements, or normalized intensities? Since the output depends on βl, what constraints distinguish attenuation from transmission distance and prevent the additive residual from absorbing either effect? Please report parameter ranges and compare with using rendered depth directly as the distance.
  3. Test-time inputs and geometric consistency. Please confirm that LTN/TCN use only rendered quantities for novel views and clarify whether the thermal image enters TCN directly or only through residual addition. Reporting raw versus corrected thermal metrics and cross-view consistency would help establish whether the gains reflect improved geometry or primarily image-space refinement.
  4. Evaluation protocol and transfer. Were ThermalGaussian results independently rerun with the stated training budget, resolution, and data split? For ThermoScenes, which parameters are optimized per scene and which, if any, transfer without retraining? Please also define the reported memory metric so that its reduction can be interpreted alongside the increased training time and lower rendering speed.
10 comments
AuthorsOpenReview ↗

Reply to Reviewer rQoG (Part 1/5)

Many thanks for your thorough and constructive review. We are encouraged that you found the paper “clear and easy to follow”, that thermal PSNR “improves consistently over ThermalGaussian”, and that the comparison between explicit consistency losses and network-based conditioning is “informative”. Our replies to your four questions follow.

Q1: Novelty and controlled comparisons.

Q1.1. What goes beyond the transmission and conduction modeling in Thermal3D-GS?

Our Reply: Thank you for pointing this out, our PRT-3D and Thermal3D-GS differ in the following respects:

1) Setting. Thermal3D-GS reconstructs from thermal images only. PRT-3D is a joint RGB-T method, and its physics modules are where the two modalities interact. The RGB branch supplies the depth and edge evidence that is unreliable in the texture-poor thermal branch.

2) Transmission attenuation. In Thermal3D-GS, an MLP maps each Gaussian's position and the capture time to (μabs,μsca,d)(\mu_{abs}, \mu_{sca}, d) and rescales that Gaussian's SH coefficients. The distance dd is not tied to the camera-to-surface distance of the rendered view. In contrast, PRT-3D applies the Beer–Lambert law per pixel after rasterization and conditions it on the rendered depth maps of both modalities. The transmission attenuation therefore follows the actual viewing geometry and is regularized by the RGB depth.

3) Conduction. The conduction module of Thermal3D-GS only takes the thermal image and its second-order gradient. In contrast, PRT-3D differs by using the Laplacian of the RGB rendering as a cross-modal edge reference, which is what the edge prior from the MAP analysis requires.

The contribution beyond Thermal3D-GS is therefore the cross-modal conditioning of both physical terms and its MAP-based derivation, which is the "RGB conditioning and image-space formulation" that you identified.

Q1.2. Please provide results for PRT-3D without either module, without RGB conditioning, and with a generic refinement network matched in inputs and capacity.

Our Reply: Thank you for this inspiring suggestion. Following it, we added the comparisons below.

a. PRT-3D without either module. We evaluate three variants: PRT-3D w/o TCN, PRT-3D w/o LTN, and PRT-3D w/o TCN&LTN. These ablations were reported in the supplementary material of the submission (Tables A1 and A2); we reproduce them here for convenience.

b. PRT-3D without RGB conditioning. We block the RGB inputs of the two modules and keep only the thermal inputs. We name this variant PRT-3D w/o RGB.

c. PRT-3D with a generic refinement network matched in inputs and capacity. Following your suggestion and those of Reviewers rBqn and cCWN, we designed an input- and capacity-matched generic refinement network. It takes the same rendered quantities (IR,IT,dR,dTI_R, I_T, d_R, d_T) as inputs, has the same number of parameters as LTN and TCN combined (around 46.5 K), and adds its output directly to the rendered thermal image, without any physics-inspired structure.We name this variant PRT-3D w/o physics.

All other experimental settings are kept unchanged. The results are summarized below.

MethodThermal PSNR↑RGB PSNR↑
PRT-3D28.9124.94
PRT-3D w/o TCN28.8324.68
PRT-3D w/o LTN28.6024.58
PRT-3D w/o TCN&LTN28.4024.44
PRT-3D w/o RGB28.6924.76
PRT-3D w/o physics28.5624.58

The results show that:

  1. Both modules contribute. Removing TCN, LTN, or both lowers thermal PSNR by 0.08, 0.31 and 0.51 dB, and RGB PSNR by 0.26, 0.36 and 0.50 dB, respectively. The full model performs best in both modalities.

  2. RGB conditioning matters. Without the RGB inputs, thermal PSNR drops by 0.22 dB and RGB PSNR by 0.18 dB, which confirms the benefit of the cross-modal information.

  3. The main gain does not come from capacity. With the same inputs and the same number of parameters, PRT-3D w/o physics is 0.35 dB (thermal) and 0.36 dB (RGB) below PRT-3D. It recovers only 0.16 dB (31%) of the 0.51 dB thermal gain brought by the two modules, and it remains below PRT-3D w/o RGB, although the latter has no access to the RGB inputs. This shows that the physics-inspired structure, rather than the added parameters or inputs, is what makes the modules effective.

These comparisons separate the three factors you listed. The base representation alone reaches 28.40 dB, and the two modules add 0.51 dB on top of it. Removing the cross-modal information loses 0.22 dB, i.e., 43% of this module gain. Replacing the physics-inspired design by a generic network of matched inputs and capacity loses 0.35 dB, i.e., 69% of this module gain.

AuthorsOpenReview ↗

Reply to Reviewer rQoG (Part 2/5)

Q2: Physical meaning of the learned quantities.

Q2.1: Are thermal images calibrated radiance, temperature measurements, or normalized intensities?

Our reply: The thermal images are normalized intensities, not calibrated radiance or temperature measurements. However, a thermal camera converts the received radiance into pixel intensity through a built-in deterministic and strictly monotonic (injective) mapping based on Planck’s law. Intensity, radiance and temperature are therefore in one-to-one correspondence:

L(T)=∫η(λ) 2hc2λ5 1ehc/(λkBT)−1 dλ(band radiance, Planck’s law)L(T) = \int \eta(\lambda)\thinspace\frac{2hc^{2}}{\lambda^{5}}\thinspace\frac{1}{e^{hc/(\lambda k_B T)}-1}\thinspace d\lambda \qquad \text{(band radiance, Planck's law)}

S=G L(T)+O(camera signal, linear detector response)S = G\thinspace L(T) + O \qquad \text{(camera signal, linear detector response)}

I=S−Smin⁡Smax⁡−Smin⁡(normalized intensity)I = \frac{S - S_{\min}}{S_{\max} - S_{\min}} \qquad \text{(normalized intensity)}

Here η(λ)\eta(\lambda) is the spectral response of the camera, GG and OO are its gain and offset, and Smin⁡S_{\min}, Smax⁡S_{\max} define the display range. L(T)L(T) increases strictly with the temperature TT, and the last two steps are increasing affine maps. The composition I=g(L(T))I = g(L(T)) is thus strictly increasing and invertible: each intensity value corresponds to exactly one radiance and one temperature.

Moreover, because the last two steps are affine in LL, an atmospheric transmittance e−βle^{-\beta l} acting on the radiance acts on the intensity II as the same multiplicative factor, up to an additive offset. The Beer–Lambert form therefore carries over to the normalized-intensity domain in which our model operates.

Q2.2: Since the output depends on βl\beta l, what constraints distinguish attenuation from transmission distance and prevent the additive residual from absorbing either effect?

Our reply: Thanks for pointing this out and we agree with your analysis. The image depends on β\beta and ll only through the optical depth βl\beta l, so the reconstruction loss cannot identify them individually, and we do not claim that it does. This is inherent to the physics: a single-band passive measurement can constrain at most the transmittance, never β\beta and ll separately, which is why thermography software asks the operator to supply the distance. We therefore interpret the product as a whole, i.e., the transmittance e−βle^{-\beta l}, which is a function of the rendered depths and thus tied to the viewing geometry. The factorization is a notation inherited from the Beer–Lambert law, and we will revise the paper to say so.

Regarding the additive residual, the two terms are separated by design. The LTN receives only depth maps and produces a distance-dependent factor. The TCN receives only edge maps and produces an edge correction. Besides, they have different forms, the attenuation is multiplicative and scales with the signal, whereas the residual is additive. Empirically, the two modules give complementary gains: 0.43 dB (thermal) & 0.24 dB (RGB) for LTN alone, 0.20 dB (thermal) & 0.14 dB (RGB) for TCN alone, and 0.51 dB (thermal) & 0.50 dB (RGB) together. The two modules therefore mostly correct different errors.

AuthorsOpenReview ↗

Reply to Reviewer rQoG (Part 3/5)

Q2: Physical meaning of the learned quantities (continued)

Q2.3: Please report parameter ranges and compare with using rendered depth directly as the distance.

Our reply: Thanks for your valuable guidance. We report both below.

Parameter ranges. Statistics are computed on all test-view pixels of each scene. Intensities are on a 0–1 scale. For the LTN we report the attenuation exponent β⋅l\beta\cdot l, so the transmittance applied to the thermal image is e−β⋅le^{-\beta\cdot l}. For the TCN we report the compensation residual DD.

Sceneβ⋅l\boldsymbol{\beta\cdot l}: meanβ⋅l\boldsymbol{\beta\cdot l}: 99th pct.Transmittance at 99th pct.
IronIngot0.0830.1260.88
Truck0.0490.1680.85
Dimsum0.0140.0710.93
DarkScene0.0080.0290.97
DailyStuff<0.0010.0200.98
Building, Ebike, LandScape, Parterre, RoadBlock, RotaryKiln<0.001<0.001>0.999
SceneD\boldsymbol{D}: 1st–99th pct.D\boldsymbol{D}: meanMagnitude of D\boldsymbol{D}, RGB edge / non-edgeRelative compensation, low- vs. high-contrast thermal edges
Dimsum[−0.048, 0.049]0.0001.5×2.3×
DailyStuff[−0.060, 0.061]0.0001.3×2.6×
Ebike[−0.066, 0.066]0.0001.5×2.4×
RoadBlock[−0.034, 0.039]0.0002.4×2.5×
Truck[−0.048, 0.049]0.0001.3×3.6×
RotaryKiln[−0.040, 0.045]0.0001.7×2.1×
Building[−0.030, 0.037]0.0002.1×2.6×
IronIngot[−0.069, 0.065]0.0001.9×1.4×
Parterre[−0.056, 0.049]0.0002.3×3.4×
LandScape[−0.073, 0.072]0.0001.4×2.0×
DarkScene[−0.034, 0.033]0.0001.7×6.0×

RGB edges are the top 10% of pixels by Laplacian magnitude in the rendered RGB image; thermal edges are defined in the same way on the thermal ground truth. The relative compensation is the magnitude of DD divided by the local thermal contrast (maximum minus minimum intensity in a 7×7 window); the last column compares the lowest-contrast quartile of thermal edges with the highest-contrast one.

These ranges show three things.

  1. Both quantities are small and bounded. For 99% of the pixels, β⋅l\beta\cdot l stays below 0.17, i.e., the LTN attenuates the rendered radiance by at most about 15%, and DD stays within ±0.08. The two modules therefore act as mild physical corrections, not as free image-space refinements.

  2. The learned transmittance is physically plausible. It stays above 0.85 everywhere and is close to 1 in most scenes, which is consistent with the weak atmospheric attenuation expected at the short ranges of these scenes.

  3. D\boldsymbol{D} behaves as an edge compensation and cannot absorb the attenuation. DD is zero-mean in all 11 scenes, so it does not change the overall brightness. It is 1.3–2.4 times larger at RGB edges than elsewhere, and low-contrast thermal edges, where small temperature differences make boundaries most blurred, receive 1.4–6.0 times more compensation relative to their amplitude than high-contrast thermal edges. The weaker the edge, the larger the fraction of it that is lost, and the larger the fraction that must be restored. The compensation learned by the TCN follows exactly this pattern, which indicates that it performs a physically meaningful diffusion compensation rather than an arbitrary image refinement.

Rendered depth as the distance. We trained a variant in which ll is set to the rendered thermal depth and the network predicts β\beta only. It reaches 28.57 dB thermal PSNR, which is 0.34 dB below the full model (28.91 dB) and on par with the model without the LTN (28.60 dB, Q1.2). Using the raw rendered depth as the distance therefore brings almost no benefit. This is consistent with our preliminary analysis (Sec. 3, Fig. 2): the depth rendered from the texture-poor thermal branch is unreliable, which is why the LTN estimates the transmittance from the depth maps of both modalities instead of taking the thermal depth as is.

Reviewer_rBqnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PRT-3D, a physics-inspired RGB-T 3D Gaussian Splatting framework that incorporates transmission and conduction-inspired modules.

Critical strengths
  1. The method is well motivated, addressing specific limitations in RGB-T reconstruction.
  2. The experimental results are encouraging, showing substantial improvements in thermal reconstruction without significantly sacrificing RGB quality.
Critical weaknesses
  1. The physical interpretation of the learned transmission parameters is not fully established.
  2. It remains unclear whether the improvements primarily come from the physics-inspired design or the additional network capacity.
Questions for authors
  1. Could the authors clarify whether the learned transmission parameters correspond to measurable physical quantities?
  2. Could a capacity-matched generic fusion module be included for comparison?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer rBqn

Thank you very much for the positive and constructive review. We are happy that you found the paper “clear and easy to follow”, the method “well motivated, addressing specific limitations in RGB-T reconstruction”, and the experimental results “encouraging, showing substantial improvements in thermal reconstruction without significantly sacrificing RGB quality”. Our replies to your two questions follow.

Q1: Do the learned transmission parameters correspond to measurable physical quantities?

Our reply: Yes, the learned transmission parameters are β\beta, ll and DD, each is defined by a physical law and is measurable in principle.

  1. β\boldsymbol{\beta} is the atmospheric extinction coefficient in the Beer–Lambert law (unit: m$^{-1}$). It is the attenuation rate of infrared radiation per unit path length in air, i.e., the sum of absorption (water vapor, CO2\mathrm{CO_2}) and scattering (aerosols, fog). It can be measured with a dedicated transmissometer, but not with an ordinary thermal camera, so no public dataset records it.

  2. l\boldsymbol{l} is the transmission distance in the Beer–Lambert law: the distance from the object surface to the camera along the line of sight (unit: m). It can be measured with a laser rangefinder. In practice, however, obtaining pixel-level distances over object surfaces takes considerable effort, and public datasets usually record only a rough range, e.g., 1–3 m.

  3. D\boldsymbol{D} is the conduction compensation residual, derived from the heat equation. The underlying physical parameter is the thermal diffusivity κ=k/(ρc)\kappa = k/(\rho c) (unit: m$^{2}$/s), which can be measured on material samples with the laser flash method (ASTM E1461). In practice, this would require measuring the material parameters of every object in every scene.

So overall, these physical quantities are hard to measure directly in practice, yet they affect RGB-T reconstruction. This is why we propose physics-inspired networks to estimate them. Our experiments show that introducing these physics-inspired modules effectively improves RGB-T 3D reconstruction quality while avoiding complex physical measurements.

Q2: Could a capacity-matched generic fusion module be included for comparison?

Our reply: Thank you for your valuable suggestion. Yes, we added this comparison. The generic module receives the rendered images and depth maps of both modalities (IR,IT,dR,dTI_R, I_T, d_R, d_T). This is a superset of the information given to LTN and TCN. It has the same number of parameters (around 46.5K) and the same training schedule, and it outputs an unconstrained additive correction: I^T=IT+G(IR,IT,dR,dT).\hat{I}_T = I_T + G(I_R, I_T, d_R, d_T).

MethodThermal PSNR↑Thermal SSIM↑Thermal LPIPS↓RGB PSNR↑RGB SSIM↑RGB LPIPS↓
PRT-3D w/o LTN and TCN28.400.9340.17824.440.8470.213
PRT-3D w/o physics (generic fusion)28.560.9280.20124.580.8490.211
PRT-3D (LTN+TCN)28.910.9360.17424.940.8520.210

The results show that, with the same inputs and the same number of parameters, the generic module is 0.35 dB below PRT-3D in thermal PSNR and 0.36 dB below in RGB PSNR, and it is worse on SSIM and LPIPS in both modalities. Compared with the model without any module (first row), it recovers only 0.16 dB (31%) of the 0.51 dB thermal gain brought by LTN and TCN. Moreover, its thermal SSIM and LPIPS are even worse than without any module (0.928 vs. 0.934 and 0.201 vs. 0.178): an unconstrained refinement raises PSNR slightly but degrades the structural and perceptual quality, whereas the physics-inspired modules improve all three metrics.

The same holds for RGB. The generic module recovers only 0.14 dB (28%) of the 0.50 dB RGB gain brought by LTN and TCN (24.58 vs. 24.94 dB). Since the RGB rendering does not pass through either module, this gain can only come from the shared geometry, which shows that the physics-inspired modules also regularize the geometry more effectively than a generic network.

The improvement therefore comes mainly from the physics-inspired structure rather than from added capacity. The two modules (LTN+TCN) together have only 46.5 K parameters, which is only 0.16% of the whole model size (28.46 M parameters).

We hope this addresses your concerns, and we are happy to provide further details.

Thank you for the detailed response and additional experiments. The clarifications on the physical interpretation of the learned parameters and the capacity-matched comparison have helped address my concerns.

AuthorsOpenReview ↗

Thank you for the positive feedback

Thank you very much for your positive feedback. We are glad that the clarifications on the physical interpretation and the capacity-matched comparison have addressed your concerns. We will revise the paper accordingly. Thank you again for your effort and constructive comments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Reply

We thank all reviewers for their careful reading and constructive suggestions. We are encouraged that the reviewers found the paper “clear and easy to follow” (rBqn, rQoG, cCWN) and “interesting or useful” (rBqn, rQoG); the method “well motivated, addressing specific limitations in RGB-T reconstruction” (rBqn) and “clearly structured”, with modules “well aligned with the identified problems” (cCWN); the results “encouraging”, with “substantial improvements in thermal reconstruction” (rBqn) that hold “consistently” (rQoG) while “largely preserving RGB reconstruction quality” (cCWN); and the comparison between explicit losses and network-based conditioning “informative” (rQoG). We address all comments below and will revise the paper accordingly.

In summary, we made the following additions and clarifications:

  • Controlled comparisons. We added a capacity-matched generic fusion network and further variants (without RGB conditioning, without the exponential attenuation form, without the edge inputs) to separate the contribution of the physics-inspired design from that of added capacity (rBqn Q2; rQoG Q1.2; cCWN Q2).
  • Physical meaning. We clarified the physical quantities behind the learned parameters, the type of the thermal images and the identifiability of β\beta and ll, and we reported the ranges of β⋅l\beta\cdot l and DD together with a comparison between explicit losses and network-based conditioning (rBqn Q1; rQoG Q2.1–Q2.4).
  • Geometry versus image-space refinement. We confirmed the test-time inputs of LTN and TCN and reported raw versus corrected outputs and an automatic cross-view consistency analysis on all 11 scenes (rQoG Q3.1–Q3.2).
  • More baselines. We compared PRT-3D with MMOne, ThermoSplat and PhysIR-Splat under a unified setting on RGBT-Scenes and ThermoScenes (cCWN Q1.1).
  • Differences from physics-based methods. We clarified the key differences between PRT-3D and Thermal3D-GS and PhysIR-Splat (rQoG Q1.1; cCWN Q1.2).
  • Robustness to edges and misregistration. We added a quantitative analysis of modality-specific edges, a misregistration test with shifts of up to 8 px, and a visual analysis in the supplementary material (cCWN Q3).
  • Evaluation protocol. We clarified the ThermalGaussian setting, which parameters are optimized per scene on ThermoScenes, and the definition of the memory metric (rQoG Q4.1–Q4.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.