acceptodds
Under review as a conference paper at ICLR 2027

PoseSpecGS: Pose-Aligned Multimodal Gaussian Splatting with Spectrum-Response Rendering

Abstract

Multimodal novel-view synthesis aims to reconstruct a unified scene representation from images acquired by sensors with distinct image-formation characteristics. However, modality-dependent appearance and inaccurate camera poses hinder learning a compact, reusable representation. We present PoseSpecGS, a pose-aligned Gaussian splatting method with spectrum-response rendering for multimodal novel-view synthesis. Inspired by sensor spectral responses, we first introduce Spectrum-Response Rendering, which synthesizes modality-specific images from alpha-composited shared Gaussian spectral features using compact modality-channel response operators. To address pose inaccuracies, we further introduce Cross-Modal Alignment, which registers auxiliary-modality cameras to an RGB structure-from-motion backbone and jointly refines their poses and scene parameters using RGB-guided reprojection consistency, without known rig extrinsics or synchronized frame correspondences. The representation supports static and time-conditioned multimodal reconstruction. Moreover, shared response operators pretrained across scenes enable zero-shot one-to-multimodal transfer under RGB-only supervision. Experiments demonstrate state-of-the-art reconstruction quality on the evaluated benchmarks, with reduced model storage and fewer training iterations. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal 3D reconstruction, 3D Gaussian splatting, novel view synthesis, latent spectral representation, cross-modal alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_joTYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents PoseSpecGS, a multimodal fusion method that integrates heterogeneous sensor observations into a unified scene representation for multimodal novel-view synthesis. It combines Cross-Modal Alignment and Spectrum-Response Rendering to enable fusion without known sensor-rig extrinsics or synchronization. Experiments demonstrate strong reconstruction quality and compact representations.

Critical strengths
  1. Important and Practical Problem: The paper addresses the challenging problem of multimodal 3D reconstruction without requiring known sensor-rig extrinsics or synchronized observations.

  2. Strong Experimental Results: The method achieves promising reconstruction quality across multiple modalities, including time-varying RGB–thermal scenes, while demonstrating compactness and cross-scene generalization.

Critical weaknesses
  1. Insufficient Validation of Cross-Modal Alignment: Despite being a major contribution, the paper lacks direct registration accuracy evaluation. Appendix C.7 reports rig consistency rather than errors against ground-truth poses. The authors question the reliability of dataset-provided poses, but provide insufficient evidence to justify excluding them as a reference. It remains unclear whether the proposed refinement improves camera alignment or merely compensates for pose errors through scene optimization.

  2. Unclear Independent Benefits of Spectrum-Response Rendering: The performance gap with MS-Splatting decreases substantially from Table 3 to Table 6 when similar camera pose protocols are used, raising questions about the independent contribution of Spectrum-Response Rendering. Moreover, the compactness evaluation lacks a per-modality Scaffold-GS baseline, making it difficult to distinguish the proposed representation's benefits from those of its underlying backbone.

Questions for authors

In addition to the concerns raised in the Weaknesses section, I have the following questions:

  1. Latent Dimension Ablation: Could the authors provide an ablation on the 32D spectral feature dimension (e.g., 8, 16, 32, 64), evaluating reconstruction quality and model storage? Is 32D still necessary when only RGB and monochrome observations are available?

  2. Cross-Modal Overlap and Registration Robustness: The method assumes sufficient cross-modal overlap without quantitatively defining it. What minimum overlap is required, and does it vary across modalities? How robust is DINOv3 retrieval to cross-modal appearance differences, and how are failed PnP initializations handled?

2 comments
AuthorsOpenReview ↗

Response to Reviewer joTY (1/2)

We thank Reviewer joTY for the detailed feedback and for recognizing the practical importance of the problem and the strong reconstruction results. We address the concerns about alignment accuracy, representation benefits, feature dimensionality, and registration robustness below.

W1. Direct validation of Cross-Modal Alignment.

Response:

Direct quantitative evaluation. We agree that reconstruction quality alone cannot establish camera-pose accuracy. As suggested, we evaluate our refined poses and two baselines, Joint SfM and Per-modality SfM + Sim(3), against the dataset-provided poses across all 32 MMS-DATA scenes (Tab. 26, revised manuscript). After a single RGB-based similarity alignment per scene, we measure the mean angular error of aligned camera orientations and nATE, the camera-center RMSE normalized by the reference RGB trajectory’s RMS radius and expressed as a percentage. PoseSpecGS achieves 0.179°–1.119° rotation error and 1.774%–2.988% nATE, outperforming both SfM baselines across all four auxiliary modalities. This reference-based evaluation provides additional evidence of pose alignment quality, complementing the rig-consistency analysis in Appendix C.7.

Rationale for the original evaluation. Reliable ground-truth camera poses are difficult to obtain for real-world multimodal captures, and dataset-provided calibration may retain residual alignment errors. Our concern about reference-pose accuracy arose from visible reconstruction artifacts with the dataset-provided poses and substantial improvements in baseline reconstruction after applying Cross-Modal Alignment (Fig. 7, Tabs. 4 and 6). The original submission therefore focused on rig consistency and reconstruction quality, which we now supplement with the direct evaluations above.

W2. Independent benefits of Spectrum-Response Rendering and comparison with Scaffold-GS.

Response:

Independent contribution of Spectrum-Response Rendering. We agree that applying our camera initialization and pose refinement substantially improves MS-Splatting, demonstrating the effectiveness of Cross-Modal Alignment. To examine the independent benefits of Spectrum-Response Rendering, we compare the two methods across all 32 MMS-DATA scenes under a matched camera-pose protocol (Tab. 6). PoseSpecGS still outperforms MS-Splatting by 2.40–3.01 dB across all five modality groups, which is a considerable improvement supporting the contribution of our representation beyond camera alignment. Parentheses below indicate PSNR gains over MS-Splatting.

MethodRGBMonoNIRPolarMS
MS-Splatting (w/ Cross-Modal Alignment)32.1429.6031.1029.5529.74
Ours34.54 (+2.40)32.61 (+3.01)34.01 (+2.91)32.12 (+2.57)32.51 (+2.77)

Table 8 further examines the rendering design through within-model ablations. Our formulation achieves higher PSNR across all five modality groups, with average gains of 1.42 dB over intensity compositing and 0.70 dB over a joint response MLP. Together, the controlled comparison and internal ablations support the independent contribution of Spectrum-Response Rendering.

Motivated by sensor image formation, our explicit separation of scene-side features and channel responses also provides a compact inductive bias for reusable channel mappings. Its practical value is demonstrated by one-to-multimodal transfer under RGB-only supervision (Sec. 4.4), cross-scene reuse (App. C.4), and few-shot channel adaptation (App. C.5).

Reconstruction and compactness beyond the Scaffold-GS backbone. As suggested, we added per-modality Scaffold-GS to Tables 3, 4, and 6. The table below reports 32-scene average PSNR for this baseline with and without Cross-Modal Alignment. Parentheses indicate each baseline's PSNR difference from Ours.

MethodRGBMonoNIRPolarMS
Scaffold-GS, per modality28.64 (−5.90)26.88 (−5.73)28.06 (−5.95)25.60 (−6.52)27.32 (−5.19)
Scaffold-GS, per modality (w/ Cross-Modal Alignment)31.22 (−3.32)28.28 (−4.33)29.61 (−4.40)27.23 (−4.89)26.08 (−6.43)
Ours34.5432.6134.0132.1232.51

PoseSpecGS retains a 3.32–6.43 dB advantage over Scaffold-GS under the matched pose-processing protocol. The updated storage comparison (Tab. 22) also shows 74.4%–77.7% less model storage across the three representative scenes. These reconstruction and storage advantages persist with the same anchor-based backbone, supporting the benefits of sharing scene parameters across modalities.

AuthorsOpenReview ↗

Response to Reviewer joTY (2/2)

Q1. Latent dimension ablation.

Response:

We thank the reviewer for suggesting this ablation. We evaluated 8D, 16D, 32D, and 64D features on three MMS-DATA scenes under five-modality and RGB+Mono-only settings. The table below reports mean PSNR across the three scenes, with 5M and 2M denoting these two settings. Full results are in Tables 27 and 28 (revised manuscript).

DimensionRGB (5M)Mono (5M)NIR (5M)Polar (5M)MS (5M)RGB (2M)Mono (2M)
8D34.9333.1434.7132.8332.8135.1533.22
16D35.4233.8935.2033.2933.4334.7432.12
32D36.3434.1535.2633.5633.7935.4833.26
64D35.2533.6334.5833.0732.9435.2032.68

With five modalities, 32D achieves the highest mean PSNR in all five groups, outperforming 8D by 0.54–1.40 dB, while increasing to 64D reduces mean PSNR across all five groups. Lower-dimensional features may limit the capacity to encode diverse multimodal appearance, whereas higher-dimensional features introduce additional degrees of freedom that may be harder to constrain with the available supervision. These results support 32D as a suitable default for five-modality reconstruction.

With RGB+Mono only, 32D is not necessary for competitive reconstruction. An 8D representation already provides sufficient capacity, trailing 32D by only 0.34 dB in RGB and 0.04 dB in Mono on average.

Measured storage varies non-monotonically with spectral feature dimension in both settings. The anchor-code dimension remains fixed at 32, following Scaffold-GS [1], while storage primarily depends on the anchor population produced by initialization and stochastic optimization. Additional modalities share this scene representation and add compact response operators, so more modalities do not necessarily require larger model files.

References

[1] Tao Lu et al. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. CVPR, 2024.

Q2. Cross-modal overlap, retrieval robustness, and failed PnP initialization.

Response:

Minimum overlap required. Overlap requirements depend on modality-specific appearance and correspondence quality. On three MMS-DATA scenes, we randomly drop views from all modalities, including RGB. We report per-modality mean 3D FoV IoU within the scene bounding box, pairing each retained auxiliary view with its three nearest retained RGB cameras. At 50% dropout, mean overlap ranges from 39.26% to 51.25% across auxiliary modalities, while 91.0% of the remaining auxiliary views register successfully (Tab. 24, revised manuscript). At 76% dropout, mean overlap falls to 25.02%–33.38%, and per-modality registration success drops to 11.1%–22.2%, showing degradation as cross-view support becomes sparser.

DINOv3 retrieval robustness. For DINOv3 retrieval, we measure whether the top-five RGB candidates contain at least one view reaching a specified MINIMA correspondence threshold. Table 25 (revised manuscript) reports coverage at 200, 500, and 1,000 correspondence pairs. At the 200-pair threshold, coverage remains at least 96.5% across all evaluated dropout rates through 80%, supporting retrieval robustness under substantial view removal.

Initialization failure handling. Across all 32 MMS-DATA scenes, initialization achieves 98.25% overall registration success, with no observed PnP failures. The remaining initialization failures arise during subsequent auxiliary-only bundle adjustment. For a failed view, we use the camera extrinsics of the nearest successfully registered view in DINOv3 feature space within the same scene and modality, providing a fallback initialization for subsequent pose refinement.

Reviewer_cK7yOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

PoseSpecGS tackles the problem of synthesizing novel views from images captured by different sensors. It uses a structure-from-motion reconstruction of the RGB images as a reference for aligning the other cameras, then jointly refines their poses and the scene representation. The method assumes known camera intrinsics and can work without rig extrinsics or synchronized frames across sensors.

For reflected-light modalities, the model shares features across modalities at each Gaussian and uses a compact response operator to render each modality. The authors extend the approach to thermal reconstruction and show that it can transfer to new scenes with RGB supervision alone. They also study how thermal appearance changes over time and how well the model adapts to new channels given only a few observations.

My main reservation is that the conceptual advance over existing feature rendering methods is modest. Even so, the method addresses a practical problem, and the extensive evaluation shows clear gains over the baselines and convincing visual results. On balance, I recommend a weak accept.

Critical strengths
  1. Method and experimental design. The approach to cross-modal alignment and multimodal scene representation is well motivated. The experiments are a good fit for the questions the paper sets out to answer.

  2. Thorough evaluation. The authors compare their method with a range of baselines across several benchmarks. The ablations help clarify how individual components contribute to the overall results.

  3. Strong results. The method shows clear quantitative gains over the baselines. The examples in both the main paper and the supplementary material are also visually convincing.

Critical weaknesses
  1. The conceptual advance is relatively modest. Shared Gaussian features and feature compositing have established precedents. The main advance appears to be a compact, modular response formulation integrated with cross-modal alignment; its conceptual distinction from existing feature-based renderers is less compelling than its empirical effectiveness.

  2. Presentation of the extensions. The paper could explain more clearly how the thermal and cross-scene transfer extensions relate to the core response formulation.

Questions for authors
  1. Could the authors clarify the main methodological advance over existing feature rendering frameworks, beyond the use of a compact linear response decoder?

  2. According to Eq. (6), viewing direction is an input to the geometry decoder but not the spectral feature decoder. Could the authors explain how this design represents specular reflections and other appearance changes with viewing direction?

4 comments
AuthorsOpenReview ↗

Response to Reviewer cK7y (1/2)

We thank Reviewer cK7y for the constructive assessment and recognition of the coherent design, extensive evaluation, and strong reconstruction results of PoseSpecGS. We appreciate the opportunity to clarify the methodological contribution, the relationship between the extensions and our core formulation, and the use of viewing direction.

W1 & Q1. Methodological advance over existing feature rendering frameworks.

Response:

Our main methodological advance lies in an image-formation-inspired factorization of multimodal observations into shared spectral scene features and independently reusable modality-channel response operators, rather than the use of a compact linear decoder alone. This explicit separation provides an inductive bias for learning transferable cross-modal relationships, allowing the learned channel mappings to be reused beyond individual scene representations. Table 8 supports this parameterization through representation ablations.

Cross-scene generalization tests whether the learned mappings remain useful in unseen scenes. As demonstrated in Table 10, our model transfers learned response mappings to unseen scenes, enabling the synthesis of unobserved modalities using only RGB supervision. Our cross-scene reuse (App. C.4) and few-shot channel adaptation (App. C.5) additionally provide evidence that the learned feature representation captures cross-modal information that remains effective beyond its original training observations.

Cross-Modal Alignment enables heterogeneous observations to jointly constrain the shared scene representation without requiring known inter-sensor extrinsics or synchronized frame correspondences, providing the geometric consistency needed to realize the rich multimodal learning capabilities of this factorization. We will clarify these methodological contributions and their supporting evidence in the revised manuscript.

W2. Relationship of the thermal and cross-scene extensions to the core formulation.

Response:

We thank the reviewer for this constructive suggestion. Both extensions build upon the core factorization of scene-side appearance features and modality-channel response operators, while addressing different aspects of multimodal reconstruction.

Thermal and time-varying appearance. Thermal imaging operates in substantially longer infrared wavelength bands beyond the visible and near-infrared bands considered in our reflected-light formulation [1, 2], where observed intensities are primarily governed by thermal emission. These distinct image-formation characteristics motivate the dedicated thermal appearance branch we introduced. At the same time, the thermal branch remains integrated within our unified scene representation, sharing Gaussian anchors, anchor codes, and base geometry while retaining the feature-compositing and response-rendering formulation (Apps. A.4 and B.3). For time-varying thermal scenes, we further introduce time-conditioned anchor codes, allowing thermal appearance to evolve while keeping geometry and response operators time-independent. Thus, the extension accommodates distinct imaging mechanisms and temporal variations within the same scene-response formulation (Tab. 7 and Fig. 6).

Cross-scene transfer. This extension directly leverages the separation between scene-specific features and reusable channel responses. After multimodal pretraining, the shared decoders and response operators are retained and frozen, while only scene-specific parameters are optimized using RGB observations from a new scene. The resulting scene features are then mapped to unobserved modalities through the pretrained response operators, enabling one-to-multimodal transfer without target-scene supervision from those modalities. The results in Table 10 demonstrate that the learned cross-modal relationships remain useful in unseen scenes, connecting cross-scene transfer directly to the core factorization (Sec. 4.4 and App. B.1).

We will clarify these connections in the revised manuscript.

References

[1] Rongfeng Lu et al. ThermalGaussian: Thermal 3D Gaussian Splatting. ICLR, 2025.

[2] Yvette Y. Lin et al. ThermalNeRF: Thermal Radiance Fields. ICCP, 2024.

AuthorsOpenReview ↗

Response to Reviewer cK7y (2/2)

Q2. Viewing direction and the representation of specular reflections.

Response:

We thank the reviewer for catching this omission. The spectral feature decoder does receive viewing direction in our implementation, and this input was inadvertently omitted from Equation (6). The corrected expression is

zj,rr=1R=Fspec(hj,dj).\\{\mathbf{z}_{j,r}\\}_{r=1}^{R} = \mathcal{F}_{\mathrm{spec}}(\mathbf{h}_j, \mathbf{d}_j).

Consequently, the spectral features of each Gaussian can vary with the viewing direction, allowing the rendered channel intensities to represent view-dependent appearance, including specular highlights.

We apologize for the confusion and have corrected Equation (6) in the revised manuscript. This correction to the manuscript description does not change the model used to obtain the reported results.

The authors' response was helpful to assess the paper more clearly. I appreciate it and keep my rating to weak acceptance.

We are glad that our response helped clarify the paper. Thank you for your support and for the thoughtful feedback that helped us improve the manuscript.

Reviewer_S3AiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PoseSpecGS, a 3D Gaussian Splatting (3DGS) framework for multimodal novel-view synthesis (NVS), which constructs a shared 3D scene representation from heterogeneous observations, including RGB, near-infrared (NIR), multispectral (MS), and monochrome (Mono) images. First, the authors introduce Spectrum-Response Rendering, which assigns a 32-dimensional latent spectral feature to each Gaussian and synthesizes modality-specific images through linear response operators. Second, the framework incorporates Cross-Modal Alignment to reduce its dependence on known inter-sensor extrinsics and synchronized frame correspondences. Experiments on four datasets, RGBT-Scenes, MMS-DATA, MS-Splatting, and RHD, demonstrate competitive performance in novel-view synthesis, zero-shot cross-scene transfer, and few-shot channel adaptation.

Critical strengths
  1. The proposed framework jointly addresses shared scene representation learning and cross-modal camera-pose alignment, without requiring known inter-sensor extrinsics or synchronized frame correspondences as inputs.

  2. The modality-channel response operators are compact and efficiently parameterized, potentially reducing overfitting in appearance decoding. This design explicitly separates shared scene features from modality-specific response mappings and demonstrates promising cross-scene transfer capabilities.

  3. The proposed method achieves competitive reconstruction performance across multiple datasets and imaging modalities.

Critical weaknesses
  1. Shared latent feature representations have already been explored in several prior works, and the main distinction of this paper appears to be the introduction of linear response operators. However, the learned spectral features are not constrained by physical wavelengths, spectral bases, or calibrated sensor response functions, making it difficult to establish whether they capture stable or physically meaningful spectral information. Furthermore, since the response operators are linear, their application commutes with alpha compositing before the final nonlinear activation. Consequently, the primary differences lie in parameter sharing and the placement of the final nonlinearity, rather than in a fundamentally new spectral rendering formulation. The authors should better clarify the novelty and representational advantages of this design.

  2. Although the proposed loss may provide useful multiview RGB consistency regularization, it does not independently identify the correct auxiliary-camera pose. In particular, for an accurately reconstructed scene, RGB proxy images rendered from arbitrary candidate camera poses can still achieve low reprojection errors when projected onto reference RGB views with valid overlap. Therefore, it is unclear whether this loss provides meaningful additional constraints on camera poses beyond improving the consistency of the reconstructed scene geometry.

  3. In Tables 1–5, PoseSpecGS benefits from RGB-backbone-based pose initialization, train-time pose refinement, and test-time pose refinement, whereas the competing baselines generally use dataset-provided camera poses without equivalent optimization. This introduces a significant confounding factor in the reported reconstruction improvements. Notably, when a matched pose-refinement protocol is applied in Table 6, the performance gaps between PoseSpecGS and the baselines become substantially smaller. Although the additional controlled experiments in the appendix partially address this concern, the main comparisons do not consistently isolate the benefits of the proposed representation from those of camera-pose optimization.

Questions for authors
  1. How should the proposed Spectrum-Response Rendering mechanism be distinguished from conventional linear feature decoding, and what evidence supports the physical interpretation of the learned response operators? Since the response mapping is linear, the proposed formulation appears mathematically equivalent, before the final nonlinear activation, to applying a shared linear layer to predict modality-specific channel logits for each Gaussian before compositing. Could the authors clarify the essential differences and provide more direct evidence of the advantages of the proposed formulation?

  2. Why does the thermal imaging branch introduce a separate 32-dimensional thermal feature representation? This design appears to weaken the claim of a unified, shared representation across modalities. Could the authors explain how the modality-specific thermal branch is consistent with the proposed unified spectral representation?

  3. What is the specific contribution of cross-modal RGB-guided reprojection consistency to camera-pose refinement? Although the proposed loss improves reconstruction quality, it remains unclear whether these improvements arise from better scene geometry, more accurate camera poses, or their interaction. Could the authors provide experiments that disentangle these effects, ideally including quantitative pose-accuracy evaluations? Additionally, how does the proposed reprojection loss handle occlusions between the proxy and reference views?

  4. Could the authors adopt a consistent camera-pose evaluation protocol for all methods in the main experimental comparisons? The matched-pose evaluation in Table 6 covers only a subset of the MMS-DATA scenes, whereas the main results compare methods under different pose-processing procedures. A more comprehensive comparison under identical pose initialization and refinement settings would be necessary to establish the improvements attributable to the proposed representation itself.

2 comments
AuthorsOpenReview ↗

Response to Reviewer S3Ai (1/2)

We thank Reviewer S3Ai for the detailed feedback and for recognizing the compact response formulation, its reuse potential, and the reconstruction results. We clarify the representation design, the role of reprojection consistency, and the camera-pose evaluation protocols below.

W1 & Q1. Physical motivation and advantages of Spectrum-Response Rendering.

Response:

Physical motivation and supporting evidence. The physical process of sensor image formation motivates our factorization into shared scene-side spectral features and compact modality-channel response operators. Their explicit separation provides an inductive bias for reusing channel mappings across scenes. We learn latent features because recovering continuous spectra from finite measurements is underdetermined, while calibrated sensor responses are difficult to obtain accurately and do not alone capture downstream digital processing [1].

On the MS-Splatting dataset, Table 5 shows improved spectral fidelity over MS-Splatting, with lower spectral angular error (SAM, from 0.109 to 0.094), higher spectral correlation (SCM, from 0.888 to 0.902), and lower divergence between normalized spectral distributions (SID, from 0.024 to 0.017). Together with cross-scene transfer and reuse (Sec. 4.4 and App. C.4), these results support the conclusion that the learned representation captures physically meaningful spectral information that remains useful across scenes.

Novelty and advantages beyond conventional linear decoding. We agree that the linear part of the response mapping commutes with alpha compositing before the final activation. Our design advantage lies in explicitly separating scene features from independently reusable channel mappings. For reflected-light modalities in our factorization, a single alpha-compositing pass produces a shared feature image for all modeled channels, providing a unified rendering interface. This separation supports cross-scene transfer with frozen mappings and additional-channel adaptation with the existing scene representation frozen (App. C.5).

To our knowledge, PoseSpecGS is the first 3DGS-based framework for cross-scene one-to-multimodal synthesis under RGB-only target-scene supervision (Sec. 4.4), demonstrating the practical value of these reuse capabilities. After pretraining on 27 multimodal scenes, PoseSpecGS freezes shared components and fits five unseen scenes using RGB alone, predicting Mono, NIR, and MS. It improves Mono and NIR PSNR over SPoILeR by 0.77 dB and 0.70 dB, respectively, with comparable MS quality (Tab. 10). Table 8 further isolates rendering choices within our model, showing an average PSNR gain of 0.70 dB over a joint response MLP across the five modality groups. Together, these results support our explicit factorization as a compact inductive bias for learning transferable cross-modal relationships.

References

[1] Rei Kawakami et al. Camera Spectral Sensitivity and White Balance Estimation from Sky Images. IJCV, 2013.

Q2. Thermal-specific features within the unified formulation.

Response:

The dedicated thermal branch is motivated by the distinct physical characteristics of thermal imaging. Unlike reflected-light imaging, thermal imaging operates in substantially longer infrared wavelength bands, where observed intensities are primarily governed by thermal emission, motivating separate appearance modeling to improve thermal reconstruction [1, 2]. We therefore introduce separate thermal features to accommodate these differences.

Importantly, the thermal branch remains integrated within our unified scene representation, sharing Gaussian anchors, anchor codes, and base geometry. It also retains the same feature-compositing and channel-response interface. For thermal imaging, unification does not require identical spectral features, but rather enables distinct image-formation mechanisms to be jointly modeled within a shared scene representation.

This shared framework achieves the best average RGB and thermal results on RGBT-Scenes (Tabs. 1–2), with thermal PSNR exceeding PhysIR-Splat by 1.95 dB. For time-varying thermal reconstruction on RHD (Tab. 7), it achieves 43.01 dB, surpassing ETGS by 2.33 dB.

References

[1] Rongfeng Lu et al. ThermalGaussian: Thermal 3D Gaussian Splatting. ICLR, 2025.

[2] Mariam Hassan et al. ThermoNeRF: A multimodal Neural Radiance Field for joint RGB-thermal novel view synthesis of building facades. Advanced Engineering Informatics, 2025.

AuthorsOpenReview ↗

Response to Reviewer S3Ai (2/2)

W2 & Q3. Role of RGB-guided reprojection consistency in joint optimization.

Response:

Joint constraints on geometry and poses. Because geometry and poses are coupled in photometric reconstruction [1], our reprojection term uses shared geometry as the bridge between RGB and auxiliary modalities to regularize their joint optimization against RGB observations. Appearance reconstruction losses alone do not guarantee accurate geometry. In sparse-view 3DGS, photometric optimization can fit input colors while leaving geometrically incorrect floaters [2], motivating additional consistency constraints beyond appearance fitting.

In the ideal case raised by the reviewer, we agree that multiple poses may indeed yield low RGB reprojection errors. However, auxiliary-image losses can still update auxiliary poses and shared geometry during joint optimization. These updates may alter RGB consistency, which the reprojection consistency term regularizes during subsequent optimization.

Quantitative evidence. Table 9 shows that reprojection consistency improves PSNR by 0.51 dB on average across the five modality groups. We also evaluate the complete alignment pipeline against the dataset-provided poses across all 32 MMS-DATA scenes (Tab. 26, revised manuscript). After RGB-based similarity alignment, we measure camera-orientation angular error and camera-center RMSE normalized by the reference RGB trajectory’s RMS radius, termed nATE. Our refined poses achieve 0.179°–1.119° and 1.774%–2.988%, respectively, compared with 2.673°–3.681° and 6.348%–8.254% for Joint SfM across the four auxiliary modalities.

Occlusion handling. We use DINOv3 to select RGB reference views with similar scene content, favoring overlapping viewpoints to reduce occlusion-related mismatches. We further restrict reprojection to valid regions through depth validity, image-boundary, and alpha-support filtering, reducing the influence of unreliable correspondences.

References

[1] Prune Truong et al. SPARF: Neural Radiance Fields From Sparse and Noisy Poses. CVPR, 2023.

[2] Gurutva Patle et al. AD-GS: Alternating Densification for Sparse-Input 3D Gaussian Splatting. SIGGRAPH Asia, 2025.

W3 & Q4. Fair comparisons under matched camera-pose protocols.

Response:

We thank the reviewer for suggesting a more comprehensive controlled comparison. We have expanded Table 6 to all 32 MMS-DATA scenes, including the additional per-modality Scaffold-GS baseline. The comparison also covers all ten RGBT-Scenes (Tab. 19). In these controlled comparisons, all methods use the suggested matched camera initialization and refinement settings. Under matched camera-pose processing, PoseSpecGS achieves the highest PSNR across all evaluated modalities, exceeding the best baseline for each modality by 0.76–2.91 dB on MMS-DATA and by 0.52 dB in RGB and 1.77 dB in thermal on RGBT-Scenes.

Mean PSNR (dB) on all 32 MMS-DATA scenes (Tab. 6).

MethodRGBMonoNIRPolarMS
3DGS, per modality33.7830.1930.1729.5825.44
Scaffold-GS, per modality31.2228.2829.6127.2326.08
MS-Splatting32.1429.6031.1029.5529.74
Ours34.5432.6134.0132.1232.51

Mean PSNR (dB) on all 10 RGBT-Scenes (Tab. 19).

MethodRGBThermal
ThermalGaussian24.3825.60
MMOne24.7825.83
PhysIR-Splat24.4426.48
Ours25.3028.25

We acknowledge that pose refinement substantially improves the baselines, highlighting the value of its contribution across multimodal Gaussian reconstruction methods. Our evaluation covers 59 scenes across four datasets, including matched camera-pose comparisons on all 42 scenes from MMS-DATA and RGBT-Scenes. Tables 8 and 9 further examine the representation and alignment components through ablations. The consistent gains under these controlled settings support the effectiveness of our representation beyond camera-pose processing alone.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We thank the reviewers for recognizing the practical importance of our problem, the compactness and reuse potential of our representation, and our strong reconstruction results, and for their constructive feedback. Below, we highlight the core design and supporting evidence, followed by a concise summary of our responses to the individual concerns.

Design and advantages of Spectrum-Response Rendering

Sensor image formation motivates our explicit separation of shared scene-side spectral features and independently reusable modality-channel response operators. A single alpha-compositing pass produces a shared feature image for all modeled channels, providing a unified rendering interface. Learned channel mappings can be reused while fitting new scenes, and additional channels can be incorporated with the existing scene representation frozen. Cross-Modal Alignment provides the geometric consistency needed for Spectrum-Response Rendering to learn a shared scene representation from heterogeneous observations, without requiring known inter-sensor extrinsics or synchronized frame correspondences.

Across four datasets comprising 59 scenes, PoseSpecGS achieves state-of-the-art multimodal reconstruction performance. More importantly, Spectrum-Response Rendering enables cross-scene transfer and channel adaptation. To our knowledge, PoseSpecGS is the first 3DGS-based method for cross-scene one-to-multimodal transfer under RGB-only target-scene supervision, demonstrated on five unseen scenes after pretraining on 27 multimodal scenes (Sec. 4.4). Cross-scene reuse of frozen mappings (App. C.4) and few-shot channel adaptation (App. C.5) further demonstrate the reusability and extensibility enabled by this design. Improved spectral fidelity in Table 5 provides complementary evidence that the reconstructed channels preserve spectral structure. These results support the factorization as a compact inductive bias for learning transferable cross-modal information.

Responses to specific concerns

  • Controlled comparisons and backbone effects (Reviewer S3Ai W3/Q4, Reviewer joTY W2). We expanded the matched-pose MMS-DATA comparison to all 32 scenes, highlighted the ten-scene RGBT controlled evaluation, and added per-modality Scaffold-GS reconstruction and storage comparisons (Tabs. 3, 4, 6, 19, and 22).
  • Direct pose evaluation (Reviewer joTY W1, Reviewer S3Ai Q3). We added comparisons against dataset-provided poses across all 32 MMS-DATA scenes, where our final poses outperform both SfM baselines in aligned rotation error and normalized camera-position error across all auxiliary modalities (Tab. 26, revised manuscript).
  • Reprojection consistency (Reviewer S3Ai W2/Q3). We clarified its role in joint scene–pose optimization through shared geometry, explained the correspondence filtering used to mitigate occlusion-related mismatches, and discussed the reconstruction ablation in Table 9.
  • Overlap and registration robustness (Reviewer joTY Q2). We quantified view overlap and per-modality registration success under view dropout, evaluated DINOv3 retrieval coverage, and explained initialization failure handling (Tabs. 24–25, revised manuscript).
  • Feature dimension ablation (Reviewer joTY Q1). We added 8D, 16D, 32D, and 64D ablations reporting reconstruction quality and storage under five-modality and RGB+Mono supervision, supporting 32D for the former and showing competitive results with 8D for the latter (Tabs. 27–28, revised manuscript).
  • Thermal and cross-scene extensions (Reviewer S3Ai Q2, Reviewer cK7y W2). We clarified the shared and modality-specific components of thermal modeling and how cross-scene transfer builds on the scene–response separation (Secs. 4.3–4.4 and Apps. A.4, B.1, and B.3).
  • Viewing-direction input (Reviewer cK7y Q2). We corrected the omitted viewing-direction input in Equation (6), matching the model used for the reported experiments.

We appreciate the Area Chair's and reviewers' consideration and welcome further questions or discussion.

Best,

Authors

Dear Area Chair and Reviewers,

We have provided detailed responses to the reviewers' concerns, additional experiments, and a General Response summarizing our contributions and supporting evidence. Reviewer cK7y found our response helpful and maintained the weak-acceptance recommendation.

As the discussion period draws to a close, we would welcome the opportunity for further discussion to understand and clarify any remaining concerns. We remain available to respond promptly.

Thank you for your time and consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.