- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents PoseSpecGS, a multimodal fusion method that integrates heterogeneous sensor observations into a unified scene representation for multimodal novel-view synthesis. It combines Cross-Modal Alignment and Spectrum-Response Rendering to enable fusion without known sensor-rig extrinsics or synchronization. Experiments demonstrate strong reconstruction quality and compact representations.
Critical strengths
-
Important and Practical Problem: The paper addresses the challenging problem of multimodal 3D reconstruction without requiring known sensor-rig extrinsics or synchronized observations.
-
Strong Experimental Results: The method achieves promising reconstruction quality across multiple modalities, including time-varying RGB–thermal scenes, while demonstrating compactness and cross-scene generalization.
Critical weaknesses
-
Insufficient Validation of Cross-Modal Alignment: Despite being a major contribution, the paper lacks direct registration accuracy evaluation. Appendix C.7 reports rig consistency rather than errors against ground-truth poses. The authors question the reliability of dataset-provided poses, but provide insufficient evidence to justify excluding them as a reference. It remains unclear whether the proposed refinement improves camera alignment or merely compensates for pose errors through scene optimization.
-
Unclear Independent Benefits of Spectrum-Response Rendering: The performance gap with MS-Splatting decreases substantially from Table 3 to Table 6 when similar camera pose protocols are used, raising questions about the independent contribution of Spectrum-Response Rendering. Moreover, the compactness evaluation lacks a per-modality Scaffold-GS baseline, making it difficult to distinguish the proposed representation's benefits from those of its underlying backbone.
Questions for authors
In addition to the concerns raised in the Weaknesses section, I have the following questions:
-
Latent Dimension Ablation: Could the authors provide an ablation on the 32D spectral feature dimension (e.g., 8, 16, 32, 64), evaluating reconstruction quality and model storage? Is 32D still necessary when only RGB and monochrome observations are available?
-
Cross-Modal Overlap and Registration Robustness: The method assumes sufficient cross-modal overlap without quantitatively defining it. What minimum overlap is required, and does it vary across modalities? How robust is DINOv3 retrieval to cross-modal appearance differences, and how are failed PnP initializations handled?
2 comments
Response to Reviewer joTY (2/2)
Q1. Latent dimension ablation.
Response:
We thank the reviewer for suggesting this ablation. We evaluated 8D, 16D, 32D, and 64D features on three MMS-DATA scenes under five-modality and RGB+Mono-only settings. The table below reports mean PSNR across the three scenes, with 5M and 2M denoting these two settings. Full results are in Tables 27 and 28 (revised manuscript).
| Dimension | RGB (5M) | Mono (5M) | NIR (5M) | Polar (5M) | MS (5M) | RGB (2M) | Mono (2M) |
|---|---|---|---|---|---|---|---|
| 8D | 34.93 | 33.14 | 34.71 | 32.83 | 32.81 | 35.15 | 33.22 |
| 16D | 35.42 | 33.89 | 35.20 | 33.29 | 33.43 | 34.74 | 32.12 |
| 32D | 36.34 | 34.15 | 35.26 | 33.56 | 33.79 | 35.48 | 33.26 |
| 64D | 35.25 | 33.63 | 34.58 | 33.07 | 32.94 | 35.20 | 32.68 |
With five modalities, 32D achieves the highest mean PSNR in all five groups, outperforming 8D by 0.54–1.40 dB, while increasing to 64D reduces mean PSNR across all five groups. Lower-dimensional features may limit the capacity to encode diverse multimodal appearance, whereas higher-dimensional features introduce additional degrees of freedom that may be harder to constrain with the available supervision. These results support 32D as a suitable default for five-modality reconstruction.
With RGB+Mono only, 32D is not necessary for competitive reconstruction. An 8D representation already provides sufficient capacity, trailing 32D by only 0.34 dB in RGB and 0.04 dB in Mono on average.
Measured storage varies non-monotonically with spectral feature dimension in both settings. The anchor-code dimension remains fixed at 32, following Scaffold-GS [1], while storage primarily depends on the anchor population produced by initialization and stochastic optimization. Additional modalities share this scene representation and add compact response operators, so more modalities do not necessarily require larger model files.
References
[1] Tao Lu et al. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. CVPR, 2024.
Q2. Cross-modal overlap, retrieval robustness, and failed PnP initialization.
Response:
Minimum overlap required. Overlap requirements depend on modality-specific appearance and correspondence quality. On three MMS-DATA scenes, we randomly drop views from all modalities, including RGB. We report per-modality mean 3D FoV IoU within the scene bounding box, pairing each retained auxiliary view with its three nearest retained RGB cameras. At 50% dropout, mean overlap ranges from 39.26% to 51.25% across auxiliary modalities, while 91.0% of the remaining auxiliary views register successfully (Tab. 24, revised manuscript). At 76% dropout, mean overlap falls to 25.02%–33.38%, and per-modality registration success drops to 11.1%–22.2%, showing degradation as cross-view support becomes sparser.
DINOv3 retrieval robustness. For DINOv3 retrieval, we measure whether the top-five RGB candidates contain at least one view reaching a specified MINIMA correspondence threshold. Table 25 (revised manuscript) reports coverage at 200, 500, and 1,000 correspondence pairs. At the 200-pair threshold, coverage remains at least 96.5% across all evaluated dropout rates through 80%, supporting retrieval robustness under substantial view removal.
Initialization failure handling. Across all 32 MMS-DATA scenes, initialization achieves 98.25% overall registration success, with no observed PnP failures. The remaining initialization failures arise during subsequent auxiliary-only bundle adjustment. For a failed view, we use the camera extrinsics of the nearest successfully registered view in DINOv3 feature space within the same scene and modality, providing a fallback initialization for subsequent pose refinement.
Response to Reviewer joTY (1/2)
We thank Reviewer joTY for the detailed feedback and for recognizing the practical importance of the problem and the strong reconstruction results. We address the concerns about alignment accuracy, representation benefits, feature dimensionality, and registration robustness below.
W1. Direct validation of Cross-Modal Alignment.
Response:
Direct quantitative evaluation. We agree that reconstruction quality alone cannot establish camera-pose accuracy. As suggested, we evaluate our refined poses and two baselines, Joint SfM and Per-modality SfM + Sim(3), against the dataset-provided poses across all 32 MMS-DATA scenes (Tab. 26, revised manuscript). After a single RGB-based similarity alignment per scene, we measure the mean angular error of aligned camera orientations and nATE, the camera-center RMSE normalized by the reference RGB trajectory’s RMS radius and expressed as a percentage. PoseSpecGS achieves 0.179°–1.119° rotation error and 1.774%–2.988% nATE, outperforming both SfM baselines across all four auxiliary modalities. This reference-based evaluation provides additional evidence of pose alignment quality, complementing the rig-consistency analysis in Appendix C.7.
Rationale for the original evaluation. Reliable ground-truth camera poses are difficult to obtain for real-world multimodal captures, and dataset-provided calibration may retain residual alignment errors. Our concern about reference-pose accuracy arose from visible reconstruction artifacts with the dataset-provided poses and substantial improvements in baseline reconstruction after applying Cross-Modal Alignment (Fig. 7, Tabs. 4 and 6). The original submission therefore focused on rig consistency and reconstruction quality, which we now supplement with the direct evaluations above.
W2. Independent benefits of Spectrum-Response Rendering and comparison with Scaffold-GS.
Response:
Independent contribution of Spectrum-Response Rendering. We agree that applying our camera initialization and pose refinement substantially improves MS-Splatting, demonstrating the effectiveness of Cross-Modal Alignment. To examine the independent benefits of Spectrum-Response Rendering, we compare the two methods across all 32 MMS-DATA scenes under a matched camera-pose protocol (Tab. 6). PoseSpecGS still outperforms MS-Splatting by 2.40–3.01 dB across all five modality groups, which is a considerable improvement supporting the contribution of our representation beyond camera alignment. Parentheses below indicate PSNR gains over MS-Splatting.
Table 8 further examines the rendering design through within-model ablations. Our formulation achieves higher PSNR across all five modality groups, with average gains of 1.42 dB over intensity compositing and 0.70 dB over a joint response MLP. Together, the controlled comparison and internal ablations support the independent contribution of Spectrum-Response Rendering.
Motivated by sensor image formation, our explicit separation of scene-side features and channel responses also provides a compact inductive bias for reusable channel mappings. Its practical value is demonstrated by one-to-multimodal transfer under RGB-only supervision (Sec. 4.4), cross-scene reuse (App. C.4), and few-shot channel adaptation (App. C.5).
Reconstruction and compactness beyond the Scaffold-GS backbone. As suggested, we added per-modality Scaffold-GS to Tables 3, 4, and 6. The table below reports 32-scene average PSNR for this baseline with and without Cross-Modal Alignment. Parentheses indicate each baseline's PSNR difference from Ours.
PoseSpecGS retains a 3.32–6.43 dB advantage over Scaffold-GS under the matched pose-processing protocol. The updated storage comparison (Tab. 22) also shows 74.4%–77.7% less model storage across the three representative scenes. These reconstruction and storage advantages persist with the same anchor-based backbone, supporting the benefits of sharing scene parameters across modalities.