- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents GeMI3D, which converts RGB images, depth maps, and point clouds into shared 3D representations. Through staged training, feature anchoring, and modality dropout, a single detector supports seven input combinations. The authors also construct TriScan3D from ScanNet++ to evaluate inputs with distinct geometric sources. The model achieves competitive results on both datasets with complete inputs.
Critical strengths
- The same model parameters support all seven input combinations without retraining, making the approach useful when input configurations change.
- TriScan3D uses distinct depth and point-cloud sources with scene-disjoint splits, providing a useful setting for evaluating varying input combinations.
- With complete inputs, the model outperforms the baselines listed in Table 1 on both mAP25 and mAP50 across the two datasets.
Critical weaknesses
- As acknowledged in Appendix D, Section 3.3 selects the highest-priority available reference using P>D>I without considering sample-specific input quality. The empirical effect of this choice when the point cloud is sparse or noisy remains unclear.
- Table 2 compares fixed-input detectors by zeroing missing inputs. Although Section 2 discusses OV-Uni3DETR, no experimental comparison with a modality-switching method is provided. Comparing the shared I, P, and I+P settings under matched training and evaluation conditions would help outperform this prior work.
- Table 7 shows that additional R50-FPN and DINOv2 image-semantic features raise complete-input mAP25 from 66.2 to 68.6. Although this ablation quantifies their contribution, their integration is described only in Appendix A.3 and should be made clear in the main method description.
Questions for authors
- How much does GeMI3D improve over a geometric-conversion, shared-detector, and modality-dropout baseline using the same backbone, image features, and training budget?
I would appreciate clarification on the concerns raised in the Weaknesses section.
4 comments
Point to Point Response to Reviewer tyjb (2/2)
W2: Modality-switching comparisons under shared input settings.
We add comparisons in the shared I, P, and I+P settings, evaluating each method’s final multimodal checkpoint. (1) We reproduce OV-Uni3DETR for closed-vocabulary modality switching on SUN RGB-D, where GeMI3D achieves higher mAP${}_{25}$ in all three settings. (2) We retrain TR3D+FF and IIFNet3D with modality dropout so that missing-modality configurations are also included during training. Both methods still obtain 0.0 mAP${}_{25}$ with I-only inputs on both datasets.
(1) Comparison with OV-Uni3DETR. The official OV-Uni3DETR repository provides open-vocabulary configurations but no dedicated recipe for reproducing its closed-vocabulary results. We therefore undertook additional reproduction work to evaluate OV-Uni3DETR in our closed-vocabulary setting on SUN RGB-D. We first train the image and point-cloud branches separately and perform multimodal training. We then evaluate its modality-switching model under the same I, P, and I+P settings. It achieves 15.2/61.4/66.6 mAP${}_{25}$ on SUN RGB-D, compared with 31.2/64.5/68.4 for GeMI3D. GeMI3D performs better in all three settings, with the largest gain in I-only (+16.0). Beyond the result, the methods differ in where inputs are unified: OV-Uni3DETR uses separate image and point-cloud feature extractors, whereas GeMI3D converts heterogeneous inputs into a common 3D point representation before shared hub encoding. This gives the image pathway explicit depth-derived geometry and access to representations learned from point clouds, which may contribute to the stronger I-only performance of our final unified model.
(2) Training fixed-input baselines for modality switching. We retrain TR3D+FF and IIFNet3D by sampling I+P, P, and I with probabilities of 50%, 25%, and 25%. TR3D+FF achieves 70.1/65.4/0.0 on SUN RGB-D and 70.0/68.9/0.0 on TriScan3D; IIFNet3D achieves 65.6/63.3/0.0 and 64.5/64.2/0.0. Despite modality-dropout training, both methods retain zero I-only performance. Their image branches augment point-cloud detection rather than provide an independent detection pathway, so adding dropout does not resolve this architectural dependence. In contrast, GeMI3D converts each available modality into a common 3D representation for shared encoding and detection, supporting all seven input combinations, including those without point clouds.
By the way, on SUN RGB-D, using DA v2-Large as the depth estimator increases GeMI3D’s I+P to 70.3, exceeding TR3D+FF with modality dropout (70.1), and further improves I-only performance to 45.4.
| Method | Training setting | SUN RGB-D I | SUN RGB-D P | SUN RGB-D I+P | TriScan3D I | TriScan3D P | TriScan3D I+P |
|---|---|---|---|---|---|---|---|
| TR3D+FF | w/o dropout | 0.0 | 8.4 | 69.2 | 0.0 | 50.3 | 69.2 |
| TR3D+FF | w/ dropout | 0.0 | 65.4 | 70.1 | 0.0 | 68.9 | 70.0 |
| IIFNet3D | w/o dropout | 0.0 | 63.7 | 68.1 | 0.0 | 66.6 | 67.2 |
| IIFNet3D | w/ dropout | 0.0 | 63.3 | 65.6 | 0.0 | 64.2 | 64.5 |
| GeMI3D | w/ dropout | 31.2 | 64.5 | 68.4 | 19.9 | 68.3 | 69.9 |
W3: Clarification of image-semantic feature integration in the main method.
Due to space constraints, these details were included in Appendix A.3. We agree that they would be better placed in the main method and we have revised Section 3.3 to clarify how DINOv2 and R50-FPN features are integrated.
Q1: Gains over a geometric-conversion baseline.
We introduce a controlled baseline that retains geometric conversion, the shared detector, and modality dropout, but replaces staged training with single-stage joint training. Both methods use the same backbone, image features, and total training budget.
The baseline achieves 66.5/49.1 mAP${}_{25}{}_{50}$, while GeMI3D achieves 68.6/52.3, yielding absolute improvements of 2.1 and 3.2 percentage points, respectively. This comparison shows that staged training provides additional gains beyond the combination of geometric conversion, a shared detector, and modality dropout.
Thank you for the additional experiments. Under severe point-cloud noise on TriScan3D, the new selection rule still performs worse than using I+D alone. I remain concerned about its ability to identify when the point cloud should be excluded, and will retain my original score.
(1) Clarification of the main focus.
Our primary objective is inference-time flexibility under missing modalities: enabling a single trained detector to accommodate different combinations of available inputs. As stated in Sec. 1 (Line 52), “We therefore study modality-independent indoor 3D object detection, which aims to use a single detector across different combinations of available image, depth, and point-cloud inputs without architectural changes or retraining.”
This objective distinguishes modality availability from measurement reliability. Our central contribution is a unified geometric representation that supports shared detection across heterogeneous input configurations. While this representation enables flexible inference when some modalities are unavailable, that capability alone does not establish broad robustness to corruption within the available modalities. In response to the reviewer’s suggestion, we added corruption experiments and a quality-aware selection rule to investigate this complementary question. These additions extend the evaluation to degraded-input conditions, with any resulting robustness conclusions restricted to the corruption settings examined.
(2) Robustness evidence on SUN RGB-D.
It is unfair to overlook the robustness results that we have reported on SUN RGB-D while placing disproportionate emphasis on the sensitivity to point-cloud noise observed on TriScan3D. On SUN RGB-D, our previously reported selection result reaches 68.0 mAP${}_{25}$ at noise level 5, matching I+D and outperforming fixed I+D+P (44.6).
(3) Better performance on TriScan3D.
(i) Threshold purpose and selection. The threshold is used to identify noisy point clouds and exclude them at inference. Our initial threshold choice 0.18 aims to preserve accuracy at best under mild noise while still recovering accuracy under severe noise. To address the current concern more directly, we tested a lower gating threshold and found that it recovered more accuracy under severe noise with only little loss under mild noise.
(ii) Results of a lower threshold. With a threshold of 0.11, the gate raises mAP${}_{25}$ at noise level 5 from 11.7 with fixed I+D+P to 48.4, close to the result of using I+D (48.6). At levels 3 and 4, the gate achieves 50.6 and 49.0, outperforming I+D alone, suggesting that the gate retains point clouds that still contribute useful geometry rather than simply switching to I+D. At level 1, it achieves 69.7, compared with 69.8 for fixed I+D+P. Thus, this simple rule preserves the benefit of point clouds under mild noise, outperforms I+D alone at intermediate noise levels, and approaches I+D performance under severe noise on TriScan3D.
(iii) Pronounced sensitivity to point-cloud noise on TriScan3D. On TriScan3D, the mesh-derived point clouds provide detailed geometric cues that are not fully captured by RGB images and sensor depth. During training, the model can learn to use these point-cloud-specific details for detection. Severe point-cloud noise can therefore disrupt these cues at inference and may lead to a larger drop in detection performance. Thus, although TriScan3D’s point clouds provide high-quality geometry under clean conditions, severe noise can disrupt the additional cues that support detection, making this dataset a particularly demanding benchmark for evaluating robustness to point-cloud corruption.
Point to Point Response to Reviewer tyjb (1/2)
W1: Fixed reference-path selection under sparse or noisy point clouds.
Our primary focus is flexible inference with different subsets of available modalities. To further evaluate robustness when the point cloud is sparse or noisy, in Appendix B.4, we added corruption experiments and evaluated a quality-aware input selection rule. We find that (i) severe coordinate noise is more damaging than point removal; (ii) quality-aware selection recovers accuracy under severe point-cloud noise with little change under mild noise.
(i) Robustness to sparse and noisy point clouds.
We tested random sparsification, local cutout, and Gaussian coordinate noise on both datasets (Table 8). At level 5, random sparsification, local cutout, and Gaussian coordinate noise yield 67.2, 64.3, and 44.6 mAP${}_{25}$ on SUN RGB-D (clean: 68.6), and 69.6, 65.3, and 11.7 on TriScan3D (clean: 70.7). On both datasets, random sparsification and local cutout have relatively small effects on detection accuracy, whereas Gaussian coordinate noise causes a much larger drop. Unlike point removal, which preserves the coordinates of the remaining points, Gaussian noise distorts local geometry and can make the point-cloud path a less reliable reference for fusion, causing larger degradation.
(ii) Quality-aware input selection.
We then use a per-frame roughness score to decide whether P participates in fusion. In this experiment, we evaluate the rule when all three modalities are available: if P is excluded, the model switches from I+D+P to I+D, with D as the reference. This requires no retraining or second detection pass and incurs only a modest overhead, which can be offset when skipping the point-cloud path (Appendix B.4). At Gaussian noise level 1, the gate changes mAP${}_{25}$ by less than 0.1 point on either dataset. At level 5 of Gaussian noise, the gate switches to I+D on 99.6% of SUN RGB-D frames and 78.1% of TriScan3D frames, improving mAP${}_{25}$ over fixed I+D+P from 44.6 to 68.0 and from 11.7 to 38.9 (Table 10). Thus, the gate largely preserves accuracy under mild noise while recovering part of the loss under severe noise.
Based on the above analysis and the reviewer’s suggestion, we will adopt quality-aware selection as an optimized version of reference-path selection to improve the module’s robustness under higher noise. The revised version is provided in Section 3.3 of the main text.