acceptodds
Under review as a conference paper at ICLR 2027

GeMI3D: Geometry-Grounded Modality-Independent Indoor 3D Object Detection

Abstract

Indoor 3D object detection localizes and recognizes objects in 3D space from images, depth maps, point clouds, or their combinations, but existing detectors are typically designed for predefined input configurations. When the available inputs change, these detectors may require architectural changes or additional training. We therefore study modality-independent indoor 3D object detection, where a single detector handles any combination of image, depth, and point-cloud inputs. To this end, we propose GEMI3D, a geometry-grounded modality-independent framework for indoor 3D object detection, which maps heterogeneous inputs into a stable 3D geometric hub. First, Cross-Modal Hub Construction learns a stable spatial representation from point-cloud data and then extends the geometry-grounded hub with depth and image inputs mapped into the same 3D space, while feature anchoring preserves the point-cloud representation. Second, Modality-Conditioned Hub Adaptation uses modality dropout to expose the detector to different input combinations and fuses the available hub features for a modality-agnostic detection head. To comprehensively evaluate modality independence with inputs from distinct sources, we further construct TriScan3D, a frame-level benchmark derived from ScanNet++. With all modalities, GEMI3D achieves 68.6/52.3 mAP25/mAP50 on SUN RGB-D and 70.7/51.4 on TriScan3D. Across all seven non-empty input combinations, it maintains robust detection performance without retraining. The code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Indoor 3D Object Detection, Modality Independence, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tyjbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents GeMI3D, which converts RGB images, depth maps, and point clouds into shared 3D representations. Through staged training, feature anchoring, and modality dropout, a single detector supports seven input combinations. The authors also construct TriScan3D from ScanNet++ to evaluate inputs with distinct geometric sources. The model achieves competitive results on both datasets with complete inputs.

Critical strengths
  1. The same model parameters support all seven input combinations without retraining, making the approach useful when input configurations change.
  2. TriScan3D uses distinct depth and point-cloud sources with scene-disjoint splits, providing a useful setting for evaluating varying input combinations.
  3. With complete inputs, the model outperforms the baselines listed in Table 1 on both mAP25 and mAP50 across the two datasets.
Critical weaknesses
  1. As acknowledged in Appendix D, Section 3.3 selects the highest-priority available reference using P>D>I without considering sample-specific input quality. The empirical effect of this choice when the point cloud is sparse or noisy remains unclear.
  2. Table 2 compares fixed-input detectors by zeroing missing inputs. Although Section 2 discusses OV-Uni3DETR, no experimental comparison with a modality-switching method is provided. Comparing the shared I, P, and I+P settings under matched training and evaluation conditions would help outperform this prior work.
  3. Table 7 shows that additional R50-FPN and DINOv2 image-semantic features raise complete-input mAP25 from 66.2 to 68.6. Although this ablation quantifies their contribution, their integration is described only in Appendix A.3 and should be made clear in the main method description.
Questions for authors
  1. How much does GeMI3D improve over a geometric-conversion, shared-detector, and modality-dropout baseline using the same backbone, image features, and training budget?

I would appreciate clarification on the concerns raised in the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Point to Point Response to Reviewer tyjb (1/2)

W1: Fixed reference-path selection under sparse or noisy point clouds.

Our primary focus is flexible inference with different subsets of available modalities. To further evaluate robustness when the point cloud is sparse or noisy, in Appendix B.4, we added corruption experiments and evaluated a quality-aware input selection rule. We find that (i) severe coordinate noise is more damaging than point removal; (ii) quality-aware selection recovers accuracy under severe point-cloud noise with little change under mild noise.

(i) Robustness to sparse and noisy point clouds.

We tested random sparsification, local cutout, and Gaussian coordinate noise on both datasets (Table 8). At level 5, random sparsification, local cutout, and Gaussian coordinate noise yield 67.2, 64.3, and 44.6 mAP${}_{25}$ on SUN RGB-D (clean: 68.6), and 69.6, 65.3, and 11.7 on TriScan3D (clean: 70.7). On both datasets, random sparsification and local cutout have relatively small effects on detection accuracy, whereas Gaussian coordinate noise causes a much larger drop. Unlike point removal, which preserves the coordinates of the remaining points, Gaussian noise distorts local geometry and can make the point-cloud path a less reliable reference for fusion, causing larger degradation.

CorruptionInputsSUN RGB-D: Level 5TriScan3D: Level 5
Random sparsificationI+D+P67.269.6
Local cutoutI+D+P64.365.3
Gaussian noiseI+D+P44.611.7

(ii) Quality-aware input selection.

We then use a per-frame roughness score to decide whether P participates in fusion. In this experiment, we evaluate the rule when all three modalities are available: if P is excluded, the model switches from I+D+P to I+D, with D as the reference. This requires no retraining or second detection pass and incurs only a modest overhead, which can be offset when skipping the point-cloud path (Appendix B.4). At Gaussian noise level 1, the gate changes mAP${}_{25}$ by less than 0.1 point on either dataset. At level 5 of Gaussian noise, the gate switches to I+D on 99.6% of SUN RGB-D frames and 78.1% of TriScan3D frames, improving mAP${}_{25}$ over fixed I+D+P from 44.6 to 68.0 and from 11.7 to 38.9 (Table 10). Thus, the gate largely preserves accuracy under mild noise while recovering part of the loss under severe noise.

DatasetNoise levelFixedGatedSwitch (%)
SUN RGB-D167.267.11.0
SUN RGB-D544.668.099.6
TriScan3D169.869.80.4
TriScan3D511.738.978.1

Based on the above analysis and the reviewer’s suggestion, we will adopt quality-aware selection as an optimized version of reference-path selection to improve the module’s robustness under higher noise. The revised version is provided in Section 3.3 of the main text.

AuthorsOpenReview ↗

Point to Point Response to Reviewer tyjb (2/2)

W2: Modality-switching comparisons under shared input settings.

We add comparisons in the shared I, P, and I+P settings, evaluating each method’s final multimodal checkpoint. (1) We reproduce OV-Uni3DETR for closed-vocabulary modality switching on SUN RGB-D, where GeMI3D achieves higher mAP${}_{25}$ in all three settings. (2) We retrain TR3D+FF and IIFNet3D with modality dropout so that missing-modality configurations are also included during training. Both methods still obtain 0.0 mAP${}_{25}$ with I-only inputs on both datasets.

(1) Comparison with OV-Uni3DETR. The official OV-Uni3DETR repository provides open-vocabulary configurations but no dedicated recipe for reproducing its closed-vocabulary results. We therefore undertook additional reproduction work to evaluate OV-Uni3DETR in our closed-vocabulary setting on SUN RGB-D. We first train the image and point-cloud branches separately and perform multimodal training. We then evaluate its modality-switching model under the same I, P, and I+P settings. It achieves 15.2/61.4/66.6 mAP${}_{25}$ on SUN RGB-D, compared with 31.2/64.5/68.4 for GeMI3D. GeMI3D performs better in all three settings, with the largest gain in I-only (+16.0). Beyond the result, the methods differ in where inputs are unified: OV-Uni3DETR uses separate image and point-cloud feature extractors, whereas GeMI3D converts heterogeneous inputs into a common 3D point representation before shared hub encoding. This gives the image pathway explicit depth-derived geometry and access to representations learned from point clouds, which may contribute to the stronger I-only performance of our final unified model.

(2) Training fixed-input baselines for modality switching. We retrain TR3D+FF and IIFNet3D by sampling I+P, P, and I with probabilities of 50%, 25%, and 25%. TR3D+FF achieves 70.1/65.4/0.0 on SUN RGB-D and 70.0/68.9/0.0 on TriScan3D; IIFNet3D achieves 65.6/63.3/0.0 and 64.5/64.2/0.0. Despite modality-dropout training, both methods retain zero I-only performance. Their image branches augment point-cloud detection rather than provide an independent detection pathway, so adding dropout does not resolve this architectural dependence. In contrast, GeMI3D converts each available modality into a common 3D representation for shared encoding and detection, supporting all seven input combinations, including those without point clouds.

By the way, on SUN RGB-D, using DA v2-Large as the depth estimator increases GeMI3D’s I+P to 70.3, exceeding TR3D+FF with modality dropout (70.1), and further improves I-only performance to 45.4.

MethodTraining settingSUN RGB-D ISUN RGB-D PSUN RGB-D I+PTriScan3D ITriScan3D PTriScan3D I+P
TR3D+FFw/o dropout0.08.469.20.050.369.2
TR3D+FFw/ dropout0.065.470.10.068.970.0
IIFNet3Dw/o dropout0.063.768.10.066.667.2
IIFNet3Dw/ dropout0.063.365.60.064.264.5
GeMI3Dw/ dropout31.264.568.419.968.369.9

W3: Clarification of image-semantic feature integration in the main method.

Due to space constraints, these details were included in Appendix A.3. We agree that they would be better placed in the main method and we have revised Section 3.3 to clarify how DINOv2 and R50-FPN features are integrated.

Q1: Gains over a geometric-conversion baseline.

We introduce a controlled baseline that retains geometric conversion, the shared detector, and modality dropout, but replaces staged training with single-stage joint training. Both methods use the same backbone, image features, and total training budget.

The baseline achieves 66.5/49.1 mAP${}_{25}/mAP/mAP{}_{50}$, while GeMI3D achieves 68.6/52.3, yielding absolute improvements of 2.1 and 3.2 percentage points, respectively. This comparison shows that staged training provides additional gains beyond the combination of geometric conversion, a shared detector, and modality dropout.

Thank you for the additional experiments. Under severe point-cloud noise on TriScan3D, the new selection rule still performs worse than using I+D alone. I remain concerned about its ability to identify when the point cloud should be excluded, and will retain my original score.

(1) Clarification of the main focus.

Our primary objective is inference-time flexibility under missing modalities: enabling a single trained detector to accommodate different combinations of available inputs. As stated in Sec. 1 (Line 52), “We therefore study modality-independent indoor 3D object detection, which aims to use a single detector across different combinations of available image, depth, and point-cloud inputs without architectural changes or retraining.”

This objective distinguishes modality availability from measurement reliability. Our central contribution is a unified geometric representation that supports shared detection across heterogeneous input configurations. While this representation enables flexible inference when some modalities are unavailable, that capability alone does not establish broad robustness to corruption within the available modalities. In response to the reviewer’s suggestion, we added corruption experiments and a quality-aware selection rule to investigate this complementary question. These additions extend the evaluation to degraded-input conditions, with any resulting robustness conclusions restricted to the corruption settings examined.

(2) Robustness evidence on SUN RGB-D.

It is unfair to overlook the robustness results that we have reported on SUN RGB-D while placing disproportionate emphasis on the sensitivity to point-cloud noise observed on TriScan3D. On SUN RGB-D, our previously reported selection result reaches 68.0 mAP${}_{25}$ at noise level 5, matching I+D and outperforming fixed I+D+P (44.6).

(3) Better performance on TriScan3D.

(i) Threshold purpose and selection. The threshold is used to identify noisy point clouds and exclude them at inference. Our initial threshold choice 0.18 aims to preserve accuracy at best under mild noise while still recovering accuracy under severe noise. To address the current concern more directly, we tested a lower gating threshold and found that it recovered more accuracy under severe noise with only little loss under mild noise.

(ii) Results of a lower threshold. With a threshold of 0.11, the gate raises mAP${}_{25}$ at noise level 5 from 11.7 with fixed I+D+P to 48.4, close to the result of using I+D (48.6). At levels 3 and 4, the gate achieves 50.6 and 49.0, outperforming I+D alone, suggesting that the gate retains point clouds that still contribute useful geometry rather than simply switching to I+D. At level 1, it achieves 69.7, compared with 69.8 for fixed I+D+P. Thus, this simple rule preserves the benefit of point clouds under mild noise, outperforms I+D alone at intermediate noise levels, and approaches I+D performance under severe noise on TriScan3D.

(iii) Pronounced sensitivity to point-cloud noise on TriScan3D. On TriScan3D, the mesh-derived point clouds provide detailed geometric cues that are not fully captured by RGB images and sensor depth. During training, the model can learn to use these point-cloud-specific details for detection. Severe point-cloud noise can therefore disrupt these cues at inference and may lead to a larger drop in detection performance. Thus, although TriScan3D’s point clouds provide high-quality geometry under clean conditions, severe noise can disrupt the additional cues that support detection, making this dataset a particularly demanding benchmark for evaluating robustness to point-cloud corruption.

Reviewer_YMBtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies indoor 3D object detection with a single checkpoint that supports all seven non-empty subsets of RGB, depth, and point-cloud inputs. GeMI3D converts the available inputs into 3D point representations and processes them with a shared PTv3 encoder. CMHC first trains on point clouds and then incorporates image and depth inputs with feature anchoring. A subsequent MCHA stage uses modality dropout and reference-guided fusion before a shared detection head. The authors also introduce TriScan3D, which pairs iPhone RGB-D observations with point clouds raycast from reconstructed ScanNet++ meshes.

Critical strengths
  1. The problem is practically relevant. Supporting different input combinations with one checkpoint could simplify deployment across sensor configurations, and the paper explicitly evaluates all seven subsets.
  2. The shared geometric representation provides a coherent framework for integrating heterogeneous inputs. TriScan3D also offers a useful evaluation setting with distinct depth and point-cloud sources and a well-documented construction pipeline.
Critical weaknesses
  1. The comparisons do not sufficiently isolate the benefit of the proposed method. Table 2(a) mainly shows that fixed-input detectors fail when their required point-cloud input is removed. It does not establish an advantage over methods trained for missing modalities. Comparisons with matched modality-dropout baselines and relevant methods such as OV-Uni3DETR are missing. TR3D+FF is also an important omitted baseline given the use of a TR3D-style head and RGB feature fusion.
  2. The accuracy cost of using one unified model remains unclear. On SUN RGB-D, I-only performance is 31.2 mAP25, compared with ImVoxelNet’s 41.0 in Table 1, and P-only performance is below the reported TR3D and SR3D results. These gaps make comparisons with the Phase 1 model and per-subset specialists important for understanding the tradeoff between flexibility and accuracy.
  3. Practical sensor robustness is only partially validated. SUN RGB-D’s depth and point-cloud geometry are largely derived from the same observation, while TriScan3D uses reconstruction-derived point clouds. Performance under noisy inputs or calibration errors remains untested. Moreover, several component gains are small and reported from a single seed, leaving their stability unclear.
Questions for authors
  1. How does GeMI3D compare with TR3D+FF and methods explicitly trained for modality switching under matched evaluation protocols?
  2. What is the P-only performance after each training stage, and how does the final model compare with per-subset specialists?
  3. Could you evaluate robustness to depth noise or calibration errors and report multi-seed results for the main component ablations?
4 comments
AuthorsOpenReview ↗

Point to Point Response to Reviewer YMBt (1/3)

W1&Q1: Comparisons with modality-dropout baselines and relevant detectors.

To address the reviewer’s concern, we directly conducted the following comparisons with the modality-dropout baseline and TR3D+FF. The details are as follows.

(1) Comparison with TR3D+FF and modality-dropout baselines.

We retrain TR3D+FF and IIFNet3D with modality dropout, sampling the I+P, P-only, and I-only input configurations with probabilities of 50%, 25%, and 25%. Dropout improves P-only performance for TR3D+FF, but both baselines still score 0.0 mAP25\mathrm{mAP}_{25} on I-only. This suggests that their image branches primarily serve to augment point-cloud-based detection, rather than provide an independent detection path. Consequently, modality dropout alone cannot overcome their inherent architectural dependence on point-cloud inputs. In contrast, GeMI3D maps each available modality into a shared 3D representation, which is then processed by a unified encoder for detection, enabling GeMI3D to flexibly handle all input modality combinations.

MethodSettingSUN RGB-D ISUN RGB-D PSUN RGB-D I+PTriScan3D ITriScan3D PTriScan3D I+P
TR3D+FFw/o dropout0.08.469.20.050.369.2
TR3D+FFw/ dropout0.065.470.10.068.970.0
IIFNet3Dw/o dropout0.063.768.10.066.667.2
IIFNet3Dw/ dropout0.063.365.60.064.264.5
GeMI3Dw/ dropout31.264.568.419.968.369.9

By the way, on SUN RGB-D, using DA v2-Large as the depth estimator increases GeMI3D’s I+P to 70.3, exceeding TR3D+FF with modality dropout (70.1), and further improves I-only performance to 45.4 mAP${}_{25}$.

(2) Comparison with methods designed for missing-modality scenarios.

The official OV-Uni3DETR repository provides open-vocabulary configurations but no dedicated recipe for reproducing its closed-vocabulary results. We therefore undertook additional reproduction work to evaluate OV-Uni3DETR in our closed-vocabulary setting on SUN RGB-D. We first train the image and point-cloud branches separately and then perform multimodal training. For I, P, and I+P, OV-Uni3DETR achieves 15.2, 61.4, and 66.6 mAP${}_{25}$, compared with 31.2, 64.5, and 68.4 for GeMI3D. GeMI3D performs better in all three settings, with the largest gain in I-only (+16.0). Architecturally, OV-Uni3DETR uses separate image and point-cloud feature extractors, whereas GeMI3D converts both inputs into a common 3D point representation before shared encoding. This gives the image pathway explicit depth-derived geometry and access to representations learned from point clouds, which may contribute to the stronger I-only performance of our final unified model.

(3) Clarification of our contribution compared with existing multimodal detectors.

Rather than designing separate encoders or modality-specific architectures for different input configurations, we employ a unified encoder as geometric backbone to accommodate diverse combinations of input modalities. This design allows our method to flexibly handle varying modality configurations while maintaining a unified geometric representation.

AuthorsOpenReview ↗

Point to Point Response to Reviewer YMBt (2/3)

W2&Q2: Accuracy–flexibility trade-off across training stages and single-modality specialists.

Our results support a favorable balance between input flexibility and single-modality accuracy. (1) Stage-wise evaluation shows complementary roles that MCHA largely restores P-only accuracy after CMHC extends the hub to image and depth inputs. (2) In P-only inference, GeMI3D lowers than TR3D and SR3D but nearly matches our Phase 1 specialist. It outperforms the I-only and D-only specialists, and an additional DA v2-Large configuration can surpasses ImVoxelNet in I-only detection.

(1) P-only performance across training stages.

We first evaluate P-only detection after each training stage. The CMHC Phase 1 checkpoint, trained only on point clouds, serves as the P-only specialist. P-only accuracy drops after Phase 2 and recovers after MCHA. The final model nearly matches the P-only specialist, while supporting all seven input combinations with a single checkpoint. This highlights the role of MCHA in recovering point-cloud detection accuracy after the hub is extended to depth and image inputs.

CheckpointmAP${}_{25}$mAP${}_{50}$
After CMHC Phase 164.848.2
After CMHC Phase 261.843.9
After MCHA (final)64.549.1

(2) Comparison with single-modality specialists.

We train separate models for I-only and D-only and use the CMHC Phase 1 checkpoint as P-only specialists. In mAP${}_{25}$, GeMI3D substantially outperforms the I-only specialist (31.2 vs. 17.7) and the D-only specialist (64.1 vs. 34.5), while nearly matching the P-only specialist (64.5/49.1 vs. 64.8/48.2). The gains on I-only and D-only may reflect the transfer of geometric representations learned from point clouds during CMHC. Overall, by retaining competitive single-modality accuracy with a single checkpoint, GeMI3D reduces the need to deploy separate models as input availability changes.

InputSingle-modality specialistGeMI3D
I-only17.731.2
D-only34.564.1
P-only64.864.5
AuthorsOpenReview ↗

Point to Point Response to Reviewer YMBt (3/3)

W3&Q3: Robustness to sensor noise and calibration errors, and stability of component gains across random seeds.

Our primary focus is flexible inference with different subsets of available modalities. Appendix B.4 further evaluates robustness to (i) point removal, (ii) Gaussian noise in point clouds and depth maps, and (iii) extrinsic calibration errors. Severe point-cloud noise causes the largest observed degradation. We therefore (iv) evaluate quality-aware input selection, which partially recovers accuracy under severe noise with little change at the lowest noise level.

We further reran all variants in Table 3(b) on SUN RGB-D, and the result supports the gains observed in the original single-seed ablations.

The details are as follows.

(1) Robustness

(i) Point removal. We tested random sparsification and local cutout (Table 8). At level 5, random sparsification and local cutout yield 67.2/64.3 mAP${}_{25}$ on SUN RGB-D (clean: 68.6), and 69.6/65.3 on TriScan3D (clean: 70.7). Overall, performance remains relatively stable under point removal.

(ii) Gaussian noise.

At level 5, Gaussian coordinate noise in point clouds reduces mAP${}_{25}$ from 68.6 to 44.6 on SUN RGB-D and from 70.7 to 11.7 on TriScan3D (Table 8). Unlike point removal, which preserves the coordinates of the remaining points, Gaussian noise distorts local geometry and make the point-cloud path a less reliable reference for fusion, causing larger degradation.

We also evaluate Gaussian noise in sensor depth separately (Table 9). With clean point-cloud input retained, I+D+P remains close to clean performance even at level 5: 68.6→68.1 on SUN RGB-D and 70.7→70.5 on TriScan3D. Without P, depth noise has a larger impact: I+D decreases from 68.0 to 55.6 and from 48.6 to 33.1. Nevertheless, these scores remain above the corresponding clean image-only results of 31.2 and 19.9, indicating that noisy observed depth still provides useful information for detection beyond the image-only pathway.

CorruptionInputsSUN RGB-D: Level 5TriScan3D: Level 5
P: Random sparsificationI+D+P67.269.6
P: Local cutoutI+D+P64.365.3
P: Gaussian noiseI+D+P44.611.7
D: Gaussian noiseI+D55.633.1
D: Gaussian noiseI+D+P68.170.5

(iii) Extrinsic calibration errors. With fixed I+D+P, we apply the same per-frame rigid offset to the camera-to-detector transform for both I and D, misaligning their lifted geometry relative to P. The raw inputs, annotations, and model weights remain unchanged. We report mAP${}_{25}$ and mAP${}_{50}$:

Rotation / TranslationSUN RGB-DTriScan3D
Clean (0° / 0 cm)68.6 / 52.370.7 / 51.4
2.5° / 5 cm68.3 / 51.770.1 / 50.8
5° / 10 cm67.8 / 50.869.3 / 50.5

Even at 5° / 10 cm, mAP${}_{25}$ decreases by at most 1.4 points and mAP${}_{50}$ by at most 1.5 points across the two datasets. Thus, detection accuracy changes relatively little under the tested extrinsic offsets.

(iv) Quality-aware selection. We then use a per-frame point-cloud roughness score to decide whether the corrupted point cloud participates in fusion. In this experiment, we evaluate the rule under I+D+P: if P is excluded, the model switches to I+D, with D as the reference. This requires no retraining or second detection pass. At Gaussian noise level 1, the gate changes mAP${}_{25}$ by less than 0.1 point on either dataset. At level 5 of Gaussian noise, the gate switches to I+D on 99.6% of SUN RGB-D frames and 78.1% of TriScan3D frames, improving mAP${}_{25}$ over fixed I+D+P from 44.6 to 68.0 and from 11.7 to 38.9 (Table 10). Thus, the gate largely preserves accuracy under mild noise while recovering part of the loss under severe noise.

DatasetNoise levelFixedGatedSwitch (%)
SUN RGB-D167.267.11.0
SUN RGB-D544.668.099.6
TriScan3D169.869.80.4
TriScan3D511.738.978.1

(2) Stability of component gains across random seeds. We reran all variants in Table 3(b) on SUN RGB-D using seed 1, 2, 3, 4. Full MCHA achieves the highest mAP${}_{25}$ both with all three modalities and averaged over the seven input combinations, supporting the gains observed in the original single-seed ablations.

VariantI+D+PAvg.
w/o feature projector67.9±0.0667.9_{\pm 0.06}61.2±0.1361.2_{\pm 0.13}
Voxel-wise mean fusion66.9±0.1266.9_{\pm 0.12}61.0±0.0361.0_{\pm 0.03}
w/o consistency supervision67.0±0.4267.0_{\pm 0.42}61.0±0.1261.0_{\pm 0.12}
w/o hub fine-tuning67.4±0.2367.4_{\pm 0.23}60.3±0.1360.3_{\pm 0.13}
Full MCHA68.6±0.0468.6_{\pm 0.04}61.5±0.0661.5_{\pm 0.06}

Dear reviewer YMBt,

Thank you for taking the time to review our work. We have carefully addressed all your comments and incorporated the suggested revisions. This is a gentle reminder that today marks the deadline for review feedback. Please let us know if you have any additional feedback or concerns.

Best regards,

Authors.

Reviewer_JenEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeMI3D has developed an indoor 3D object detection method that is compatible with multiple modalities of input. Through the designed three-dimensional geometric hub, it achieves stable detection of any modal input. The experimental results have proved its effectiveness.

Critical strengths
  1. The article proposes a stable three-dimensional geometric hub that can handle a set of weighted processing for seven different modal combinations of inputs. This effectively solves the problem of any missing modalities in the three-dimensional detection model when it is applied in practice, and it is highly innovative and has practical application value.
  2. The overall experimental setup is quite comprehensive. The paper also presents the full-modal accuracy, seven input combinations, comparison with fixed input detectors, as well as ablations of each module and modal discarding, and uses different sources of depth and point clouds to check whether the geometric core can truly be reused across inputs. Therefore, the conclusion is supported by a set of mutually contrasting experiments, rather than relying solely on a single score on a single dataset.
Critical weaknesses
  1. From the experimental results, it can be observed that the modality independence is mainly reflected in the input containing point clouds. When the point clouds are removed, the accuracy significantly decreases. Only in the case of images, both localization and classification become unreliable. The average value of the seven combinations masks this gap. For example, in Table 2, only when the image input is used, the performance is only 31.2 and 19.9, which is lower than other detectors that originally use images as input.
  2. The performance contributions of each module in Table 3(b) seem to be relatively small, which might instead lead to additional computational overhead.
  3. There are problems with the fixed anchor select branch in the method design. Both point clouds and depth may have noise issues caused by the sensors. In some cases, images can obtain better quality depth through a large-scale trained depth estimation model. Dynamically selecting the appropriate one would be more in line with the actual situation.
Questions for authors
  1. Please explain why the performance of the image-only input method is lower than that of other image detectors. Is it because this method heavily relies on the point cloud mode?
  2. The geometry of the image path comes from the frozen depth estimator. The 2D features are only supplementary. Please compare "estimating depth" and "improving with the true depth", and in both cases, remove the two-dimensional semantic features to compare the impact of depth estimation quality on performance and whether the improvement brought by the image modality is due to the depth estimation or the additional 2D semantic features.
6 comments
AuthorsOpenReview ↗

Point to Point Response to Reviewer JenE (1/4)

W1&Q1: Dependence on point-cloud inputs and image-only performance.

Sorry for this confusion. We address this concern from two aspects:

(1) Relationship between point-cloud inputs and detection performance.

We would like to clarify that the statement, “When the point clouds are removed, the accuracy significantly decreases,” reflects a misunderstanding of GeMI3D’s performance across input configurations. On SUN RGB-D, removing point clouds while retaining RGB and depth reduces performance from 68.6 to 68.0 mAP${}_{25}$—only a 0.6-point decrease. This provides direct evidence that GeMI3D can maintain high accuracy without point-cloud inputs when geometric information remains available through depth.

This distinction matters because 3D object detection requires accurate estimates of object positions and spatial extents. Both point clouds and depth provide explicit metric geometry, whereas RGB images require this geometry to be inferred. The relevant factor is therefore the quality of the available geometric evidence, rather than point-cloud availability alone.

Our image-only experiment further supports this interpretation: training GeMI3D with DA v2-Large improves performance on SUN RGB-D from 31.2/9.2 to 45.4/16.2 mAP${}_{25}/mAP/mAP{}_{50}$ while the external input remains RGB-only. The improvement with a stronger depth estimator is consistent with better geometric reconstruction benefiting detection. Together, these results indicate that reliable 3D geometry is a key factor in GeMI3D’s accuracy, and that this geometry can be obtained from depth or inferred from images without requiring point-cloud inputs.

(2) Image-only performance: geometric limitations and comparison with image-based detectors.

RGB images provide rich appearance and semantic cues but do not directly measure metric depth. GeMI3D must therefore infer geometry to ground these cues in 3D. Errors in estimated depth can distort object positions and spatial extents, limiting detection accuracy even when semantic information is informative.

Our depth-estimator ablation supports this explanation. Without 2D semantic features, using DA v2-Large instead of DA v2-Small improves performance on SUN RGB-D from 26.4/7.3 to 36.4/11.3 mAP${}_{25}/mAP/mAP{}_{50}$. This improvement demonstrates that stronger depth estimation benefits detection independently of additional 2D semantic features, supporting the quality of recovered geometry as a key factor in image-only performance.

Regarding the observation that image-only performance is “lower than other detectors,” GeMI3D with the stronger estimator achieves 45.4/16.2, exceeding ImVoxelNet’s 41.0/13.5 by 4.4/2.7 points, while inference still requires only RGB images. Thus, improving estimated geometry both strengthens the image-only path and reverses the original gap to ImVoxelNet. These results support a geometric reconstruction bottleneck, rather than an inherent inability of GeMI3D to perform competitive detection without point-cloud inputs. Recovering reliable metric geometry from RGB is a shared challenge in image-based 3D detection, and we will further improve this capability in future work.

AuthorsOpenReview ↗

Point to Point Response to Reviewer JenE (2/4)

W2: Effectiveness and computational overhead of MCHA components.

Our results indicate that MCHA offers a favorable trade-off between accuracy and computational cost, since (1) MCHA as a whole yields a clear improvement; (2) across 4 training seeds, full MCHA achieves higher mean mAP${}_{25}$ than the ablated variants both with I+D+P and when averaged over the seven input combinations, and (3) these gains come with acceptable additional training cost and little inference overhead.

The following is a detailed analysis:

(1) Overall benefit of MCHA. Starting from the same CMHC basis, we compare full MCHA with a baseline that skips MCHA and combines the three modality features using voxel-wise mean fusion. With all three modalities available, MCHA improves mAP${}_{25}$ from 60.5 to 68.6. This comparison measures the contribution of the stage as a whole, complementing the single-component ablations in Table 3(b). The additional whole-stage comparison is reported in Appendix B.2.

(2) Stability across seeds. First, the gains in Table 3(b) are more pronounced when all three modalities are available. Additionally, across 4 seeds (1,2,3,4), full MCHA retains higher mAP${}_{25}$ than the ablated variants, both with all three modalities and when averaged over all seven input combinations. The four-seed stability results are reported in Appendix Table 8.

VariantI+D+PAvg.
w/o feature projector67.9±0.0667.9_{\pm 0.06}61.2±0.1361.2_{\pm 0.13}
Voxel-wise mean fusion66.9±0.1266.9_{\pm 0.12}61.0±0.0361.0_{\pm 0.03}
w/o consistency supervision67.0±0.4267.0_{\pm 0.42}61.0±0.1261.0_{\pm 0.12}
w/o hub fine-tuning67.4±0.2367.4_{\pm 0.23}60.3±0.1360.3_{\pm 0.13}
Full MCHA68.6±0.0468.6_{\pm 0.04}61.5±0.0661.5_{\pm 0.06}

(3) Computational cost. We benchmark MCHA and its component ablations on SUN RGB-D using a single RTX 5880 Ada. After warm-up, for training, we measured 240 timed iterations per variant, and for inference, we time each variant on the same 240 timed iterations of 400 frames.

Training: Full MCHA takes about 1.0 s per iteration at batch size 6. Hub fine-tuning increases the mean iteration time from 664.7 to 1017.3 ms. Reference-guided fusion and feature projection add 36.0 ms and 8.1 ms per iteration. The 0.8 ms difference for consistency supervision is too small to distinguish from timing variability.

Inference: Full MCHA takes 216.6 ms/frame with all three modalities and batch size 1. Replacing reference-guided fusion with voxel-wise mean fusion reduces latency to 214.5 ms/frame, so the overhead is about 1%. Additional latency from feature projection is too small to distinguish from timing variability, and neither consistency supervision nor hub fine-tuning adds inference-time operations.

In summary, although MCHA’s reference-guided fusion incurs only about 1% inference overhead (216.6 vs. 214.5 ms/frame relative to voxel-wise mean fusion), full MCHA improves mAP${}_{25}$ from 60.5 to 68.6 (a +8.1-point gain with all three modalities over the same CMHC basis without MCHA), so considering this favorable accuracy–cost trade-off, we retain the current MCHA design.

AuthorsOpenReview ↗

Point to Point Response to Reviewer JenE (3/4)

W3: Fixed reference-path selection under noisy inputs.

Thank you for raising this concern. Our primary focus is flexible inference with different subsets of available modalities. To further evaluate robustness, in Appendix B.4, we added point-cloud and sensor-depth corruption experiments and evaluated a quality-aware input selection rule. We find that (1) severe point-cloud noise causes the largest degradation while I+D with noisy depth still outperforms clean I-only inference; and (2) quality-aware selection recovers accuracy under severe point-cloud noise with little change under mild noise. The following is a detailed analysis:

(1) Point-cloud and depth corruption.

We tested random sparsification, local cutout, and Gaussian coordinate noise on point clouds (Table 8). At level 5, random sparsification, local cutout, and Gaussian noise yield 67.2, 64.3, and 44.6 mAP${}_{25}$ on SUN RGB-D (clean: 68.6), and 69.6, 65.3, and 11.7 on TriScan3D (clean: 70.7). Unlike point removal, which preserves the coordinates of the remaining points, Gaussian noise distorts local geometry and can make the point-cloud path a less reliable reference for fusion, causing larger degradation.

We also perturbed sensor depth separately (Table 9). With clean point clouds available, I+D+P remains close to the clean scores (68.6 → 68.1 and 70.7 → 70.5). Without P, depth noise has a larger impact: I+D decreases from 68.0 to 55.6 and from 48.6 to 33.1. Nevertheless, these scores exceed the clean image-only results of 31.2/19.9 in Table 2(b). On SUN RGB-D, the score of 55.6 also exceeds the 45.4 achieved by the image-only baseline using DA v2-Large. These results indicate that noisy observed depth still provides useful information for detection beyond the image-only path.

CorruptionInputsSUN RGB-D: Level 5TriScan3D: Level 5
P: Random sparsificationI+D+P67.269.6
P: Local cutoutI+D+P64.365.3
P: Gaussian noiseI+D+P44.611.7
D: Gaussian noiseI+D55.633.1
D: Gaussian noiseI+D+P68.170.5

(2) Quality-aware selection.

We use a per-frame roughness gate to decide whether the corrupted modality participates in fusion. In this experiment, we evaluate the rule under I+D+P: if P is excluded, the model switches to I+D, with D as the reference. This requires no retraining or second detection pass. At Gaussian noise level 1, the gate changes mAP${}_{25}$ by less than 0.1 point on either dataset. At level 5 of Gaussian noise, the gate switches to I+D on 99.6% of SUN RGB-D frames and 78.1% of TriScan3D frames, improving mAP${}_{25}$ over fixed I+D+P from 44.6 to 68.0 and from 11.7 to 38.9. Thus, the gate largely preserves accuracy under mild noise while recovering part of the loss under severe noise.

DatasetNoise levelFixedGatedSwitch (%)
SUN RGB-D167.267.11.0
SUN RGB-D544.668.099.6
TriScan3D169.869.80.4
TriScan3D511.738.978.1

Based on the above analysis and the reviewer’s suggestion, we will adopt quality-aware selection as an optimized version of reference-path selection to improve the module’s robustness under higher noise. The revised version is provided in Section 3.3 of the main text.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response by Authors

We sincerely thank all three reviewers for their careful evaluations and constructive feedback.

(1) Comparisons beyond fixed-input baselines (YMBt, tyjb).

We directly conducted the following comparisons with the modality-dropout baseline and OV-Uni3DETR, a modality-switching method. The details are as follows.

(i) We retrained TR3D+FF and IIFNet3D with modality dropout. Dropout improves P-only accuracy for TR3D+FF, but neither fixed-input baseline can detect objects with images alone. This suggests that their image branches primarily serve to augment point-cloud-based detection, rather than provide an independent detection path.

(ii) Compared with the reproduced OV-Uni3DETR, GeMI3D achieves higher mAP${}_{25}$ in all three shared settings (I, P, and I+P), with the largest gain on I-only. Architecturally, OV-Uni3DETR uses separate image and point-cloud feature extractors, whereas GeMI3D converts both inputs into a common 3D point representation before shared encoding. This gives the image path explicit depth-derived geometry and access to representations learned from point clouds, which may contribute to the stronger I-only performance of our final unified model.

By the way, on SUN RGB-D, using DA v2-Large as the depth estimator increases GeMI3D’s I+P mAP${}_{25}$ to 70.3, exceeding TR3D+FF with modality dropout (70.1), and further improves I-only performance to 45.4 mAP${}_{25}$.

MethodSettingSUN RGB-D ISUN RGB-D PSUN RGB-D I+PTriScan3D ITriScan3D PTriScan3D I+P
TR3D+FFw/o dropout0.08.469.20.050.369.2
TR3D+FFw/ dropout0.065.470.10.068.970.0
IIFNet3Dw/o dropout0.063.768.10.066.667.2
IIFNet3Dw/ dropout0.063.365.60.064.264.5
GeMI3Dw/ dropout31.264.568.419.968.369.9

(2) Robustness to degraded inputs (JenE, YMBt, tyjb).

Appendix B.4 now evaluates point removal, Gaussian noise in point clouds and depth maps, and extrinsic calibration errors. We additionally test quality-aware input selection under point-cloud noise.

Severe point-cloud noise causes the largest degradation while I+D with noisy depth still outperforms clean I-only inference, and clean point clouds mitigate the effect of depth noise. Accuracy changes relatively little under the tested extrinsic offsets. Quality-aware input selection largely recovers accuracy under severe point-cloud noise with little change under mild noise.

(3) Image-only detection and semantic features (JenE, tyjb).

On SUN RGB-D, stronger depth estimation and 2D semantic features each improve image-only detection, and combining them enables GeMI3D to outperform ImVoxelNet. Without 2D semantic features, switching from DA v2-Small to DA v2-Large raises mAP${}_{25}$ from 26.4 to 36.4 (Table 8). With DA v2-Small held fixed, adding semantic features raises mAP${}_{25}$ from 26.4 to 31.2 (Table 7). With DA v2-Large and semantic features, GeMI3D achieves 45.4/16.2 mAP${}_{25}/mAP/mAP{}_{50}$, outperforms 41.0/13.5 for ImVoxelNet.

We have also revised Section 3.3 to clarify how DINOv2 and R50-FPN features are integrated.

Input pathDepth source2D semantic featuresmAP${}_{25}$mAP${}_{50}$
I-onlyDA v2-SmallNone26.47.3
I-onlyDA v2-LargeNone36.411.3
I-onlyDA v2-SmallR50-FPN + DINOv231.29.2
I-onlyDA v2-LargeR50-FPN + DINOv245.416.2

(4) Accuracy–flexibility trade-off (JenE, YMBt).

The final GeMI3D model supports all seven input combinations while nearly matching our P-only specialist and outperforming the evaluated I-only and D-only specialists. On SUN RGB-D, MCHA largely recovers P-only accuracy after CMHC extends the geometric hub learned from point clouds to image and depth inputs. The final model scores 0.3 points lower in mAP${}_{25}$ and 0.9 points higher in mAP${}_{50}$ than Phase 1.

We compare full MCHA with a baseline that skips MCHA and combines the three modality features using voxel-wise mean fusion. With all three modalities available, MCHA improves [email protected] from 60.5 to 68.6. This comparison measures the contribution of the stage as a whole, complementing the ablations in Table 3(b).

(5) Additional results across random seeds (JenE, YMBt).

We report that across 4 training seeds, full MCHA achieves higher mAP${}_{25}$ than the ablated variants both with I+D+P and when averaged over the seven input combinations.

VariantI+D+PAvg.
w/o feature projector67.9±0.0667.9_{\pm 0.06}61.2±0.1361.2_{\pm 0.13}
Voxel-wise mean fusion66.9±0.1266.9_{\pm 0.12}61.0±0.0361.0_{\pm 0.03}
w/o consistency supervision67.0±0.4267.0_{\pm 0.42}61.0±0.1261.0_{\pm 0.12}
w/o hub fine-tuning67.4±0.2367.4_{\pm 0.23}60.3±0.1360.3_{\pm 0.13}
Full MCHA68.6±0.0468.6_{\pm 0.04}61.5±0.0661.5_{\pm 0.06}

We thank the Program Chairs, Senior Area Chairs, Area Chairs, and reviewers for their time and constructive feedback.

We appreciate the reviewers’ recognition of the strengths of our work:

  • Practical input flexibility. A single checkpoint supports all seven non-empty combinations of image, depth, and point-cloud inputs without retraining.
  • A coherent shared geometric representation. The geometric hub maps heterogeneous inputs into a common 3D space, enabling shared detection across different input combinations.
  • A useful benchmark and comprehensive evaluation. TriScan3D provides distinct depth and point-cloud sources, complementing the evaluations across datasets, input combinations, and model components.

Based on the reviewers’ feedback, we conducted additional experiments and provided clarifications during the rebuttal. We summarize them below and incorporate them into the revised manuscript.

  • Comparisons with modality-dropout baselines and modality-switching methods. After dropout training, TR3D+FF and IIFNet3D still score zero on I-only. GeMI3D also outperforms OV-Uni3DETR in all three shared settings—I, P, and I+P.
  • Contribution beyond geometric conversion and dropout. Under the same backbone, image features, and training budget, staged training improves full-modality mAP${}_{25}/mAP/mAP{}_{50}$ from 66.5/49.1 to 68.6/52.3 over the single-stage baseline.
  • Accuracy–flexibility trade-off. One checkpoint supports all seven input combinations while outperforming the tested I-only and D-only specialists (31.2 vs. 17.7 and 64.1 vs. 34.5 mAP${}_{25}$) and nearly matching the P-only specialist (64.5 vs. 64.8).
  • Separate contributions of geometry and image semantics. Without semantic features, stronger depth estimation raises I-only mAP₂₅ from 26.4 to 36.4. Combining both achieves 45.4/16.2 mAP${}_{25}/mAP/mAP{}_{50}$, exceeding ImVoxelNet’s 41.0/13.5.
  • MCHA effectiveness, reproducibility, and overhead. MCHA raises full-modality mAP${}_{25}$ from 60.5 to 68.6, with four-seed ablations supporting its component gains. Inference overhead is small: reference-guided fusion adds only about 1% latency. Hub fine-tuning adds some training cost but adds no inference-time operations.
  • Robustness and quality-aware selection. We added point-cloud and depth corruptions, camera extrinsic perturbations, and quality-aware input selection. Accuracy remains relatively stable under point removal and the tested extrinsic offsets, but drops markedly under severe point-cloud noise. On both datasets, our newly added selection rule preserves accuracy under mild noise and approaches I+D-only performance under severe noise. Our central claim remains flexible inference under missing modalities, not broad robustness to arbitrary sensor corruption.
  • Method presentation. Section 3.3 now explains how DINOv2 and R50-FPN features are integrated and clarifies the default reference priority and quality-aware selection.

Reviewer JenE confirmed that our replies addressed most concerns and maintained a positive rating.

To address Reviewer tyjb’s follow-up on severe point-cloud noise, we clarified the distinction between corruption robustness and missing-modality flexibility and resolved the specific concern through additional threshold analysis. The results show that the gate approaches I+D-only performance under severe noise while preserving the benefit of point clouds under mild noise.

We also provided detailed responses to Reviewer YMBt on baseline comparisons, the accuracy–flexibility trade-off, sensor robustness, and multi-seed stability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.