acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Aware 2D-3D Multimodal Fusion for 3D Segmentation

Abstract

Multimodal 2D-3D segmentation leverages complementary 2D semantic priors and 3D geometric structures for scene understanding. However, geometric inconsistency across the 2D-3D fusion pipeline remains a key challenge. To address this challenge, we propose GAMFusion, a geometry-aware 2D-3D multimodal fusion framework designed to preserve geometric consistency throughout the 2D-to-3D feature flow. First, we construct an extensible parallel fusion topology that decouples heterogeneous modality pathways and introduces pixel-wise adaptive interaction to mitigate feature interference caused by modality discrepancies. Second, we introduce a corrective feature consistency alignment module that restores local surface structure before voxelization using geometric consistency constraints. Finally, we design a normal-aware 3D contextual fusion mechanism that incorporates surface normal constraints into neighborhood aggregation to reduce cross-surface feature leakage caused by Euclidean distance-based retrieval. Extensive experiments on ScanNet200, S3DIS, and AI2THOR demonstrate the effectiveness of the proposed method, with strong improvements over existing approaches, particularly on boundary-sensitive metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Semantic Segmentation, 3D Instance Segmentation, Multimodal Fusion, Geometry-Aware Learning, Normal-Aware Context Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P3J8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose the method called GAMFusion and demonstrate its effectiveness on 3 3D segmentation datasets. The main architecture is a 2-branch fusion structure with 2D and 3D branch. The idea is to maintain the geometric consistency of the 2D and 3D branch. This work aims to address a problem that may occur after lifting 2D features to 3D. If Euclidean distance is used for neighborhood construction, features from different surfaces may easily be mixed together. It may make it difficult to maintain geometric consistency. So, the method has four stages: the explicit 2D stage, the pre-voxelization stage, which is similar to geometry-guided bilateral filtering, the post-voxelization stage, which uses an improved distance metric, and the decoder which follows a standard transformer design. However, the modules proposed by the authors are all combinations of existing techniques. From the perspective of network learning, this work's novelty is limited. The narrative of the paper is not very good, the experiments are simple and insufficient. This may not be enough to meet the accept standard of ICLR. I suggest that the authors improve the writing and make the experiment more complete. In particular, the problems identified in the motivation section currently lack supporting experimental data. After these improvements, I suggest submitting the manuscript to other top conference or journal in the computer vision field.

Critical strengths
  1. The authors explain the problem clearly, making it easier for readers to understand the problems with current methods based on Euclidean neighborhoods. However, supporting experimental data are lacking (see the Weaknesses 2 below).
  2. The figures are well drawn and clear. Readers can understand the method clearly just by looking at the figures.
  3. The method section is written clearly. So it should be relatively easy to reproduce the method.
Critical weaknesses
  1. I think the main problem is that the contribution of the proposed method is insufficient. In my view, the most valuable part is the use of normal-aware k-NN. The other parts are all relatively common. This small innovation may not be enough for ICLR. Of course, I think a small innovation can be acceptable. However, the insights provided by normal-aware k-NN for the field of 2D-3D modality fusion and related fields may not be sufficient.
  2. I have some suggestions about the narrative of the paper. Although the method section is written clearly and makes it easier for readers to reproduce the method, the main thread of the paper may have been weakened. As shown in Section 1 (Introduction), the authors organize the paper around an overall framework. This is acceptable, but it does not highlight the problem with Euclidean k-NN retrieval. This is the part that this type of work should discuss in detail, as in Figure 2. Figure 2 is quite well drawn, but the related discussion is too limited. It also lacks supporting experimental data. More visualizations would be better. I suggest that the authors add more relevant content.
  3. The experimental section also has problems. First, too few architectures are compared. The current baselines are relatively classic, and some recent works are missing, including some well-known architectures such as PTv3. Judging from the experimental section, the paper does not look very professional. Discussing only inference cost without discussing model size may not be enough. The authors mention "boundary-sensitive" in the conclusion, but the related experiments are missing. The current evaluation metrics do not seem to reflect boundary sensitivity.
  4. I also have some suggestions about the structure and formatting of the paper. Section 2 Related Work is currently too long. This may make it difficult for the authors to clearly explain the problems with existing methods. Tables 2 to 4 contain very little data but take up a lot of space. This may also leave the authors with insufficient space to explain the motivation. In addition, for a comparison table like Table 1, it would be better to include the publication year of each method.
  5. From Table 1, the improvement on Tail is quite large. The authors could further analyze the reasons for this in the future.
Questions for authors

The motivation and method of this paper are simple, so I can understand this work. I do not have any questions for the authors. For suggestions on improving the paper, I suggest referring to the Weaknesses part.

Reviewer_Bm3nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GAMFusion, a geometry-aware 2D–3D multimodal fusion framework for semantic and instance segmentation, aiming to mitigate modality interference and cross-surface feature mixing. The framework consists of three components: (1) an Extensible Parallel Fusion Topology (EPFT) with a Pixel-wise Interaction Block (PIB) for adaptive multimodal fusion; (2) a Corrective Feature Consistency Alignment (CFCA) module that refines lifted features using spatial distance, depth continuity, and surface-normal consistency before voxelization; and (3) a Normal-Aware 3D Contextual Fusion module that incorporates surface orientation into neighborhood retrieval. Experiments on ScanNet200, S3DIS, AI2-THOR, and DeLiVER report competitive segmentation results, with ablations examining the geometric components and different normal sources.

Critical strengths
  1. The paper addresses a relevant limitation of 2D–3D fusion: neither 2D adjacency nor 3D Euclidean proximity necessarily reflects actual surface connectivity, potentially causing feature mixing across object boundaries. Combining spatial distance, normal consistency, and depth continuity provides a practical way to constrain aggregation. Applying geometric constraints before and after voxelization is also a reasonable design choice.

  2. GAMFusion reports improvements across several indoor 3D segmentation benchmarks. For example, Table 1 reports 48.2 mAP50 on ScanNet200, compared with 45.3 for ODIN, while Table 3 shows an improvement from 43.5 to 48.2 mAP50 after introducing the geometric components. These results suggest practical benefits.

Critical weaknesses
  1. The proposed components largely build upon existing techniques. EPFT/PIB resembles prior pixel-wise multimodal fusion methods such as GeminiFusion, while CFCA and Normal-Aware aggregation rely on established geometric weighting and surface-aware neighborhood construction. In particular, Eq. (7) combines spatial and normal differences in a weighted distance metric. The main distinction appears to be applying these geometric constraints before and after voxelization, but the paper does not sufficiently demonstrate a substantive methodological advance over prior work.

  2. The claimed reduction in cross-surface feature leakage lacks direct quantitative validation, such as boundary-sensitive metrics or cross-surface neighborhood analysis. The ablations are also incomplete, particularly the missing Normal-Aware-only configuration and separate evaluation of EPFT/PIB. Furthermore, unclear ScanNet200 evaluation splits, incomplete mAP comparisons, and insufficient comparisons with strong multimodal fusion baselines weaken the reported performance advantages. Missing details on geometric inputs, normal estimation, and hyperparameter settings further limit reproducibility.

Questions for authors
  1. What are the key technical differences between EPFT/PIB and GeminiFusion (ICML 2024), particularly in modality relation weighting, pixel-wise interaction, and branch organization? How do CFCA and Normal-Aware aggregation advance beyond existing surface-aware or boundary-aware methods? In particular, how does the formulation in Eq. (2), which computes query-key interactions separately for each modality, differ from conventional gated feature fusion?

  2. Could the authors provide direct quantitative evidence that the proposed modules reduce cross-surface feature leakage, such as Boundary F1 or cross-instance neighborhood connection rates? Additionally, could the authors report complete ablations of CFCA and Normal-Aware aggregation (including the Normal-Aware-only configuration), as well as separate evaluations of EPFT and PIB on the final 3D segmentation task?

  3. Could the authors clarify the dataset splits and evaluation protocols, particularly for ScanNet200, and provide consistent comparisons using mAP, mAP50, and mAP25? How does the proposed fusion approach compare with stronger baselines such as CMNeXt under matched experimental settings?

  4. How are HSurf-Net normals obtained and aligned with the point/voxel representations used by the geometric modules? How were σx\sigma_x and σn\sigma_n selected, and how sensitive is performance to λ=0.5\lambda=0.5 and alternative geometric distance formulations? Finally, could the authors clarify the "None" configuration in Table 4 and whether the reported latency includes normal estimation?

Reviewer_w5iTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the issue of local structural inconsistency that arises when fusing 2D image features with 3D geometric information, this paper proposes the GAMFusion framework for 3D semantic and instance segmentation. The framework consists of three primary modules: First, the Extensible Parallel Fusion Topology (EPFT) and the Pixel-wise Interaction Block (PIB) integrate 2D multi-modal features through independent modal branches and adaptive weighting. Second, the Corrective Feature Consistency Alignment (CFCA) module refines 3D point features prior to voxelization by applying constraints based on spatial distance, surface normals, and depth continuity. Finally, the Normal-Aware 3D Contextual Fusion module incorporates a normal-constrained neighborhood search after voxelization, which helps mitigate inappropriate feature aggregation across distinct surfaces. The authors evaluated the framework’s 3D segmentation performance on the ScanNet200, S3DIS, and AI2-THOR datasets, and utilized DeLiVER to validate the effectiveness of the 2D multi-modal fusion. Experimental results demonstrate strong segmentation performance, particularly regarding certain long-tail categories in ScanNet200 and various instance segmentation metrics.

Critical strengths
  1. This paper addresses the issue where, upon projecting 2D semantic features into 3D space, Euclidean-distance-based neighborhoods may inadvertently connect distinct physical surfaces. CFCA and Normal-Aware Aggregation are applied before and after voxelization, respectively, with clear design motivations and provide a reasonable approach to improving the blending of boundary features in 3D scenes.

  2. Table 1 shows that GAMFusion achieves 48.2% mAP50 for instance segmentation on ScanNet200, which is higher than SegDINO3D's 45.4%, and 67.4% mAP on AI2-THOR, which is higher than ODIN's 64.3%. Furthermore, the authors analyze various surface normal estimation methods and their associated inference overheads, providing preliminary evidence for evaluating the role of geometric information.

Critical weaknesses
  1. The core argument of the paper is to reduce cross-surface feature leakage through geometric consistency constraints, but existing experiments mainly rely on overall segmentation metrics and qualitative visualizations, lacking direct quantitative evidence. Furthermore, Table 3 does not evaluate Normal-Aware Aggregation in isolation, making it difficult to substantiate the independent contributions and complementarity of the two geometric modules.

  2. The paper proposes a unified 2D-3D multimodal fusion framework; yet, the EPFT and PIB components are only ablated within the context of DeLiVER, lacking independent validation on primary 3D segmentation tasks. Consequently, the current experiments fail to distinguish whether performance gains stem from multimodal fusion or geometric constraints, nor do they sufficiently demonstrate the necessity of each component within the complete framework.

Questions for authors
  1. Table 3 only reports results for the incremental addition of CFCA and Normal-Aware Aggregation. Are there experiments that evaluate Normal-Aware Aggregation in isolation? Furthermore, is there any existing quantitative analysis regarding its performance at object boundaries or in scenarios involving the blending of features across different surfaces?

  2. The ablation studies for EPFT and PIB are primarily conducted on the DeLiVER dataset. Could the authors provide further details on the individual contributions of these two modules to the overall 3D segmentation performance on datasets such as ScanNet200, S3DIS, or AI2-THOR? Are there any experiments that distinguish between the effects of 2D-modality fusion and 3D-geometric constraints?

  3. Regarding the implementation, Eq. (6) expresses neighborhood features as a simple summation weighted by geometric factors. Please clarify whether weight normalization is applied in the actual implementation, and how the method accounts for potential variations in feature scale resulting from a changing number of valid neighbors. Additionally, since Eq. (4) only incorporates camera intrinsic parameters, how are 3D points from multiple keyframes transformed into a unified coordinate system?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.