acceptodds
Under review as a conference paper at ICLR 2027

IHF-ReLoc: An Instance-Aware Hierarchical Framework for 2D-3D Visual Relocalization

Abstract

Existing visual relocalization methods typically follow a coarse-to-fine pipeline, establishing patch-level correspondences between a query image and a pre-built 3D map before refining them into point-to-pixel matches. However, in scenes with weak textures and occlusions, visually similar patches produce ambiguous responses. While recent methods exchange contextual information between image and point features to alleviate this ambiguity, they overlook the topological consistency among neighboring correspondences. In this paper, we propose IHF-ReLoc, an instance-aware hierarchical framework for 2D-3D visual relocalization, which progressively refines matching from instance-level associations to fine-grained point-to-patch correspondences. Specifically, IHF-ReLoc consists of two core modules, namely Scene Graph Matching (SGM) and Candidate Graph Reasoner (CGR). SGM performs matching over 2D and 3D instance graphs and exploits instance topology to enhance feature discriminability, thereby reducing cross-modal ambiguity. The CGR models candidate point-to-patch correspondences as graph nodes and their local geometric relations as edges. By propagating geometric information among neighboring candidates, CGR suppresses visually similar but incorrect matches. Experiments show that IHF-ReLoc achieves state-of-the-art performance and remains robust to temporal scene changes without relying on pose priors obtained through image retrieval.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Location awareness, Training, Visual relocalization, Three-dimensional displays, Pose estimation, Intelligent robots, Candidate Edge Graph

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SAxmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents IHF-ReLoc, an instance-aware framework for image-to-point-cloud camera relocalization. It establishes instance-level associations through scene graph matching, progressively refines point-to-patch correspondences, and uses a Candidate Graph Reasoner to incorporate local geometric consistency into candidate selection. Experiments on ScanNet and 3RScan evaluate localization accuracy and robustness to scene changes.

Critical strengths
  1. The progression from instance-level association to fine-grained matching provides a coherent approach to reducing cross-modal ambiguity.

  2. Retaining multiple correspondence candidates and using geometric consistency to select among them is a reasonable design for ambiguous regions.

Critical weaknesses

The novelty of the second contribution is limited. Scene graphs and object relationships have already been used for cross-modal association: GOReloc establishes correspondences between 2D and 3D object graphs for camera pose estimation [1], while SceneGraphLoc matches query images against 3D scene graphs for coarse localization [2]. Using scene graphs and instance topology to support matching is therefore not new. Moreover, filtering map points using positive-depth, viewing-frustum, and instance-mask constraints constitutes standard geometric and visibility checking, rather than a new occlusion model or visibility reasoning method. The combination presented here does not establish sufficient methodological novelty to justify a major contribution.

The experimental evaluation is insufficient. The ScanNet v2 comparison omits Top-I2P [3], a relevant method that also exploits topology for image-to-point-cloud registration and reports ScanNet results. More importantly, the 3RScan experiments compare only different training configurations of the proposed method, without external localization baselines. They therefore do not establish an advantage over existing methods in changing scenes. PnP success rate also cannot substitute for localization recall under specified pose-error thresholds.

References

[1] Yutong Wang, Chaoyang Jiang, and Xieyuanli Chen. GOReloc: Graph-based Object-Level Relocalization for Visual SLAM. IEEE Robotics and Automation Letters, 2024.

[2] Yang Miao et al. SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs. ECCV, 2024.

[3] Pei An et al. Top-I2P: Explore Open-Domain Image-to-Point Cloud Registration Using Topology Relationship. IJCAI, 2025.

Questions for authors

The appendix describes three training stages. Which stages use each loss in Section 3.5, and which network outputs do these losses supervise? Which modules are frozen or jointly trained at each stage? Please also specify the CGR architecture and its training procedure.

Table 1 reports a median translation error of 0.32 m and recall of 60.4% at 0.5 m / 15° for PlanaReLoc, compared with 0.20 m and 73.1% in its original paper. Given the different evaluation protocols, could the authors explain the differences in query selection, model training, map construction, and post-refinement, and confirm whether the complete PlanaReLoc pipeline was used?

Could the authors provide a comparison with Top-I2P on ScanNet under the same evaluation protocol?

Could the authors provide comparisons with external localization baselines on 3RScan under comparable input conditions, including localization recall at explicit translation and rotation thresholds?

1 comment

Q1: The novelty of the second contribution is limited.

A1: Our main contribution lies in coupling instance association, local verification, and pose-guided correspondence refinement. Scene-graph context retrieves candidate map instances, and instance features are injected into local matching to reduce ambiguity among similar objects. Local point-to-patch evidence then verifies the candidate instance pairs, providing instance associations for progressive refinement.

Q2: The experimental evaluation is insufficient ... pose-error thresholds.

A2: Top-I2P is added to Related Work. Its source code was unavailable for our comparison; Hg-I2P provides a related graph-based baseline.

We have added external baseline comparisons in Section 4.2. All methods are evaluated on validation rescans using reference-scan maps. Baselines and ours* train on training reference scans and rescans; ours uses reference scans only.

MethodsLocal MapΔR Mean (°) ↓ΔR Med. (°) ↓Δt Mean (m) ↓Δt Med. (m) ↓Pose Recall (0.2 m, 10°) (%) ↑Pose Recall (0.5 m, 15°) (%) ↑Pose Recall (1.0 m, 30°) (%) ↑
2D3D-MATR✓28.515.91.290.5318.739.163.3
CoFiI2P✓24.313.21.110.4320.850.369.4
Hg-I2P✓26.615.11.310.5019.741.666.1
PlanaReLoc21.99.91.940.4228.357.372.8
ours25.08.51.860.4429.258.970.6
ours*17.66.21.730.3630.161.578.0

Results and analysis are provided in Section 4.2 (Table 3).

Q3: The appendix describes three training stages.

A3: Training consists of segmentation pretraining, SGM training, and joint matching training. Appendix C.2 specifies the stages and which parameters are frozen or optimized. Appendix C.1 details CGR architecture and message passing, and Appendix C.4 defines the supervision and loss weights. CGR is trained jointly with matching, without separate pretraining.

Q4: Table 1 reports a median translation error of 0.32 m ... complete PlanaReLoc pipeline was used?

A4: We implemented PlanaReLoc from the paper and use its basic pipeline without subsequent refinement, as stated in Table 1.

1. Query selection. We follow our ScanNet split and frame sampling, additionally removing frames with fewer than three visible map planes.

2. Training. Frozen MoGe-2 predicts query depth; sequential RANSAC extracts planes and visual features are pooled within each region. Two PointNet branches encode individual map-plane shape and global layout, followed by self-/cross-attention plane matching.

Stage one uses GT planes with normal/offset noise standard deviations of 5°/5 cm and query-plane dropout probability 0.1. Stage two uses planes recovered online by MoGe-2 and RANSAC. Predicted and GT regions are associated at IoU 0.3; annotated map-plane IDs provide labels for correct and unmatched planes. The visual-feature head, map encoders, and matcher are trained with AdamW (learning rate and weight decay 10⁻⁴), batch 2 accumulated to 16. The optimizer and scheduler are reset for stage two.

3. Map construction. Sequential RANSAC extracts planes within plane-supporting semantic instance regions, followed by merging compatible planes and removing fragments.

4. Pose estimation. Query plane extraction uses a 10 cm distance threshold, 0.9 normal dot product, and at most 16 planes, stopping when the current plane has fewer inliers than 1% of pixels. Matches are filtered by mutual nearest neighbors and confidence 0.1. Rotation RANSAC samples two nonparallel plane correspondences, uses a 10° normal-angle threshold, and re-estimates rotation from all inliers. No subsequent refinement is applied.

Reviewer_axFPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes IHF-ReLoc, an instance-aware framework for camera relocalization by matching a query image to a 3D point cloud map without retrieval-derived pose initialization. Scene Graph Matching establishes instance associations through cross-modal graph interaction and local correspondence verification. A Candidate Graph Reasoner then models point-to-patch correspondence hypotheses as nodes and uses projected geometric relations to refine matching scores. Coarse-to-fine matching alternates with PnP-based pose estimation and instance-mask filtering. The paper reports improved localization accuracy over selected baselines on ScanNet v2, evaluates cross-temporal localization on an instance-rich subset of 3RScan, and provides component ablations and qualitative results.

Critical strengths
  1. The paper is well-written and clearly organized, making the proposed framework easy to follow from motivation to methodology and experiments.
  2. The method is well motivated by the sources of ambiguity in cross-modal matching. It combines object-level context with local geometric consistency and pose feedback, providing a coherent approach to resolving ambiguous correspondences and limiting error propagation during refinement.
Critical weaknesses
  1. The methodological novelty is insufficiently established relative to closely related work. SceneGraphLoc (Miao et al., ECCV 2024) already exploits object-level scene graphs for cross-modal localization, while GraphI2P (Bie et al., CVPR 2025) learns correspondence consistency through graph-based filtering; the paper needs a sharper conceptual and empirical distinction to establish the significance of its instance hierarchy and candidate-level reasoning.
  2. The experimental validation is insufficient to establish the method’s generality. Comparisons against competing methods are limited to ScanNet v2, while the 3RScan evaluation compares only the authors’ own training configurations on a filtered, instance-rich subset. This leaves unclear whether the gains extend beyond one benchmark and whether the proposed framework is more robust to temporal changes than existing approaches.
  3. The baseline coverage leaves important methodological comparisons unresolved. EP2P-Loc [1] provides a closely related coarse-to-fine localization pipeline, while P2-Net [2] offers an established direct pixel-to-point matching baseline. A comparison with graph-based correspondence filtering, such as an appropriately adapted GraphI2P [3], would further clarify whether CGR offers benefits beyond existing correspondence-consistency mechanisms.

[1] Minjung Kim, Junseo Koo, and Gunhee Kim. “EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization.” ICCV, 2023. [2] Bing Wang et al. “P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching.” ICCV, 2021. [3] Lin Bie et al. “GraphI2P: Image-to-Point Cloud Registration with Exploring Pattern of Correspondence via Graph Learning.” CVPR, 2025.

Questions for authors
  1. How are the test-time 3D instance partitions obtained: from dataset annotations or a prediction model?
  2. Can you provide comparisons against competing methods on 3RScan, including performance across different numbers of visible reference-map instances?
  3. What thresholds and perturbation ranges define the training-time pose fallback, and how frequently is it activated?
1 comment

Q1: The methodological novelty is insufficiently established ... graph-based filtering.

A1: SceneGraphLoc primarily addresses coarse localization in a database of 3D scene graphs. GraphI2P bridges the modality gap using virtual point clouds and selects consistent correspondences through graph learning. CGR instead operates within instance-constrained progressive matching: each point retains multiple candidate patches, each hypothesis becomes a graph node, and projected relative geometry guides message passing and candidate scoring before the next scale. Our contribution is the integration of instance association, local verification, and pose-guided candidate refinement.

Q2: The experimental validation is insufficient ... visible reference-map instances?

A2: We have added external baseline comparisons in Section 4.2. All methods are evaluated on validation rescans using reference-scan maps. Baselines and ours* train on training reference scans and rescans; ours uses reference scans only.

MethodsLocal MapΔR Mean (°) ↓ΔR Med. (°) ↓Δt Mean (m) ↓Δt Med. (m) ↓Pose Recall (0.2 m, 10°) (%) ↑Pose Recall (0.5 m, 15°) (%) ↑Pose Recall (1.0 m, 30°) (%) ↑
2D3D-MATR✓28.515.91.290.5318.739.163.3
CoFiI2P✓24.313.21.110.4320.850.369.4
Hg-I2P✓26.615.11.310.5019.741.666.1
PlanaReLoc21.99.91.940.4228.357.372.8
ours25.08.51.860.4429.258.970.6
ours*17.66.21.730.3630.161.578.0

Results and analysis are provided in Section 4.2 (Table 3).

Evaluation uses queries with more than four valid visible reference-map instances, as specified in Appendix B.2.

Q3: The baseline coverage leaves important methodological comparisons unresolved.

A3: Some baselines cannot be reproduced directly from their public implementations. For example, the EP2P-Loc repository provides preprocessing but lacks the training/testing implementation. Our comparison covers related approaches: 2D3D-MATR represents coarse-to-fine cross-modal matching and directly compares P2-Net in its original paper; Hg-I2P uses graph relationships to filter inconsistent correspondences. These baselines provide references for evaluating our hierarchical matching and candidate reasoning.

Q4: How are the test-time 3D instance partitions obtained ... a prediction model?

A4: The test-time 3D instance partitions are obtained from the instance annotations provided by the dataset and are prepared offline during map construction. This paper studies cross-modal association and pose estimation given an instance-annotated 3D map, rather than 3D instance segmentation itself.

Q5: What thresholds and perturbation ranges define the training-time pose fallback ... activated?

A5: The fallback stabilizes training when an intermediate pose cannot provide reliable projection geometry. Both SGM and Progressive Matching use it when PnP fails or the visible regions under the estimated and GT poses do not overlap. A perturbed GT pose then provides the next geometric context. All stages use bounds of 10° rotation and 1 m translation, with random axes and directions.

Fallback is disabled during validation/testing: GT poses never enter the prediction path. If geometric feedback fails, predicted correspondence coordinates and valid point support are retained. Appendix C.2 gives the trigger and perturbation settings.

Reviewer_pLbpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IHF-ReLoc estimates camera pose from a query image and a 3D point-cloud map. SGM matches image and map instances, then local matching refines point-to-patch correspondences. CGR scores candidate correspondences using neighborhood geometry projected under an intermediate pose. PnP estimates and instance-mask filtering provide feedback for the next scale. The paper reports better ScanNet pose errors and recalls than the listed baselines, supported by component ablations. On 3RScan, adding training rescans improves cross-temporal localization.

Critical strengths
  1. SGM narrows the search to matching instances, and CGR checks the geometry of local candidates. Adding CGR reduces mean rotation/translation errors from 17.2°/1.21 m to 10.7°/0.92 m with the other modules fixed (Table 3).
  2. ScanNet pose recalls improve over Hg-I2P by 4.5, 12.6 and 9.1 percentage points (Table 1). The method achieves these gains without a retrieval-based initial pose or local-map truncation.
Critical weaknesses
  1. GraphI2P already uses correspondences as graph nodes, so the novelty distinction on p. 3 is unconvincing. The paper should explain what CGR adds and compare it with a GraphI2P-style module using the same features, candidates and pose solver.
  2. Table 2 compares only two training settings, while instance filtering removes about half the validation queries. This leaves the robustness claim weakly supported. The paper needs baseline comparisons and CGR/confidence ablations on 3RScan, with pose recall also reported for queries containing four or fewer valid shared instances.
Questions for authors
  1. Do the 3RScan gains come from rescan training or the proposed modules? Please compare baselines under the same training conditions, ablate CGR and confidence filtering.
  2. How sensitive is refinement to the initial SGM/PnP pose? Please report its initial accuracy and refinement results across pose-error ranges, and specify the fallback settings, score formulas, loss definitions and weights.
1 comment

Q1: GraphI2P already uses correspondences as graph nodes ... same features, candidates and pose solver.

A1: Both methods use correspondence nodes. CGR differs in 3D-neighborhood connectivity, edges comparing projected-point and candidate-patch displacements, and pose-dependent updates across scales. In Section 4.3, the topology-augmented CGM alternative gives mean RRE/RTE of 12.5°/1.14 m, versus 10.7°/0.92 m for CGR, with SGM and occlusion filtering enabled. This compares indirect feature-level topology propagation with direct candidate-level reasoning.

Q2: Table 2 compares only two training settings ... half the validation queries.

A2: Table 2 evaluates both reference and rescan queries under two training settings. Training rescans improve cross-temporal localization. Queries require more than four valid visible reference-map instances; the evaluation therefore concerns this instance-rich subset.

Q3: The paper needs baseline comparisons ... ablate CGR and confidence filtering.

A3: We have added external baseline comparisons in Section 4.2. All methods are evaluated on validation rescans using reference-scan maps. Baselines and ours* train on training reference scans and rescans; ours uses reference scans only.

MethodsLocal MapΔR Mean (°) ↓ΔR Med. (°) ↓Δt Mean (m) ↓Δt Med. (m) ↓Pose Recall (0.2 m, 10°) (%) ↑Pose Recall (0.5 m, 15°) (%) ↑Pose Recall (1.0 m, 30°) (%) ↑
2D3D-MATR✓28.515.91.290.5318.739.163.3
CoFiI2P✓24.313.21.110.4320.850.369.4
Hg-I2P✓26.615.11.310.5019.741.666.1
PlanaReLoc21.99.91.940.4228.357.372.8
ours25.08.51.860.4429.258.970.6
ours*17.66.21.730.3630.161.578.0

Results and analysis are provided in Section 4.2 (Table 3).

We also added the confidence ablation in Appendix D.3:

MethodsΔR Mean (°) ↓ΔR Med. (°) ↓Δt Mean (m) ↓Δt Med. (m) ↓Pose Recall (0.2 m, 10°) (%) ↑Pose Recall (0.5 m, 15°) (%) ↑Pose Recall (1.0 m, 30°) (%) ↑
w/o Confidence21.37.91.770.3929.759.774.1
Full Model17.66.21.730.3630.161.578.0

Confidence-ablation results are reported in Appendix D.3 (Table D.3).

Q4: How sensitive is refinement to the initial SGM/PnP pose ... pose-error ranges.

A4: Appendix D.4 reports stage-wise errors. From 32×32 to 16×16 to 4×4 matching, mean RRE decreases from 20.85° to 13.84° to 10.71°, and mean RTE from 1.52 m to 0.97 m to 0.92 m. Both medians also decrease. These results show improvement along the model's prediction trajectory.

Q5: specify the fallback settings, score formulas, loss definitions and weights.

A5: Appendix C.2 specifies the training-only pose fallback, including its triggers and perturbation bounds; validation/testing do not use GT poses. Appendix C.1 details instance retrieval, local verification, and CGR-refined matching scores. Appendix C.4 provides the loss definitions, supervision sets, and level-specific weights, covering bidirectional matching, confidence, and relative geometry. Gradients do not pass through PnP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.