acceptodds
Under review as a conference paper at ICLR 2027

Matched, but Where? Query-Conditioned Point Prediction for 3DGS-based Visual Localization

Abstract

3D Gaussian Splatting (3DGS) has emerged as a powerful scene representation for visual localization, enabling direct matching between 2D image keypoints and 3D Gaussian primitives as well as efficient rendering for pose refinement. Existing direct matching methods typically assign the center of a matched Gaussian to the query keypoint. However, a Gaussian represents a volumetric region, and its center generally differs from the observed scene point. That is to say, a correct Gaussian match can therefore yield an inaccurate 3D coordinate, compromising PnP-based pose estimation. To address this limitation, we propose ReCoLoc, a region-to-coordinate framework that explicitly separates region retrieval from point prediction by equipping Gaussian landmarks with query-conditioned coordinate functions. Specifically, contribution-weighted identity encoding aggregates multiview descriptors using Gaussian rendering contributions, yielding reliable regional identities for retrieval. Query-conditioned coordinate decoding then predicts the corresponding 3D position within the retrieved Gaussian region from the query descriptor, thereby reducing the geometric error introduced by center-based assignment. Considering that retrieval and coordinate prediction both determine correspondence quality, correspondence-aware landmark selection jointly evaluates their reliability when constructing the localization map. Also, we introduce shared-subspace map compression, which jointly encodes identity descriptors and coordinate functions in a common low-dimensional basis, reducing direct-matching map storage substantially with little loss of localization accuracy. Experiments on Cambridge Landmarks, 7-Scenes, and 12-Scenes demonstrate that ReCoLoc outperforms current state-of-the-arts in terms of localization accuracy and recall.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual localization, 3D Gaussian splatting, 2D-3D correspondence, correspondence learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yRuvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReCoLoc, a 3DGS-based visual localization framework that separates Gaussian region retrieval from query-conditioned 3D coordinate prediction. Each Gaussian is associated with an identity descriptor for retrieval and an affine coordinate decoder that predicts a 3D point based on the query feature. The framework constructs these representations from mapping-view observations and introduces a correspondence-aware landmark selection strategy that considers both retrieval and coordinate prediction quality. A shared-subspace compression scheme is further proposed to reduce map storage. At inference time, the predicted 2D–3D correspondences are used for camera pose estimation via RANSAC-PnP, with optional pose refinement.

Critical strengths
  1. Insightful problem formulation and intuitive method design. The paper identifies an interesting limitation of 3DGS-based visual localization, i.e., assigning a single fixed 3D coordinate to each Gaussian may introduce ambiguity because different image keypoints can correspond to different locations within the same Gaussian region. The proposed region-to-coordinate formulation addresses this issue by separating Gaussian identity retrieval from query-conditioned coordinate prediction. The perspective is well motivated, intuitive, and potentially useful for improving 2D–3D correspondences in Gaussian-based localization.

  2. Strong empirical results and well-designed ablation studies. The method achieves competitive localization performance across three established benchmarks. More importantly, the experiments provide targeted evidence for the central contribution. In particular, the comparisons against Gaussian centers and weighted mean coordinates under identical retrieval results help isolate the benefit of query-conditioned prediction. Additional analyses of shared Gaussian identities and alternative coordinate representations further strengthen the empirical support for the proposed approach.

Critical weaknesses
  1. Insufficient justification for the proposed dimensionality reduction strategy. While the paper provides ablations on compression dimensions and the joint basis construction, it remains unclear why this post-hoc compression strategy is preferable to directly fitting the landmark representations in a low-dimensional feature space. A comparison with such a straightforward alternative would help establish the necessity and benefits of the proposed design, beyond demonstrating that compression can preserve localization performance.

  2. Limited analysis of the coordinate learning mechanism and its generalization. The query-conditioned coordinate decoder is a central component of the proposed method, yet its fitting quality and generalization across views are not directly evaluated. Although the paper reports improvements in correspondence accuracy and downstream localization, it does not provide a comparison of coordinate prediction errors between fitting and held-out scoring views, nor does it analyze how prediction quality varies with the number of observations per Gaussian. Given that the decoders are independently fitted using pseudo-coordinate targets that may be inconsistent across views, these analyses would be valuable for understanding the reliability of the learned mappings and whether the improvements primarily reflect accurate coordinate prediction or the effectiveness of subsequent landmark selection and robust pose estimation.

Questions for authors
  1. Could the authors clarify the motivation for the specific scoring formulations in Eqs. (7) and (8), and whether alternative scoring functions were considered?

  2. Why is SplitGS-Loc omitted from the 12-Scenes benchmark and recall comparisons in Tables 1–4? Are the pose refinement protocols comparable across methods?

  3. Given the relatively low Top-1 Gaussian association accuracy (21.26%) in Table 6, could the authors explain how incorrect Gaussian retrieval affects coordinate prediction and final localization performance?

Reviewer_UY6hOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ReCoLoc, a 3D Gaussian Splatting-based visual localization framework that separates Gaussian region retrieval from query-conditioned 3D coordinate prediction. Instead of assigning each matched keypoint to a Gaussian center, it uses a lightweight per-Gaussian affine decoder to predict the corresponding 3D point for PnP pose estimation. The framework also incorporates correspondence-aware landmark selection and shared-subspace map compression.

Critical strengths
  1. The method is well motivated and easy to follow. Separating Gaussian region retrieval from query-conditioned coordinate prediction directly addresses the limitations of center-based correspondences. The lightweight affine decoder, fitted through closed-form ridge regression, provides a simple and efficient implementation.

  2. The paper evaluates the method on three standard benchmarks and provides extensive ablations. Additional experiments on alternative coordinate representations, feature extractors, landmark selection, and map compression further support the proposed design.

Critical weaknesses

The paper aims to address the geometric discrepancy between Gaussian centers and observed scene points. However, its coordinate supervision is derived from the maximum-density point of each Gaussian along individual observation rays. These targets need not lie on the physical surface and can assign different 3D coordinates to observations of the same physical point across views. While the localization gains are convincing, pose and reprojection metrics alone do not establish improved 3D accuracy or cross-view consistency. Without direct geometric validation, the paper’s central explanation of correcting correspondence geometry remains insufficiently supported.

Questions for authors
  1. Cross-view consistency of pseudo-coordinate supervision. Equation (3) uses the point of maximum Gaussian density along an observation ray as the coordinate target. Consequently, observations of the same physical point from different viewpoints may receive different target coordinates. Could the authors clarify how this inconsistency affects coordinate decoding and final localization accuracy? For example, could they quantify the positional error between these maximum-density points and the corresponding SfM-reconstructed 3D points?

  2. Advantages of Gaussian regions over generic spatial partitions. The proposed separation of region retrieval and within-region coordinate prediction is conceptually related to region-conditioned coordinate regression methods such as HSCNet and NeuMap. Could the authors clarify the specific advantages of using Gaussians as local regions rather than voxels or clusters of 3D points? With observations, local regression models, and storage budgets kept as comparable as possible, could experiments distinguish the benefits of Gaussian geometry from those of local coordinate regression more generally?

  3. Sensitivity of coordinate prediction to appearance changes and observation coverage. Query descriptors may vary not only with the corresponding spatial location but also with illumination, viewpoint, and occlusion. Could the authors evaluate how the method’s performance changes under two conditions: (a) reducing the number of mapping images, resulting in sparse mapping views, and (b) increasing the viewpoint differences between query and mapping images?

1 comment

Thank you for the detailed response and the additional experiments. Most of my questions have been adequately addressed, in particular the analysis of 3D error and cross-view consistency, and the controlled comparison between Gaussian regions and generic partitions. I maintain my positive rating.

Reviewer_5wZpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReCoLoc, a 3DGS-based localization framework that matches 2D query keypoints to Gaussians and then predicts the corresponding 3D point inside each matched Gaussian with a closed-form decoder. It also selects landmarks by retrieval and reprojection quality and compresses the map with a shared low-dimensional basis. ReCoLoc outperforms recent 3DGS localizers on Cambridge Landmarks, 7-Scenes and 12-Scenes.

Critical strengths

The problem is clear and well motivated. The paper observes that using the Gaussian center as the 3D point is a concrete error source in 3DGS-based 2D-3D matching, and it is the first in 3DGS-based localization to predict a query-specific 3D point inside the matched Gaussian with a closed-form decoder. The results are solid.

Critical weaknesses

The paper does not show the need for a 3DGS map. SfM points have no center bias, and the same landmark selection, descriptor aggregation, and GS-CPR refinement could be applied to the SfM point cloud used to initialize the 3DGS. Without this direct-matching baseline, the gains may come from fixing an error introduced by the 3DGS map itself.

Questions for authors
  1. How does direct matching on that SfM point cloud perform with the same landmark selection, descriptor aggregation, pose solver and refinement?
  2. How does the decoder's accuracy change with the distance between the query and the nearest mapping view?
  3. Tiny-ReCoLoc is on par with or slightly better than the full model. Would a full-dimensional decoder with stronger ridge regularization perform similarly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.