acceptodds
Under review as a conference paper at ICLR 2027

Where Is Here? Spatial Frame-of-Reference Mismatch in Egocentric Text-to-Point-Cloud Localization

Abstract

Textual queries specify target locations through surrounding scene cues when exact coordinates are unavailable. Text-to-point-cloud (T2P) localization estimates the target position in a 3D point-cloud map from such queries. Existing T2P query protocols typically use map-frame directional relations, which can be interpreted as map-space constraints once the referenced objects are grounded. However, localization requests can instead describe local observations relative to an implicit observer-centered reference frame. An observer may know that a tree is on the left while the corresponding map-frame direction remains unknown. Even with correct object grounding, the same egocentric relation can imply different map-space constraints under different reference directions. We formalize this problem as Spatial Frame-of-Reference Mismatch. To study this setting, we introduce EgoT2P, a controlled benchmark with paired map-frame and egocentric queries over matched local maps and target positions. To test whether explicit reference-state modeling helps in this setting, we propose FoRLoc, which predicts a target-centered reference state consisting of a facing anchor and a coarse target-to-anchor bearing, together with auxiliary object correspondences, before estimating the target position. In the paired EgoT2P evaluation, all five evaluated T2P methods yield lower recall with egocentric queries. FoRLoc achieves a 20.6% relative improvement in 5 m localization recall over the strongest compared method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Point-Cloud Localization, Egocentric Localization, Frame-of-Reference Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2buWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the EgoT2P benchmark allowing for the text-to-point cloud localization. Additionally, the benchmark is extended by the introduced FoRLoc method designed for the setup. The method outperforms the baselines on the introduced benchmark.

Critical strengths
  • Well identified issue in the text to localization domain where the localization relativity poses a significant problem
  • Simple and effective solution to the presented problem
Critical weaknesses
  • The introduced benchmark is claimed as one of the novelties, also the motivation for this research comes from the relativity of the text prompt to an actual location. The design here is rather simple and only building an extra layer on the top of an existing dataset, which is rather an engineering step. It is not a weakness on its own - the most prominent issue lies in a conceptual naive approach here - the definition of “left” “right” “behind” “facing” is quite a simple design, and it is quite fuzzy definition on its own. Why has it been decided (Fig B1) that “left” is just approx. 33%? The left can mean 50% of the diagram. There is little literature supporting it (linguistic, behavioral studies?), rather ad hoc created labels to simplify the problem.
  • Also, even though the benchmark claims point clouds it actually operates only in the BEV on 2D images. Another concern pertains to the experimental setup. What is the performance of the introduced method on the other benchmarks already introduced in the community. The reproduced baselines indeed have a lower performance on the introduced dataset, but how does it compare to the already established ones? This leaves the generalisation question open. Since the claim in the novelty is both on the method and benchmark - this does not seem to be substantiated.
Questions for authors
  • As in the above-mentioned weaknesses, I would ask to compare the introduced method on at least one (or two) relevant benchmarks to showcase the capability of the method.
  • And also, I would like to ask about conceptual clarification on the design of the benchmark (potentially with numerical evaluation wrt other benchmarks - essentially why do we need this benchmark?) Or maybe, is this really scalable conceptually (hard labels on left/right/behind)? I would like to ask more conceptual discussion here.
5 comments
AuthorsOpenReview ↗

Responses to Reviewer 2buW (1/3)

We sincerely thank the reviewer for their time and thoughtful feedback. We appreciate their constructive comments, which help us better clarify the motivation and contributions of our work. We address the specific concerns below.


1. Q1 & W2 FoRLoc and evaluation on other benchmarks

1.1 Performance on other benchmarks

Answer: We appreciate the suggestion to evaluate FoRLoc beyond the main EgoT2P setting. Table 4 of the original submission reports results on the native cells of KITTI360Pose. We retain the original submaps, construct egocentric descriptions for the retained targets, and train all methods on the same split. This comparison evaluates fine localization in a different local map configuration within KITTI-360.

To also examine scenes outside KITTI-360, we evaluated FoRLoc on Toronto-3D [1] and Soufflot03 from Paris-CARLA-3D [2]. We retain the maps, target positions, and referenced objects, and construct egocentric queries using the EgoT2P protocol. For each target, all methods receive the same query and target-containing submap.

SceneMethodR@5 m ↑R@10 m ↑R@15 m ↑
KITTI360PoseText2Loc13.17%40.93%63.88%
KITTI360PoseVLM-Loc24.36%54.71%73.29%
KITTI360PoseFoRLoc29.02%58.94%79.18%
Toronto-3DText2Loc11.76%30.32%48.87%
Toronto-3DVLM-Loc15.38%42.53%61.99%
Toronto-3DFoRLoc21.27%45.70%62.90%
Soufflot03Text2Loc3.64%25.45%61.82%
Soufflot03VLM-Loc7.27%25.45%54.55%
Soufflot03FoRLoc10.91%32.73%65.45%

FoRLoc has higher recall than the two baselines at all three thresholds in these comparisons. The additional scenes provide evidence that its benefit carries over to other point-cloud maps when evaluated with the same egocentric query protocol.

[1] Toronto-3D: A large-scale mobile LiDAR dataset for semantic segmentation of urban roadways. CVPRW, 2020.

[2] Paris-CARLA-3D: A real and synthetic outdoor point cloud dataset for challenging tasks in 3D mapping. Remote Sensing, 2021.

1.2 Methodological Contribution of FoRLoc

Answer: In the map-frame setting used by prior T2P benchmarks, “I am east of the road” tells us which side of the road to search once we identify it. Now consider “the road is on my left.” Even with the correct road identified, a north-facing speaker could be east of it, while a south-facing speaker could be west of it.

The same grounded objects and the same words can therefore point to opposite sides of the map.

FoRLoc addresses this ambiguity by estimating the speaker’s reference state from the full query and local map. It predicts the facing anchor and the coarse target-to-anchor bearing, then uses this reference to interpret “on my left.”

Table 5 of the original submission shows that adding the predicted reference state improves localization even when the same predicted object correspondences are available. In our additional comparison for Reviewer Y1uh (Q2), predicting the target-to-anchor bearing also yields higher recall at all three thresholds than predicting alternative coarse geometric intermediate representations. Together, these results further highlight FoRLoc's advantages in egocentric localization through explicit reference-state modeling.

1.3 Role of the BEV representation

Answer: As defined in §3.1, our task is to localize a target on the ground plane of a given 3D point-cloud map. EgoT2P is built from such maps. Following VLM-Loc [1], FoRLoc uses both a BEV image and a graph of the map’s semantic instances for fine localization (§3.3). Even in this view, the model still needs to infer the observer’s reference direction to interpret egocentric relations in map coordinates. This is the spatial frame-of-reference mismatch we study.

[1] VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models. CVPR, 2026.

AuthorsOpenReview ↗

Responses to Reviewer 2buW (2/3)

2. Q2 & W1 Motivation and design of EgoT2P

2.1 Why Do We Need EgoT2P?

Answer: Existing T2P benchmarks primarily focus on map-frame localization, without directly evaluating the challenges introduced by observer-relative spatial descriptions. Unlike map-frame queries, which directly constrain target positions through grounded objects and map-frame directions, egocentric queries additionally require inferring the observer's reference direction to interpret spatial relations in the map.

T2P BenchmarkMain Focus of the Original EvaluationPaired Map/Ego Evaluation
KITTI360Pose [1]Text-guided submap retrieval, object matching, and fine localizationNo
CityLoc-K / CityLoc-C [2]Localization with larger local maps, multiple point-cloud sources, and complex spatial structuresNo
Text2Loc++ evaluation datasets [3]Localization across different levels of linguistic complexity and diverse environmentsNo
EgoT2PControlled evaluation of reference-frame differences in T2P localizationYes

EgoT2P addresses this evaluation gap through paired Map/Ego queries with matched scene evidence, including the same map, target, referenced objects, correspondences, and mention order (Appendix B.5). This enables a controlled comparison of how different spatial reference frames affect localization.

[1] Text2Pos: Text-to-Point-Cloud Cross-Modal Localization. CVPR, 2022.

[2] VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models. CVPR, 2026.

[3] Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language. TPAMI, 2026.

2.2 Interpreting directional terms

Answer: We agree that directional terms such as Left/Right do not have uniquely fixed angular boundaries in natural language, and interpreting Left as the entire left half-plane is a reasonable convention. However, two distinct issues should be distinguished: the context-dependent semantics of directional language and the explicit geometric rules required for consistent benchmark evaluation. EgoT2P addresses the latter, rather than attempting to define universal human directional semantics.

Specifically, EgoT2P's query construction is grounded in prior spatial cognition, language, and behavioral research:

  • Zhang et al. (ICLR'25, §3.4) [1] discuss why human acceptance regions can depart from disjoint 90° quadrants or uniform 180° half-planes. They evaluate directional understanding using both a binary half-plane function and a continuous cosine-based function.

  • Landau (CDPS'25) [2] describes Left, Right, and Behind in terms of orthogonal axes centered on a reference object and relative position vectors between objects. This motivates our use of the facing anchor to establish a direction and of relative object positions to define the regions.

  • Furthermore, Zhang et al. (ICLR’25) cite behavioral experiments by Franklin et al. (M&C’95) [3], who found mean conceptual widths of 124°, 110°, 91°, and 92° for Front, Back, Left, and Right in Experiment 1. These widths sum to 417°, exceeding 360° and indicating overlapping directional regions. In Experiment 3, participants used Left and Right over wider angular ranges of 144° and 140° when describing positions.

These studies provide a conceptual basis for our use of reference axes and potentially overlapping directional regions. The specific angular boundaries operationalize these principles for consistent and reproducible Map/Ego evaluation, with the goal of studying Spatial Frame-of-Reference Mismatch rather than defining universal directional semantics.

[1] Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities. ICLR, 2025.

[2] Geometry and Force Dynamics in Simple Spatial Terms: Two Theories, One Resolution. CDPS, 2025.

[3] Parsing surrounding space into regions. Mem. Cognit., 1995.

AuthorsOpenReview ↗

Responses to Reviewer 2buW (3/3)

2.3 The “left” region in Fig. B1

Answer: The studies discussed in §2.2 motivate allowing directional regions to overlap. In Fig. B1, a support object on the speaker’s left can be a candidate for “left” even if it is also somewhat ahead of or behind the speaker. To make this rule reproducible, we allow its forward or backward offset to be up to twice its leftward offset. This gives the roughly one-third candidate region (127°) shown in the figure, including its overlap with “behind.” To address your concern about whether the observed Map/Ego gap depends on this boundary choice, we repeated the comparison using the four 90° sectors of the map-frame convention in both frames.

Unified 90° Directional Encoding. We used identical four-way sectors and boundary rules in both frames, relative to the map axes for map-frame queries and to the speaker’s facing direction for egocentric queries. We kept the other pairing conditions fixed and retrained both baselines.

MethodDirectional encodingMap-frame R@5 ↑Egocentric R@5 ↑Δ Drop
VLM-LocOriginal59.61%26.01%33.6%
VLM-LocUnified 90°45.93%30.70%15.23%
Text2LocOriginal18.36%12.64%5.72%
Text2LocUnified 90°16.55%11.81%4.74%

Both conventions yield the same qualitative finding. Under either convention, both models achieve lower R@5 m when the same targets are described from the speaker’s perspective. The shared 90° partition narrows the gap, especially for VLM-Loc, but the gap persists for both models.

180° half-plane interpretation. Following your suggestion, we also examine an alternative interpretation where “left” and “right” each cover a 180° half-plane, while keeping the query text unchanged. We then measure the feasible area satisfying all spatial relations in each query.

MetricResult
Analyzed test queries3,092
Queries with strictly enlarged feasible regions99.94%
Median increase in feasible-region area+62.81 m²

As shown in the table above, this alternative interpretation allows more positions to satisfy the same query, reducing positional discrimination without necessarily better reflecting natural directional language.

2.4 Scalability Beyond Hard Labels

Answer: EgoT2P's directional labels provide a consistent way to represent observer-relative spatial relations, rather than defining the scope of the benchmark itself. The central evaluation concerns reference-frame differences, not a particular angular partition. As shown in the previous section, the Map/Ego localization gap persists under an alternative 90° directional convention.

To further examine flexibility in query composition, we evaluate 200 user-authored descriptions in which participants independently select 2–5 referenced objects and their spatial relations, as detailed in our response to Reviewer 7ezL (Q3). FoRLoc outperforms VLM-Loc across all three localization thresholds on user-authored queries. These results suggest that FoRLoc remains competitive even beyond the fixed six-object template, when users independently select referenced objects and spatial relations.

Reviewer_7ezLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies text-to-point-cloud localization when queries describe objects using egocentric directions such as left and right, while the observer’s heading is unknown. It introduces EgoT2P, which pairs egocentric and map-frame descriptions of the same locations, and FoRLoc, which first predicts a facing anchor and a coarse bearing before estimating the target position. The results show that existing methods struggle with egocentric descriptions and that the proposed intermediate representation helps.

Critical strengths
  1. I find the problem well motivated. People can usually describe what is on their left or right more easily than give cardinal directions, so this seems like a useful setting to study.
  2. The paired-query setup is a nice part of the paper. Keeping the maps, targets, and referenced objects fixed makes the comparison easier to interpret, and the fact that all five retrained baselines show a drop makes the observation convincing.
  3. The improvement over VLM-Loc is meaningful, with test R@5m increasing from 26.01% to 31.37%. I also appreciate the direct-prediction controls and the experiments that remove or replace intermediate predictions, which help explain why the approach works.
Critical weaknesses
  1. I am not entirely sure how much of the paired-query gap comes from difficulty reasoning about reference frames and how much comes from differences in the information provided by the two formulations. It would help to know whether the egocentric queries tend to admit larger or more ambiguous sets of possible locations.
  2. The intermediate state seems useful, although its benefit could partly come from breaking localization into easier geometric steps. A comparison with a simple geometric search using predicted object correspondences would make this aspect more convincing.
  3. The evaluation is still fairly narrow, since both main settings come from KITTI-360 and the manual rewrites preserve the original objects and relations. The rewrite experiment is helpful, but I would be interested in seeing whether the gains carry over to descriptions collected directly from users.
  4. The additional inference cost is noticeable, increasing from 0.85 to 2.14 seconds per query, and the gains are less consistent with retrieved submaps. Some discussion of when this trade-off is worthwhile would be useful.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Responses to Reviewer 7ezL (1/2)

We thank the reviewer for the constructive suggestions, which have helped us better clarify the contributions of our method. We address the concerns as follows.


Q1 & W1

Answer: To examine whether differences in positional ambiguity can explain the Map/Ego performance gap, we compare the feasible regions of paired queries describing the same targets. Specifically, for fully grounded Map/Ego pairs, we identify all locations satisfying all query relations on a 0.25 m grid and compare the area, spatial span, and connectivity of the resulting feasible regions.

Query RepresentationMean Area (m²)Median Area (m²)Median Spatial Span (m)Multiple Components (%)
Map-frame59.1334.139.860.00%
Egocentric36.9916.0014.7897.07%

Although Egocentric queries have smaller feasible areas, their feasible locations are more spatially dispersed. This suggests that feasible area alone does not fully capture positional ambiguity.

To further examine whether this remaining ambiguity accounts for the localization gap, we evaluate R@5m on queries where all feasible locations in both frames lie within 5 m of the GT target.

MethodMap-frameEgocentric
Text2Loc22.70%21.28%
VLM-Loc84.40%46.10%

Even when all feasible locations under both query formulations lie within 5 m of the ground-truth target, a substantial Map/Ego localization gap is still observed. Together with the unified 90° comparison in our response to Reviewer Y1uh (Q1), these results suggest that differences in positional ambiguity and directional encoding alone cannot fully explain the observed gap, further supporting the importance of Spatial Frame-of-Reference Mismatch in T2P localization.


Q2 & W2

Answer: To examine whether FoRLoc's gains mainly come from decomposing localization into simpler geometric steps, we compare it with a geometric search baseline on all test queries. The baseline uses the same predicted object correspondences and query relations. For each candidate location, the reference direction points toward the predicted facing anchor, and candidates are scored by their geometric consistency with matched spatial relations.

Localization MethodR@5m ↑R@10m ↑R@15m ↑
Geometric Search (Predicted Correspondences)26.72%57.00%76.07%
FoRLoc (Ours)31.37%65.11%82.28%

Simple geometric search does not match the localization performance of the full FoRLoc pipeline. Together with the Two-Stage Direct and assignment-only ablations (Table 5 in the original submission) , these results suggest that FoRLoc's gains are not solely due to decomposing localization into simpler stages, and that reference-state modeling can further help improve localization by facilitating the interpretation of egocentric spatial relations in the map.

AuthorsOpenReview ↗

Responses to Reviewer 7ezL (2/2)

Q3 & W3

Answer: To evaluate FoRLoc on descriptions collected directly from users, we collect user-authored queries for the same 200 test locations. Unlike the Rewritten queries, participants independently select 2–5 referenced objects and describe their spatial relations in their own words from a first-person perspective. An example is provided in the table below and more examples are provided in the revised Appendix D.4.

Original (Template-based)User-authored
The target location is a gray building on its left, a gray-green road on its right, a green sidewalk on its right, facing a black fence, a gray-green vegetation on its left, and a green wall on its left.I have a black fence straight ahead of me. Over on my left is a green wall, and I have a gray building on my left.
MethodQuery SettingR@5m ↑R@10m ↑R@15m ↑
VLM-LocOriginal27.00%61.50%78.50%
VLM-LocRewritten16.00%48.00%70.50%
VLM-LocUser-authored13.50%43.50%62.50%
FoRLocOriginal31.00%67.00%82.50%
FoRLocRewritten26.50%58.00%74.00%
FoRLocUser-authored17.50%44.50%65.50%

FoRLoc achieves better localization performance on user-authored queries across all three thresholds. This complements the original rewrite evaluation by further examining localization performance when users independently select referenced objects and spatial relations, rather than only varying the wording.


Q4 & W4

Answer: To further clarify the inference cost of FoRLoc, we discuss it from the following two perspectives.

  1. When a reliable local map is available. Consider human-guided robotic relocalization. An operator may know which area the robot is in and describe its position using nearby objects. For an occasional relocalization request, locating the robot accurately within that area may matter more than minimizing response time. Our accuracy–cost comparison supports this use of additional computation, as FoRLoc outperforms VLM-Loc with additional position samples in R@5m at the closest tested inference budget (Reviewer Y1uh, Q6).

  2. When the local map must be retrieved. FoRLoc’s gains are less consistent when all methods receive the same top-1 submap. Retrieving a submap that contains the target is an upstream challenge shared by coarse-to-fine T2P pipelines. Our work focuses on the subsequent fine-localization stage, where FoRLoc interprets egocentric spatial relations within the supplied point-cloud map. This is consistent with the scope defined in Section 3.1.

Reviewer_Y1uhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies egocentric text-to-point-cloud localization, where spatial relations such as “on the left” are expressed relative to an implicit observer-centered frame. The authors introduce EgoT2P, a benchmark with paired map-frame and egocentric queries, and propose FoRLoc, a two-stage method that predicts an intermediate reference state and object assignments before localizing the target. The experiments show that existing methods perform worse on egocentric queries and that FoRLoc improves over the strongest evaluated baseline.

Critical strengths
  1. The paper addresses a meaningful problem in language-guided localization. The distinction between grounding referenced objects and interpreting their relations in the map frame is clearly motivated.

  2. The paired-query protocol is a useful benchmark design. By keeping the local map, target position, and referenced objects fixed, it enables a controlled comparison between map-frame and egocentric formulations.

  3. The evaluation includes several helpful analyses, including component ablations, query rewrites, challenging subsets, and an experiment on native KITTI360Pose submaps.

  4. The paper is generally clear and acknowledges important limitations, including the use of template-generated queries, annotated semantic instances, and a single underlying dataset.

Critical weaknesses
  1. The paired-query comparison changes not only the reference frame but also the directional partitioning and label-assignment rules. As a result, the performance gap in Table 1 may reflect both reference-frame uncertainty and differences in relation encoding.

  2. The predicted bearing is derived from the target-to-anchor direction and therefore contains coarse target geometry. Although the ablations support the usefulness of this intermediate signal, they do not yet isolate the benefit of reference-state modeling from that of coarse geometric supervision.

  3. EgoT2P relies on template-generated queries and annotated semantic-instance maps from KITTI-360. The current evaluation therefore provides limited evidence about performance on naturally authored queries, imperfect map inputs, or other environments.

  4. FoRLoc improves localization accuracy but has higher inference cost than VLM-Loc. The evaluation would be stronger with repeated runs or variability estimates, and with a more direct comparison of accuracy under matched computation budgets.

Questions for authors
  1. The paired-query protocol controls the local map, target position, and referenced objects, which is valuable for comparing the two query formulations. However, Appendices B.2 and B.5 use different directional partitioning and label-assignment rules. An additional comparison using consistent angular partitions and assignment rules would help distinguish the effects of reference-frame uncertainty from those of relation encoding. This would strengthen the interpretation of the performance gap observed in Table 1.

  2. The Two-Stage Direct baseline provides useful evidence that adding a second localization stage alone does not reproduce FoRLoc’s gains. Nevertheless, the proposed bearing label also encodes coarse target geometry, as acknowledged in Section 3.4 and Appendix C.2. Comparing the reference state with alternative coarse geometric intermediate representations, such as target-grid classification or quantized relative displacement, would further clarify the specific benefit of reference-state modeling beyond that of intermediate geometric supervision.

  3. It would be useful to quantify the positional ambiguity remaining after all query relations are considered. Although a reference state can correspond to multiple positions, this does not necessarily imply that the complete query remains ambiguous at the evaluation thresholds. An analysis of the spatial extent and connectivity of regions satisfying all described relations could help distinguish model errors from ambiguity inherent in the input. Multiple hypotheses or geometric-consistency metrics could then be explored for highly ambiguous cases.

  4. Training Stage 2 with predicted intermediate states and allowing assignment correction are sensible measures for handling Stage 1 errors. However, these training inputs are generated by Stage 1 on its own training split, and their error distribution may differ from that on unseen data. Please report Stage 1 performance on the training and validation splits and, if a substantial gap exists, examine out-of-fold predictions or controlled perturbations. Perturbing bearing estimates while preserving valid node assignments would also provide a more targeted robustness analysis than jointly shuffling intermediate outputs across samples.

  5. The object-level graph control in Appendix C.1 simultaneously removes primitive-type annotations, semantic groups, and primitive groups. Consequently, Table D3 establishes their combined benefit but does not isolate the contribution of each component. Separate ablations would clarify which aspects of SP-SHG improve localization. Given the relatively small differences between some variants, reporting variability across repeated training runs would also help assess the robustness of these gains.

  6. Appendix D.6 transparently reports the accuracy–cost trade-off and acknowledges that Two-Stage Direct is not a matched-compute control. FoRLoc requires approximately 2.5 times the amortized inference time of VLM-Loc under the reported setting. A comparison under matched inference budgets, or an accuracy–cost curve, would further clarify the value of the proposed intermediate representation relative to alternative uses of additional computation, such as candidate sampling or localization refinement. Single-query latency would also be informative if online deployment is discussed.

  7. The limitations concerning template-generated queries, annotated semantic instances, and evaluation within KITTI-360 are appropriately acknowledged. To better characterize sensitivity to the current assumptions, a small controlled robustness study would be valuable. Possible tests include missing support hints, perturbations to map instances, or offsets between the observer’s heading and the target-to-anchor bearing. Such experiments would help establish the method’s practical operating range without requiring a substantially expanded benchmark.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer Y1uh (1/3)

Thank you for your thoughtful review and for explaining your concerns in detail. In the time available, we prioritized the experiments that most directly address those concerns. We respond to each question below.


Q1 & W1

Answer: We have added a controlled Map/Ego comparison using identical 90° directional partitions and label-assignment rules, with map-frame queries rebuilt to match the egocentric queries at the same target locations. The R@5/10/15m results are shown in the table below.

MethodMap-frameEgocentricΔ Drop
VLM-Loc45.93 / 79.67 / 91.8730.70 / 61.67 / 78.9615.23 / 18.00 / 12.91
Text2Loc16.55 / 46.52 / 71.8011.81 / 39.20 / 66.004.74 / 7.32 / 5.80

Both methods retain a Map/Ego gap at all three thresholds under these shared rules. This suggests that the need to infer the observer's reference direction may contribute to the remaining performance gap. The persistent gap supports spatial frame-of-reference mismatch as a challenge in T2P localization.


Q2 & W2

Answer: To further examine the benefits of reference-state modeling beyond coarse geometric supervision, we further compare it with two alternative geometric representations under identical experimental settings.

  • Target-grid classification (4×4) encodes coarse absolute position as the target's cell in a uniform 4×4 grid on the 224×224 BEV map.

  • Quantized relative displacement (4×4) encodes the 2D displacement from the target to the facing-anchor centroid using four bins per axis.

Intermediate RepresentationR@5m ↑R@10m ↑R@15m ↑
Target Grid (4×4)27.63%60.14%79.12%
Quantized relative displacement (4×4)26.12%58.56%77.13%
Target-to-anchor Bearing (16 bins, FoRLoc)31.37%65.11%82.28%

The reference state outperforms both alternative representations across all three thresholds, supporting its benefit for egocentric localization. This comparison further clarifies the specific value of reference-state modeling over alternative forms of intermediate geometric supervision.


Q3

Answer: To quantify the remaining positional ambiguity, we examine grounded Map/Ego query pairs on a 0.25 m grid to identify locations satisfying all query relations. We then report the mean and median feasible area, median spatial span (maximum pairwise distance between feasible locations), and the percentage of queries with multiple 8-connected components.

Query RepresentationMean Area (m²)Median Area (m²)Median Spatial Span (m)Multiple Components (%)
Map-frame59.1334.139.860.00%
Egocentric36.991614.7897.07%

Although Egocentric queries have smaller feasible areas than Map-frame queries, their feasible locations are more spatially dispersed, with 97.07% exhibiting disconnected regions. This suggests that feasible area alone does not fully capture positional ambiguity.

To examine whether this remaining ambiguity accounts for the localization gap at the 5 m evaluation threshold, we further evaluate R@5m on queries where all feasible locations in both frames lie within 5 m of the GT target.

MethodMap-frameEgocentric
Text2Loc22.70%21.28%
VLM-Loc84.40%46.10%

Even when all feasible locations lie within 5 m of the GT target, a substantial Map/Ego localization gap can still persist. This suggests that input ambiguity alone cannot explain the observed gap in this subset, further supporting our motivation for studying Spatial Frame-of-Reference Mismatch in T2P localization.

AuthorsOpenReview ↗

Responses to Reviewer Y1uh (2/3)

Q4

Answer: We compare Stage 1 performance on the training and validation splits and conduct controlled bearing-only perturbations to examine how bearing estimation errors affect Stage 2 localization.

  1. Stage 1 Prediction Quality
SplitAnchor Acc.Bearing Acc.Assignment Acc.
Train88.03%67.69%69.25%
Validation84.42%63.08%65.03%

Stage 1 shows relatively modest train–validation gaps of 3.61–4.61 percentage points across the three prediction metrics, without a substantial degradation on validation.

  1. Bearing-only Perturbation

We perturb only the predicted bearing by ±1 or ±2 bins (22.5°/45°), while keeping all other inputs unchanged. Results are averaged over positive and negative offsets.

Bearing PerturbationR@5m ↑R@10m ↑R@15m ↑
Original Prediction31.37%65.11%82.28%
±1 bin (22.5°)30.91%64.85%81.37%
±2 bins (45°)28.12%61.83%79.64%

These results show that Stage 2 can tolerate small bearing estimation errors with limited impact on localization. Larger bearing offsets lead to a larger decrease in localization accuracy, consistent with the role of the estimated reference direction in interpreting egocentric spatial relations. FoRLoc combines this estimate with the full query and map, allowing it to tolerate small bearing errors, as shown above.


Q5

Answer: To better isolate the contribution of each SP-SHG component beyond the combined ablation, we separately remove primitive-type annotations, semantic groups, and primitive groups and evaluate their effects on localization performance.

Map RepresentationPrimitive TypeSemantic GroupsPrimitive GroupsR@5m ↑R@10m ↑R@15m ↑
SP-SHG w/o Primitive Type❌️✅✅31.25%64.83%81.92%
SP-SHG w/o Semantic Groups✅❌️✅31.11%64.49%81.45%
SP-SHG w/o Primitive Groups✅✅❌️31.18%64.65%81.68%
Full SP-SHG (FoRLoc )✅✅✅31.37%65.11%82.28%

Removing any one of the three components lowers recall at all three thresholds. Removing semantic groups gives the largest observed drop, followed by primitive groups and then primitive-type annotations. The drops are modest in each case and smaller than when all three components are removed together as shown in Table D3 of the original submission.


Q6 & W4

Answer: To examine whether additional inference computation can reproduce FoRLoc's gains, we evaluate VLM-Loc by generating 1–5 position candidates per query, selecting the candidate with the smallest total distance to the other valid candidates.

MethodCandidatesR@5m ↑R@10m ↑R@15m ↑Time (s/query)
VLM-Loc126.01%59.02%77.43%0.85
VLM-Loc226.60%58.90%77.31%1.60
VLM-Loc328.55%61.58%80.58%2.39
VLM-Loc429.56%62.23%81.09%3.19
VLM-Loc530.06%62.98%81.11%3.99
FoRLoc131.37%65.11%82.28%2.14

Although generating additional position candidates improves VLM-Loc's R@5m, it still trails FoRLoc by 2.82 pp with 3 candidates under the closest tested inference budget, and by 1.31 pp even with 5 candidates. These results suggest that simply increasing direct position sampling does not reproduce FoRLoc's gains within the evaluated range, supporting the value of reference-state modeling as an alternative use of inference computation. The accuracy–cost curve plot is included in the revised manuscript (Fig. 5).

AuthorsOpenReview ↗

Responses to Reviewer Y1uh (3/3)

Q7 & W3

Answer: To assess FoRLoc's sensitivity to variations in query structure and available spatial cues, we collect user-authored descriptions for 200 test locations. Participants freely select 2–5 referenced objects and describe the locations in their own words. Examples are provided in the revised Appendix D.4.

MethodQuery SettingR@5m ↑R@10m ↑R@15m ↑
VLM-LocOriginal27.00%61.50%78.50%
VLM-LocRewritten16.00%48.00%70.50%
VLM-LocUser-authored13.50%43.50%62.50%
FoRLocOriginal31.00%67.00%82.50%
FoRLocRewritten26.50%58.00%74.00%
FoRLocUser-authored17.50%44.50%65.50%

Despite reduced accuracy on user-authored queries, FoRLoc still retains its advantage over VLM-Loc across all three localization thresholds.

The Partial Context experiment (Fig. 4 in the original submission) further shows that FoRLoc does not require every referenced object to be matched in the map, while the bearing-only perturbation in Q4 demonstrates tolerance to small reference-direction errors. Together, these results indicate that FoRLoc can achieve good localization performance under imperfect spatial evidence, although the availability of spatial cues and the accuracy of reference-direction estimates still influence localization accuracy. This is consistent with our original analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.