acceptodds
Under review as a conference paper at ICLR 2027

BEARING: A Benchmark for Egocentric–Allocentric Correspondence in Map-Guided Indoor Navigation

Abstract

Map-guided navigation draws on the spatial intelligence of multimodal models, since a model must relate an allocentric map to its egocentric view and act on the result. Although maps tailored to a particular system improve its navigation, they are judged only by that navigation, so a general model's ability to relate map and view has received little systematic study. We introduce BEARING, a benchmark that tests this ability through scale, pose and identity correspondences and pairs each test with navigation from the same pose. Its 5 task categories share 60 routes in 9 Matterport3D buildings, from closed-loop navigation of a route to open-loop planning of its segments and static questions at their starts. We evaluate 9 closed-source, open-source and embodied models and find that all remain well below a human reference, with the widest gap in self-localisation. They lose directions rather than distances, in estimating their own heading, reading bearings on the map or converting them into their own frame, and only the strongest models read the map's compass and agent marker as intended. Relating the tasks to one another, we find that identity tests predict the progress of the segment that starts at their pose but not of other segments on the same route, so a navigation failure can be diagnosed with tests at the pose where it occurs.

Primary area:
datasets and benchmarks
Keywords:
benchmark, vision-and-language navigation, spatial reasoning, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bhmWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes BEARING, a benchmark for evaluating egocentric–allocentric correspondence in map-guided indoor navigation. The benchmark defines five categories of tasks—Scale, Pose, Identity, Transition, and Navigation—covering static correspondence questions, segment-level planning, and full-route navigation. The benchmark is built from 60 routes in 9 Matterport3D buildings, with 1,418 static items, 119 transition segments, and 60 navigation episodes. The paper evaluates 9 multimodal models and compares them against a human reference on a subset.

The paper’s main claims are that current models remain far below humans, especially on self-localization; that models lose directional information more than distance information; that identity tests predict transition performance only at the same pose; and that weaker models benefit more from maps expressed in text or in the agent’s own frame.

Critical strengths

The paper targets a meaningful gap in current evaluation practice. It is reasonable and useful to ask whether map-guided navigation failures come from poor map–view correspondence rather than only measuring final navigation success. The attempt to decompose this capability into multiple task types is conceptually worthwhile.

The paper is clearly written and includes more analysis than a typical benchmark submission. The benchmark setup, task taxonomy, and most analyses are presented clearly. The authors also include several diagnostic experiments beyond simple leaderboard reporting.

Critical weaknesses

The benchmark scale and dependence structure are too limited to support the paper’s stronger interpretive claims. Although the benchmark is carefully designed, it is built from only 9 buildings, 60 routes, and 119 transition segments, and many static questions are anchored at the same segment starts and therefore share the same pose, map, and view context. This limits the effective number of independent evaluation states. As a result, the paper’s stronger claims—especially about diagnosing navigation failures at the pose where they occur, model ranking stability, and broad conclusions about spatial reasoning failures—seem insufficiently supported at the current scale.

The evaluation is substantially entangled with a specific map interface and protocol, which weakens the interpretation of the results. A major empirical finding of the paper is that performance changes markedly when the same map content is rewritten into text or aligned to the agent’s frame, and that many models do not reliably use the compass, scale bar, or agent marker as intended. This suggests that BEARING may be measuring competence with one particular map representation at least as much as general egocentric–allocentric correspondence. In addition, there are protocol issues that materially affect credibility of the comparisons, including the acknowledged Pose2View filename leakage for GPT-5.6 Sol and the apparent mismatch between some embodied models and the benchmark interaction protocol.

Questions for authors

Can you rerun GPT-5.6 Sol after removing the Pose2View filename leakage and update all Pose-related tables/analyses that depend on it?

Can you report uncertainty estimates for the main benchmark comparisons, especially confidence intervals or ranking stability analyses at the route/building level for Table 2?

Can you provide a stronger analysis for the “diagnosis” claim, e.g., predicting segment failure from same-pose static tests while controlling for route/building difficulty and shared context?

Can you clarify the intended scope of the benchmark claims: is BEARING primarily a benchmark for general spatial correspondence, or for competence with this specific map representation and prompting convention?

7 comments

We thank the reviewer for the careful reading and the concrete questions. We answer in four parts: measurement precision with items nested in routes and buildings (W1, Q2), the link between tests at a segment's start and navigation (W1, Q3), robustness to the map representation (W2, Q4), and the Pose2View result of GPT-5.6 Sol and the evidence on embodied models (W2, Q1).

1. Measurement precision with items nested in routes and buildings (W1, Q2)

Intervals and ranking stability (W1, Q2)

For every category and for Overall, we compute the score with a 95% confidence interval and will add these intervals to Table 2 in the paper. Because items that share a segment start, with its pose, map and view, are not independent, the intervals resample whole routes: each score is computed on the actual data, and its interval comes from a bootstrap in which we resample whole routes within each building 1,000 times, recompute the score each time, and take the 2.5th and 97.5th percentiles (Appendix E.1 in the paper). The items of a start are therefore always drawn together, and each interval reflects the number of routes rather than the number of items; all intervals in this response are computed this way. GPT-5.6 Sol's Pose and Overall use its corrected Pose2View answers (part 4), which moves it from third to fourth on Pose; all other ranks are as in the paper.

Rebuttal Table 1. Category and Overall scores with 95% confidence intervals.

ModelScalePoseIdentityTransitionNavOverall
Gemini-3.8-Flash64.8 [61.1, 68.0]72.8 [68.9, 76.7]56.0 [51.1, 61.1]64.5 [60.8, 68.1]68.7 [61.1, 75.7]60.9 [58.3, 63.3]
GPT-5.6 Sol63.2 [59.4, 66.7]46.8 [42.3, 51.4]55.3 [51.3, 59.6]59.5 [54.8, 64.4]45.8 [37.2, 54.0]48.7 [45.6, 51.7]
Claude-Opus-552.1 [48.3, 55.5]61.7 [57.3, 65.9]44.9 [40.3, 50.5]49.7 [44.7, 54.9]58.1 [49.8, 66.5]47.1 [44.1, 50.4]
Qwen3.8-27B59.5 [56.4, 62.2]47.1 [42.2, 52.1]43.6 [39.1, 48.4]48.3 [43.5, 52.8]43.3 [34.0, 52.4]42.2 [39.4, 44.9]
Gemma-4-31B47.4 [43.9, 50.8]32.3 [28.1, 36.8]28.1 [23.9, 32.3]41.0 [36.3, 45.7]33.3 [24.6, 42.3]28.8 [26.0, 31.7]
Hy-Embodied-A3B46.7 [43.1, 50.2]18.9 [16.0, 22.3]18.2 [14.8, 21.9]20.6 [17.4, 23.9]11.6 [5.3, 18.4]14.4 [12.4, 16.6]
InternVL3.5-38B41.2 [38.0, 44.1]14.8 [11.5, 18.2]14.0 [11.4, 17.5]15.7 [12.2, 19.5]1.3 [0.0, 3.8]8.2 [6.8, 9.9]
Cosmos-Reason2-32B40.0 [37.0, 42.8]13.8 [10.9, 16.9]11.6 [8.8, 15.0]13.9 [11.4, 16.3]2.7 [0.0, 6.7]7.2 [6.1, 8.7]
RoboBrain2.0-32B32.6 [29.0, 36.0]12.4 [9.5, 15.3]11.4 [8.1, 15.1]19.8 [16.3, 23.3]4.1 [0.9, 8.1]6.0 [4.4, 7.8]

Most differences between consecutively ranked models are statistically significant. For each pair of models with consecutive Overall ranks, the 95% confidence interval of their difference excludes zero, except for GPT-5.6 Sol and Claude-Opus-5 (1.6 [−1.7, 4.7]) and among the three lowest models (InternVL3.5-38B, Cosmos-Reason2-32B and RoboBrain2.0-32B). We will state these exceptions in §4.2 of the revised paper.

Stable across buildings. The top-model ranking is consistent across all nine buildings, while several closely performing models exchange positions. Removing any one building and recomputing all results from the remaining eight leaves all nine Overall ranks unchanged and keeps the paper's three main findings in all nine runs:

  • closed-source models take the top two places in every category (§4.2);
  • for all models except Gemini-3.8-Flash and Claude-Opus-5, most Self-Pose headings align with the map axes (63–100%, compared with 67–100% when all buildings are included; §4.3);
  • Identity accuracy correlates positively with progress at the segment's own start (0.15–0.22, compared with 0.19 with all buildings), but shows little correlation with progress at other starts (−0.01–0.03, compared with 0.00; §4.4).

Eight external buildings (W1)

We add 8 buildings from HM3D (6 of them non-residential, complementing the mostly residential MP3D scenes; 30 routes, 57 Transition segments (open-loop segment planning), 393 static items), generated with the same pipeline and reported separately, and run all nine tasks on them. Rebuttal Table 2 gives the five categories of the paper (each the mean of its tasks, as in Table 2 in the paper); each cell is HM3D / MP3D, with models ordered by MP3D Overall. GPT-5.6 Sol's MP3D Pose uses its corrected Pose2View answers (part 4). The last row counts the pairs of models whose order on HM3D is the same as on MP3D. We will add the nine tasks with intervals to the appendix.

Rebuttal Table 2. HM3D / MP3D scores by category.

ModelScalePoseIdentityTransitionNav-Text
GPT-5.6 Sol69.5 / 63.260.5 / 46.857.5 / 55.357.3 / 59.540.4 / 45.8
Qwen3.8-27B58.9 / 59.554.4 / 47.143.1 / 43.635.3 / 48.330.3 / 43.3
Gemma-4-31B56.5 / 47.438.6 / 32.328.5 / 28.133.7 / 41.018.1 / 33.3
Hy-Embodied-A3B45.9 / 46.729.0 / 18.916.1 / 18.218.8 / 20.66.1 / 11.6
Cosmos-Reason2-32B43.8 / 40.021.1 / 13.812.3 / 11.612.6 / 13.92.4 / 2.7
RoboBrain2.0-32B29.4 / 32.69.7 / 12.48.2 / 11.417.0 / 19.86.8 / 4.1
Model pairs in the same order as on MP3D15/1514/1515/1515/1514/15

Model rankings on HM3D are largely consistent with MP3D. Across six evaluated models and five task categories, 73 of 75 pairwise model orderings are preserved on HM3D. Although absolute performance varies across datasets, the relative ranking is largely consistent, suggesting that the benchmark's comparative findings are not specific to the original MP3D buildings.

2. Linking tests at a segment's start to navigation (W1, Q3)

We agree that correlations alone cannot establish the cause of an individual navigation failure. We will therefore state the diagnostic claim more precisely, and we add complementary predictive and interventional analyses.

To check whether the static items asked at a segment's start can diagnose navigation from that start, we rule out two alternative explanations of their relation to navigation:

  • A difficult start, with its pose, map and view context, could make both the static tests and the navigation hard. We predict segment failure from the same-pose tests on held-out buildings while controlling for the segment's difficulty and shared context, and compare models on the same segment.
  • The tested correspondence could accompany progress without being used for it. We supply it at the start and measure whether navigation improves.

Predicting segment failure with difficulty and shared context controlled (Q3)

We predict complete failure of a segment (progress = 0) from the Identity score at its start, leaving each building out of fitting in turn, with the logistic regression

logit⁡P(failms)=γm+δ,pˉ−m,s+β,Identityms,\operatorname{logit} P(\text{fail}_{ms}) = \gamma_m + \delta \\, \bar{p}_{-m,s} + \beta \\, \text{Identity}_{ms},

where

  • failms\text{fail}_{ms} indicates that model mm makes no progress on segment ss
  • γm\gamma_m is a term for model mm
  • pˉ−m,s\bar{p}_{-m,s} is the mean progress of the other eight models on ss, which captures the segment's difficulty and the context shared at its start;
  • Identityms\text{Identity}_{ms} is model mm's accuracy on the Identity items at the start of ss; as a control, we replace it with Identitymsother\text{Identity}^{\text{other}}_{ms}, its accuracy at the other starts of the route.

We fit two baselines and add each Identity score to both. Rebuttal Table 3 gives the AUC (area under the ROC curve; 0.5 is chance and 1 is perfect) on held-out buildings; intervals use the same bootstrap as above.

Rebuttal Table 3. Predicting segment failure on held-out buildings (AUC).

BaselineBaseline alone+ β,Identityms\beta \\, \text{Identity}_{ms} (at the start)+ β,Identitymsother\beta \\, \text{Identity}^{\text{other}}_{ms} (elsewhere on the route)
Model only: γm\gamma_m0.7080.750 (+0.041 [0.024, 0.059])0.706 (−0.003 [−0.007, 0.001])
Model and difficulty: γm+δ,pˉ−m,s\gamma_m + \delta \\, \bar{p}_{-m,s}0.7490.767 (+0.018 [0.003, 0.032])0.746 (−0.002 [−0.004, −0.001])

Identity accuracy at the segment's start provides additional information for predicting segment failure, while accuracy elsewhere on the route does not. Adding the score at the start raises the AUC of the model-only baseline from 0.708 to 0.750, close to the 0.749 of the baseline that already includes segment difficulty, and still raises that baseline to 0.767. Adding the score from elsewhere on the route raises neither (0.706 and 0.746). Leaving out routes instead of buildings gives the same result.

Within-segment comparison (W1: shared pose, map and view context)

Complementing the held-out prediction analysis, we compare models within exactly the same segment to remove variation in the shared navigation context. We fit a two-way fixed-effects regression of the progress of model mm on segment ss on its accuracy on the Identity items at the segment's start:

progressms=β⋅Identityms+αs+γm+εms,\text{progress}_{ms} = \beta \cdot \text{Identity}_{ms} + \alpha_s + \gamma_m + \varepsilon_{ms},

where

  • progressms\text{progress}_{ms} is the progress of model mm on segment ss;
  • Identityms\text{Identity}_{ms} is model mm's accuracy on the Identity items asked at the start of ss (the items of Fig. 2b in the paper);
  • αs\alpha_s is a segment effect that absorbs everything all models share on ss: building, route, map, view and difficulty;
  • γm\gamma_m is a model effect that absorbs each model's overall level.

β\beta is therefore estimated only from differences between models on the same segment. As a placebo, we replace Identityms\text{Identity}_{ms} with Identitymsother\text{Identity}^{\text{other}}_{ms}, the same model's accuracy at the other starts of the route.

Rebuttal Table 4. Within-segment association of Identity accuracy with progress.

Regressor in the model aboveβ\beta [95% CI]
Identityms\text{Identity}_{ms}: at the segment's start+0.13 [0.05, 0.21]
Identitymsother\text{Identity}^{\text{other}}_{ms}: at the other starts of the route (placebo)+0.03 [−0.07, 0.13]

β\beta is estimated on the actual data; its interval uses the same bootstrap as above.

Identity accuracy at the start remains associated with progress when the shared context is held fixed. On the same segment, a model whose Identity accuracy at the start is 10 points higher makes about 1.3 more points of progress, while the placebo coefficient is small and its interval includes zero.

Reviewer_52V8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BEARING, a benchmark for diagnosing how multimodal models align global indoor maps with first-person observations. Using paired views from 60 routes across nine buildings, it separates scale estimation, pose localization, target identity matching, local planning, and closed-loop navigation. The best model achieves an overall score of 60.87, compared with 77.92 for humans. The main difficulty lies in orientation rather than distance estimation, with models misreading map bearings, estimating their own heading incorrectly, and confusing left and right when translating map coordinates into egocentric directions.

Critical strengths
  1. The main idea is to pair map–view correspondence with navigation at the same location, so navigation splits into scale, pose, and identity, and those static skills can be checked against local planning and full-route progress. The paper finds that every model trails humans, and direction transforms are the main break. Scale-bar, compass, and agent-marker interventions show that only the strongest model reads those elements as expected.
  2. Beyond the total score, the paper analyzes direction error, map-element reading, view removal, and a text-map control.
  3. The error analysis is actionable: direction is lost in self-heading, map bearing, or the coordinate transform, not in distance.
  4. Overall, the generation, task definitions, and the eval protocol are clear enough to reproduce.
Critical weaknesses
  1. All data are Matterport3D with one map-render style. A model may adapt to that symbol system, so external validity is limited. This public scenes do not rule out pretraining contamination, and there is no private test on new scenes.
  2. Each model is essentially run once. Closed-loop navigation is stochastic, and there are no repeated runs, variance, or task-level confidence intervals.
Questions for authors
  1. Is it possible to add scenes with more variations and keep a private test set?
  2. Run closed-loop tasks over multiple seeds.
  3. Besides the composite score, report raw-unit error, task-specific utility, and how the ranking changes under different weights.
4 comments

We thank the reviewer for the careful reading and the concrete suggestions. We answer in three parts: generalisation beyond MP3D and the original symbol system (W1, Q1), variability across repeated runs (W2, Q2), and errors in physical units, task utility and weighting (Q3).

1. Generalisation beyond MP3D and the original symbol system (W1, Q1)

Eight external buildings from HM3D (W1, Q1: scenes with more variation)

We add 8 buildings from HM3D (2 residential, 6 non-residential; 30 routes, 57 Transition segments (open-loop segment planning), 393 static items), generated with the same pipeline and reported separately, and run all nine tasks on them. Each cell is HM3D / MP3D for the five categories of the paper, with models ordered by MP3D Overall; the last row counts the model pairs whose order is the same as on MP3D. We will add the nine tasks with intervals to the appendix.

Rebuttal Table 2. HM3D / MP3D scores by category.

ModelScalePoseIdentityTransitionNav-Text
GPT-5.6 Sol69.5 / 63.260.5 / 46.857.5 / 55.357.3 / 59.540.4 / 45.8
Qwen3.8-27B58.9 / 59.554.4 / 47.143.1 / 43.635.3 / 48.330.3 / 43.3
Gemma-4-31B56.5 / 47.438.6 / 32.328.5 / 28.133.7 / 41.018.1 / 33.3
Hy-Embodied-A3B45.9 / 46.729.0 / 18.916.1 / 18.218.8 / 20.66.1 / 11.6
Cosmos-Reason2-32B43.8 / 40.021.1 / 13.812.3 / 11.612.6 / 13.92.4 / 2.7
RoboBrain2.0-32B29.4 / 32.69.7 / 12.48.2 / 11.417.0 / 19.86.8 / 4.1
Model pairs in the same order as on MP3D15/1514/1515/1515/1514/15

Model rankings on HM3D are largely consistent with MP3D. Across six evaluated models and five task categories, 73 of 75 pairwise model orderings are preserved on HM3D. Although absolute performance varies across datasets, the relative ranking is largely consistent, suggesting that the benchmark's comparative findings are not specific to the original MP3D buildings.

Three new map styles (W1: adaptation to one symbol system)

To test whether the results depend on adapting to one symbol system, we build three variants with geometry and questions fixed:

  • A replaces the whole symbol system: a metric grid instead of the scale bar, border bearings instead of the compass, and numbered points with a legend instead of text labels.
  • B renders the floor as an abstract black-and-white plan of the kind used by indoor map applications.
  • C changes the angle convention to compass bearings clockwise from north.

Four models spanning the benchmark are run under all four maps on every item of one representative task per static dimension (Dist-Obj 239, Self-Pose 206, Ego-Loc 234) and on all 119 Transition segments. Row O re-runs the original map in the same batch as the variants.

Rebuttal Table 6. Scores under the original map and three new styles.

MapModelScale (Dist-Obj)Pose (Self-Pose)Identity (Ego-Loc)Transition
O (original)GPT-5.6 Sol63.348.153.659.4
Qwen3.8-27B54.038.337.447.5
Gemma-4-31B44.718.016.237.6
Hy-Embodied-A3B46.412.115.918.6
A (symbol system)GPT-5.6 Sol58.745.254.555.0
Qwen3.8-27B55.530.130.348.6
Gemma-4-31B37.918.912.236.8
Hy-Embodied-A3B45.610.216.321.3
B (abstract plan)GPT-5.6 Sol58.235.049.959.1
Qwen3.8-27B53.127.234.348.0
Gemma-4-31B46.515.516.234.0
Hy-Embodied-A3B47.47.317.418.5
C (clockwise from north)GPT-5.6 Sol58.654.455.763.2
Qwen3.8-27B53.641.338.949.4
Gemma-4-31B44.921.823.939.5
Hy-Embodied-A3B47.611.720.622.6

Relative model rankings remain consistent across the tested map variants, while absolute performance on several static tasks is sensitive to representation. Within each column, the model order under every variant is the same as under the original map, except that Gemma-4-31B and Hy-Embodied-A3B, which differ by 0.3 points on the original map, swap on Ego-Loc under A and B. Transition scores change by less than 4.5 points.

  • A (symbol system): Qwen3.8-27B scores lower on both tasks that read labels (Self-Pose and Ego-Loc) and Gemma-4-31B on Ego-Loc; Gemma-4-31B also underestimates distances read from the metric grid.
  • B (abstract plan): with furniture appearance removed, Self-Pose scores drop for all four models, most for GPT-5.6 Sol and Qwen3.8-27B.
  • C (clockwise from north): Ego-Loc scores rise most for Gemma-4-31B and Hy-Embodied-A3B, the two models that mirror left and right, because the clockwise convention removes the sign error they make.

We will add the full results to the appendix.

Contamination and a private test set (W1, Q1: private test set)

The nine buildings were taken from the R2R/VLN-CE val-unseen split, so navigation training sets built on its training splits do not contain them. The test items themselves (questions, routes, segment starts, instructions and rendered maps) are generated by our pipeline and were not released before this submission; the same holds for the HM3D items. Therefore, the val-unseen split and newly generated evaluation items reduce the risk of overlap with navigation training data and previously published benchmark questions. However, because the underlying scans are public, we cannot rule out exposure during general-purpose pretraining. We therefore identify evaluation on privately collected, unreleased scenes as an important limitation and future extension.

2. Variability across repeated runs (W2, Q2)

We measure two sources of variation: how much scores change when the closed-loop evaluation is run again with new sampling seeds, and how much they depend on which routes are evaluated, using confidence intervals over routes. Under both, the main order of the models is largely preserved, although models with close scores may change places.

Repeated closed-loop runs (Q2: multiple seeds)

To measure how much closed-loop results vary between runs, we report three runs of the full 60-route Nav-Text evaluation for four representative models. Inputs, prompts, action rules, budget and scoring are frozen, and only the sampling seed changes (for GPT-5.6 Sol, whose interface exposes no seed, each run is an independent invocation).

Rebuttal Table 7. Nav-Text over three runs.

ModelNav-Text sDTW, mean ± SD
Gemini-3.8-Flash63.7 ± 4.4
GPT-5.6 Sol47.2 ± 2.5
Qwen3.8-27B45.7 ± 3.2
Gemma-4-31B27.7 ± 4.9

Run-to-run variation does not change the main Nav-Text ranking, but leaves closely performing models unresolved. The standard deviations are much smaller than the gaps between Gemini-3.8-Flash, the pair of GPT-5.6 Sol and Qwen3.8-27B, and Gemma-4-31B, but exceed the difference within that pair. We will report these repeats in §4.2 of the revised paper.

Task-level intervals (W2: confidence intervals)

To quantify how much a score depends on which routes are evaluated, we compute a 95% confidence interval for every task score. The score itself is computed on the actual data; its interval resamples whole routes within each building 1,000 times and takes the 2.5th and 97.5th percentiles of the recomputed scores (Appendix E.1 in the paper). Rebuttal Table 8 shows the closed-loop task, Nav-Text; we will add all nine tasks to the appendix.

Rebuttal Table 8. Nav-Text with 95% confidence intervals.

ModelNav-Text sDTW [95% CI]
Gemini-3.8-Flash68.7 [61.1, 75.7]
Claude-Opus-558.1 [49.8, 66.5]
GPT-5.6 Sol45.8 [37.2, 54.0]
Qwen3.8-27B43.3 [34.0, 52.4]
Gemma-4-31B33.3 [24.6, 42.3]
Hy-Embodied-A3B11.6 [5.3, 18.4]
RoboBrain2.0-32B4.1 [0.9, 8.1]
Cosmos-Reason2-32B2.7 [0.0, 6.7]
InternVL3.5-38B1.3 [0.0, 3.8]

On Nav-Text, route sampling leaves closely ranked models unresolved, but not the gap between the general and the embodied models. The intervals of closely ranked models overlap, with half-widths of about 7–9 points, while those of the five strongest models, all general-purpose, lie above those of the four weakest, which include all three embodied models.

3. Beyond the composite score (Q3)

We have already reported several metrics beyond the composite score in Appendix C.1 (Table 9 in the paper), including the per-task scores, with chance levels and the human reference. In our paper, we report the Overall score as a quick summary. To conduct a detailed analysis, we add more discussions from three perspectives: errors in physical units, the contribution of each task, and the ranking under other weights.

Errors in physical units (Q3: raw-unit error)

To make the chance-normalised scores easier to interpret, we convert part of them into errors in physical units, each as a median over items:

  • Distance error (Dist-Ego): the difference between the predicted and the true distance from the agent to the named object, in metres and relative to the true distance.
  • Object bearing error (Ego-Loc, Abs-Loc): the angle between the predicted and the true bearing of the named object, measured from the agent's heading (Ego-Loc) or from map north (Abs-Loc), in degrees.
  • Self-heading error (Self-Pose): the angle between the predicted and the true heading of the agent, in degrees; a uniform guess gives 90°.
  • Navigation success and endpoint error (Nav-Text): the share of routes on which the agent stops within 3 m of the goal, and the median distance from its stopping point to the goal, in metres.

Rebuttal Table 9. Errors in physical units.

ModelDistance error, Dist-Ego (m / relative)Object bearing error, Ego-Loc (°)Object bearing error, Abs-Loc (°)Self-heading error, Self-Pose (°)Navigation success / endpoint error, Nav-Text (m)
Gemini-3.8-Flash0.39 / 14%6.55.31085% / 1.1
GPT-5.6 Sol0.29 / 10%8.04.03163% / 2.3
Claude-Opus-50.63 / 21%12.08.71375% / 1.5
Qwen3.8-27B0.44 / 14%12.49.92355% / 2.3
Gemma-4-31B0.62 / 24%28.314.45647% / 3.2
InternVL3.5-38B0.88 / 32%58.552.1902% / 6.3
Hy-Embodied0.87 / 27%29.423.87318% / 5.4
Cosmos-Reason2-32B1.19 / 37%45.543.4963% / 6.3
RoboBrain2.0-32B1.49 / 45%44.755.3767% / 6.7

As shown in Rebuttal Table 9, raw-unit errors reveal a particularly pronounced degradation in orientation estimation among weaker models. Their median bearing and heading errors are substantially larger than those of stronger models, reinforcing the directional failure patterns identified by the normalised task scores.

Contribution of each task to Overall (Q3: task-specific utility)

The overall score (Overall) reported in Table 2 in the paper is an averaged score over the five chance-normalised categories with equal weight (20% each), where the tasks within a single category are assigned equal weight as well. To measure what each task contributes to the comparison between models, we decompose the variance of Overall across the nine models into one share per task: the covariance of the task's contribution with Overall, divided by the variance of Overall (the factor-component decomposition of Shorrocks, 1982). The nine shares sum to one; intervals use the same bootstrap as above.

Rebuttal Table 10. Share of each task in between-model variance.

TaskCategoryWeight in OverallShare of between-model variance [95% CI]
Dist-EgoScale10%7.6% [6.5, 8.7]
Dist-ObjScale10%5.3% [4.4, 6.2]
Self-PosePose10%12.6% [11.5, 13.7]
Pose2ViewPose10%10.5% [9.4, 11.7]
Ego-LocIdentity6.70%5.0% [4.5, 5.6]
Abs-LocIdentity6.70%6.0% [5.5, 6.6]
Loc2CatIdentity6.70%6.8% [5.2, 8.4]
TransitionTransition20%21.6% [19.9, 23.2]
Nav-TextNav-Text20%24.6% [22.3, 26.8]

Every task contributes to the differences between models. Each task accounts for at least 5% of the differences between models, and no share falls below 3.8% in any bootstrap sample. Self-Pose and Nav-Text account for more than their weight (1.26 and 1.23 times), and the two distance tasks and Ego-Loc for less (0.53–0.76 times). A share reflects how much the models differ on a task and whether that difference agrees with Overall, not the importance of the task. We will add the share of usable answers per task to the appendix.

Weighting of Overall (Q3: ranking under different weights)

Rebuttal Table 11 shows how the ranking responds to other weightings.

Rebuttal Table 11. Ranking under other weightings.

WeightingChange in the nine-model ranking
By task instead of by categoryNone
By item countNone
Without chance normalisationNone
Dropping any one of the five categoriesGPT-5.6 Sol and Claude-Opus-5 swap 2nd and 3rd places; the bottom three reorder within 7th–9th places
10,000 random category weightsGemini first, Gemma fifth and Hy-Embodied sixth in every draw; Qwen fourth in 87–89% of draws

The main ranking structure is largely unchanged under the tested alternative weightings. The changes occur between GPT-5.6 Sol and Claude-Opus-5 and among the three lowest models (InternVL3.5-38B, Cosmos-Reason2-32B and RoboBrain2.0-32B), whose Overall differences are not statistically significant. In the revised paper, we will report category results as primary and Overall as a summary.

Reviewer_7udNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BEARING, a benchmark for evaluating how multimodal models relate an allocentric map to an egocentric view in indoor navigation. It tests scale, pose, and object-identity correspondences and pairs these tests with open-loop segment planning and closed-loop route navigation at shared physical states. The benchmark is constructed from 60 routes in nine Matterport3D buildings. The authors evaluate nine models, compare them with a human reference, analyze relationships between correspondence scores and navigation progress, and study how changes to map representation and map elements affect performance.

Critical strengths
  1. A useful task design that connects spatial correspondence to navigation. The benchmark evaluates map–view correspondence and navigation at shared physical states, making it possible to study spatial abilities in the context where they may be needed for action.

  2. The paper goes beyond aggregate benchmark scores. The map-format comparisons and interventions on elements such as the compass, scale bar, and agent marker help characterize how models use map information and where their behavior changes.

Critical weaknesses
  1. Generalization beyond the selected environments is not yet clear. Although the benchmark contains many static questions, they are derived from 60 routes in nine Matterport3D buildings using a common generation pipeline. The paper therefore provides limited evidence that the findings transfer to different buildings, map styles, incomplete maps, or naturally occurring navigation conditions. This matters because broad claims about map-guided navigation rely on diversity at the environment level, not only the number of generated questions.

  2. The evidence for diagnosing navigation failures is mainly correlational. The same-pose identity score has a modest correlation with segment progress, but this does not by itself establish that an identity error caused a particular navigation failure. Shared local difficulty may affect both tasks. The paper’s map interventions are informative, but a targeted intervention that supplies correct correspondence information and measures the change in navigation would more directly support the diagnostic claim.

Questions for authors
  1. Can the authors report results across held-out buildings or another navigation dataset, and quantify variability across buildings and routes? How sensitive are the conclusions to the selected environments and map presentation?

  2. Does the same-pose identity–progress relationship remain after accounting for local scene or route difficulty? Is the difference between same-pose and other-pose correlations statistically supported?

  3. Can the authors intervene on a suspected failure by supplying correct object identity, bearing, or pose information at the relevant state and measuring whether navigation improves? Such an experiment would help distinguish prediction of progress from diagnosis of its cause.

  4. Can the authors report human and model performance on exactly the same subset, together with uncertainty and annotator disagreement? This would make the claimed human–model gap easier to assess.

5 comments

We thank the reviewer for the constructive questions. We answer in four parts: generalisation across environments (W1, Q1), robustness to the map representation (W1, Q1), the link between tests at a segment's start and navigation (W2, Q2, Q3), and the human reference (Q4).

1. Generalisation across environments (W1, Q1)

We analyse generalisation at the level of buildings, the unit at which environments differ. This part answers the environment questions of Q1 in (a)–(c); the map presentation, the other half of Q1, is answered in part 2.

(a) Eight external buildings from HM3D (Q1: another dataset)

We add 8 buildings from HM3D (6 of them non-residential, complementing the mostly residential MP3D scenes; 30 routes, 57 Transition segments (open-loop segment planning), 393 static items), generated with the same pipeline and reported separately, and run all nine tasks on them. Rebuttal Table 2 gives the five categories of the paper (each the mean of its tasks, as in Table 2 in the paper); each cell is HM3D / MP3D, with models ordered by MP3D Overall. The last row counts the pairs of models whose order on HM3D is the same as on MP3D. We will add the nine tasks with intervals to the appendix.

Rebuttal Table 2. HM3D / MP3D scores by category.

ModelScalePoseIdentityTransitionNav-Text
GPT-5.6 Sol69.5 / 63.260.5 / 46.857.5 / 55.357.3 / 59.540.4 / 45.8
Qwen3.8-27B58.9 / 59.554.4 / 47.143.1 / 43.635.3 / 48.330.3 / 43.3
Gemma-4-31B56.5 / 47.438.6 / 32.328.5 / 28.133.7 / 41.018.1 / 33.3
Hy-Embodied-A3B45.9 / 46.729.0 / 18.916.1 / 18.218.8 / 20.66.1 / 11.6
Cosmos-Reason2-32B43.8 / 40.021.1 / 13.812.3 / 11.612.6 / 13.92.4 / 2.7
RoboBrain2.0-32B29.4 / 32.69.7 / 12.48.2 / 11.417.0 / 19.86.8 / 4.1
Model pairs in the same order as on MP3D15/1514/1515/1515/1514/15

Model rankings on HM3D are largely consistent with MP3D. Across six evaluated models and five task categories, 73 of 75 pairwise model orderings are preserved on HM3D. Although absolute performance varies across datasets, the relative ranking is largely consistent, suggesting that the benchmark's comparative findings are not specific to the original MP3D buildings.

(b) Variability across buildings and routes (Q1: variability)

For each model mm and building bb, we compute Overallm,b\text{Overall}_{m,b}, the score of Table 2 in the paper restricted to the items of building bb, and likewise Overallm,r\text{Overall}_{m,r} for each route rr. Rebuttal Table 12 reports the standard deviation SDb(Overallm,b)\text{SD}_b(\text{Overall}_{m,b}) over the nine buildings, SDr(Overallm,r)\text{SD}_r(\text{Overall}_{m,r}) over the 60 routes, and the range of the model's rank across buildings.

Rebuttal Table 12. Variability across buildings and routes.

ModelOverallSD over buildingsSD over routesRank range over buildings
Gemini-3.8-Flash60.96.813.51–1
GPT-5.6 Sol48.78.214.72–4
Claude-Opus-547.18.614.22–5
Qwen3.8-27B42.27.113.82–4
Gemma-4-31B28.83.712.54–5
InternVL3.5-38B8.24.37.16–9
Hy-Embodied-A3B14.43.68.86–8
Cosmos-Reason2-32B7.23.36.47–9
RoboBrain2.0-32B6.03.35.26–9

The top-model ranking is consistent across all nine buildings, while several closely performing models exchange positions. The larger route-level variability further motivates reporting uncertainty at the route level.

(c) Leaving out each building (Q1: sensitivity to the selected environments)

To test whether any single building drives the results, we remove one building at a time and recompute all results from the remaining eight. In all nine runs:

  • Model ranking (Table 2 in the paper): every model keeps its Overall rank, and within a category at most two pairs of closely ranked models swap.
  • The paper's three main findings still hold:
    • closed-source models take the top two places in every category (§4.2);
    • for all models except Gemini-3.8-Flash and Claude-Opus-5, most Self-Pose headings align with the map axes (63–100%, compared with 67–100% when all buildings are included; §4.3);
    • Identity accuracy correlates positively with progress at the segment’s own start (0.15–0.22, compared with 0.19 with all buildings), but shows little correlation with progress at other starts (−0.01–0.03, compared with 0.00; §4.4).

With any single building removed, every model keeps its Overall rank and the paper's three main findings still hold.

2. Robustness to the map representation (W1, Q1)

BEARING’s map follows a common format for indoor navigation. It combines a top-down metric floor plan with a north reference, a scale, a marker indicating the agent’s position and heading (similar to the arrow in mobile navigation apps), and labels for rooms and objects. This format is also used in map-guided indoor navigation research (e.g., Floorplan-VLN, FloNa and PM-Nav; all cited in §2). These elements support the correspondences tested in our benchmark: the scale for distance, the north reference and agent marker for pose, and the labels for identity. Our rendering is one instance of this format.

We address the map representation in three parts: the representation changes already in the paper, three new map styles evaluated for this response, and the scope with respect to incomplete maps and natural conditions.

Representation changes already in the paper (W1)

Our paper already includes experiments on map presentation (§4.5 in the paper). Figure 3 in the paper changes the form in which the map is given, as text or turned to the agent's heading, and Table 4 changes single map elements: the scale bar, compass and agent marker.

  • As shown in Figure 3 in the paper:
    • Adding the map as text (VT−V) raises Transition progress by about 10 to 20 points for Gemma-4-31B, InternVL3.5-38B, Hy-Embodied-A3B and Cosmos-Reason2-32B, and little for the three strongest models.
    • Turning the map heading-up (V-hu−V-nu) raises it by 11 points for InternVL3.5-38B and 6 for Gemma-4-31B, and little for Gemini-3.8-Flash.
    • Thus, the weaker models can use the map's content once it is given relative to their own position and heading.
  • As shown in Table 4 in the paper:
    • When the scale bar, compass or agent marker is changed, Gemini-3.8-Flash and GPT-5.6 Sol change their answers as reading the element requires, whereas the other models do so partly or not at all.
    • Thus, the weaker models do not fully use the elements that relate a map to the agent's position and heading.

Together, these results indicate that the score differences reflect whether a model can convert the map into its own view, which is the correspondence BEARING targets, rather than familiarity with our particular rendering. The three new map styles below vary the rendering itself.

Three new map styles (Q1: sensitivity to map presentation)

We build three new variants with geometry and questions fixed:

  • A replaces the whole symbol system: a metric grid instead of the scale bar, border bearings instead of the compass, and numbered points with a legend instead of text labels.
  • B renders the floor as an abstract black-and-white plan of the kind used by indoor map applications.
  • C changes the angle convention to compass bearings clockwise from north.

Four models spanning the benchmark are run under all four maps on every item of one representative task per static dimension (Dist-Obj 239, Self-Pose 206, Ego-Loc 234) and on all 119 Transition segments. Row O re-runs the original map in the same batch as the variants.

Rebuttal Table 6. Scores under the original map and three new styles.

MapModelScale (Dist-Obj)Pose (Self-Pose)Identity (Ego-Loc)Transition
O (original)GPT-5.6 Sol63.348.153.659.4
Qwen3.8-27B54.038.337.447.5
Gemma-4-31B44.718.016.237.6
Hy-Embodied-A3B46.412.115.918.6
A (symbol system)GPT-5.6 Sol58.745.254.555.0
Qwen3.8-27B55.530.130.348.6
Gemma-4-31B37.918.912.236.8
Hy-Embodied-A3B45.610.216.321.3
B (abstract plan)GPT-5.6 Sol58.235.049.959.1
Qwen3.8-27B53.127.234.348.0
Gemma-4-31B46.515.516.234.0
Hy-Embodied-A3B47.47.317.418.5
C (clockwise from north)GPT-5.6 Sol58.654.455.763.2
Qwen3.8-27B53.641.338.949.4
Gemma-4-31B44.921.823.939.5
Hy-Embodied-A3B47.611.720.622.6

Relative model rankings remain consistent across the tested map variants, while absolute performance on several static tasks is sensitive to representation. Within each column, the model order under every variant is the same as under the original map, except that Gemma-4-31B and Hy-Embodied-A3B, which differ by 0.3 points on the original map, swap on Ego-Loc under A and B. Transition scores change by less than 4.5 points.

  • A (symbol system): Qwen3.8-27B scores lower on both tasks that read labels (Self-Pose and Ego-Loc) and Gemma-4-31B on Ego-Loc; Gemma-4-31B also underestimates distances read from the metric grid.
  • B (abstract plan): with furniture appearance removed, Self-Pose scores drop for all four models, most for GPT-5.6 Sol and Qwen3.8-27B.
  • C (clockwise from north): Ego-Loc scores rise most for Gemma-4-31B and Hy-Embodied-A3B, the two models that mirror left and right, because the clockwise convention removes the sign error they make.

We will add the full results to the appendix.

Incomplete maps and natural conditions (W1)

We agree that incomplete maps and natural, changing conditions matter for deployment, and that generalisation to them remains untested. BEARING deliberately isolates map–view correspondence under complete-map conditions, because with an incomplete map or under natural conditions, an error could also come from insufficient map understanding, missing map content, sensing noise or changes to the scene. We will make this scope explicit and identify robustness under imperfect maps and natural conditions as future work.

3. Linking tests at a segment's start to navigation (W2, Q2, Q3)

To further diagnose navigation failures beyond the correlation in §4.4 (Figure 2b in the paper), we report three analyses: a two-way fixed-effects regression that controls for segment difficulty (Q2), a test of whether the same-pose and other-pose correlations differ (Q2), and a new intervention experiment that supplies the correct correspondence at the segment's start (Q3).

Comments on the paper

1 comment

We thank the reviewers and the AC. Below, R1, R2 and R3 refer to Reviewers 7udN, 52V8 and bhmW. The reviews raise four main concerns; we summarise our response to each, with details and numbers in the individual responses.

Findings largely carry over to new buildings and map styles (R1, R2, R3). We add 8 external buildings from HM3D and three new map styles: a different symbol system, an abstract floor plan and a different angle convention. On the new buildings, 73 of 75 model pairs keep their order; under the new styles, one pair of models with near-equal scores swaps on one task, and Transition scores change by less than 4.5 points.

The main ranking structure is stable (R2, R3). We add route-level 95% confidence intervals for every task, category and Overall, leave-one-building-out analyses, repeated closed-loop runs and alternative weightings. Under all of them the main ranking structure holds; the differences that are not statistically significant are between models with close scores (GPT-5.6 Sol and Claude-Opus-5, and the three lowest models).

The diagnostic link is supported beyond correlation (R1, R3). The same-pose tests relate to navigation progress after controlling for segment difficulty and predict segment failure in held-out buildings, and supplying the tested correspondence in a targeted intervention improves progress (by 6.3–20.5 points over the placebo for open-weight models), while the same information about distant objects gives no detectable gain.

Humans and models are compared on the same items (R1). On the 430 items answered independently by five annotators, every model remains below the human reference, the strongest by 19.9 [15.4, 24.6] points, and annotator disagreement is small relative to this gap.

We will state the claims more precisely in the revised paper. We will state what the diagnosis delivers (the sub-segment where progress is lost and the correspondences that are weak at its start) and the scope of the map claims, describe the left–right mirroring as a sign error in frame conversion, and report how the closed-loop results of the embodied models depend on the interaction protocol. We also correct the Pose2View input for GPT-5.6 Sol as R3 requested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.