- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes BEARING, a benchmark for evaluating egocentric–allocentric correspondence in map-guided indoor navigation. The benchmark defines five categories of tasks—Scale, Pose, Identity, Transition, and Navigation—covering static correspondence questions, segment-level planning, and full-route navigation. The benchmark is built from 60 routes in 9 Matterport3D buildings, with 1,418 static items, 119 transition segments, and 60 navigation episodes. The paper evaluates 9 multimodal models and compares them against a human reference on a subset.
The paper’s main claims are that current models remain far below humans, especially on self-localization; that models lose directional information more than distance information; that identity tests predict transition performance only at the same pose; and that weaker models benefit more from maps expressed in text or in the agent’s own frame.
Critical strengths
The paper targets a meaningful gap in current evaluation practice. It is reasonable and useful to ask whether map-guided navigation failures come from poor map–view correspondence rather than only measuring final navigation success. The attempt to decompose this capability into multiple task types is conceptually worthwhile.
The paper is clearly written and includes more analysis than a typical benchmark submission. The benchmark setup, task taxonomy, and most analyses are presented clearly. The authors also include several diagnostic experiments beyond simple leaderboard reporting.
Critical weaknesses
The benchmark scale and dependence structure are too limited to support the paper’s stronger interpretive claims. Although the benchmark is carefully designed, it is built from only 9 buildings, 60 routes, and 119 transition segments, and many static questions are anchored at the same segment starts and therefore share the same pose, map, and view context. This limits the effective number of independent evaluation states. As a result, the paper’s stronger claims—especially about diagnosing navigation failures at the pose where they occur, model ranking stability, and broad conclusions about spatial reasoning failures—seem insufficiently supported at the current scale.
The evaluation is substantially entangled with a specific map interface and protocol, which weakens the interpretation of the results. A major empirical finding of the paper is that performance changes markedly when the same map content is rewritten into text or aligned to the agent’s frame, and that many models do not reliably use the compass, scale bar, or agent marker as intended. This suggests that BEARING may be measuring competence with one particular map representation at least as much as general egocentric–allocentric correspondence. In addition, there are protocol issues that materially affect credibility of the comparisons, including the acknowledged Pose2View filename leakage for GPT-5.6 Sol and the apparent mismatch between some embodied models and the benchmark interaction protocol.
Questions for authors
Can you rerun GPT-5.6 Sol after removing the Pose2View filename leakage and update all Pose-related tables/analyses that depend on it?
Can you report uncertainty estimates for the main benchmark comparisons, especially confidence intervals or ranking stability analyses at the route/building level for Table 2?
Can you provide a stronger analysis for the “diagnosis” claim, e.g., predicting segment failure from same-pose static tests while controlling for route/building difficulty and shared context?
Can you clarify the intended scope of the benchmark claims: is BEARING primarily a benchmark for general spatial correspondence, or for competence with this specific map representation and prompting convention?
7 comments
Eight external buildings (W1)
We add 8 buildings from HM3D (6 of them non-residential, complementing the mostly residential MP3D scenes; 30 routes, 57 Transition segments (open-loop segment planning), 393 static items), generated with the same pipeline and reported separately, and run all nine tasks on them. Rebuttal Table 2 gives the five categories of the paper (each the mean of its tasks, as in Table 2 in the paper); each cell is HM3D / MP3D, with models ordered by MP3D Overall. GPT-5.6 Sol's MP3D Pose uses its corrected Pose2View answers (part 4). The last row counts the pairs of models whose order on HM3D is the same as on MP3D. We will add the nine tasks with intervals to the appendix.
Rebuttal Table 2. HM3D / MP3D scores by category.
| Model | Scale | Pose | Identity | Transition | Nav-Text |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 69.5 / 63.2 | 60.5 / 46.8 | 57.5 / 55.3 | 57.3 / 59.5 | 40.4 / 45.8 |
| Qwen3.8-27B | 58.9 / 59.5 | 54.4 / 47.1 | 43.1 / 43.6 | 35.3 / 48.3 | 30.3 / 43.3 |
| Gemma-4-31B | 56.5 / 47.4 | 38.6 / 32.3 | 28.5 / 28.1 | 33.7 / 41.0 | 18.1 / 33.3 |
| Hy-Embodied-A3B | 45.9 / 46.7 | 29.0 / 18.9 | 16.1 / 18.2 | 18.8 / 20.6 | 6.1 / 11.6 |
| Cosmos-Reason2-32B | 43.8 / 40.0 | 21.1 / 13.8 | 12.3 / 11.6 | 12.6 / 13.9 | 2.4 / 2.7 |
| RoboBrain2.0-32B | 29.4 / 32.6 | 9.7 / 12.4 | 8.2 / 11.4 | 17.0 / 19.8 | 6.8 / 4.1 |
| Model pairs in the same order as on MP3D | 15/15 | 14/15 | 15/15 | 15/15 | 14/15 |
Model rankings on HM3D are largely consistent with MP3D. Across six evaluated models and five task categories, 73 of 75 pairwise model orderings are preserved on HM3D. Although absolute performance varies across datasets, the relative ranking is largely consistent, suggesting that the benchmark's comparative findings are not specific to the original MP3D buildings.
2. Linking tests at a segment's start to navigation (W1, Q3)
We agree that correlations alone cannot establish the cause of an individual navigation failure. We will therefore state the diagnostic claim more precisely, and we add complementary predictive and interventional analyses.
To check whether the static items asked at a segment's start can diagnose navigation from that start, we rule out two alternative explanations of their relation to navigation:
- A difficult start, with its pose, map and view context, could make both the static tests and the navigation hard. We predict segment failure from the same-pose tests on held-out buildings while controlling for the segment's difficulty and shared context, and compare models on the same segment.
- The tested correspondence could accompany progress without being used for it. We supply it at the start and measure whether navigation improves.
Predicting segment failure with difficulty and shared context controlled (Q3)
We predict complete failure of a segment (progress = 0) from the Identity score at its start, leaving each building out of fitting in turn, with the logistic regression
where
- indicates that model makes no progress on segment
- is a term for model
- is the mean progress of the other eight models on , which captures the segment's difficulty and the context shared at its start;
- is model 's accuracy on the Identity items at the start of ; as a control, we replace it with , its accuracy at the other starts of the route.
We fit two baselines and add each Identity score to both. Rebuttal Table 3 gives the AUC (area under the ROC curve; 0.5 is chance and 1 is perfect) on held-out buildings; intervals use the same bootstrap as above.
Rebuttal Table 3. Predicting segment failure on held-out buildings (AUC).
| Baseline | Baseline alone | + (at the start) | + (elsewhere on the route) |
|---|---|---|---|
| Model only: | 0.708 | 0.750 (+0.041 [0.024, 0.059]) | 0.706 (−0.003 [−0.007, 0.001]) |
| Model and difficulty: | 0.749 | 0.767 (+0.018 [0.003, 0.032]) | 0.746 (−0.002 [−0.004, −0.001]) |
Identity accuracy at the segment's start provides additional information for predicting segment failure, while accuracy elsewhere on the route does not. Adding the score at the start raises the AUC of the model-only baseline from 0.708 to 0.750, close to the 0.749 of the baseline that already includes segment difficulty, and still raises that baseline to 0.767. Adding the score from elsewhere on the route raises neither (0.706 and 0.746). Leaving out routes instead of buildings gives the same result.
Within-segment comparison (W1: shared pose, map and view context)
Complementing the held-out prediction analysis, we compare models within exactly the same segment to remove variation in the shared navigation context. We fit a two-way fixed-effects regression of the progress of model on segment on its accuracy on the Identity items at the segment's start:
where
- is the progress of model on segment ;
- is model 's accuracy on the Identity items asked at the start of (the items of Fig. 2b in the paper);
- is a segment effect that absorbs everything all models share on : building, route, map, view and difficulty;
- is a model effect that absorbs each model's overall level.
is therefore estimated only from differences between models on the same segment. As a placebo, we replace with , the same model's accuracy at the other starts of the route.
Rebuttal Table 4. Within-segment association of Identity accuracy with progress.
| Regressor in the model above | [95% CI] |
|---|---|
| : at the segment's start | +0.13 [0.05, 0.21] |
| : at the other starts of the route (placebo) | +0.03 [−0.07, 0.13] |
is estimated on the actual data; its interval uses the same bootstrap as above.
Identity accuracy at the start remains associated with progress when the shared context is held fixed. On the same segment, a model whose Identity accuracy at the start is 10 points higher makes about 1.3 more points of progress, while the placebo coefficient is small and its interval includes zero.
We thank the reviewer for the careful reading and the concrete questions. We answer in four parts: measurement precision with items nested in routes and buildings (W1, Q2), the link between tests at a segment's start and navigation (W1, Q3), robustness to the map representation (W2, Q4), and the Pose2View result of GPT-5.6 Sol and the evidence on embodied models (W2, Q1).
1. Measurement precision with items nested in routes and buildings (W1, Q2)
Intervals and ranking stability (W1, Q2)
For every category and for Overall, we compute the score with a 95% confidence interval and will add these intervals to Table 2 in the paper. Because items that share a segment start, with its pose, map and view, are not independent, the intervals resample whole routes: each score is computed on the actual data, and its interval comes from a bootstrap in which we resample whole routes within each building 1,000 times, recompute the score each time, and take the 2.5th and 97.5th percentiles (Appendix E.1 in the paper). The items of a start are therefore always drawn together, and each interval reflects the number of routes rather than the number of items; all intervals in this response are computed this way. GPT-5.6 Sol's Pose and Overall use its corrected Pose2View answers (part 4), which moves it from third to fourth on Pose; all other ranks are as in the paper.
Rebuttal Table 1. Category and Overall scores with 95% confidence intervals.
Most differences between consecutively ranked models are statistically significant. For each pair of models with consecutive Overall ranks, the 95% confidence interval of their difference excludes zero, except for GPT-5.6 Sol and Claude-Opus-5 (1.6 [−1.7, 4.7]) and among the three lowest models (InternVL3.5-38B, Cosmos-Reason2-32B and RoboBrain2.0-32B). We will state these exceptions in §4.2 of the revised paper.
Stable across buildings. The top-model ranking is consistent across all nine buildings, while several closely performing models exchange positions. Removing any one building and recomputing all results from the remaining eight leaves all nine Overall ranks unchanged and keeps the paper's three main findings in all nine runs: