acceptodds
Under review as a conference paper at ICLR 2027

REMAP: Evaluating Cross-Perspective Spatial Reasoning

Abstract

Building coherent spatial representations requires relating what is seen from different viewpoints to a shared global layout. Frontier multimodal models are becoming increasingly capable at this problem in realistic environments, but it is unclear whether these gains reflect robust spatial reasoning or dependence on familiar correspondences between representations. We introduce REMAP, a benchmark for cross-perspective spatial reasoning in which models identify a target location on an allocentric map from multiple egocentric views. REMAP is based on settings developed for cognitive studies of human spatial reasoning. It combines FLOORPLAN, which is based on indoor layouts, and TRIANGLE, a controlled diagnostic that isolates robustness to changes in how local views correspond to the global map. REMAP exhibits high discriminative power. Across 24 vision-language models and human participants, capability ranges from below random to surpassing average human performance on FLOORPLAN. TRIANGLE enables deeper analysis, showing that simple variations in the geometric correspondence between views can lead to significant performance drops, whereas humans are largely robust.

Primary area:
datasets and benchmarks
Keywords:
spatial reasoning, vision-language models, benchmark, cross-perspective reasoning, allocentric-egocentric correspondence, map reading, cognitively inspired evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_sEZSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

REMAP tests whether a model can match a top-down map to a first-person view, using texture-free indoor layouts and simple triangular enclosures. FloorPlan has 3,603 questions from 291 homes, Triangle has 3,888 questions crossing three map formats with three view formats. The nice part is that views, labels, and answers are held fixed while the map is redrawn, so the comparison is controlled. 24 models and a human subset are evaluated, and a set of graded guidance prompts recovers a lot of the drop seen in the strongest model (pp. 3-8; Appendices A-E).

Critical strengths
  1. The answer-preserving intervention is much more informative than comparing across unrelated benchmarks. With Corners views fixed, redrawing the map as Sides drops Astra from 92.1% to 19.7%, and even the Full map version drops it to 60.9%. That is strong evidence of representation sensitivity and it doesn't require any guess about what is happening inside the model (p. 7, Figure 3).
  2. The guidance experiment is done on the same 576 questions for every model, with the three added paragraphs matched at 45 words. Combined with the breakdown by target type and location error, you can actually tell apart "can't do it" from "can do it with a hint." The human-model comparisons are matched, including the 600-item FloorPlan comparison (pp. 8-9, Figures 4-6; p. 24).
Critical weaknesses
  1. I am not convinced the humans and models got the same information. Participants had a Triangle demo, screening questions, and repeated trials, and the FloorPlan instructions include a legend. But participant counts, what was in the demo and screening, whether feedback was given, trial order, and total exposure are not really spelled out (pp. 21-23). So saying there was "no explicit convention statement" doesn't show equal learning opportunity, and doesn't show that the convention was inferred spontaneously. Also, full-benchmark numbers and human-subset numbers should be kept apart. Table 1 by itself can't support a matched Congruent human advantage.
  2. The benchmark doesn't say how much of the task is actually within-type localization vs just recognizing the target type. For example supplementary Q40 and Q1240 only have one candidate of the required type. This doesn't break the benchmark, and Figure 5 already restricts to a diagnostic subset, but I'd want the prevalence over the full data and a type-aware baseline. The repeated geometric blocks and the participant/item dependence also need proper uncertainty estimates, and Astra only has one full run (pp. 13-14, 23). The broader claims about general spatial competence should stay closer to what was actually measured.
Questions for authors
  1. Can you give participant counts, trial allocation, the demo and screening materials, feedback and ordering details, and the sampling manifest? Are the map-redrawing blocks preserved in the human subset, and do paired items show up in the same session?
  2. Can you release item/pairing IDs, raw predictions, exact inference settings, candidate-type counts per item, and block- and participant-aware uncertainty?
  3. Is the representation penalty still large when restricted to questions that need within-type localization? New shape families or photorealistic scenes are fine to leave for future work.
4 comments
AuthorsOpenReview ↗

Response to Reviewer sEZS (1/2)

Thank you for the thorough review and for recognizing our design and analysis.

Please find our responses below, with revisions in the updated PDF marked in blue.

Questions

  1. Can you give participant counts, trial allocation, [...] Are the map-redrawing blocks preserved in the human subset, and do paired items show up in the same session?

All 9 map and view pairings are preserved for every combination in the Triangle human subset. Also, no participant answered two differently drawn versions of the same question.

Each of the 64 triangle, target, and camera combinations appears in all 9 pairings (576 questions). Each 9-question set draws from 9 different combinations, with one set per participant.

Across 372 non-overlapping participants, 192 answered the 64 Triangle sets of 9 and 180 answered the 60 FloorPlan sets of 10 (three participants per set, with questions in a fixed order). Before the scored questions, participants saw instructions (with a map-reading paragraph in FloorPlan), one Congruent demo with Full walls in Triangle (on a non-benchmark triangle, with feedback), and two one-attempt screening questions (both Congruent in Triangle), with no feedback on screening or scored questions.

We updated the details in Appendix B.

We will release the list of sampled questions (the "sampling manifest"), set assignments, and all the other materials upon acceptance.

  1. Can you release item/pairing IDs, raw predictions, exact inference settings, candidate-type counts per item, and block- and participant-aware uncertainty?

We will release question, pairing, and group IDs, per-question counts of corner and side-midpoint candidates, and raw predictions upon acceptance.

We listed all inference settings in Table 7 in Appendix C. GPT-6 Astra now uses two full independent runs (Table 1).

In Appendix B.1, we added 95% bootstrap intervals (5,000 resamples) and between-participant SDs (13.0 points on Triangle, 14.8 on FloorPlan).

On the 576 subset, resampling the 16 target–camera groups ("blocks" in the reviewer's question, each pooling all 4 triangles and 9 pairings) gives Δ=32.0\Delta = 32.0 [27.7, 36.2] for Astra, 12.612.6 [8.0, 17.0] for humans, and difference 19.419.4 [14.7, 24.8] (19.419.4 [13.1, 25.4] resampling questions; full-set Astra Δ=29.4\Delta = 29.4 [27.3, 31.5] across all 108 groups).

Resampling participants gives human Triangle Δ=12.6\Delta = 12.6 [8.5, 16.6] and FloorPlan accuracy 80.1%80.1\% [77.9, 82.2].

3a. Is the representation penalty still large when restricted to questions that need within-type localization?

Yes. 80.3% of Triangle questions (3,123/3,888) have at least two visible candidates of the target's type, corner or side midpoint (corresponding to within-type localization). The Congruent-to-Incongruent drop stays large on this subset.

On this subset, the group-resampled drop is 30.2 points [27.9, 32.5] for Astra and 16.0 [13.8, 18.3] for Gemini 3.8 Flash on the full set, and 12.5 [7.1, 18.1] for humans vs. 32.6 for Astra on the 576 subset.

Uniform guessing among visible candidates of the target's type (the type-aware baseline) gives 59.4% overall on Triangle and 49.4% on this subset (vs. 29.6% chance).

On Incongruent questions here, models fall below that baseline (full set: Astra 37.3%, Gemini 3.8 Flash 25.5% vs. 49.4%; human subset: Astra 44.8% vs. 50.0%, humans 66.9%). On the 19.7% of questions with one candidate of the target's type, Astra drops 26.2 points and the four Gemini models drop 18–34 points.

We added this analysis to Appendix D.

3b. New shape families or photorealistic scenes are fine to leave for future work.

In a 3-run experiment (mean ±\pm SD) on 774 Trapezoid and 810 Arrowhead questions (remap-new-polygons), Gemini 3.8 Flash drops 10.5±1.510.5 \pm 1.5 and 18.8±2.718.8 \pm 2.7 points (Δ\Delta) and Qwen3.8-27B drops 10.7±0.210.7 \pm 0.2 and 19.3±2.419.3 \pm 2.4 points (losing 72%-75% of their Congruent margin above chance).

Full numbers are in our response (Question 2b) to Reviewer S7mH.

AuthorsOpenReview ↗

Response to Reviewer sEZS (2/2)

Concerns

  1. I am not convinced [...] Also, full-benchmark numbers and human-subset numbers should be kept apart. [...]

We do not claim equal prior exposure or spontaneous rule induction, and Sec. 5.1 and Table 8 (Appendix D) now separate full-benchmark and matched human-subset results.

Without a demo or the FloorPlan map-reading paragraph (the "legend" in the human instructions), Astra is similar to humans on Congruent Triangle (83.1% vs. 84.0%) and exceeds them on FloorPlan (89.4% vs. 80.1%), diverging on Incongruent questions (51.0% vs. 71.4%; Δ=32.0\Delta = 32.0 vs. 12.612.6).

Humans saw no Incongruent pre-task question or format explanation (Question 1). Adding the solved demo or two solved Congruent examples shifts Incongruent accuracy by −0.4 and +0.9 points on Gemini 3.8 Flash and +6.5 and +3.1 on Qwen3.8-27B (≤49.5% vs. 71.4% humans). On FloorPlan, the three human-instruction variants shift both models by −0.9 to +1.0 points (Table 9).

  1. The benchmark doesn't say how much of the task is actually within-type localization [...].

Please see Question 3a for the share of questions with multiple same-type candidates and the corresponding guessing baseline, and Question 2 for group- and participant-resampled intervals and Astra's second run. We also scoped claims in Secs. 1, 5.1, and 6 to abstract geometric and untextured map–view representations.

In light of these additions, would you consider raising your score, or let us know if anything remains unclear?

Reviewer_sEZSOpenReview ↗

Official Review of Reviewer sEZS

Thank you for the detailed review. The response has addressed my concerns, but it has not changed my overall assessment enough to warrant a high score. I keep the score of 3.

AuthorsOpenReview ↗

Reply to Reviewer sEZS

Thank you for your reply! We’re glad our response has addressed your concerns and appreciate your careful consideration of our submission and rebuttal.

Reviewer_7HibOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a new benchmark for testing cross-spatial reasoning called REMAP. The feature of REMAP is it's probe on how models accommodate different views. Tests on 24 VLMs and human subject baselines (approx. 2 x c.600 questions on 3 adults via Prolific) provides insights into model failure modes around spatial representation (realistic versus controlled enclosure). Targeted interventions in some cases compensate for drops.

The paper is well written, easy to read, and references well. The introduction is compelling, and the results figures as part of Figure 1 tell a clear story that engages the reader (it would be interesting to speculate what has triggered the leap in performance since August 2025).

Results show pretty much catastrophic failure for most of the 24 models tested (at or below random) which somewhat weakens the headline 24 model comparisons, but there is strong and differentiated performance among Gemini and GPT models, at least for FloorPlan. I think though this is the point of the story: VLMs were weak, they are becoming stronger but most are not yet capable of human-level consistency when considering different spatial views.

Critical strengths
  • Strong-ish benchmark paper with 'room to grow'; -ish because I'd like to see a larger human population but I appreciate that there are already nearly 3600 response items if my maths is correct. I do wonder whether letting a larger pool of people label smaller slices of data with some cross-over between slices might have been a better design..
  • Near random performance could be listed as a weakness but I'd view this as an informative finding and it doesn't invalidate the conclusions.
  • Nice experimental design to isolate representational effects (Triangle and FloorPlan).
  • Detailed error analysis (nice to see wrong-room versus wrong-location discussion).
Critical weaknesses
  • Just need to clarify if it was three people labelling c.1200 items or whether it was actually sliced among multiple human subjects and what the cross over actually was. If it was three people in total, what do the authors think about the issue of the raters becoming progressively adapted to the task? (e.g. raters get stronger as they perform more tasks - check with rater performance versus quartile position of question).
  • More clarity on the prompt construction would be good and how sensitive the results are to alternative prompts.
Questions for authors
  • Can the authors speculate on the likely cause of failure when comparing rendering versus visual completeness?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 7Hib

We appreciate your thoughtful review and kind feedback on our presentation and error analysis.

Below we respond to each of your comments and questions. Changes in the revised PDF are marked in blue.

Concerns

Just need to clarify if it was three people labelling c.1200 items or [...] If it was three people in total, what do the authors think about the issue of the raters becoming progressively adapted to the task? [...]

The human experiments involved 372 unique participants. Each participant answered one session of 9 or 10 questions, so every question was answered by three different people.

The table below shows the exact distribution, with no overlap across sets or studies. We updated the details in Appendix B.

StudyParticipantsQuestion setsResponses per question
Triangle19264 sets of 9 (576 questions)3
FloorPlan18060 sets of 10 (600 questions)3

More clarity on the prompt construction would be good and how sensitive the results are to alternative prompts.

All models receive one fixed prompt across both settings (Appendix C), and accuracy is stable across generic prompt additions.

For prompt construction, we iteratively refined a minimal prompt on pilot items to prevent format ambiguities: clarifying the four-image setup, stating that repeated numbers across views mark the same physical location, requiring a single number, and fixing a parseable two-line output (Explanation: followed by Location:).

We tested prompt sensitivity in three ways:

  1. Length-matched neutral text (Sec. 5.3, Fig. 4): We added a 45-word neutral paragraph changes Incongruent accuracy by only −1.6 to +4.4 points across four models.

  2. Pre-task examples: Adding the solved human demo or two solved Congruent examples changes Gemini 3.8 Flash's Incongruent accuracy by −0.4 and +0.9 points (and lifts Qwen3.8-27B overall without shrinking its Congruent-Incongruent gap).

  3. Human FloorPlan instructions (Appendix D, Table 9): Adding the human map-reading paragraph changes accuracy by −0.9 to +1.0 points for Gemini 3.8 Flash and Qwen3.8-27B.

Among the tested variants, accuracy moves substantially only when the prompt points to or states how the drawings relate. For instance, GPT-6 Astra's Incongruent accuracy rises from 55.5% (Neutral) to 75.0% (Attention) and 79.9% (Explicit convention).

Questions

Can the authors speculate on the likely cause of failure when comparing rendering versus visual completeness?

Our hypothesis is that the failures come from the map being drawn differently from the views, and not from missing information in the map.

Below is GPT-6 Astra's accuracy on Triangle with Corners views held fixed (mean of 2 independent runs), grouped by map format:

Map (GPT-6 Astra, Corners views)Accuracy
Corners map (drawn like the views)91.9%
Full map (complete outline, contains those corners)59.6%
Sides map19.7%

The results show that first, visual completeness hurts rather than helps. Adding the full outline cuts accuracy from 91.9% to 59.6% even though the corners remain drawn, while Sides maps fall below the 29.6% random baseline.

Second, Astra appears to use the adjacent wall piece as a literal cue (Fig. 5): on the 96 Incongruent Corners$\leftrightarrow$Sides items, wrong-type errors (e.g., picking a side midpoint for a corner) drop from 65 under Neutral to 3 under Explicit convention.

These points suggest that failure is driven by literal matching of rendered wall pieces rather than map completeness.


We hope these clarifications resolve your remaining questions. Would you kindly consider raising your score, or let us know if we can clarify anything further?

Reviewer_pqbTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces REMAP, a new benchmark dataset where the task is to locate a target point from allocentric and egocentric views. The motivation behind this dataset is to quantify the spatial reasoning capabilities of state-of-the-art VLMs by ensuring whether VLMs are robust enough or if they are matching on familiar correspondences between representations in the scene. The paper proposes two variants of the dataset: 1) FLOORPLAN, which tests on realistic, texture-free home layouts and 2) TRIANGLE, where the representation is adapted from cognitive science to represent the scene. The paper demonstrates extensive study across 24 VLMs and a human study done on the users from Prolific. On the Floorplan variant, GPT6-Astra beats human performance, while it lags by several points on the other variant, Triangle. Additionally, the prompts that provide the drawing convention recover most of the loss for GPT-6-Astra but not for other models.

Critical strengths
  1. This paper presents a clean and clear motivated problem and has provided a good benchmark by running on a substantial number of VLMs.
  2. The use of the Triangle variation is interesting, as it is grounded in and taken from cognitive studies of human spatial reasoning.
  3. The paper provides a deeper analysis on the proposed dataset with the various benchmark models.
Critical weaknesses
  1. This paper demonstrates that the results on the Triangle variant by the frontier models are low compared to the Floorplan variant. But could the difference, or the negative results, be due to a domain shift issue? Before asking the models to evaluate, can we either show them some of the demo examples with the solutions and then begin evaluating? Also, a similar approach was used during a human evaluation, where they were shown/provided with some demo sequences before the test began.
Questions for authors
  1. Can the authors provide details on whether the reasoning effort was on/off for the VLMs during evaluation?
  2. Can the authors conduct experiments to evaluate how models perform with the human demo item or a few in-context examples added to the prompt?
1 comment
AuthorsOpenReview ↗

Response to Reviewer pqbT

Thank you for your review and for highlighting our clear motivation and cognitively grounded Triangle design.

We address each point below. Updates in the revised PDF are marked in blue.

Questions

"Can the authors provide details on whether the reasoning effort was on/off for the VLMs during evaluation?"

Models generally followed default or recommended settings. Built-in thinking was active for all API models and Qwen3.8-27B, and not enabled for the remaining open-weight models, which were run in their default mode.

GPT-6 Astra used medium reasoning effort in both runs. All Gemini models used dynamic thinking, Gemini 3.8 Flash at its default medium level and Gemini 3.1 Pro, Gemini 3 Flash and Gemini Robotics-ER 1.6 without a thinking budget.

Please see the new Table 7 in Appendix C for each model's reasoning setting, sampling parameters, and output limit.

"Can the authors conduct experiments to evaluate how models perform with the human demo item or a few in-context examples added to the prompt?"

We ran this experiment on both a commercial and an open-weight model: adding the human demo or two solved in-context examples does not close the Congruent-to-Incongruent gap (Δ\Delta), and both models remain over 20 points below humans on Incongruent questions.

We tested Gemini 3.8 Flash and Qwen3.8-27B (the strongest open-weight model) on the human subset of Triangle, prepending either the solved human demo item (a Congruent Full-outline right isosceles triangle) or the two solved Congruent human screening questions to the prompt (attaching up to 12 full-resolution images per query. GPT-6 Astra was omitted due to multi-image API cost, and Sec. 5.3 instead evaluates graded prompt guidance on Astra).

Original-prompt rows report the three-run mean ± SD from Table 8 in Appendix D. The rows with added examples report one run each:

ModelConditionCongruent (%)Incongruent (%)Δ\Delta
Gemini 3.8 FlashOriginal prompt58.5 ± 1.635.8 ± 1.222.7 ± 2.5
Gemini 3.8 Flash+ human demo item with answer57.335.421.9
Gemini 3.8 Flash+ two in-context examples with answers56.236.719.5
Qwen3.8-27BOriginal prompt53.5 ± 2.043.0 ± 1.610.5 ± 0.4
Qwen3.8-27B+ human demo item with answer59.949.510.4
Qwen3.8-27B+ two in-context examples with answers63.046.116.9
Humans (average)Pre-task demo & screening (no convention explanation)84.071.412.6

For Gemini 3.8 Flash, Incongruent accuracy barely changes (−0.4 and +0.9 points). For Qwen3.8-27B, examples lift overall accuracy (+6.4 and +9.5 Congruent, and +6.5 and +3.1 Incongruent), yet a large Δ\Delta remains (10.4 and 16.9 vs. 10.5) and Incongruent accuracy peaks at 49.5%, i.e. still 21.9 points below humans (71.4%).

Concerns

[...] could the difference, or the negative results, be due to a domain shift issue? [...]

A shift to the synthetic Triangle scene does not explain the failure on its own. Rather, the shift lies in changing how the same space is drawn, which is the exact robustness REMAP measures.

On the human study subsets (Table 8 that we added), GPT-6 Astra exceeds the average human response on FloorPlan (89.4% vs. 80.1%) and matches humans on Congruent Triangle without any demo (83.1% vs. 84.0%), dropping sharply only on Incongruent Triangle (51.0% vs. 71.4%). Holding Corners views, labels, and answers fixed while redrawing only the map as Sides drops Astra from 91.9% to 19.7%, below 29.6% chance (Fig. 3).

Human participants likewise never saw an Incongruent redraw or convention explanation. So whether called a representation shift or convention unfamiliarity (tested via graded guidance in Sec. 5.3), models lose far more accuracy across drawings of the same space than humans.


We hope these new experiments and clarifications address your questions. Would you consider raising your score, or let us know if any concerns remain?

Reviewer_KtXTOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

REMAP tests cross-perspective spatial reasoning: given a top-down map marking a target and three egocentric views with numbered candidates, a model must pick the matching location. FLOORPLAN uses 3,603 questions on 291 reconstructed home layouts; TRIANGLE adapts a developmental-psychology paradigm into 3,888 questions where each "Incongruent" item has a "Congruent" twin differing only in how the map is drawn, with answers derived deterministically from geometry. Across 24 VLMs and human participants, GPT-6 Astra beats the average human on FLOORPLAN (88.2% vs 80.1%) yet loses 29.1 points when only the map is redrawn (humans: 12.6), falling below chance on the hardest redraws; graded prompt guidance recovers most of Astra's loss but little for other models. The paper argues evaluation should measure the assistance a system needs, not only outcomes.

Critical strengths
  1. Causal isolation of representation sensitivity. Congruent/Incongruent twins hold views, labels, and the answer fixed while only the map's drawing changes, with chance and target balance identical across cells. The results are dramatic and internally consistent (Astra: 92.1→19.7 on Corners→Sides redraws; Δ 29.1 = 65% of its above-chance margin vs 23% for humans), and the finding that a more complete map (Full outline) still hurts (92.1→60.9) rules out the trivial "less information" explanation.

  2. Assistance-gradient evaluation against a human reference on identical questions. Length-controlled prompt additions that humans never received separate models sharply (Astra's Incongruent accuracy 55.5→79.9 under Neutral→Explicit vs ≤50.8 for other models, vs human 71.4% unaided), and the type-vs-place error decomposition localizes what guidance fixes.

Critical weaknesses
  1. Statistical reporting is absent exactly where claims are quantitative. GPT-6 Astra — the subject of every headline result — was run once, no confidence intervals appear anywhere, and the human reference (3 respondents/question) is reported without participant counts or variance. Moreover, Astra scores 8–10 points higher on the 576-question subset (Fig. 4: 84/52) than on the full benchmark (Table 1: 74.2/45.2) — beyond item-sampling noise — and this unexplained gap anchors the Section 5.3 recovery claims.

  2. Human–model protocol asymmetries partially confound the comparisons. Human FLOORPLAN instructions teach map-reading conventions ("light gray areas are floor space", "gaps usually indicate openings") absent from the model prompt; the TRIANGLE demo item's content is unspecified and could have exposed the drawing convention before scored items. "Humans achieve X without guidance" is thus a behavioral reference, not a matched-condition comparison.

Questions for authors
  1. Multi-seed runs or item-level bootstrap CIs for Astra, and between-participant variance for the human averages — Δ 29.1 vs 12.6 currently carries no uncertainty.

  2. Why is Astra ~8–10 points higher on the 576-question subset than on the full benchmark? Does the gap appear for other models?

  3. Per-model invalid/unparseable response rates — could format failures contribute to below-chance accuracy?

  4. What did the TRIANGLE demo item show, and was feedback given?

  5. Either give models the FLOORPLAN map-reading paragraph humans received, or remove it from the human protocol.

  6. A small pilot with other polygon families or more realistic wall renderings would address whether the below-chance collapses generalize beyond one abstract shape and rendering style.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KtXT (1/2)

Thank you for the positive feedback and the questions.

We answer the questions below. The changes in the revised PDF are marked in blue.

Questions

Q1: Multi-seed runs or item-level bootstrap CIs for Astra, and between-participant variance for the human averages — Δ 29.1 vs 12.6 currently carries no uncertainty.

(1) We ran Astra a second time for both FloorPlan and Triangle, and now report the standard deviation between the two independent runs in Table 1.

Below are the results, with 95% item-level bootstrap confidence intervals in square brackets. They are obtained by resampling questions with replacement 5,000 times.

The values for Astra are the average over the two runs. On Triangle, the two runs give Δ = 29.1 and Δ = 29.8. On FloorPlan, the accuracy is 88.2% for both runs.

QuestionsAstraHumans
Triangle Δ, full benchmark (3,888)29.4 [26.6, 32.3]N/A
Triangle Δ, subset of 576 questions32.0 [25.1, 38.6]12.6 [7.4, 17.4]
FloorPlan accuracy, full benchmark (3,603)88.2 [87.3, 89.2]N/A
FloorPlan accuracy, subset of 600 questions89.4 [87.1, 91.6]80.1 [77.7, 82.4]

On the same questions, Astra's Triangle Δ is 19.4 points larger than the human Δ [13.1, 25.4]. For FloorPlan, Astra's accuracy is 9.4 points higher than the human accuracy [6.8, 11.9]. Both difference intervals above exclude zero.

Resampling matched Congruent and Incongruent questions together, or resampling participants instead of questions, gives narrower intervals (Appendix B).

We limited Astra to two runs because each full run costs about $500.

(2) We also computed the between-participant variance of the human averages and added it to Appendix B. 192 participants each answered 9 Triangle questions and 180 participants each answered 10 FloorPlan questions.

The between-participant standard deviation of accuracy is 13.0 percentage points on Triangle and 14.8 on FloorPlan.

Q2: Why is Astra ~8–10 points higher on the 576-question subset than on the full benchmark? Does the gap appear for other models?

Other models are also higher on these 576 questions, and on these same questions the Δ difference between Astra and humans is larger and not smaller. All 24 models are reported on these questions in Table 8. On them, Astra's Δ is 32.0 and the human Δ is 12.6, a difference of 19.4 points [13.1, 25.4] (see Q1).

The 576 questions are complete sets (all four triangles and all nine map and view pairings) for a subset of the target and camera placements, so their composition differs from the full benchmark. For example, the target is visible in only one of the three views in 21.9% of these questions, versus 31.3% in the full benchmark.

Sec. 5.1 now reports Astra's Δ and the human Δ on these 576 questions, and every result in Sec. 5.3 is on these same questions.

Q3: Per-model invalid/unparseable response rates — could format failures contribute to below-chance accuracy?

Format failures have little effect on the reported accuracy and do not explain the below-chance results. GPT-6 Astra produced no invalid response on Triangle, and Gemini 3.8 Flash and Gemini Robotics-ER 1.6, which are below chance on Incongruent questions, have invalid rates of 0.03% and 0.19% there.

The table below summarises all 24 models. Invalid responses are excluded and accuracy and the random baseline are recomputed on the remaining questions.

ScopePer-model invalid rateMax. accuracy change after excluding invalid (pp)# models moving above chance
FloorPlan0.01–3.44%+0.733 (0.05-0.41 points above chance)
Triangle overall0–4.64%+1.110
Triangle Congruent0–4.19%+1.021 (0.31 points above chance)
Triangle Incongruent0–4.86%+1.160

After excluding invalid responses, accuracy changes by less than one percentage point for all 24 models on FloorPlan and for 23 of 24 models in each Triangle analysis. The few models crossing above chance exceed their recomputed baselines by only 0.05-0.41 percentage points.

All models originally below chance remain numerically below it on Triangle overall and Incongruent questions.

Per-model rates are in Table 10 of the Appendix.

We note that producing a valid answer is also part of completing the task, so invalid or unparseable responses represent task failures and count as incorrect in all reported accuracies.

AuthorsOpenReview ↗

Response to Reviewer KtXT (2/2)

Q4: What did the TRIANGLE demo item show, and was feedback given?

The Triangle demo item shows a simple right isosceles triangle that is not in the benchmark, and feedback was limited to correct or incorrect to check task understanding. It is a Congruent question, with Full walls in both the map and the views.

The demo therefore does not show a map drawn differently from the views, and no explanation of how the drawings relate was given.

We have added this description to Appendix B.

Q5: Either give models the FLOORPLAN map-reading paragraph humans received, or remove it from the human protocol.

To test the influence of the map-reading paragraph on FloorPlan accuracy, we conducted an additional experiment: we added it to the prompts of two models (one open-source and one commercial, both above chance), and accuracy changes by −0.9 to +1.0 points.

Each model received three prompt versions: the original prompt with the map-reading paragraph added, the same prompt plus the sentence on matching the floor plan to the views that humans also received, and the full human instructions in place of the original prompt.

ModelOriginal prompt (Table 1)+ map-reading paragraph+ map-reading paragraph and matching sentenceFull human instructions
Gemini 3.8 Flash59.9% ± 0.459.6%59.0%59.6%
Qwen3.8-27B43.5% ± 0.443.7%44.2%44.5%

The original-prompt column lists the mean and standard deviation of three runs from Table 1, while the other columns each report one run on all 3,603 questions.

In every version, both models remain about 20 points or more below the 80.1% average human response. We also note that GPT-6 Astra, which did not receive the paragraph, already exceeds the average human response on FloorPlan.

We tested two models because re-running every model in Table 1 would require a very large amount of API and GPU costs (we ran 2-3 seeds for each model, and for a single model such as Astra, two runs cost about $1,000).

We report this comparison in Table 9 of the Appendix.

Q6: A small pilot with other polygon families or more realistic wall renderings would address whether the below-chance collapses generalize beyond one abstract shape and rendering style.

We ran a pilot with two new distinct polygons with the Triangle pipeline and find that the Congruent-Incongruent drop persists across both new shapes for both a commercial and an open-weight model.

We provide example questions and full data in the anonymized links below:

  • Trapezoid extends Triangle's design from three to four distinguishable angles (45°, 75°, 105°, 135°), contributing 774 questions.
  • Arrowhead introduces mirror symmetry and a 220° concave angle (50° tip and two 45° mirrored angles), contributing 810 questions.

For both shapes, we cross 3 map and 3 egocentric view formats as in Triangle.

We evaluate Gemini 3.8 Flash and Qwen3.8-27B over three runs each (with mean ±\pm sample SD):

TriangleTrapezoidArrowhead
Chance (%)29.629.627.927.926.726.7
Gemini 3.8 Flash
Overall (%)34.9±0.334.9 \pm 0.335.1±0.635.1 \pm 0.639.7±0.739.7 \pm 0.7
Congruent (%)47.9±0.747.9 \pm 0.742.1±1.642.1 \pm 1.652.2±1.652.2 \pm 1.6
Incongruent (%)28.3±0.228.3 \pm 0.231.6±0.131.6 \pm 0.133.5±1.333.5 \pm 1.3
Δ\Delta19.6±0.719.6 \pm 0.710.5±1.510.5 \pm 1.518.8±2.718.8 \pm 2.7
Qwen3.8-27B
Overall (%)36.7±1.136.7 \pm 1.135.7±1.935.7 \pm 1.939.6±1.339.6 \pm 1.3
Congruent (%)42.0±0.842.0 \pm 0.842.8±1.842.8 \pm 1.852.5±1.952.5 \pm 1.9
Incongruent (%)34.0±1.334.0 \pm 1.332.1±1.932.1 \pm 1.933.2±1.733.2 \pm 1.7
Δ\Delta8.0±0.78.0 \pm 0.710.7±0.210.7 \pm 0.219.3±2.419.3 \pm 2.4

Both models show clear Congruent-to-Incongruent drops on both new shapes. Qwen3.8-27B's drop in particular is even larger on Trapezoid (10.7) and Arrowhead (19.3) than on Triangle (8.0). The sensitivity to how the map is drawn is therefore not specific to Triangle.


Given these revisions, would you kindly consider raising your score on our paper? If any concerns remain, could you please let us know what additional changes you would like to see in order to recommend a clear accept?

Reviewer_S7mHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces REMAP, a benchmark for evaluating cross-perspective spatial reasoning in vision-language models. It comprises two settings, FloorPlan and Triangle, which assess the correspondence between egocentric observations and allocentric maps. Experiments across 24 VLMs and human participants reveal substantial differences in robustness to changes in map representations. The results suggest that strong performance on spatial tasks does not necessarily imply robust reasoning across different geometric representations.

Critical strengths
  1. Clear presentation. The paper is well written and organized, with effective figures and a clear experimental narrative.

  2. Thoughtful benchmark design. The controlled construction of map-view pairs is original and allows the authors to isolate the effects of changes in map representation.

  3. Insightful experimental analysis. The ablation studies reveal interesting differences across models, particularly their sensitivity to map representations compared with the relative robustness of human participants.

Critical weaknesses
  1. Limited support for the central claims. The experiments mainly reveal sensitivity to specific map representations, rather than directly examining general cross-perspective spatial reasoning. The observed failures may also reflect unfamiliarity with drawing conventions, making the broader conclusions insufficiently supported.

  2. Limited benchmark diversity and generalizability. FloorPlan builds on an existing dataset, while Triangle contains only four basic geometric configurations. Despite the number of generated questions, the limited geometric diversity, lack of progressive challenges, and absence of real-world validation weaken the benchmark's novelty and the generalizability of its findings.

Questions for authors
  1. Scope of spatial reasoning. REMAP primarily evaluates geometric correspondence under changes in map representation. How does this limited setting support broader conclusions about cross-perspective spatial reasoning and consistency?

  2. Geometric diversity and task progression. FloorPlan and Triangle differ considerably in their geometric structures, without a clear progression of spatial challenges. Why were these two settings selected, and how can their results be interpreted within a unified evaluation framework? Could more diverse geometries and spatial relationships strengthen the benchmark?

  3. Generalizability and evaluation scale. Although the benchmark contains thousands of questions, many are generated from a small number of underlying geometric configurations. How representative are these variations of independent spatial reasoning challenges? Can the observed findings generalize to realistic environments with more complex geometry, visual features, and occlusions?

  4. Interpretation of representation sensitivity. The substantial performance recovery following explicit guidance suggests that some errors may arise from unfamiliar drawing conventions rather than deficiencies in spatial reasoning itself. How can the authors distinguish these factors and establish that the observed failures reflect limitations in cross-perspective reasoning?

  5. The Introduction suggests that interpreting missing corners in a room sketch requires recognizing drawing conventions rather than physical openings. However, this interpretation may depend on prior knowledge of common drawing conventions rather than spatial reconstruction itself. Humans may already be familiar with such abstractions, whereas VLMs may interpret them differently. How do the authors disentangle learned representational conventions from genuine spatial reasoning ability? Could the substantial improvement under explicit guidance indicate a problem of convention understanding rather than spatial reasoning?

3 comments
AuthorsOpenReview ↗

Response to Reviewer S7mH (1/2)

Thank you for your feedback and for engaging.

We answer the points raised below. The changes in the revised PDF are marked in blue.

Ethics statement

The ETHICS STATEMENT lacks a declaration regarding whether any necessary information was withheld [...] and educational background.

We have updated the declaration and the participant information in the Ethics Statement.

No necessary information was withheld when human participants were involved.

Recruitment was restricted to ages 21 to 40 and to current residents of the United Kingdom, the United States, Ireland, Australia, Canada, and New Zealand. Across the 372 unique participants (180 in FloorPlan and 192 in Triangle, with no overlap), mean ± SD age was 31.3 ± 5.3 years, and residence was distributed across the United States (44.4%), the United Kingdom (31.5%), Canada (21.2%), and Australia, Ireland, and New Zealand (3.0% combined).

Following data minimization principles, educational background was not collected.

Could you please let us know whether the revised Ethics Statement resolves your ethics concern?

Questions

1: Scope of spatial reasoning. [...] How does this limited setting support broader conclusions about cross-perspective spatial reasoning and consistency?

We do not claim to test all of spatial reasoning. Our conclusion is that high accuracy on realistic layouts does not guarantee consistent egocentric-allocentric alignment across representations of the same geometry.

In REMAP, maps and first-person views connect only through 3D geometry, so alignment based on geometry should stay above chance when only the map is redrawn in another equivalent, human-readable way. However, even GPT-6 Astra, which beats humans in the realistic FloorPlan setup, falls below chance on Corners$\leftrightarrow$Sides map swaps in Triangle (Fig. 3).

We revised Section 1 to scope GPT-6 Astra's milestone to FloorPlan and Section 5.1 to describe the measured gap between settings rather than general spatial competence.

2a (Geometric diversity & progression): Why were these two settings selected, [...] a unified evaluation framework?

FloorPlan and Triangle share one map-to-view task and serve complementary roles rather than forming a single difficulty ladder.

FloorPlan evaluates correspondence on complex multi-room layouts, whereas Triangle (motivated by developmental map-reading studies) provides a simplified, controlled setting where map representation can change while the environment, views, candidates, and answer stay fixed.

Within each setting, difficulty varies across candidate numbers in FloorPlan (2-23 candidates) and 4 angle setups in Triangle whose angles grow progressively harder to tell apart (Fig. 2).

2b: Could more diverse geometries and spatial relationships strengthen the benchmark?

On more diverse geometries, we built two new polygon families with the Triangle pipeline. The redraw drop holds across both shapes for both a commercial and an open-weight model.

We provide example images and full data for both shapes in the anonymized links below:

  • Trapezoid extends Triangle from three to four distinguishable angles (45°, 75°, 105°, 135°), including 774 questions.
  • Arrowhead adds mirror symmetry and a 220° concave angle (50° tip, two 45° mirrored angles), including 810 questions.

Both shapes cross the 3 map and 3 view formats (Full, Corners, and Sides, the same ones used in Triangle).

Evaluating Gemini 3.8 Flash and Qwen3.8-27B over three runs each (with mean ± sample SD):

TriangleTrapezoidArrowhead
Chance (%)29.629.627.927.926.726.7
Gemini 3.8 Flash
Overall (%)34.9±0.334.9 \pm 0.335.1±0.635.1 \pm 0.639.7±0.739.7 \pm 0.7
Congruent (%)47.9±0.747.9 \pm 0.742.1±1.642.1 \pm 1.652.2±1.652.2 \pm 1.6
Incongruent (%)28.3±0.228.3 \pm 0.231.6±0.131.6 \pm 0.133.5±1.333.5 \pm 1.3
Δ\Delta19.6±0.719.6 \pm 0.710.5±1.510.5 \pm 1.518.8±2.718.8 \pm 2.7
Qwen3.8-27B
Overall (%)36.7±1.136.7 \pm 1.135.7±1.935.7 \pm 1.939.6±1.339.6 \pm 1.3
Congruent (%)42.0±0.842.0 \pm 0.842.8±1.842.8 \pm 1.852.5±1.952.5 \pm 1.9
Incongruent (%)34.0±1.334.0 \pm 1.332.1±1.932.1 \pm 1.933.2±1.733.2 \pm 1.7
Δ\Delta8.0±0.78.0 \pm 0.710.7±0.210.7 \pm 0.219.3±2.419.3 \pm 2.4

Both models show clear Congruent-to-Incongruent drops on both new shapes, losing 72%–75% of their Congruent margin above chance across Trapezoid and Arrowhead (the same metric as in Sec. 5.1), with Qwen3.8-27B's raw drop even larger on both (10.7 and 19.3) than on Triangle (8.0).

AuthorsOpenReview ↗

Response to Reviewer S7mH (2/2)

3a (Generalizability & evaluation scale): How representative are these variations of independent spatial reasoning challenges?

Triangle's 3,888 questions are matched comparisons by design rather than independent geometries, while FloorPlan spans diverse and distinct home layouts.

In Triangle, 4 angle setups × 6 targets × 18 camera triplets form 432 base combinations, each evaluated across all 9 map × view pairings. Within each view format, repeating the same target and egocentric views across map formats isolates the map drawing as the sole cause of the drop, and GPT-6 Astra's redraw penalty holds consistently across all 4 triangles (Δ=26.2\Delta = 26.2–33.333.3 points).

FloorPlan provides structural diversity across 291 Structured3D home layouts.

3b: Can the observed findings generalize to realistic environments with [...] occlusions?

FloorPlan evaluates complex multi-room geometry and partial scene occlusion, while surface textures and objects are omitted by design.

FloorPlan includes doorways, windows, and cross-room wall occlusions (Sec. 5.4 analyzes 203 targets outside the camera room, each visible in at least one view).

REMAP deliberately removes textures and objects so semantic cues cannot shortcut geometric correspondence, leaving textured or photographic scenes outside the target scope of this benchmark. We updated this point in Section 6.

4 (Interpretation of representation sensitivity): How can the authors distinguish these factors [...] cross-perspective reasoning?

Giving the Explicit convention prompt helps distinguish convention reading from spatial localization across the models tested: it resolves most of GPT-6 Astra's drop, whereas the other tested models still fail localization.

On the 576-question subset (Fig. 4), Explicit convention shrinks Astra's Congruent-to-Incongruent gap from 31 points (Neutral) to about 4 points (83.9% vs. 79.9%), consistent with convention interpretation accounting for most of Astra's loss.

By contrast, Gemini 3.8 Flash, Gemini 3 Flash, and Qwen3.8-27B reach only 47.4%-50.8% Incongruent accuracy (and 53.6%-56.8% even on Congruent). On the 96 Incongruent Corners <-> Sides items (Fig. 5), among questions where Explicit convention corrects a wrong-type error (i.e., picking a corner for a side midpoint, or vice versa), Gemini 3.8 Flash and Qwen3.8-27B choose the right location between the two same-type candidates only 33% and 47% of the time—at or below the 50% chance rate (vs. 68% for Astra).

5: The Introduction suggests that [...] rather than physical openings.

Yes. For GPT-6 Astra, most of the gain under guidance comes from convention reading.

In practical tasks, convention reading and spatial alignment happen together, and our graded prompts help distinguish them. For instance, when assembling furniture from a diagram or reading a room sketch, people work out how parts are drawn while matching them in 3D space.

It is difficult to disentangle in real tasks, but because we designed the Triangle convention, we can supply it in graded steps (cf. Section 5.3). The experiment shows that Astra's drop comes mainly from convention inference, while the other three models still fail localization even after the convention is stated.

Concerns

  1. Limited support for the central claims

Please see our response to Questions 1, 4 and 5.

  1. Limited benchmark diversity and generalizability

Please see Questions 2a-2b and Questions 3a-3b.


In light of these revisions, would you kindly consider increasing your score for our paper? If any concerns remain, could you please let us know any additional changes you would like to see in order to recommend acceptance?

AuthorsOpenReview ↗

Gentle follow-up on our rebuttal

Dear reviewer S7mH,

As the discussion phase comes to a close, we wanted to gently follow up and see if our response and new experimental results have resolved your questions.

If anything remains unclear, we would be glad to elaborate. We would greatly appreciate your feedback as you finalize your assessment.

Thank you again for your time and thoughtful review!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response to reviewers

Thank you to all reviewers for their thoughtful feedback. We were encouraged that they found our setting well motivated and grounded in "cognitive studies of human spatial reasoning" (Reviewer pqbT), that REMAP's "thoughtful benchmark design" (Reviewer S7mH) enables "causal isolation of representation sensitivity" (Reviewer KtXT) through an "answer-preserving intervention" that is "much more informative than comparing across unrelated benchmarks" (Reviewer sEZS), and that the paper offers "detailed error analysis" (Reviewer 7Hib).

Questions raised across reviews are summarized below alongside our new experiments and manuscript revisions (in blue). For additional points not covered here, please see our individual responses.

  1. Generalization beyond Triangle (Reviewers S7mH, KtXT, sEZS): the Congruent-Incongruent performance drop persists in new polygon families. We extended the Triangle framework to two new shapes, Trapezoid (774 questions) and Arrowhead (810 questions). Across three runs each, both Gemini 3.8 Flash and Qwen3.8-27B show 10.5-19.3 point drops, losing 72-75% of their above-chance Congruent accuracy margin (see our responses to Reviewers S7mH, KtXT, and sEZS). We also clarify how Triangle's controlled map redraws and FloorPlan's multi-room layouts with occlusion serve complementary roles (Sections 3.2, 3.3, and 5.4).

  2. Human-model comparability (Reviewers KtXT, sEZS, pqbT, 7Hib): giving models the instructions and examples provided to human participants does not close the gap with humans. We expanded Appendix B.1 to clarify the human study procedures, confirming that no participant saw two drawings of the same question and that Triangle participants received no explanation of how the drawing formats relate. Giving tested models the human FloorPlan instructions (Appendix D, Table 9) or solved Triangle demo and in-context examples (see our responses to Reviewers pqbT and sEZS) still leaves them over 20 points below humans on both FloorPlan and Incongruent Triangle questions.

  3. Statistical reliability and within-type localization (Reviewers KtXT, sEZS, 7Hib, pqbT): our main findings hold across independent runs, grouped resampling, and multi-candidate subsets. We added a second independent run of GPT-6 Astra with consistent results (on FloorPlan, 88.2% in both runs and Triangle drops of 29.1 and 29.8 points), alongside bootstrap confidence intervals and between-participant variability in Appendix B.1. In addition, on the 80.3% of Triangle questions with at least two visible candidates matching the target's type, Astra and Gemini 3.8 Flash still show substantial drops (30.2 and 16.0 points) despite a 49.4% type-aware random baseline (Appendix D).

  4. Scope and interpretation of findings (Reviewers S7mH, sEZS, pqbT, 7Hib): we clarify the scope of our conclusions and distinguish convention interpretation from target localization. The revised manuscript focuses its conclusions on cross-perspective alignment across equivalent geometric representations. We further distinguish difficulties interpreting drawing conventions from difficulties localizing the target after those conventions are explicitly supplied.

  5. Ethics disclosures (Reviewer S7mH): the statement is updated, and the protocol and demographics are fully documented. The Ethics Statement and Appendix B.1 confirm that no necessary information was withheld, alongside informed consent, IRB exemption, and pay. We also provide the statistics on participants' age and geographical distribution. Following data minimization principles, we didn't collect information on educational background.

We have also expanded reporting of inference settings (Appendix C, Table 7), invalid responses (Appendix D, Table 10), and experimental limitations (Section 6) in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.