- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
REMAP tests whether a model can match a top-down map to a first-person view, using texture-free indoor layouts and simple triangular enclosures. FloorPlan has 3,603 questions from 291 homes, Triangle has 3,888 questions crossing three map formats with three view formats. The nice part is that views, labels, and answers are held fixed while the map is redrawn, so the comparison is controlled. 24 models and a human subset are evaluated, and a set of graded guidance prompts recovers a lot of the drop seen in the strongest model (pp. 3-8; Appendices A-E).
Critical strengths
- The answer-preserving intervention is much more informative than comparing across unrelated benchmarks. With Corners views fixed, redrawing the map as Sides drops Astra from 92.1% to 19.7%, and even the Full map version drops it to 60.9%. That is strong evidence of representation sensitivity and it doesn't require any guess about what is happening inside the model (p. 7, Figure 3).
- The guidance experiment is done on the same 576 questions for every model, with the three added paragraphs matched at 45 words. Combined with the breakdown by target type and location error, you can actually tell apart "can't do it" from "can do it with a hint." The human-model comparisons are matched, including the 600-item FloorPlan comparison (pp. 8-9, Figures 4-6; p. 24).
Critical weaknesses
- I am not convinced the humans and models got the same information. Participants had a Triangle demo, screening questions, and repeated trials, and the FloorPlan instructions include a legend. But participant counts, what was in the demo and screening, whether feedback was given, trial order, and total exposure are not really spelled out (pp. 21-23). So saying there was "no explicit convention statement" doesn't show equal learning opportunity, and doesn't show that the convention was inferred spontaneously. Also, full-benchmark numbers and human-subset numbers should be kept apart. Table 1 by itself can't support a matched Congruent human advantage.
- The benchmark doesn't say how much of the task is actually within-type localization vs just recognizing the target type. For example supplementary Q40 and Q1240 only have one candidate of the required type. This doesn't break the benchmark, and Figure 5 already restricts to a diagnostic subset, but I'd want the prevalence over the full data and a type-aware baseline. The repeated geometric blocks and the participant/item dependence also need proper uncertainty estimates, and Astra only has one full run (pp. 13-14, 23). The broader claims about general spatial competence should stay closer to what was actually measured.
Questions for authors
- Can you give participant counts, trial allocation, the demo and screening materials, feedback and ordering details, and the sampling manifest? Are the map-redrawing blocks preserved in the human subset, and do paired items show up in the same session?
- Can you release item/pairing IDs, raw predictions, exact inference settings, candidate-type counts per item, and block- and participant-aware uncertainty?
- Is the representation penalty still large when restricted to questions that need within-type localization? New shape families or photorealistic scenes are fine to leave for future work.
4 comments
Response to Reviewer sEZS (2/2)
Concerns
- I am not convinced [...] Also, full-benchmark numbers and human-subset numbers should be kept apart. [...]
We do not claim equal prior exposure or spontaneous rule induction, and Sec. 5.1 and Table 8 (Appendix D) now separate full-benchmark and matched human-subset results.
Without a demo or the FloorPlan map-reading paragraph (the "legend" in the human instructions), Astra is similar to humans on Congruent Triangle (83.1% vs. 84.0%) and exceeds them on FloorPlan (89.4% vs. 80.1%), diverging on Incongruent questions (51.0% vs. 71.4%; vs. ).
Humans saw no Incongruent pre-task question or format explanation (Question 1). Adding the solved demo or two solved Congruent examples shifts Incongruent accuracy by −0.4 and +0.9 points on Gemini 3.8 Flash and +6.5 and +3.1 on Qwen3.8-27B (≤49.5% vs. 71.4% humans). On FloorPlan, the three human-instruction variants shift both models by −0.9 to +1.0 points (Table 9).
- The benchmark doesn't say how much of the task is actually within-type localization [...].
Please see Question 3a for the share of questions with multiple same-type candidates and the corresponding guessing baseline, and Question 2 for group- and participant-resampled intervals and Astra's second run. We also scoped claims in Secs. 1, 5.1, and 6 to abstract geometric and untextured map–view representations.
In light of these additions, would you consider raising your score, or let us know if anything remains unclear?
Official Review of Reviewer sEZS
Thank you for the detailed review. The response has addressed my concerns, but it has not changed my overall assessment enough to warrant a high score. I keep the score of 3.
Reply to Reviewer sEZS
Thank you for your reply! We’re glad our response has addressed your concerns and appreciate your careful consideration of our submission and rebuttal.
Response to Reviewer sEZS (1/2)
Thank you for the thorough review and for recognizing our design and analysis.
Please find our responses below, with revisions in the updated PDF marked in blue.
Questions
All 9 map and view pairings are preserved for every combination in the Triangle human subset. Also, no participant answered two differently drawn versions of the same question.
Each of the 64 triangle, target, and camera combinations appears in all 9 pairings (576 questions). Each 9-question set draws from 9 different combinations, with one set per participant.
Across 372 non-overlapping participants, 192 answered the 64 Triangle sets of 9 and 180 answered the 60 FloorPlan sets of 10 (three participants per set, with questions in a fixed order). Before the scored questions, participants saw instructions (with a map-reading paragraph in FloorPlan), one Congruent demo with Full walls in Triangle (on a non-benchmark triangle, with feedback), and two one-attempt screening questions (both Congruent in Triangle), with no feedback on screening or scored questions.
We updated the details in Appendix B.
We will release the list of sampled questions (the "sampling manifest"), set assignments, and all the other materials upon acceptance.
We will release question, pairing, and group IDs, per-question counts of corner and side-midpoint candidates, and raw predictions upon acceptance.
We listed all inference settings in Table 7 in Appendix C. GPT-6 Astra now uses two full independent runs (Table 1).
In Appendix B.1, we added 95% bootstrap intervals (5,000 resamples) and between-participant SDs (13.0 points on Triangle, 14.8 on FloorPlan).
On the 576 subset, resampling the 16 target–camera groups ("blocks" in the reviewer's question, each pooling all 4 triangles and 9 pairings) gives Δ=32.0 [27.7, 36.2] for Astra, 12.6 [8.0, 17.0] for humans, and difference 19.4 [14.7, 24.8] (19.4 [13.1, 25.4] resampling questions; full-set Astra Δ=29.4 [27.3, 31.5] across all 108 groups).
Resampling participants gives human Triangle Δ=12.6 [8.5, 16.6] and FloorPlan accuracy 80.1% [77.9, 82.2].
Yes. 80.3% of Triangle questions (3,123/3,888) have at least two visible candidates of the target's type, corner or side midpoint (corresponding to within-type localization). The Congruent-to-Incongruent drop stays large on this subset.
On this subset, the group-resampled drop is 30.2 points [27.9, 32.5] for Astra and 16.0 [13.8, 18.3] for Gemini 3.8 Flash on the full set, and 12.5 [7.1, 18.1] for humans vs. 32.6 for Astra on the 576 subset.
Uniform guessing among visible candidates of the target's type (the type-aware baseline) gives 59.4% overall on Triangle and 49.4% on this subset (vs. 29.6% chance).
On Incongruent questions here, models fall below that baseline (full set: Astra 37.3%, Gemini 3.8 Flash 25.5% vs. 49.4%; human subset: Astra 44.8% vs. 50.0%, humans 66.9%). On the 19.7% of questions with one candidate of the target's type, Astra drops 26.2 points and the four Gemini models drop 18–34 points.
We added this analysis to Appendix D.
In a 3-run experiment (mean ± SD) on 774 Trapezoid and 810 Arrowhead questions (remap-new-polygons), Gemini 3.8 Flash drops 10.5±1.5 and 18.8±2.7 points (Δ) and Qwen3.8-27B drops 10.7±0.2 and 19.3±2.4 points (losing 72%-75% of their Congruent margin above chance).
Full numbers are in our response (Question 2b) to Reviewer S7mH.