acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Single Drawing: MultiFloorQA for Cross-View Spatial Reasoning in Architectural Drawings

Abstract

Spatial reasoning in multimodal large language models (MLLMs) remains insufficiently understood, particularly when spatial evidence must be integrated across multiple views. MultiFloorQA is introduced as a benchmark of 608 question–answer pairs over multiple architectural drawings, spanning six spatial task categories and Single-floor, Cross-floor, and Cross-projection evidence. Drawing generation, question construction, and reference-answer computation are grounded in a shared building representation, enabling precise and reproducible evaluation. Evaluation of representative MLLMs reveals substantial variation across models and spatial operations, while precise geometric computation and larger-scale spatial composition remain challenging even for strong systems. Controlled input-condition experiments further indicate that recovering and organizing geometry from the drawings themselves is a major source of difficulty. These findings motivate further advances in visual grounding, geometric computation, and cross-view spatial integration.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Spatial Reasoning, Architectural Drawings, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_WUubOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MultiFloorQA, a benchmark for evaluating spatial reasoning over architectural drawings. It contains 608 question-answer pairs across six task categories, covering single-floor, cross-floor, and cross-projection reasoning. The authors evaluate several MLLMs and show that precise geometric computation and complex multi-floor reasoning remain challenging.

Critical strengths
  1. The benchmark provides a meaningful setting for evaluating spatial reasoning over architectural drawings, particularly across multiple floors.

  2. The unified building representation enables automatic question generation and programmatically verifiable ground-truth answers.

  3. The experiments cover multiple MLLMs and provide detailed evaluations across different spatial reasoning tasks.

Critical weaknesses
  1. Insufficient analysis of the visual perception bottleneck. The structured JSON experiments in Section 4.4 and Appendix F show substantial performance improvements when drawings are replaced with structured representations. However, JSON provides explicit coordinates, geometric relationships, object structures, and potentially topological and cross-floor information. Therefore, the improvements cannot be attributed solely to visual information extraction. More fine-grained ablations, such as providing coordinates without topological relationships, would help identify the primary bottleneck.

  2. Insufficient clarification of cross-floor connectivity. Section B.2 states that cross-floor correspondence between stairs and elevators does not necessarily imply a traversable connection. However, Section C.3 describes navigation over rooms, doors, cross-floor connections, and traversable regions. The authors should provide additional explanations of how cross-floor correspondences are converted into traversable connections and how the navigation graph is constructed.

  3. Lack of a human baseline. The benchmark does not provide human performance for comparison, making it difficult to assess the difficulty of the tasks and how close current MLLMs are to human-level architectural spatial reasoning. Including a small group of participants with relevant professional backgrounds would provide a useful reference for evaluating model capabilities.

Questions for authors
  1. Could the authors provide more fine-grained ablations to distinguish visual information extraction from subsequent spatial reasoning?

  2. How are traversable cross-floor connections determined from the building representation?

  3. Is it possible to include a small-scale human baseline, preferably involving participants with architectural expertise?

1 comment

Response to Reviewer WUub

We thank Reviewer WUub for recognizing the research value of MultiFloorQA, its unified building representation, and its verifiable data construction methodology. In response to your suggestions, we have added structured information ablation experiments and a human baseline evaluation, and further clarified the rules for cross-floor connectivity. We address each point below.

W1/Q1. Distinguishing Visual Information Extraction from Spatial Reasoning

We thank the reviewer for suggesting a clearer distinction between visual information extraction and subsequent spatial reasoning. We added structured information ablation experiments in Section 4.5 to further analyze how different types of spatial information affect model performance. The experimental settings and complete results are provided in Appendix F.3.

Using the same 64 MultiFloorQA-Main questions as in the input-condition experiments in Section 4.5, we supplemented the original architectural drawings with three types of structured information:

  • Geometry: Coordinates and dimensions of rooms, objects, doors, and windows.
  • Relations: Spatial relationships, including entity containment and room connectivity.
  • Full JSON: The complete structured building representation.

The results are summarized below:

ModelOriginal+Geometry+Relations+Full JSON
GPT-5.6 Sol79.69%96.88%85.94%100.00%
DeepSeek-V4.1-Flash45.31%98.44%71.88%100.00%
GLM-5.3-Flash62.50%92.19%70.31%96.88%

The results show that providing supplementary structured information substantially improves model performance, with geometric information generally yielding larger improvements than relational information. This suggests that accurately extracting geometric information from drawings may be an important bottleneck in spatial reasoning.

W2/Q2. Cross-Floor Connectivity and Navigation Graph Construction

We thank the reviewer for pointing out that the cross-floor connectivity rules were insufficiently clear. We have added specific criteria and computation methods in Appendices B.2 and C.3.

Cross-floor connectivity: We examine stairwell and elevator spaces on adjacent floors within a unified building coordinate system. When two spaces have the same type, satisfy geometric alignment requirements in plan position, and admit a unique correspondence, we establish a bidirectional traversable connection according to the benchmark's navigation conventions.

Navigation graph and path computation: We construct the navigation graph using rooms and traversable spaces as nodes, connected by doors, open passages, and the cross-floor connections defined above, while excluding doors specified as closed in the question. Minimum-door tasks count the number of physical doors traversed, with cross-floor transitions and open passages contributing no door cost. Shortest traversable-distance tasks compute path lengths using doorway positions, room boundaries, visible obstacles, and the floor heights specified in the question.

During evaluation, models receive only the architectural drawings, questions, and task conventions. They must infer spatial connectivity and perform path planning without access to the internal navigation graph or reference routes.

W3/Q3. Human Baseline Evaluation

We thank the reviewer for suggesting a human evaluation involving participants with architectural backgrounds. We added a human baseline experiment in Section 4.4, and compared human and model performance on the same questions.

We invited two participants with architectural backgrounds to answer questions using only the provided architectural drawings, questions, and necessary task conventions, without calculators, pen and paper, or AI assistance. Their answers were evaluated using the same scoring rules as the models.

The experiment included 92 questions: 64 from MultiFloorQA-Main and 28 from MultiFloorQA-Hard.

The results are presented below:

ModelMultiFloorQA-MainMultiFloorQA-Hard
GPT-6 Astra90.62%67.86%
GPT-5.6 Sol79.69%42.86%
Gemini 3.1 Pro Preview60.94%46.43%
Qwen3.8 Flash60.94%28.57%
DeepSeek-V4.1-Flash45.31%21.43%
Human78.12%25.00%

We find that the human participants perform in the upper-middle range on MultiFloorQA-Main, while their performance falls toward the lower end on MultiFloorQA-Hard. This may indicate that humans and models exhibit different generalization capabilities as task difficulty increases.

These results provide a human baseline reference for MultiFloorQA, helping us better understand task difficulty and performance differences between models and human participants.

Reviewer_mVChOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

Paper introduces MultiFloorQA, which is a benchmark desinged to assess multimodal large language models (MLLMs) spatial reasoning capability on architectural drawings. The benchmark consists of 608 questions (482 Main, 126 Hard) created from 186 drawings of 25 buildings (16 for Main; 9 for Hard) spanning six spatial task categories (Recognition, Navigation, Distance, Geometry, Matching, Counting) and three evidence families (Single-floor, Cross-floor, Cross-projection). The benchmark is designed to satisfy three principles (1) visual evidence must being present in the drawings, (2) correspondence between related entities being well-defined, and (3) reference answers being computable and verifiable under spatial rules. Evaluation on 10 MLLMs demonstrate that there are performance variations across models and spatial operations, while precise geomtric computation and larger-scale spatial composition still remains as a challenging task for modern MLLMs.

Critical strengths
  1. The paper addresses an interesting problem of spatial reasoning over architectural drawings. The benchmark evaluates spatial reasoning across Single-floor, Cross-floor, and Cross-projection which is valuable to examine how models interpret spatial information within each architectural drawing and also for evaluating how they integrate information when distributed across multiple floors.

  2. The benchmark construction process is well-designed in a manner that supports reproducibility. The paper uses a unified building representation to create architectural drawings, create questions, and then compute answers deterministically (using explicit geometric and topological operations). This reduces the potential ambiguity in the answer and is especially valuable, as it allows the model's output to be evaluated against well-defined spatial rules rather than subjective judgments.

Critical weaknesses
  1. One of the findings of the paper (Table 4) is that replacing architectural drawing with JSON improves model performance. However, this result seems somewhat expected as structured JSON provides explicit geometric information and spatial relationship unlike the images where the model first will need to extract them from the drawings. While it is an interesting experiment, it is unclear what additional insights this experiment provides regarding model's spatial reasoning capability.

  2. Some of the models already achieve 100% or very high performance on the MultiFloorQA-Main set, which raises the question of whether some tasks (Matching task) are sufficiently challenging to distinguish capabilities between models. The Hard subset doesn't include the Matching task (and Recognition task), which makes it difficult to evaluate whether the model's performance is impacted as the complexity of the architectural diagram increases.

  3. The questions are derived from 25 synthetic buildings, which leads to many questions sharing the same architectural drawing. The count does not seem sufficient to capture the diversity of architectural layout and spatial configurations. Additionally, as the drawings are created through some rendering pipeline, whether similar results will be shown in more complex real-world buildings is questionable. Some subjective evaluation on the quality of diagrams from these synthetic buildings seems to be needed.

Questions for authors
  1. Please provide further justification on why a 5% relative error tolerance was selected for distance and area questions. It is unclear why specifically the 5% value was selected and how sensitive the evaluation results are to the tolerance value. Also, as the benchmarks include categorical and Boolean questions, what is the random-chance baseline for each answer type? This will help better understand the model accuracies.

  2. What were measures used for quality control on the architectural drawings? Did it go through human or expert validation? Can we ensure that all questions (given the 5% tolerance) are actually answerable from the visual drawings alone? If human validation was conducted, what are the details on the annotators?

  3. To what extend will the findings found through MultiFloorQA expect to be generalized to real world floor plans? Would MLLMs exhibit similar performance pattern when the six task categories are created from actual floor plan diagrams? Additional evaluation on real floor plans or discussion supported by existing evidence would help better understand the benchmark findings.

6 comments

Response to Reviewer mVCh (1/2)

We thank Reviewer mVCh for recognizing MultiFloorQA's research value, cross-drawing spatial reasoning tasks, and reproducible benchmark construction. In response to your suggestions, we conducted structured JSON ablations, expanded MultiFloorQA-Hard, evaluated real-world buildings, analyzed numerical tolerance and random baselines, and clarified drawing quality control and question answerability. We address each point below.

W1. Structured JSON Ablation

We thank the reviewer for suggesting further analysis of the JSON experiment. We added structured information ablations in Section 4.5, Input-Condition Comparisons, with detailed settings in Appendix F.3, Structured JSON Ablation.

Using the same 64 MultiFloorQA-Main questions, we supplemented the original drawings with three types of structured information:

  • Geometry: Coordinates and dimensions of rooms, objects, doors, and windows.
  • Relations: Spatial relationships, including entity containment and room connectivity.
  • Full JSON: The complete structured building representation.
ModelOriginal+Geometry+Relations+Full JSON
GPT-5.6 Sol79.69%96.88%85.94%100.00%
DeepSeek-V4.1-Flash45.31%98.44%71.88%100.00%
GLM-5.3-Flash62.50%92.19%70.31%96.88%

Additional structured information substantially improves accuracy, with geometric information yielding larger gains than relational information. This suggests that extracting precise spatial information from drawings may be an important bottleneck in current models.

W2. Expanding MultiFloorQA-Hard

We thank the reviewer for questioning the ability of some tasks to distinguish model capabilities and the task coverage of MultiFloorQA-Hard. We further analyzed task discrimination in MultiFloorQA-Main and expanded MultiFloorQA-Hard. The revisions are included in Sections 3.3 and 4.3, with details in Appendix D.

Task discrimination in MultiFloorQA-Main. Matching represents a fundamental capability in architectural drawing understanding. Although some leading models achieve 100% accuracy, other models exhibit substantial differences. For example, Qwen3.8 Flash, DeepSeek-V4.1-Flash, and MiniMax M3 achieve 83.93%, 75.00%, and 46.43%, respectively. Thus, Matching still distinguishes models with different spatial reasoning capabilities.

Expanding MultiFloorQA-Hard. To better distinguish leading models, we added Recognition and Matching tasks, expanding MultiFloorQA-Hard from 126 questions across four categories to 180 questions across all six categories.

ModelRecognitionMatching
GPT-6 Astra100.00%97.22%
GPT-5.6 Sol83.33%77.78%
Gemini 3.1 Pro Preview5.56%27.78%
Qwen3.8 Flash5.56%41.67%
DeepSeek-V4.1-Flash27.78%41.67%

Compared with MultiFloorQA-Main, the added Recognition and Matching tasks in MultiFloorQA-Hard reveal substantial performance differences among leading models, providing stronger discrimination under more complex architectural configurations.

W3. Architectural Diversity and Real-World Applicability

We thank the reviewer for raising concerns about architectural diversity and real-world applicability. We first address architectural and task coverage, while drawing quality control and real-world evaluation are discussed in Q2 and Q3, respectively.

Architectural diversity. MultiFloorQA includes 25 buildings spanning 11 building types, 2–10 floors, and 1,202 labeled rooms and functional spaces. These cover residential, cultural, recreational, transportation, and other everyday environments.

Task diversity. The expanded benchmark contains 662 questions across 32 templates, covering Single-floor, Cross-floor, and Cross-projection reasoning. Among the architectural drawing benchmarks compared in Table 1, MultiFloorQA provides the largest number of evaluation instances and broad task coverage.

Extensibility. Based on MansionWorld's unified building representation, we developed a pipeline for drawing generation, question instantiation, and answer verification. This pipeline supports incorporating additional building types and expanding architectural diversity.

Building sources and drawing construction are described in Section 3.2, with coverage statistics in Appendix B.7.

Response to Reviewer mVCh (2/2)

Q1. Numerical Tolerance and Random-Chance Baselines

We thank the reviewer for requesting justification of the 5% relative error tolerance, sensitivity analysis, and random-chance baselines. We conducted additional experiments to examine the robustness of our evaluation.

Justification of the 5% tolerance. We adopted a 5% relative error tolerance based on common engineering practice. Furthermore, our sensitivity analysis below shows that the choice of tolerance has little impact on the overall model rankings.

Tolerance sensitivity. We rescored the original responses of 10 models on all 482 MultiFloorQA-Main questions using tolerances from 1% to 10%. Rankings remain unchanged from 1% to 7%. At 8%–10%, only the third- and fourth-ranked models exchange positions. The main model comparisons therefore remain stable across tolerance settings (Appendix E.7).

Random-chance baselines. Using the same 64 MultiFloorQA-Main questions as in Section 4.5, we performed 10,000 random-answer trials (640,000 responses), applying answer-type-specific sampling rules for discrete and continuous answers.

Answer TypeRandom Accuracy
Boolean49.91%
Category/Direction18.93%
Identifier11.93%
Integer Count9.40%
Continuous Distance3.91%
Continuous Area7.07%

These baselines provide additional context for interpreting model accuracy across answer types. Sampling procedures and complete results are provided in Appendix F.2.

Q2. Drawing Quality Control and Question Answerability

We thank the reviewer for asking about quality control, human or expert validation, annotator details, and whether questions are answerable from visual drawings alone.

Drawing quality control.

We combined programmatic checks with human review of legends, label readability, entity-identifier correspondence, annotation occlusion, and cross-drawing consistency. Starting from 1,000 MansionWorld buildings, we generated and checked drawings for 947, retaining 302 compliant buildings. Automated screening selected 120 candidates based on question-generation completeness and room-type diversity, followed by manual selection of 16 buildings for MultiFloorQA-Main and 9 complex high-rise buildings for MultiFloorQA-Hard.

Expert validation. Architectural professionals reviewed generated drawings and representative cases, checking label readability, annotation accuracy, and spatial information. Based on their review, we improved the drawing quality standards and then used automated checks to screen drawings on a large scale.

Question answerability. During QA generation, automated checks ensure that the evidence required to answer each question is present and unobstructed in the drawings, and that each question has a valid answer.

To further investigate question answerability, we analyzed failure results and found that errors mainly occur in entity localization, spatial relationship recognition, and geometric computation (Section 4.6). Based on these errors, we developed specific solution guidance and reminders about common mistakes and incorporated them into prompts, improving accuracy across three models (Section 4.5). Additionally, providing geometric or relational JSON information substantially improved model accuracy (Section 4.5).

These results indicate that our questions can be correctly answered when models follow appropriate solution procedures and accurately extract the required spatial evidence.

Further details on quality control and question validation are provided in Appendices B.6 and C.4, while the complete prompting and JSON ablation settings are provided in Appendices F.6 and F.3.

Q3. Generalization to Real-World Floor Plans

We thank the reviewer for suggesting evaluation on real-world floor plans. We extended our drawing generation, six-category question construction, and evaluation pipeline to real-world building data.

Specifically, using Matterport3D scans and structural annotations, we constructed a supplementary benchmark of 4 real-world buildings, 211 questions, six task categories, and 26 templates, following the same unified drawing pipeline. We evaluated three models:

ModelOverallRecognitionCountingDistanceGeometryMatchingNavigation
GPT-5.6 Sol61.14%90.00%57.50%50.00%22.50%95.65%67.86%
DeepSeek-V4.1-Flash43.60%52.50%40.00%27.50%22.50%82.61%57.14%
GLM-5.3-Flash44.55%75.00%45.00%37.50%15.00%69.57%32.14%

The results demonstrate that our drawing generation, QA construction, and evaluation pipeline can be extended to real-world building scan data. We also observe similar task-level difficulty patterns to MultiFloorQA-Main, with Geometry and Distance remaining challenging.

Experimental details, results, and examples are provided in Appendix G.

Reviewer_mVChOpenReview ↗

Response to Authors on Rebuttal

Thank you for the detailed response and additional experiment results. The rebuttal addresses many of my initial concerns through the ablation, tolerance testing, and real-world building data.

I have two new questions regarding the newly added floor plan set. (1) Would the authors be releasing the newly designed 'Real-world floor plan' set? How do the authors plan to release the newly designed benchmark set relative to the original Main and Hard sets? (2) Why are elevation-matching and cross-floor navigation not evaluated for the new set? The current results seem to show generalization to certain spatial reasoning tasks, but not to all of those evaluated in the overall benchmark.

Reviewer_RmGROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MultiFloorQA, a new benchmark for evaluating MLLMs’ understanding of multiple architectural drawings of the same building through six tasks: Recognition, Counting, Distance, Geometry, Matching, and Navigation. Questions cover three evidence scopes: Single-floor, Cross-floor, and Cross-projection. The benchmark is constructed from MansionWorld building data using question templates and programmatically computed reference answers. It contains 608 questions divided into Main and Hard subsets. The authors evaluate multiple MLLMs spanning open-weight and closed-source models. The results reveal challenges in precise geometric computation even on the Main subset, including for GPT-6 Astra. The authors also conduct input-condition comparisons to investigate how different input representations and prompting conditions affect model performance.

Critical strengths
  • The proposed dataset covers a range of tasks and difficulty levels, supporting analysis of how MLLMs answer questions about multi-floor buildings. Its Main and Hard subsets further enable investigation of performance on more complex inputs.
  • The methodology clearly describes dataset construction and evaluation, making the approach easier to understand and supporting reproducibility.
  • The paper evaluates a range of modern MLLMs, including GPT-6 Astra, spanning open-weight and closed-source MLLMs.
  • The paper provides results for each task category, identifying areas where models struggle, alongside average token usage and request time to characterize inference costs.
Critical weaknesses
  • The failure analysis provided in the paper seems limited. A more detailed examination of how models fail on each subtask would better clarify their limitations and strengthen the paper’s findings.
  • Single-floor, Cross-floor, and Cross-projection are discussed in both the dataset construction and the results. However, the full results table appears only in the appendix. More discussion of these results in the main text, particularly for Cross-projection questions, would be helpful.
  • The performance difference between Single-floor and Cross-floor questions requires further investigation beyond the explanation involving dense visual reading. Because these settings have different task distributions, a controlled comparison would help clarify whether the difference stems from evidence scope or task composition.
  • Although the paper discusses increased spatial complexity, it does not systematically analyze the number of reasoning steps required by each question and relate it to the model results. Such an analysis would help clarify whether reasoning depth affects model performance.
  • The paper only provides the results without a hypothesis-driven approach or small experiments, such as prompt engineering, that could improve the current model results. This could strengthen the paper further.
  • This is mostly an additional experiment. Evaluating smaller models would also be useful. This could help establish the benchmark’s difficulty under more constrained computational resources and inform the development of future pipelines for this task.
Questions for authors

Please address the weaknesses raised above; most require responses. The only one that may remain for future improvement is the small model evaluation.

4 comments

Response to Reviewer RmGR (1/2)

We thank Reviewer RmGR for recognizing MultiFloorQA's task design, data construction, model coverage, and result analysis. In response to your six suggestions, we have expanded the failure analysis and discussion of evidence scopes, conducted controlled single-floor and cross-floor comparisons, analyzed reasoning complexity, performed additional input-condition experiments, and evaluated smaller models. We address each point below.

W1. Further Failure Analysis

We thank the reviewer for suggesting a more detailed failure analysis. We have expanded Section 4.6, Failure Analysis, to examine concrete failures from four aspects:

  1. Entity localization: In Recognition, Distance, Navigation, and Matching tasks, models may initially identify incorrect objects or spaces, causing subsequent reasoning errors. For example, GLM-5.3-Flash mistakes a table for a bed and continues reasoning based on the incorrect identification.

  2. Connectivity reasoning: In Recognition and Navigation tasks involving accessibility, models may assume nonexistent doors or overlook valid passages. For example, Qwen3.8 Max 0902 incorrectly assumes a direct door between an elevator and a stairwell, returning 3 m instead of the correct 6 m through an intermediate space.

  3. Geometric reconstruction and computation: In Distance and Geometry tasks, we observe arithmetic errors, incorrect geometric reconstruction, and inaccurate dimension estimates. MiniMax M3 calculates "45 + 3.2 = 47.2 meters." GPT-6 Astra treats a 16 m² room with a recess as a complete 18 m² rectangle. Qwen3.8 Max 0902 sometimes assumes standard furniture dimensions instead of extracting measurements from drawings.

  4. Cross-projection correspondence: In Matching tasks involving cross-projection reasoning, models may make directional errors or incorrectly match identifiers from different numbering systems. Qwen3.8 Flash incorrectly states that facing west places north on the left. MiniMax M3 assumes that elevation identifier E-W002 corresponds to floor-plan identifier W002 solely because they share the same number.

W2. Results Across Different Evidence Scopes

We thank the reviewer for highlighting the need for further analysis across evidence scopes. We have provided additional results and discussion in Appendix E.3, together with controlled single-floor and cross-floor comparisons in Appendix E.5.

Since our single-floor and cross-floor questions partly differ in their task settings, we therefore conducted a controlled comparison using paired tasks, finding consistently lower cross-floor accuracy (see W3).

Cross-projection tasks exhibit substantial model differences: GPT-6 Astra, GPT-5.6 Sol, and Qwen3.8 Max 0902 achieve 100% accuracy, whereas DeepSeek-V4.1-Flash and MiniMax M3 achieve 50.0% and 25.0%, respectively. These results demonstrate that some models reliably perform cross-view reasoning, while others struggle with viewpoint transformations and component correspondence. Representative failure cases are discussed in Section 4.6.

W3. Controlled Single-Floor vs. Cross-Floor Comparison

We selected five pairs of corresponding tasks from MultiFloorQA-Main. Each pair follows the same task pattern, primarily differing in whether the required information spans one or multiple floors. For example, N2 finds the shortest room route on one floor, while N6 extends this task across floors.

We calculated the average accuracy across 10 models for each pair:

Task PairSingle-FloorCross-FloorΔ (pp)
N1/N5: Minimum Doors91.3%81.3%−10.1
N2/N6: Shortest Room Route92.7%80.0%−12.7
N3/N7: Traversable Distance39.3%31.9%−7.5
D3/D9: AP Signal Coverage68.3%59.2%−9.1
D8/M4: Spatial Relationships98.1%87.5%−10.6

Cross-floor accuracy is lower in all five pairs, with decreases of 7.5–12.7 percentage points. These results indicate that corresponding tasks become more challenging under cross-floor conditions. Additional discussion is provided in Appendix E.5.

Response to Reviewer RmGR (2/2)

W4. Reasoning Steps and Model Performance

We thank the reviewer for suggesting a quantitative analysis of reasoning steps. We analyzed observed reasoning steps, reasoning token consumption, and accuracy across 64 MultiFloorQA-Main questions (12 models) and 28 MultiFloorQA-Hard questions (5 models).

Inspired by GQA (Hudson and Manning, CVPR 2019), we define an observed step as a complete semantic operation in visible reasoning traces, such as entity localization, relational judgment, or geometric computation. We consistently annotated these traces using GPT-6 Terra. The methodology and full results are provided in Appendix E.6, Reasoning Complexity Analysis.

ModelObserved Steps (Main → Hard)Tokens, k (Main → Hard)Accuracy (Main → Hard)
GPT-6 Astra7.30 → 9.994.67 → 8.7792.19% → 78.57%
GPT-5.6 Sol24.31 → 41.038.60 → 16.0079.69% → 50.00%
Qwen3.8 Flash89.25 → 127.2010.46 → 13.6157.81% → 17.86%
DeepSeek-V4.1-Flash156.12 → 230.1319.32 → 27.5143.75% → 17.86%

All four models exhibit more observed steps and higher token consumption on MultiFloorQA-Hard, accompanied by lower accuracy. These findings indicate that more complex spatial tasks generally involve more reasoning operations and lower accuracy, although task-specific requirements also affect difficulty.

W5. Additional Input-Condition Experiments

We conducted two additional experiments to investigate performance improvements and spatial reasoning bottlenecks.

Task-guided prompting. Inspired by Plan-and-Solve Prompting (Wang et al., ACL 2023), we designed two strategies: Plan, which provides an explicit solution procedure, and Detailed, which additionally includes task-specific operational guidance and reminders about common errors. We evaluated both on the same 64 MultiFloorQA-Main questions used in Section 4.5.

ModelOriginalPlanDetailed
GPT-5.6 Sol79.69%81.25%85.94%
DeepSeek-V4.1-Flash45.31%54.69%62.50%
GLM-5.3-Flash62.50%60.94%64.06%

Plan produces model-dependent results, while Detailed improves accuracy for all three models, demonstrating the benefits of task-specific guidance and error reminders.

Structured JSON ablation. Using the same 64 MultiFloorQA-Main questions, we supplemented the original drawings with three information types:

  • Geometry (coordinates and dimensions of rooms, objects, doors, and windows)
  • Relations (entity containment and room connectivity)
  • Full JSON (complete structured building representation).
ModelOriginal+Geometry+Relations+Full JSON
GPT-5.6 Sol79.69%96.88%85.94%100.00%
DeepSeek-V4.1-Flash45.31%98.44%71.88%100.00%
GLM-5.3-Flash62.50%92.19%70.31%96.88%

Additional structured information substantially improves accuracy, with geometric information yielding larger gains than relational information.

The prompting results are incorporated into Section 4.5, with category-wise results in Appendix F.6 and full prompts in Appendix I.1. The structured JSON ablation and Navigation-specific results are detailed in Appendix F.3.

W6. Evaluation of Smaller Models

We additionally evaluated Qwen3.5-9B and Qwen3.5-27B on all 482 MultiFloorQA-Main questions:

ModelOverallRecognitionCountingDistanceGeometryMatchingNavigation
Qwen3.5-9B30.91%52.38%24.68%21.17%27.42%48.21%32.41%
Qwen3.5-27B45.85%61.90%42.86%35.04%33.87%62.50%53.70%

Qwen3.5-27B outperforms Qwen3.5-9B overall and across all six categories. However, both exhibit relatively low overall accuracy, establishing useful baselines for future research on smaller models.

Both models have been added to the main results table in Section 4.2, with experimental settings documented in Appendix E.1.

I appreciate the author's response. They have resolved most of my initial concerns regarding the paper. The analysis of the reasoning step could be extended to show how difficulty increases as the reasoning step increases, even when the reasoning trace is annotated by the model. The drop from single-floor to across-floor seems consistent, but more analysis can also be provided to analyze what really happens with the model when moving from single-floor to across-floor. This is really a minor thing, though. Overall, I still appreciate all the responses. I would like to keep my score, which still leans toward acceptance of the paper.

Thank you for taking the time to review our responses and additional experiments. We sincerely appreciate your thoughtful and constructive suggestions, which have helped us make our paper more comprehensive and convincing.

Comments on the paper

1 comment

Follow-up on Author Responses

We thank the reviewers for their constructive feedback and the Area Chair for overseeing the review process. In response to the reviewers’ feedback, we have provided detailed responses and updated the manuscript, including new ablation and controlled experiments, dataset expansion, small-model and human baselines, real-world building evaluation, and further analyses and methodological clarifications.

With less than one day remaining in the discussion period, we would greatly appreciate any further feedback on our responses and new results. We would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.