- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MultiFloorQA, a benchmark for evaluating spatial reasoning over architectural drawings. It contains 608 question-answer pairs across six task categories, covering single-floor, cross-floor, and cross-projection reasoning. The authors evaluate several MLLMs and show that precise geometric computation and complex multi-floor reasoning remain challenging.
Critical strengths
-
The benchmark provides a meaningful setting for evaluating spatial reasoning over architectural drawings, particularly across multiple floors.
-
The unified building representation enables automatic question generation and programmatically verifiable ground-truth answers.
-
The experiments cover multiple MLLMs and provide detailed evaluations across different spatial reasoning tasks.
Critical weaknesses
-
Insufficient analysis of the visual perception bottleneck. The structured JSON experiments in Section 4.4 and Appendix F show substantial performance improvements when drawings are replaced with structured representations. However, JSON provides explicit coordinates, geometric relationships, object structures, and potentially topological and cross-floor information. Therefore, the improvements cannot be attributed solely to visual information extraction. More fine-grained ablations, such as providing coordinates without topological relationships, would help identify the primary bottleneck.
-
Insufficient clarification of cross-floor connectivity. Section B.2 states that cross-floor correspondence between stairs and elevators does not necessarily imply a traversable connection. However, Section C.3 describes navigation over rooms, doors, cross-floor connections, and traversable regions. The authors should provide additional explanations of how cross-floor correspondences are converted into traversable connections and how the navigation graph is constructed.
-
Lack of a human baseline. The benchmark does not provide human performance for comparison, making it difficult to assess the difficulty of the tasks and how close current MLLMs are to human-level architectural spatial reasoning. Including a small group of participants with relevant professional backgrounds would provide a useful reference for evaluating model capabilities.
Questions for authors
-
Could the authors provide more fine-grained ablations to distinguish visual information extraction from subsequent spatial reasoning?
-
How are traversable cross-floor connections determined from the building representation?
-
Is it possible to include a small-scale human baseline, preferably involving participants with architectural expertise?
Response to Reviewer WUub
We thank Reviewer WUub for recognizing the research value of MultiFloorQA, its unified building representation, and its verifiable data construction methodology. In response to your suggestions, we have added structured information ablation experiments and a human baseline evaluation, and further clarified the rules for cross-floor connectivity. We address each point below.
W1/Q1. Distinguishing Visual Information Extraction from Spatial Reasoning
We thank the reviewer for suggesting a clearer distinction between visual information extraction and subsequent spatial reasoning. We added structured information ablation experiments in Section 4.5 to further analyze how different types of spatial information affect model performance. The experimental settings and complete results are provided in Appendix F.3.
Using the same 64 MultiFloorQA-Main questions as in the input-condition experiments in Section 4.5, we supplemented the original architectural drawings with three types of structured information:
The results are summarized below:
The results show that providing supplementary structured information substantially improves model performance, with geometric information generally yielding larger improvements than relational information. This suggests that accurately extracting geometric information from drawings may be an important bottleneck in spatial reasoning.
W2/Q2. Cross-Floor Connectivity and Navigation Graph Construction
We thank the reviewer for pointing out that the cross-floor connectivity rules were insufficiently clear. We have added specific criteria and computation methods in Appendices B.2 and C.3.
Cross-floor connectivity: We examine stairwell and elevator spaces on adjacent floors within a unified building coordinate system. When two spaces have the same type, satisfy geometric alignment requirements in plan position, and admit a unique correspondence, we establish a bidirectional traversable connection according to the benchmark's navigation conventions.
Navigation graph and path computation: We construct the navigation graph using rooms and traversable spaces as nodes, connected by doors, open passages, and the cross-floor connections defined above, while excluding doors specified as closed in the question. Minimum-door tasks count the number of physical doors traversed, with cross-floor transitions and open passages contributing no door cost. Shortest traversable-distance tasks compute path lengths using doorway positions, room boundaries, visible obstacles, and the floor heights specified in the question.
During evaluation, models receive only the architectural drawings, questions, and task conventions. They must infer spatial connectivity and perform path planning without access to the internal navigation graph or reference routes.
W3/Q3. Human Baseline Evaluation
We thank the reviewer for suggesting a human evaluation involving participants with architectural backgrounds. We added a human baseline experiment in Section 4.4, and compared human and model performance on the same questions.
We invited two participants with architectural backgrounds to answer questions using only the provided architectural drawings, questions, and necessary task conventions, without calculators, pen and paper, or AI assistance. Their answers were evaluated using the same scoring rules as the models.
The experiment included 92 questions: 64 from MultiFloorQA-Main and 28 from MultiFloorQA-Hard.
The results are presented below:
We find that the human participants perform in the upper-middle range on MultiFloorQA-Main, while their performance falls toward the lower end on MultiFloorQA-Hard. This may indicate that humans and models exhibit different generalization capabilities as task difficulty increases.
These results provide a human baseline reference for MultiFloorQA, helping us better understand task difficulty and performance differences between models and human participants.