- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SpatialAct, a simulator-grounded benchmark designed to evaluate the gap between static spatial reasoning and closed-loop interactive refinement in 3D environments. The benchmark adopts a hierarchical framework comprising basic spatial abilities, single-step error detection and correction, and multi-turn interactive refinement. Through evaluations of multiple vision-language models and comparisons with human performance, the authors demonstrate substantial performance degradation in complex interactive tasks. The paper further analyzes limitations in spatial reasoning, corrective action generation, and state tracking through trajectory-level diagnostics and controlled interventions. Additional experiments investigate how simulator feedback and external memory can improve interactive refinement performance.
Critical strengths
(1) Structured evaluation framework. SpatialAct organizes foundational spatial abilities, single-step correction, and multi-turn refinement within a hierarchical evaluation framework. This design provides a useful basis for assessing VLM performance across different spatial tasks and examining the challenges associated with interactive scene refinement.
(2) Informative empirical analysis. The extensive model evaluations and trajectory-level diagnostics reveal substantial limitations in interactive scene refinement. By examining error recognition, corrective action effectiveness, and state tracking, the paper identifies important failure patterns, particularly the gap between correctly diagnosing spatial errors and successfully resolving them. These findings provide useful empirical evidence for understanding the challenges faced by current VLM agents.
Critical weaknesses
(1) Limited novelty beyond existing interactive evaluation frameworks. Although SpatialAct differs from prior goal-driven 3D editing benchmarks in its task formulation, it remains unclear whether these distinctions lead to sufficiently novel scientific insights. The paper would benefit from a more convincing demonstration of its contributions beyond existing interactive 3D evaluation settings.
(2) Insufficient experimental evidence and reporting inconsistencies. Several key intervention experiments rely on relatively small samples, including 20 scenes for simulator feedback and 15 for memory evaluation, limiting the robustness and generalizability of the findings. Moreover, differences in task complexity, output requirements, and interaction protocols complicate the attribution of performance degradation to specific spatial capabilities. Although external feedback and memory improve performance, these results do not conclusively establish deficiencies in internal spatial world modeling as the primary causes of failure. Additionally, Appendix Table 6 reports identical action counts and MAU metrics for DeepSeek-V4.1-Flash and Qwen3.8-27B, while the reported Scene MAU for DeepSeek-V4.1-Flash is inconsistent with Table 2. These discrepancies require clarification and raise concerns about the accuracy of the reported results.
Questions for authors
-
Novelty clarification: What specific scientific insights does SpatialAct provide beyond existing interactive 3D editing benchmarks, particularly BlenderBench? A more explicit comparison would help clarify the significance and distinct contributions of the proposed framework.
-
Statistical robustness and generalizability: How robust are the improvements observed in the simulator-feedback and memory experiments, given their limited sample sizes? Furthermore, several diagnostic analyses focus primarily on Gemini-3.1 Pro. To what extent do these findings generalize across different model families and capability levels? Could the authors provide uncertainty estimates or additional supporting evidence?
-
Causal interpretation: How can the authors distinguish limitations in internal spatial world modeling from other factors, such as task complexity, action generation difficulty, or the additional structured information provided by external feedback and memory? More controlled comparisons would help substantiate the proposed explanations.
-
Reporting consistency: Appendix Table 6 reports identical action counts and MAU metrics for DeepSeek-V4.1-Flash and Qwen3.8-27B, while the reported Scene MAU for DeepSeek-V4.1-Flash differs from Table 2. Could the authors clarify these discrepancies and verify the correctness of the reported results?
3 comments
Author Response to Reviewer pTcM (Part 2/3)
For W2 & Q2: Expanded Evaluations and Cross-Model Generalizability
We thank the reviewer for raising these important concerns regarding the statistical robustness and generalizability of our findings. In the revised manuscript, we have substantially expanded the experimental coverage in terms of both scene numbers and model diversity to provide stronger supporting evidence.
First, regarding scene coverage, we have substantially expanded the two main intervention studies. Specifically, the Simulator-Guided Lookahead evaluation has been expanded from 20 to 100 scenes, covering three representative VLMs: Qwen3.8-27B, Gemini-3.1 Pro, and GPT-6 Astra. The results remain consistent with our original findings, with simulator feedback improving Repair Rate by 16.4, 13.8, and 7.1 percentage points, respectively. Similarly, the main Memory Design evaluation has been expanded from 15 to 100 scenes, covering Qwen3.8-27B and Gemini-3.1 Pro. Both models benefit from externally maintained spatial maps, with Current Map achieving the highest Repair Rate for both models (69.61% and 29.17%, respectively). These results confirm that the observed improvements persist under substantially larger evaluation subsets.
Second, beyond expanding scene coverage, we have further broadened model coverage to examine the generalizability of these findings. For Simulator-Guided Lookahead, the extended evaluation in Appendix A.9 further includes GPT-5.4 and GPT-5.4 mini, bringing the total to five models. Simulator feedback consistently improves Repair Rate across all five models. For Memory Design, the extended evaluation in Appendix A.12 examines six memory configurations using Gemini-3.1 Pro and GPT-6 Astra on an additional 15-scene subset. We have also expanded other diagnostic experiments to cover models with different capability levels. Specifically, Task Structure Analysis now includes GPT-6 Astra in the main text and Qwen3.8-27B in Appendix A.6, in addition to Gemini-3.1 Pro. Goal Completion Feedback has been extended from two to three models. Furthermore, building upon the Diagnosing the Sources of Failure experiments presented in Section 3.2 (Figure 3b), we have extended the same diagnostic interventions to Qwen3.8-27B and GPT-6 Astra, with the cross-model results reported in Appendix A.13.2. Despite differences in the magnitude of performance changes, these models exhibit consistent trends across the diagnostic and intervention experiments.
Author Response to Reviewer pTcM (Part 3/3)
For Q3: Distinguishing Task Demands, Internal Capabilities, and External Support
We thank the reviewer for raising this important question. We would like to clarify that the factors mentioned by the reviewer operate at different conceptual levels rather than representing mutually exclusive explanations.
Specifically, task complexity and action generation difficulty characterize the demands imposed by the benchmark, whereas internal spatial world modeling refers to an intrinsic capability of the model to anticipate action-induced spatial changes and maintain a coherent representation of the evolving environment. In contrast, structured information from external feedback and memory represents additional system-level support intended to compensate for potential limitations in the model's internal capabilities.
Our motivation follows this distinction. As VLMs become increasingly capable, we expect them to handle more complex spatial tasks beyond isolated spatial judgments. Accordingly, SpatialAct progressively increases task demands, from basic spatial abilities and single-step correction to open-ended, multi-turn refinement. Our task-structure analysis in Figure 3a explicitly examines how error disclosure, error count, and action output space affect correction difficulty. These factors define the challenges posed by the task, rather than directly explaining the underlying limitations of the models.
We then investigate why models struggle under these demands. Our trajectory-level analysis shows that even after correctly diagnosing a spatial error, only 41.58% of repair attempts successfully resolve the target error. Furthermore, action-composition analysis reveals that individually beneficial actions may introduce additional errors when combined. These findings suggest that the difficulty extends beyond spatial diagnosis and action generation, involving the ability to anticipate action consequences and maintain an accurate understanding of evolving spatial states.
Finally, we explore whether external support can mitigate these potential limitations through simulator-guided lookahead, goal-completion feedback, and different memory designs. These mechanisms provide complementary support for anticipating action consequences, monitoring task progress, and maintaining spatial states across interactions. The improvements demonstrate that external structured information can partially compensate for limitations in VLMs' internal spatial representations and enhance closed-loop refinement.
Together, these experiments support our central interpretation: task complexity and open-ended action generation expose the limitations of current VLMs, while the observed failures suggest that internal spatial world modeling, particularly action consequence prediction and persistent state tracking, remains an important bottleneck. External feedback and memory can partially compensate for these limitations and improve interactive refinement. We do not claim that internal world modeling is the sole cause of the performance gap; rather, our experiments provide converging behavioral evidence for its importance in reliable closed-loop spatial interaction.
For Q4: MAU Reporting Inconsistencies
We thank the reviewer for carefully identifying these inconsistencies. They were caused by a transcription error during table preparation, rather than different evaluation configurations. Specifically, the five numerical entries for Qwen3.8-27B (atomic action count, actionable-round count, and the three MAU values) were inadvertently copied into the DeepSeek-V4.1-Flash row. We apologize for this mistake and clarify the corrections below.
The correct Scene MAU for DeepSeek-V4.1-Flash is 0.0843, which rounds to 0.084 in Table 2. The previously reported value of 0.0997 actually belongs to Qwen3.8-27B. Therefore, the results in Table 2 are correct and remain unchanged. After rechecking the underlying evaluation results, the corrected entries in the Appendix MAU table are:
| Model | Atomic Actions | Actionable Rounds | Atomic MAU | Round MAU | Scene MAU |
|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | 5,666 | 2,321 | 0.0113 | 0.0234 | 0.0843 |
| Qwen3.8-27B | 2,234 | 1,271 | 0.0277 | 0.0445 | 0.0997 |
Although both models were evaluated on the same scenes under the same evaluation protocol, their action counts and MAU values differ because they generated different refinement trajectories. We apologize again for the confusion. We have corrected the duplicated entries and verified the consistency of the reported results across the main results table and the Appendix MAU table.
Author Response to Reviewer pTcM (Part 1/3)
We are delighted that the reviewer recognizes the value of SpatialAct's hierarchical evaluation framework and informative empirical analysis. We sincerely thank the reviewer for the constructive feedback, which have helped us further strengthen our work. In the revised manuscript, we have made corresponding updates and highlighted all changes in blue for your convenience. We hope the following responses address the reviewer's remaining concerns:
We thank the reviewer for raising this important question regarding the scientific novelty of SpatialAct. We agree that BlenderBench is a closely related benchmark, and we have already included a direct comparison in Table 1. However, their primary objectives and the scientific questions they investigate are different.
Specifically, BlenderBench focuses on target-driven 3D scene reconstruction and editing, evaluating how agents iteratively modify scenes toward predefined target configurations. In contrast, SpatialAct starts from existing erroneous scenes without providing a target layout or prescribed repair trajectory. Agents must autonomously identify spatial violations, determine appropriate corrective actions, and reassess the evolving scene through successive interactions. Different valid final layouts and repair trajectories are accepted as long as the predefined geometric constraints are satisfied.
More importantly, SpatialAct goes beyond evaluating whether agents can complete interactive editing tasks by investigating three fundamental questions: what capability gap emerges, why it arises, and how it can be bridged. We provide four specific scientific insights:
Therefore, the contribution of SpatialAct is not simply another multi-turn 3D editing benchmark. Rather, it provides a systematic framework that goes beyond evaluating task completion to reveal capability gaps, diagnose their underlying causes, and investigate potential mechanisms for bridging them. Through this progression from revealing, to diagnosing, and finally to bridging, SpatialAct provides new empirical insights into the requirements and limitations of VLM agents for closed-loop spatial intelligence.