acceptodds
Withdrawn from ICLR 2027

SpatialAct: Probing the Gap Between Static Spatial Reasoning and Closed-Loop Action of VLM agents in 3D Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Humans exhibit remarkable spatial intelligence, enabling them to perceive spatial structure, reason about their surroundings, and adapt effectively as environments change. As frontier models such as GPT-6 Astra continue to advance, an important question arises as to how far current vision-language models (VLMs) have progressed from classical spatial understanding toward more complex interactive spatial intelligence. To study this transition, we introduce , a simulator-grounded benchmark for evaluating VLMs from static spatial reasoning to closed-loop action in 3D scenes. SpatialAct adopts Multi-turn Interactive Refinement as its primary task and connects foundational spatial abilities and single-step correction within a hierarchical evaluation framework. Experiments across leading proprietary and open-source VLMs reveal a substantial performance gap as tasks progress from isolated spatial judgments to complex multi-turn interaction. While current VLMs perform relatively well on static spatial tasks, their performance drops markedly in closed-loop refinement, where even GPT-6 Astra achieves a Repair Rate of 0.681 and a Scene Success Rate of 0.332, compared with 0.911 and 0.763 for human participants. We further investigate the sources of this cross-task capability gap by decomposing the refinement loop into spatial reasoning, corrective action, and spatial state tracking, revealing distinct limitations that emerge during interaction. Finally, we explore external support for bridging this gap and find that feedback and memory can partially compensate for possible limitations in current VLMs' internal spatial representations, particularly in anticipating action consequences and updating spatial states over time.

Primary area:
datasets and benchmarks
Keywords:
Spatial Intelligence, Vision-Language Model, Reasoning-to-Action

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_pTcMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpatialAct, a simulator-grounded benchmark designed to evaluate the gap between static spatial reasoning and closed-loop interactive refinement in 3D environments. The benchmark adopts a hierarchical framework comprising basic spatial abilities, single-step error detection and correction, and multi-turn interactive refinement. Through evaluations of multiple vision-language models and comparisons with human performance, the authors demonstrate substantial performance degradation in complex interactive tasks. The paper further analyzes limitations in spatial reasoning, corrective action generation, and state tracking through trajectory-level diagnostics and controlled interventions. Additional experiments investigate how simulator feedback and external memory can improve interactive refinement performance.

Critical strengths

(1) Structured evaluation framework. SpatialAct organizes foundational spatial abilities, single-step correction, and multi-turn refinement within a hierarchical evaluation framework. This design provides a useful basis for assessing VLM performance across different spatial tasks and examining the challenges associated with interactive scene refinement.

(2) Informative empirical analysis. The extensive model evaluations and trajectory-level diagnostics reveal substantial limitations in interactive scene refinement. By examining error recognition, corrective action effectiveness, and state tracking, the paper identifies important failure patterns, particularly the gap between correctly diagnosing spatial errors and successfully resolving them. These findings provide useful empirical evidence for understanding the challenges faced by current VLM agents.

Critical weaknesses

(1) Limited novelty beyond existing interactive evaluation frameworks. Although SpatialAct differs from prior goal-driven 3D editing benchmarks in its task formulation, it remains unclear whether these distinctions lead to sufficiently novel scientific insights. The paper would benefit from a more convincing demonstration of its contributions beyond existing interactive 3D evaluation settings.

(2) Insufficient experimental evidence and reporting inconsistencies. Several key intervention experiments rely on relatively small samples, including 20 scenes for simulator feedback and 15 for memory evaluation, limiting the robustness and generalizability of the findings. Moreover, differences in task complexity, output requirements, and interaction protocols complicate the attribution of performance degradation to specific spatial capabilities. Although external feedback and memory improve performance, these results do not conclusively establish deficiencies in internal spatial world modeling as the primary causes of failure. Additionally, Appendix Table 6 reports identical action counts and MAU metrics for DeepSeek-V4.1-Flash and Qwen3.8-27B, while the reported Scene MAU for DeepSeek-V4.1-Flash is inconsistent with Table 2. These discrepancies require clarification and raise concerns about the accuracy of the reported results.

Questions for authors
  1. Novelty clarification: What specific scientific insights does SpatialAct provide beyond existing interactive 3D editing benchmarks, particularly BlenderBench? A more explicit comparison would help clarify the significance and distinct contributions of the proposed framework.

  2. Statistical robustness and generalizability: How robust are the improvements observed in the simulator-feedback and memory experiments, given their limited sample sizes? Furthermore, several diagnostic analyses focus primarily on Gemini-3.1 Pro. To what extent do these findings generalize across different model families and capability levels? Could the authors provide uncertainty estimates or additional supporting evidence?

  3. Causal interpretation: How can the authors distinguish limitations in internal spatial world modeling from other factors, such as task complexity, action generation difficulty, or the additional structured information provided by external feedback and memory? More controlled comparisons would help substantiate the proposed explanations.

  4. Reporting consistency: Appendix Table 6 reports identical action counts and MAU metrics for DeepSeek-V4.1-Flash and Qwen3.8-27B, while the reported Scene MAU for DeepSeek-V4.1-Flash differs from Table 2. Could the authors clarify these discrepancies and verify the correctness of the reported results?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer pTcM (Part 1/3)

We are delighted that the reviewer recognizes the value of SpatialAct's hierarchical evaluation framework and informative empirical analysis. We sincerely thank the reviewer for the constructive feedback, which have helped us further strengthen our work. In the revised manuscript, we have made corresponding updates and highlighted all changes in blue for your convenience. We hope the following responses address the reviewer's remaining concerns:

For W1 & Q1: Scientific Contributions Beyond Existing 3D Editing Benchmarks

We thank the reviewer for raising this important question regarding the scientific novelty of SpatialAct. We agree that BlenderBench is a closely related benchmark, and we have already included a direct comparison in Table 1. However, their primary objectives and the scientific questions they investigate are different.

Specifically, BlenderBench focuses on target-driven 3D scene reconstruction and editing, evaluating how agents iteratively modify scenes toward predefined target configurations. In contrast, SpatialAct starts from existing erroneous scenes without providing a target layout or prescribed repair trajectory. Agents must autonomously identify spatial violations, determine appropriate corrective actions, and reassess the evolving scene through successive interactions. Different valid final layouts and repair trajectories are accepted as long as the predefined geometric constraints are satisfied.

More importantly, SpatialAct goes beyond evaluating whether agents can complete interactive editing tasks by investigating three fundamental questions: what capability gap emerges, why it arises, and how it can be bridged. We provide four specific scientific insights:

  1. Revealing the gap between static spatial competence and closed-loop refinement. Through a hierarchical evaluation covering basic spatial abilities, single-step correction, and multi-turn refinement, we reveal that strong performance on isolated spatial tasks does not necessarily translate into reliable interactive behavior. Even GPT-6 Astra achieves only a 0.332 Scene Success Rate, compared with 0.763 for human participants.
  2. Diagnosing the gap between correct diagnosis and effective repair. Our trajectory-level analysis reveals that only 41.58% of repair attempts following a correct diagnosis successfully resolve the target error. Moreover, 61.96% of successful target repairs occur in rounds that introduce new errors. This demonstrates that identifying spatial violations and executing globally safe corrections are fundamentally different challenges.
  3. Identifying limitations in persistent spatial state tracking. We further show that models struggle to maintain an accurate understanding of evolving scenes across interaction rounds. For Gemini-3.1 Pro, only 37.80% of resolved errors are recognized as fixed in the subsequent round, while only 48.78% of newly introduced errors are detected. Premature stopping occurs in 54.61% of measurable trajectories, even when errors remain.
  4. Exploring mechanisms for bridging the capability gap. Beyond diagnosing model failures, we investigate how external support can improve closed-loop refinement. Through simulator-guided lookahead, goal-completion feedback, and different memory designs, we show that external consequence feedback and spatial representations can partially compensate for limitations in action consequence modeling and persistent state tracking.

Therefore, the contribution of SpatialAct is not simply another multi-turn 3D editing benchmark. Rather, it provides a systematic framework that goes beyond evaluating task completion to reveal capability gaps, diagnose their underlying causes, and investigate potential mechanisms for bridging them. Through this progression from revealing, to diagnosing, and finally to bridging, SpatialAct provides new empirical insights into the requirements and limitations of VLM agents for closed-loop spatial intelligence.

AuthorsOpenReview ↗

Author Response to Reviewer pTcM (Part 2/3)

For W2 & Q2: Expanded Evaluations and Cross-Model Generalizability

We thank the reviewer for raising these important concerns regarding the statistical robustness and generalizability of our findings. In the revised manuscript, we have substantially expanded the experimental coverage in terms of both scene numbers and model diversity to provide stronger supporting evidence.

First, regarding scene coverage, we have substantially expanded the two main intervention studies. Specifically, the Simulator-Guided Lookahead evaluation has been expanded from 20 to 100 scenes, covering three representative VLMs: Qwen3.8-27B, Gemini-3.1 Pro, and GPT-6 Astra. The results remain consistent with our original findings, with simulator feedback improving Repair Rate by 16.4, 13.8, and 7.1 percentage points, respectively. Similarly, the main Memory Design evaluation has been expanded from 15 to 100 scenes, covering Qwen3.8-27B and Gemini-3.1 Pro. Both models benefit from externally maintained spatial maps, with Current Map achieving the highest Repair Rate for both models (69.61% and 29.17%, respectively). These results confirm that the observed improvements persist under substantially larger evaluation subsets.

Second, beyond expanding scene coverage, we have further broadened model coverage to examine the generalizability of these findings. For ​Simulator-Guided Lookahead, the extended evaluation in Appendix A.9 further includes GPT-5.4 and GPT-5.4 mini, bringing the total to five models. Simulator feedback consistently improves Repair Rate across all five models. For ​Memory Design​, the extended evaluation in Appendix A.12 examines six memory configurations using Gemini-3.1 Pro and GPT-6 Astra on an additional 15-scene subset. We have also expanded other diagnostic experiments to cover models with different capability levels. Specifically, Task Structure Analysis now includes GPT-6 Astra in the main text and Qwen3.8-27B in Appendix A.6, in addition to Gemini-3.1 Pro. Goal Completion Feedback has been extended from two to three models. Furthermore, building upon the Diagnosing the Sources of Failure experiments presented in Section 3.2 (Figure 3b), we have extended the same diagnostic interventions to Qwen3.8-27B and GPT-6 Astra, with the cross-model results reported in Appendix A.13.2. Despite differences in the magnitude of performance changes, these models exhibit consistent trends across the diagnostic and intervention experiments.

AuthorsOpenReview ↗

Author Response to Reviewer pTcM (Part 3/3)

For Q3: Distinguishing Task Demands, Internal Capabilities, and External Support

We thank the reviewer for raising this important question. We would like to clarify that the factors mentioned by the reviewer operate at different conceptual levels rather than representing mutually exclusive explanations.

Specifically, task complexity and action generation difficulty characterize the demands imposed by the benchmark, whereas internal spatial world modeling refers to an intrinsic capability of the model to anticipate action-induced spatial changes and maintain a coherent representation of the evolving environment. In contrast, structured information from external feedback and memory represents additional system-level support intended to compensate for potential limitations in the model's internal capabilities.

Our motivation follows this distinction. As VLMs become increasingly capable, we expect them to handle more complex spatial tasks beyond isolated spatial judgments. Accordingly, SpatialAct progressively increases task demands, from basic spatial abilities and single-step correction to open-ended, multi-turn refinement. Our task-structure analysis in Figure 3a explicitly examines how error disclosure, error count, and action output space affect correction difficulty. These factors define the challenges posed by the task, rather than directly explaining the underlying limitations of the models.

We then investigate why models struggle under these demands. Our trajectory-level analysis shows that even after correctly diagnosing a spatial error, only 41.58% of repair attempts successfully resolve the target error. Furthermore, action-composition analysis reveals that individually beneficial actions may introduce additional errors when combined. These findings suggest that the difficulty extends beyond spatial diagnosis and action generation, involving the ability to anticipate action consequences and maintain an accurate understanding of evolving spatial states.

Finally, we explore whether external support can mitigate these potential limitations through simulator-guided lookahead, goal-completion feedback, and different memory designs. These mechanisms provide complementary support for anticipating action consequences, monitoring task progress, and maintaining spatial states across interactions. The improvements demonstrate that external structured information can partially compensate for limitations in VLMs' internal spatial representations and enhance closed-loop refinement.

Together, these experiments support our central interpretation: task complexity and open-ended action generation expose the limitations of current VLMs, while the observed failures suggest that internal spatial world modeling, particularly action consequence prediction and persistent state tracking, remains an important bottleneck. External feedback and memory can partially compensate for these limitations and improve interactive refinement. We do not claim that internal world modeling is the sole cause of the performance gap; rather, our experiments provide converging behavioral evidence for its importance in reliable closed-loop spatial interaction.

For Q4: MAU Reporting Inconsistencies

We thank the reviewer for carefully identifying these inconsistencies. They were caused by a transcription error during table preparation, rather than different evaluation configurations. Specifically, the five numerical entries for Qwen3.8-27B (atomic action count, actionable-round count, and the three MAU values) were inadvertently copied into the DeepSeek-V4.1-Flash row. We apologize for this mistake and clarify the corrections below.

The correct Scene MAU for DeepSeek-V4.1-Flash is ​0.0843​, which rounds to ​0.084 in Table 2​. The previously reported value of 0.0997 actually belongs to Qwen3.8-27B. Therefore, the results in Table 2 are correct and remain unchanged. After rechecking the underlying evaluation results, the corrected entries in the Appendix MAU table are:

ModelAtomic ActionsActionable RoundsAtomic MAURound MAUScene MAU
DeepSeek-V4.1-Flash5,6662,3210.01130.02340.0843
Qwen3.8-27B2,2341,2710.02770.04450.0997

Although both models were evaluated on the same scenes under the same evaluation protocol, their action counts and MAU values differ because they generated different refinement trajectories. We apologize again for the confusion. We have corrected the duplicated entries and verified the consistency of the reported results across the main results table and the Appendix MAU table.

Reviewer_KrBvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers whether VLMs that perform well on static spatial questions can also reliably repair 3D scenes. It introduces SpatialAct benchmark, where models identify geometric errors, generate corrective actions, and inspect the resulting scenes. The experiments compare several models and investigate whether feedback and memory help.

Critical strengths
  1. I like the repair setting because models must identify what is wrong and reassess the consequences of their own actions. Allowing different valid final layouts also makes the task less dependent on a particular reference solution.
  2. The paper makes a useful effort to understand failures rather than simply compare scores. The oracle-diagnosis experiments, simulator lookahead, and trajectory analysis are strengths.
Critical weaknesses
  1. I had difficulty reconciling some results. Table 6 gives DeepSeek and Qwen3.8 identical values across all five quantities, while DeepSeek’s Scene MAU differs between Tables 2 and 6. I also could not understand why Table 5 contains only 46 and 110 non-empty responses for two models across 555 cases.
  2. I am not convinced that the interventions isolate memory or world-modeling limitations. The task levels change several difficulty factors together, and M1 adds geometric information as well as temporal context. The small intervention subsets make it difficult to judge how consistent these findings are.
Questions for authors
  1. Could you clarify the duplicated Table 6 values and the Scene MAU discrepancy? Are these reporting errors only?
  2. What explains the low non-empty-response counts? Were the other outputs empty action lists, truncated responses, refusals, or something else?
  3. How much of the observed gap comes from free-form action generation rather than interaction? Similarly, how much of M1’s benefit comes from additional geometry rather than memory?
  4. Could you provide uncertainty estimates for the main interventions and explain how the spatial-state judge was validated?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer KrBv (Part 1/2)

We are delighted that the reviewer’s recognition of our efforts to investigate model failures beyond simply comparing performance scores. We sincerely thank the reviewer for the careful questions, which have helped us identify a reporting error and further clarify our experimental results. In the revised manuscript, we have corrected the relevant table entries and added further clarifications, with all changes highlighted in blue for your convenience. We hope the following responses address the reviewer’s concerns:

For W1 & Q1: MAU Reporting Inconsistencies

We thank the reviewer for carefully identifying these inconsistencies. They were caused by a transcription error during table preparation, rather than different evaluation configurations. Specifically, the five numerical entries for Qwen3.8-27B (atomic action count, actionable-round count, and the three MAU values) were inadvertently copied into the DeepSeek-V4.1-Flash row. We apologize for this mistake and clarify the corrections below.

The correct Scene MAU for DeepSeek-V4.1-Flash is ​0.0843, which rounds to ​0.084 in Table 2​. The previously reported value of 0.0997 actually belongs to Qwen3.8-27B. Therefore, the results in Table 2 are correct and remain unchanged.

The corrected entries in the MAU table are:

ModelAtomic ActionsActionable RoundsAtomic MAURound MAUScene MAU
DeepSeek-V4.1-Flash5,6662,3210.01130.02340.0843
Qwen3.8-27B2,2341,2710.02770.04450.0997

Although both models were evaluated on the same scenes under the same evaluation protocol, their action counts, actionable-round counts, and MAU values differ because they generated different refinement trajectories. The duplicated entries were solely a copy-and-paste error and did not reflect different evaluation configurations. We have corrected the table and verified the consistency of the reported results across Tables 2 and 6.

For W1 & Q2: Non-empty Response Statistics and Truncation

We thank the reviewer for raising this question. The low number of non-empty responses is primarily caused by excessive reasoning generation and response truncation, rather than empty action lists or refusals.

Specifically, the total numbers of recorded response rounds were 596 for Qwen3.6-27B and 649 for Kimi-K2.5. Both models frequently generated excessively long and repetitive reasoning traces, repeatedly revisiting previous considerations without reaching a final decision. Under our unified inference setting, each response was subject to a maximum completion length of 8,096 tokens. Consequently, many responses reached this limit before producing a final answer, resulting in truncated outputs without a complete action response. The response statistics are summarized below:

ModelEvaluated CasesRecorded Response RoundsTruncated ResponsesNon-empty ResponsesAbnormal Non-empty Responses
Qwen3.6-27B555596550464
Kimi-K2.555564953911010

Importantly, these truncated outputs are distinct from valid empty action arrays ([]), which explicitly indicate that the model has decided to stop refinement. They are also not refusal responses. Instead, the models failed to produce a complete final answer before reaching the generation limit.

The abnormal non-empty responses reported in Table 5 are a subset of the completed non-empty outputs. Specifically, 4 out of 46 responses (8.696%) for Qwen3.6-27B and 10 out of 110 responses (9.091%) for Kimi-K2.5 were identified as abnormal according to our predefined output-format validation criteria.

AuthorsOpenReview ↗

Author Response to Reviewer KrBv (Part 2/2)

For W2 & Q3: Expanded Interaction and Memory Analysis

We thank the reviewer for these insightful questions. To better disentangle the effects of interaction and memory, we have added two complementary analysis to the revised manuscript.

First, regarding the contribution of multi-turn interaction beyond free-form action generation, we have added multi-turn results to Figure 3a under the same correction configurations. For undisclosed double-error cases, allowing multiple correction attempts improves Gemini-3.1 Pro's success rate from 83.3% to 100.0% with 4-option outputs and from 26.7% to 50.0% with free-form outputs. These results demonstrate that iterative interaction provides additional opportunities for error correction and recovery beyond single-turn action generation under the simple setting. However, ​the benefits of interaction come with additional challenges in more complex tasks​, which requires models to continuously track how previous actions have changed the scene, distinguish resolved errors from newly introduced ones, and maintain a coherent spatial state across rounds. Our state-tracking analysis in Section 3.3 further reveals these difficulties.

Second, regarding the contribution of geometric information versus temporal context in M1 (Temporal Map), we have introduced a new Current Map condition in Section 3.5 (Figure 6c). Both Current Map and Temporal Map serve as externally maintained spatial memory representations, reconstructed from Blender scene states and provided to the model in subsequent interaction rounds. Current Map retains the same geometric representation as Temporal Map but removes historical information, including previous-state outlines, translation arrows, and movement trajectories. This allows us to compare the contribution of current geometric information with that of additional temporal context. Furthermore, we have expanded the main memory evaluation to 100 scenes using both Qwen3.8-27B and Gemini-3.1 Pro. Current Map achieves the highest Repair Rates for both models, reaching 69.61% and 29.17%, respectively. These results suggest that explicit current-state geometry provides substantial benefits, while additional temporal information does not necessarily yield further improvements.

For W2 & Q4: Expanded Evaluations and Validation of Spatial-State Assessment

We thank the reviewer for raising these important concerns regarding the robustness of our intervention results and the reliability of our spatial-state evaluation. To address them, we have substantially expanded our experiments in terms of both scene coverage and model diversity.

First, we have extended several experiments to cover three representative VLMs with different capability levels, namely Qwen3.8-27B (relatively weak), Gemini-3.1 Pro (intermediate), and GPT-6 Astra (strong), as introduced in Section 3.2.

Specifically, for ​Task Structure Analysis​, we have expanded the main-text evaluation to include GPT-6 Astra alongside Gemini-3.1 Pro, while retaining the extended evaluation of Qwen3.8-27B in Appendix A.6 (Figure 10). All three models exhibit consistent trends that correction becomes more challenging with undisclosed errors, multiple errors, and larger output spaces. We also extended the cross-model Failure diagnosis in Appendix A.13.2 to all three models, observing consistent effects of perception, reasoning, and memory interventions across different model capabilities. For ​Simulator-Guided Lookahead​, we expanded the evaluation to 100 scenes and all three representative models. Simulator feedback consistently improves Repair Rate, with gains of 16.4, 13.8, and 7.1 percentage points for Qwen3.8-27B, Gemini-3.1 Pro, and GPT-6 Astra, respectively. For ​Goal Completion Feedback, we also extended the evaluation to all three models, observing improvements in both Repair Rate and Scene Success Rate across all of them. Notably, GPT-6 Astra's Scene Success Rate increases from 30.0% to 75.0%. We also expanded the Memory Design evaluation. The main comparison now covers 100 scenes using Qwen3.8-27B and Gemini-3.1 Pro, while the extended evaluation in Appendix A.13 examines six memory configurations using Gemini-3.1 Pro and GPT-6 Astra on an additional 15-scene subset.

Regarding the reliability of spatial-state assessment, we have supplemented the original GPT-5.6-sol evaluation with two additional evaluators. First, 10 human annotators cross-rated 420 interaction rounds, providing independent human assessments. Second, we introduced Qwen3.8-27B as an independent VLM evaluator in Appendix A.12 (Figure 13). Despite differences in absolute scores, human ratings and both VLM-based evaluators reveal consistent trends that spatial-state tracking deteriorates during early interaction rounds and subsequently fluctuates at lower levels, while Gemini-3.1 Pro consistently receives higher scores than GLM-5V-Turbo.

Reviewer_CVbFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a simulator-grounded benchmark SpatialAct to evaluate VLMs across spatial reasoning, single-step error detection and correction, and multi-turn 3D scene refinement. The benchmark contains 2,600 foundational questions, 1,200 single-step questions, and 555 interactive refinement cases across indoor and urban scenes. In the interactive task, models identify geometric violations and generate movement, rotation, and scaling operations, receiving updated scene renderings after execution.

Ten general-purpose VLMs and three spatially specialized models are reported with substantial difficulties in multi-turn refinement, including unsuccessful repairs following correct diagnoses, newly introduced errors, and premature stopping.

Critical strengths
  1. The benchmark extends from single-turn tasks to multi-turn interactions without requiring additional layout operations. Actions directly modify the scene, and the resulting scene is used for subsequent decisions. This provides a useful setting for investigating whether agents can make cumulative progress while avoiding the introduction of new violations.

  2. The benchmark provides rich analyses of model behavior and performance. Figure 4(a) distinguishes among error recognition, repair following correct diagnosis, and subsequent state recognition. For Gemini-3.1 Pro, only 41.58% of repair attempts following a correct diagnosis successfully resolve the target error, while 61.96% of successful target repairs occur in rounds that introduce new errors. The fixed-cohort analysis in Appendix A.9 further examines whether the observed temporal trends are merely attributable to changes in the trajectory populations.

  3. The benchmark evaluates 10 general-purpose VLMs and 3 specialized spatial VLMs, providing relatively broad model coverage.

Critical weaknesses
  1. Some experiments involve very few scenes. For example, the memory design study uses only 15 scenes. This small sample size may be insufficient to establish the robustness of the findings.

  2. The paper reports MAU at the atomic, round, and scene levels, but the link between individual repair actions and overall loop performance remains unclear. Could the authors isolate the individual actions within multi-action rounds and evaluate them separately, then compare their individual and joint effects on the final scene? This would help reveal how individual repair capabilities translate into closed-loop performance, and whether combining actions introduces additional difficulties.

  3. Appendix A.1.2 primarily groups tasks by the number of initial errors (1–3, 4–6, and 7+), while Section 2.1 mentions injecting errors with different correction steps. However, the relationship between the initial error count and the actual number of steps required for successful repair is not sufficiently clarified. Multiple errors may be resolved by a single action, whereas even a single error may require several dependent actions. The paper does not discuss these issues. Is there a specific reason for not considering them?

Questions for authors
  1. Does the Move action support only horizontal (planar) translation, with no vertical movement? If so, can a combination of Move/ Rotate/Scale achieve vertical displacement, or is this capability outside the current action space?

  2. In the paper, “Starting from round 2, GPT-5.6-sol is used as an external judge to assign a holistic score between 0 and 100 based on the model response at the target round.” I’m not very familiar with this field. How do the authors ensure the reliability and consistency of these scores from GPT-5.6-sol? Has the judge's evaluation been validated against human judgments, for example, through inter-rater agreement or correlation analysis?

  3. Are these transcription errors or results from different evaluation configurations? Please reconcile the following: (a) DeepSeek-V4.1-Flash has Scene MAU 0.084 in Table 2 but 0.0997 in Table 6. (b) In Table 6, DeepSeek-V4.1-Flash and Qwen3.8-27B have identical action counts, actionable-round counts, and all three MAU values.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer CVbF (Part 1/3)

We are delighted that the reviewer recognizes that our benchmark provides rich analysis of model behavior and performance. We also sincerely thank the reviewer for the constructive feedback, which has helped us further strengthen our work. In the revised manuscript, we have made corresponding updates and highlighted all changes in blue for your convenience. We hope the following additional experiments and clarifications address the reviewer's remaining concerns:

For W1: Finding Robustness Across More Scenes and Models

We thank the reviewer for raising this concern about the robustness of our experimental findings. To address it, ​we have expanded the primary Memory Design and Simulator-Guided Lookahead experiments from 15/20 to 100 scenes and extended the evaluations to additional models, including Qwen3.8-27B and GPT-6 Astra.

​For Memory Design (Section 3.5, Figure 6c)​, we now evaluate Qwen3.8-27B and Gemini-3.1 Pro on 100 scenes. The results remain consistent with our original findings: ​externally maintained spatial representations improve refinement performance, while self-updated cognitive maps perform worse than the default setting​. In particular, the external current-state map achieves the highest Repair Rate of 69.61% for Gemini-3.1 Pro and 29.17% for Qwen3.8-27B. We also extend the 15-scene evaluation to GPT-6 Astra and additional memory configurations in Appendix A.12, further supporting these observations.

​For Simulator-Guided Lookahead (Section 3.4, Figure 6a)​, we evaluate Qwen3.8-27B, Gemini-3.1 Pro, and GPT-6 Astra on 100 scenes. Simulator feedback consistently improves Repair Rate across all three models, with gains of ​16.4, 13.8, and 7.1 percentage points, respectively. These results reinforce our original conclusion that external action-consequence feedback can improve closed-loop refinement.

Overall, ​the expanded evaluations preserve our original conclusions across larger scene sets and different model capabilities​, providing stronger empirical support for the effectiveness of external feedback and spatial memory in interactive refinement.

For W2: Action Composition and Closed-Loop Outcomes

We thank the reviewer for this valuable suggestion. To address this concern, we have added a new analysis, ​“Action Composition and Closed-Loop Outcomes,” in Section 3.4 (Figure 5)​, examining how individual repair actions interact within multi-action rounds and influence final refinement outcomes.

Specifically, we sample 250 multi-action rounds from Gemini-3.1 Pro trajectories and exhaustively evaluate 5,950 nonempty action subsets from their respective pre-round states. We find that the original full action set produces greater geometric severity than the oracle-best singleton in 63.6% of rounds and the best nonempty proper subset in 74.8% of rounds. Furthermore, 12.0% of rounds introduce new errors that are absent from all individual action outcomes, while 15.9% of individually beneficial actions exhibit negative leave-one-out utility when combined with other actions. These results reveal that combining individually beneficial actions does not necessarily yield better overall corrections.

To further examine the impact on closed-loop performance, we conduct downstream refinement experiments on 20 stratified cases. The mean final Scene MAU reaches 0.438 for singleton interventions and 0.391 for immediate-best action subsets, compared with 0.323 for the original full action sets. Overall, these findings demonstrate that individual action effectiveness does not necessarily translate into better joint or final refinement outcomes. Action composition can introduce additional geometric conflicts and influence subsequent interactions, highlighting the importance of evaluating both local action quality and closed-loop performance.

AuthorsOpenReview ↗

Author Response to Reviewer CVbF (Part 2/3)

For W3: Initial Error Count and Repair Complexity

We thank the reviewer for pointing out this ambiguity. We would like to clarify that our intention was to characterize the initial complexity of each scene, rather than specify the actual number of actions required for successful repair.

During benchmark construction, we introduce spatial errors through controlled geometric perturbations, with each injection targeting one predefined error. The resulting scenes are then programmatically validated using the predefined geometric predicates. Therefore, the groups in Appendix A.1.2 (1–3, 4–6, and 7+ errors) refer to the number of validated initial errors, rather than the number of required corrective actions. We agree that these two quantities are not necessarily equivalent. SpatialAct is an open-ended, multi-path task without a unique repair trajectory that correcting one error may use several dependent actions. Furthermore, models may introduce new errors or need to revise previous actions during interaction. Consequently, the actual number of repair steps depends on the selected trajectory and the model's behavior, rather than being a fixed property determined by the initial error count.

To avoid this ambiguity, we have revised the description in Section 2.1, replacing "correction steps" with "the number of initial errors" to consistently distinguish initial scene complexity from the actual repair process.

For Q1: Scope of the Action Space

We thank the reviewer for raising this point. The current action space of SpatialAct is designed for layout-level refinement rather than unrestricted 6-DoF manipulation. This design is consistent with prior 3D layout and object-placement works that PlaceIt3D [1] assumes objects have fixed vertical orientations and are placed on horizontal support surfaces with yaw-only rotation, while ScanEdit [2] initializes object heights according to their supporting surfaces and restricts its final optimization to planar translation and rotation around the vertical axis.

We acknowledge that vertical displacement is important for more general 3D manipulation tasks. We thank the reviewer for highlighting this limitation and will extend the action space to support vertical translation in future work.

[1] Ahmed Abdelreheem, et al. "PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes." ICCV 2025

[2] Mohamed El Amine Boudjoghra, et al. "ScanEdit: Hierarchically-Guided Functional 3D Scan Editing." ICCV 2025

AuthorsOpenReview ↗

Author Response to Reviewer CVbF (Part 3/3)

For Q2: Reliability and Consistency of LLM-based Evaluation

We thank the reviewer for raising this important concern. We agree that relying solely on an LLM-based evaluator may introduce potential evaluation bias. To further validate the reliability of our spatial state tracking assessment, we have supplemented the original GPT-5.6-sol evaluation with both human ratings and an independent VLM evaluator.

Specifically, as described in Section 3.5 (Figure 4b), we collected human ratings from ​10 annotators​, who cross-rated a subset of ​420 interaction rounds​. The ratings were averaged over early, middle, and late interaction stages. The human evaluations support the same overall observation that spatial state tracking deteriorates as interaction progresses, with Gemini-3.1 Pro generally maintaining higher scores than GLM-5V-Turbo.

Furthermore, in ​Appendix A.11 (Figure 13)​, we introduce Qwen3.8-27B as an independent evaluator to assess the same spatial state tracking dimension. Despite differences in absolute scores, both GPT-5.6-sol and Qwen3.8-27B reveal consistent trends that scores decline during early interaction rounds and subsequently fluctuate at lower levels, while consistently assigning higher scores to Gemini-3.1 Pro than GLM-5V-Turbo.

Together, the human evaluations and cross-evaluator analysis provide complementary validation of our findings, indicating that the observed degradation in spatial state tracking is not specific to a single LLM judge.

For Q3: MAU Table Inconsistencies

We thank the reviewer for carefully identifying these inconsistencies. They were caused by a transcription error during table preparation, rather than different evaluation configurations. Specifically, the five numerical entries for Qwen3.8-27B (atomic action count, actionable-round count, and the three MAU values) were inadvertently copied into the DeepSeek-V4.1-Flash row. We apologize for this mistake and clarify the corrections below.

Regarding (a). The correct Scene MAU for DeepSeek-V4.1-Flash is ​0.0843, which rounds to ​0.084 in Table 2​. The previously reported value of 0.0997 actually belongs to Qwen3.8-27B. Therefore, the results in Table 2 are correct and remain unchanged.

Regarding (b). The duplicated entries have been corrected. The accurate results for the two models are:

ModelAtomic ActionsActionable RoundsAtomic MAURound MAUScene MAU
DeepSeek-V4.1-Flash5,6662,3210.01130.02340.0843
Qwen3.8-27B2,2341,2710.02770.04450.0997

Although both models were evaluated on the same scenes under the same evaluation protocol, their action counts, actionable-round counts, and MAU values differ because they generated different refinement trajectories. We apologize again for the confusion. We have corrected the duplicated entries and verified the consistency of the reported results across Tables 2 and 6.

Reviewer_BRFcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpatialAct, a benchmark that focuses on current VLM's spatial reasoning capabilities. The benchmark is built with a series of 3D synthetic scenes, and the VLMs are asked to complete a series of easy-to-hard tasks. The easy tasks are usually single-step questions requiring the model to understand the contents/spatial relations/distances of the layouts in the scenes to answer, and the more difficult, multi-step tasks require the model to propose editing steps to fix the potential issues in the scenes, such as object overlapping. The paper analyzes a series of models, finding that even strong models like GPT-6 Astra may fail especially in the multi-step tasks. The key claim of this paper is that, the proposed SpatialAct and its experiment results demonstrate an important gap between static spatial reasoning and more advanced closed-loop action in a scene.

Critical strengths
  • Beyond proposing a benchmark, the paper digs into a series of details regarding the mechanisms of the spatial reasoning capabilities through experiments.
  • The benchmark covers the latest models and demonstrate even the strongest models might still suffer from the issues studied in the paper.
  • The paper discusses the design rationale behind the benchmark, which helps readers understand the necessity of the proposed tasks.
Critical weaknesses
  1. My leading concern is the analysis of the benchmark seems bring not-surprising results. The key takeaway from the benchmark is that, there is a large gap between spatial reasoning and closed-loop action of VLM agents. My understanding is, this is demonstrated by the large performance difference between single step tasks and multi-step tasks in the benchmark. But this seems natural, because multi-step process accumulates errors - the success rate of a multi-step task is bound, or largely determined by the single step success rate. My point is: how important is the finding of this gap mentioned in this paper?
  2. Also, I am hesitate about what new challenges these tasks introduce compared with existing 2D spatial benchmarks. Many existing benchmarks already probe models’ single-step spatial understanding and multi-step reasoning capabilities. Although the proposed benchmark is built in 3D scenes, the model observes them through top-down and isometric images, and the editing operations focus mainly on layout adjustments. So I am worried if these tasks require 3D understanding abilities of models and cannot be measured by 2D benchmarks. Some examples about this difference that this benchmark introduced in 3D scene would be helpful.
  3. Another minor concern is that some important details regarding how the benchmark is built are missing. For example, this benchmark seems to use blender to build the scenarios, but this is not clearly stated in the benchmark construction section. Also, I think it is important to describe how you introduced errors into the scenes. I learned some details about how errors are defined, such as overlapping areas exceeding certain thresholds, but I don't understand the construction pipeline. Are these errors automatically produced? Do human evaluators check each generated case for false positives and false negatives?
Questions for authors

I understand this is a late emergency review, so I do not expect any new systematic experimental results. However, I am mostly interested in the first two weaknesses mentioned above, since they concern the significance of this paper. Especially for the first question, consider the following thought experiment: an agent attempts a specific multi-step task in your benchmark and fails to complete it. Then, in a “teacher-forcing” setting, at each step, we provide the correct intermediate state and ask the agent to produce the next action. This would give us the success rate at each individual step. Is the observed multi-step success rate approximately the product of these individual step success rates, or is it substantially lower? If the former, the result seems expected, and the paper might need to focus more on identifying the bottlenecks in individual steps. If the latter, it would be more interesting, and the paper might need to focus more on explaining the additional difficulty introduced by multi-step interaction.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer BRFc (Part 1/3)

We are delighted that the reviewer recognizes our effort to investigate the mechanisms underlying VLMs' spatial reasoning capabilities. We sincerely thank the reviewer for the thoughtful and insightful questions. In the revised manuscript, we have made corresponding updates and highlighted all changes in blue for ease of reference. We hope the following additional experiments and clarifications address the reviewer's concerns.

For W1 & Q1: Can Multi-turn Success Be Explained by Multiplying Single-step Success Rates?

We thank the reviewer for this interesting and thought-provoking question, particularly the suggested teacher-forcing experiment. In fact, we considered a similar stepwise analysis when designing SpatialAct, which motivated our investigation of single-turn correction performance in Figure 3a in the paper. We agree that errors can accumulate across steps and naturally reduce the probability of completing a long-horizon task. However, we believe that open-ended interactive refinement cannot be adequately characterized by simply multiplying isolated single-step success rates. Our reasoning is as follows.

  1. Multi-turn refinement is not equivalent to a fixed sequence of independent single-step successes.

The reviewer's intuition is closely related to the chain rule of conditional probability:

= P(C_1)\prod_{t=2}^{T}P(C_t \mid C_1,\ldots,C_{t-1})$$ where $C_t$ denotes successful completion of step $t$. For a task requiring every step in a fixed sequence to succeed, the overall probability is naturally constrained by these conditional probabilities. However, SpatialAct does not require every intermediate action to succeed. Unlike a fixed-step task, interactive refinement allows models to revisit unresolved errors, recover from unsuccessful actions, and revise previous decisions based on updated observations. Consequently, scene-level success is not equivalent to the intersection of successful actions along a predetermined trajectory. To illustrate this distinction, we have added ​multi-turn results to Figure 3a​. For Gemini-3.1 Pro on controlled double-error cases without error disclosure, allowing multiple correction attempts increases Scene Success Rate from 83.3% to 100.0% in the 4-option setting and from 26.7% to 50.0% in the free-form setting. These improvements demonstrate that additional interaction can provide opportunities for recovery rather than merely accumulating failures in the simple setting. Therefore, while the conditional probability identity remains mathematically valid, directly multiplying isolated single-step success rates does not adequately characterize success in our open-ended refinement setting. 2. **The main challenge lies not only in accumulating errors, but also in managing their consequences across interactions.** The multiplicative interpretation assumes a prescribed sequence of successful intermediate states. However, this assumption is restrictive for current VLMs, which already struggle with correcting complex scenes in a single attempt. More importantly, their actions can change the subsequent problem itself. As analyzed in ​Section 3.4 (Translating Correct Diagnosis into Effective Repair)​, even when Gemini-3.1 Pro correctly identifies an error, only 41.58% of the corresponding repair attempts resolve the target error. Furthermore, ​61.96% of successful target repairs introduce at least one new error​, meaning that a locally successful action may still deteriorate the overall scene. This introduces challenges beyond simply executing a sequence of predefined steps. Models must continuously reassess which errors remain, distinguish resolved errors from newly introduced ones, and anticipate how corrective actions affect surrounding spatial relationships. Our analysis further shows that Gemini-3.1 Pro explicitly recognizes only 37.80% of resolved errors as fixed in the next turn and detects only 48.78% of newly introduced errors. Moreover, 54.61% of measurable trajectories terminate prematurely despite remaining errors. These observations motivate our analysis of spatial state tracking as a distinct capability required for reliable closed-loop refinement. Models must not only generate effective individual actions but also maintain and update spatial beliefs as their own actions transform the environment.
AuthorsOpenReview ↗

Author Response to Reviewer BRFc (Part 2/3)

  1. We additionally conducted the suggested teacher-forcing experiment, while recognizing its limitations for open-ended refinement.

Another important characteristic of SpatialAct is that ​there is no unique correct repair trajectory​. Multiple action sequences can lead to a geometrically valid final scene, and an action that differs from a reference trajectory may still be beneficial. This makes conventional teacher forcing difficult to interpret as a direct estimate of autonomous task success.

Nevertheless, following the reviewer's suggestion, we conducted an additional ​Teacher-Forced Repair Diagnostic. Specifically, we evaluated Qwen3.8-27B on 50 scenes using simulator-validated reference repair trajectories of 2–7 steps. Both teacher-forced and autonomous conditions used the same observation, reasoning configuration, and a maximum of one atomic action per round. In the teacher-forced condition, the model received a validated intermediate state. We sampled five responses per state, obtaining ​875 responses across 175 reference states​. A step was considered successful if its action removed at least one existing error without introducing new errors, regardless of whether the action matched the reference trajectory. The results are summarized below:

MetricResult
Teacher-forced step success rate18.97%
Mean teacher-forced trajectory success product2.67%
Autonomous scene success rate0.00%

The teacher-forced trajectory estimate was obtained by multiplying the estimated stepwise success rates along each reference trajectory and averaging the resulting products across scenes. The teacher-forced estimate is based on reference intermediate states, whereas autonomous refinement follows model-generated states and potentially different repair trajectories. Importantly, ​we do not interpret the difference between 2.67% and 0.00% as conclusive evidence​.

  1. The significance of SpatialAct extends beyond observing the performance gap itself.

Finally, we appreciate the reviewer's concern regarding the significance of the observed gap. We agree that, viewed in isolation, lower success rates on long-horizon tasks than on simpler tasks may not be surprising. However, ​the existence of this gap is the starting point of our investigation, rather than its sole contribution​.

In practical embodied applications, we ultimately expect intelligent agents to perform sustained tasks in evolving environments, such as rearranging objects, correcting spatial configurations, and adapting their actions when previous attempts fail. These tasks require more than isolated spatial judgments or single-step corrections.

Motivated by this objective, SpatialAct is organized around three connected questions:

  • WHAT — Revealing the gap. We establish a unified evaluation framework connecting foundational spatial abilities, single-step correction, and multi-turn interactive refinement, quantifying how spatial competence translates into closed-loop performance.
  • WHY — Diagnosing the gap. Rather than attributing the performance decline solely to error accumulation, we decompose interactive refinement into ​spatial reasoning, corrective action, and spatial state tracking​, identifying concrete bottlenecks in error recognition, action effectiveness, consequence assessment, and persistent state maintenance.
  • HOW — Bridging the gap. We further investigate whether external simulator feedback, goal-completion feedback, and spatial memory representations can mitigate these limitations, demonstrating practical directions for improving interactive refinement.

Therefore, ​our central contribution is not simply showing that multi-turn tasks are harder, but systematically characterizing where and why current VLM agents struggle during closed-loop interaction, and identifying mechanisms that can improve their reliability. We believe this perspective provides a more actionable understanding of the transition from static spatial reasoning toward interactive spatial intelligence, with implications for developing more capable long-horizon spatial agents.

AuthorsOpenReview ↗

Author Response to Reviewer BRFc (Part 3/3)

For W2: What 3D Challenges Does SpatialAct Introduce?

We thank the reviewer for this insightful question. We agree that SpatialAct uses top-down and isometric images as visual observations. However, the dimensionality of visual inputs does not necessarily determine the spatial capabilities being evaluated. Although the observations are 2D projections, the underlying environments, spatial constraints, and action consequences are grounded in 3D scenes, requiring models to understand and reason about 3D spatial configurations.

A concrete example is collision detection in indoor scenes. Consider a window and a desk whose rectangular footprints appear to overlap in the top-down view, suggesting a collision. However, the isometric view may reveal that they occupy different heights and do not actually intersect in 3D space. The model must integrate information across viewpoints to distinguish apparent 2D overlap from genuine 3D geometric conflict. Furthermore, SpatialAct requires models to anticipate how moving, rotating, or scaling objects changes their 3D spatial relationships, potentially resolving existing errors while introducing new ones.

This image-based evaluation of 3D spatial intelligence is also consistent with recent benchmarks such as ​3DSRBench [1] and ESI-Bench [2]​, which assess 3D spatial understanding and interaction through visual observations. Unlike conventional static spatial benchmarks, ​SpatialAct further evaluates whether such understanding can support reliable closed-loop refinement​, requiring models to diagnose geometric errors, execute corrective transformations, and continuously update their understanding of the evolving 3D environment.

[1] Ma, Wufei, et al. "3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark." ICCV 2025

[2] Hong, Yining, et al. "ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop. " arXiv 2026

For W3: Scenes and Spatial Errors Construction

We thank the reviewer for pointing out the need for clearer benchmark construction details.

​Regarding Blender​, we confirm that SpatialAct uses Blender as the underlying 3D environment for scene rendering, geometric manipulation, and simulator-based evaluation. Its use was previously mentioned in Section 3.4 and Appendix A.12, and our publicly released code and representative Blender scene files (.blend) also document this implementation. To improve clarity, we have explicitly added this information to ​Section 2.1 (Benchmark Construction).

​Regarding error generation and validation​, spatial errors are introduced automatically into clean scenes. As described in Section 2.1, we first perform programmatic cleaning and manual quality control on the original scenes, examining object dimensions, placement plausibility, and existing abnormalities. We then automatically inject spatial errors, which are validated against predefined geometric criteria. Since these errors are defined by deterministic geometric predicates and explicit thresholds, each generated case can be objectively verified using the underlying 3D geometry. So the injected errors are verified programmatically case by case​.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.