acceptodds
Under review as a conference paper at ICLR 2027

RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

Abstract

Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images. See the https://www.modelscope.cn/studios/san23333/projdemoonline demo and https://anonymous.4open.science/r/RoTA-Bench-7235source code.

Primary area:
datasets and benchmarks
Keywords:
spatial reasoning, vision-language models, interactive benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FNYkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoTA-Bench, which evaluates vision-language models’ interactive spatial control capabilities by requiring them to rotate 3D objects until their 2D projections match target silhouettes. The authors automatically construct 674 tasks and evaluate nine models, two agent configurations, and human participants on a 100-task test set. The experiments reveal a substantial human–model performance gap and investigate model failure modes and the role of interaction history through greedy baselines, action-level analyses, and input ablations.

Critical strengths
  1. The task is well motivated and combines 2D visual observations, 3D rotation actions, and interactive feedback in a clear, controlled setting. It provides a useful benchmark for evaluating current models’ interactive spatial control capabilities.

  2. The automated construction pipeline preserves target silhouettes and guarantees task solvability through reversible initialization. This supports reproducible evaluation and provides a practical basis for expanding the dataset.

  3. The experiments cover multiple models, human participants, greedy baselines, and input ablations. The analyses of rotation direction, angle selection, and interaction history help characterize model behavior and identify specific failure modes.

Critical weaknesses
  1. The interaction is restricted to discrete rotations of a single object about a fixed center. While this helps control experimental variables, it does not cover translation, joint position and orientation adjustment, or multi-object spatial relations. The applicability of the conclusions to more general interactive spatial manipulation therefore remains limited and should be clarified.

  2. Task success is defined by 2D projection IoU, but high IoU does not necessarily imply alignment with the target 3D pose. Table 9 reports final pose error averaged across successful and failed tasks, without separately examining its distribution among successful tasks. It therefore remains unclear how closely successful silhouette alignment corresponds to 3D pose alignment, limiting the interpretation of model spatial reasoning behavior.

  3. The authors generate 3D objects from 2D vector icons to reduce reliance on priors about familiar 3D shapes. However, the source icons and their associated semantics may also have been encountered during pretraining, and the effectiveness of this design choice lacks direct validation. The lower performance on icon-derived objects reported in the appendix does not distinguish the effects of familiarity from additional difficulty introduced by the generated geometries.

  4. Although the automated pipeline generates 674 tasks, the main evaluation uses only 100, the input ablations use only 30, and each object is evaluated from a single initial pose. The paper does not sufficiently justify the choice of evaluation scale.

The paper is good overall, and I would be willing to raise my score if the authors satisfactorily address the questions and concerns above.

Questions for authors

See Weaknesses.

9 comments

W1. The interaction is restricted to discrete rotations of a single object about a fixed center. While this helps control experimental variables, it does not cover translation, joint position and orientation adjustment, or multi-object spatial relations. The applicability of the conclusions to more general interactive spatial manipulation therefore remains limited and should be clarified.

We also considered this limitation regarding more general interactive spatial manipulation and included a similar discussion in Section 5.5 (Limitations and Future Work) of the original manuscript: "Second, each task features only a single target under discrete rotation actions, making the interaction dynamics relatively simplified compared to continuous physical environments... In addition, the benchmark can be extended by introducing continuous rotation control, external occlusions, and multi-object alignment or assembly tasks to further heighten the spatial reasoning challenge." To address your concern more explicitly, we have also added the following clarification to the Motivation and Design Rationale subsection (Section 3.1) in the revised manuscript to clearly define the scope of our conclusions and outline future extensions:

"By focusing on discrete rotations of a single object about a fixed center to control experimental variables, our benchmark does not yet cover translation, joint position and orientation adjustment, or multi-object spatial relations; therefore, the applicability of our conclusions to more general interactive spatial manipulation remains limited, and we leave these extensions to future work."

W2. Task success is defined by 2D projection IoU, but high IoU does not necessarily imply alignment with the target 3D pose. Table 9 reports final pose error averaged across successful and failed tasks, without separately examining its distribution among successful tasks. It therefore remains unclear how closely successful silhouette alignment corresponds to 3D pose alignment, limiting the interpretation of model spatial reasoning behavior.

Thank you for raising this important issue. Indeed, high 2D projection IoU does not guarantee exact alignment with the target 3D pose. We therefore separately analyzed the distribution of final pose errors across all successful runs:

Evaluation groupSuccessful runsMean pose error (°)Maximum (°)Second highest (°)Error ≤10°
Human4004.44179.94179.72394/400
GPT-5.6-Sol1711.95178.405.2316/17
ReAct162.063.413.1816/16
Gemini 3.6 Flash1020.23179.944.479/10
GPT-6-Astra101.743.632.5010/10
GPT-5.6-Terra82.566.143.948/8
GPT-5.6-Luna42.164.121.594/4
Magentic22.073.021.122/2
Total4674.85179.94179.94459/467

All statistics include every successful run in each evaluation group. Human includes all 400 successful runs across four passes of the 100-task test set.

Of the 467 successful runs, 459 (98.3%) have final pose errors no greater than 10°. The mean final pose error across all 467 successful runs is 4.85°, while the remaining 459 runs excluding the eight near-180° cases have a mean error of only 1.81°, indicating that successful silhouette alignment corresponds to highly accurate 3D pose alignment in the vast majority of cases.

For the eight cases with pose errors close to 180°, we compared seven target projections with their own 180°-rotated versions and compared the broken-heart target projection with its left–right reflection (corresponding to an out-of-plane 180° rotation about the vertical axis). Using the 320 × 320 evaluation masks, the resulting IoUs were 0.938 and 0.953 for the two model cases, and 0.953, 0.946, 0.920, 0.946, 0.871, and 0.953 for the six human cases, respectively. Seven comparisons exceed the success threshold of 0.90. Although the broken-heart reflection falls slightly below the threshold, its actual terminal projection achieves an IoU of 0.9005 with the target. Comparing the target and terminal projections shows that their main outer contours and most of their projected areas overlap substantially, with differences concentrated in a small number of patterns. Thus, although the patterns are not fully aligned, the main contours account for most of the area, allowing the overall IoU to reach the success threshold. The model and human examples are provided in the model figure and the human figure, respectively.

Therefore, IoU faithfully reflects 3D pose alignment in the vast majority of cases and does not affect our interpretation of model spatial reasoning behavior. However, these eight cases show that the 0.90 threshold can still admit a small amount of ambiguity. We have added this distributional analysis to Appendix K.9 of the revised manuscript. Future work will use a higher IoU threshold or incorporate 3D pose error for more precise alignment assessment.

W3. The authors generate 3D objects from 2D vector icons to reduce reliance on priors about familiar 3D shapes. However, the source icons and their associated semantics may also have been encountered during pretraining, and the effectiveness of this design choice lacks direct validation. The lower performance on icon-derived objects reported in the appendix does not distinguish the effects of familiarity from additional difficulty introduced by the generated geometries.

Our central conclusion is that sphere-bounded extrusion directly breaks the association between 2D icon semantics and real-world 3D depth priors. Because we cannot independently erase a black-box model's pretrained memory without changing the 3D geometry, weakening familiar 3D priors and introducing synthetic geometric difficulty are naturally coupled aspects of the geometry-generation process.

First, although the source 2D icons and their semantics may appear in pretraining data, the 3D structures generated through sphere-bounded extrusion are unrelated to the corresponding real-world forms. For example, the T-shirt icon in Figure 11 becomes a thick, sphere-truncated solid after extrusion, rather than the thin structure of real clothing. Even if the model recognizes the icon's semantics in 2D, its pretrained 3D priors for real objects cannot transfer to the sphere-bounded 3D structure.

Second, regarding the distinction between familiarity and synthetic geometric difficulty, blocking reliance on familiar 3D shape priors requires disrupting the corresponding 3D structure. This necessarily introduces additional difficulty from the generated geometry. These factors are coupled by construction, so we do not attribute the performance decrease to a single isolated factor.

To further investigate the effects of 3D shape priors and 2D contour sources and address your concerns, we conducted three additional controlled evaluations with GPT-5.6-Sol, each comprising 100 tasks with matched reference-path length distributions. Two sets were constructed by randomly sampling real 3D assets from Objaverse 1.0 and Poly Haven, respectively; the third used 2D contours extracted from Poly Haven objects and reconstructed through sphere-bounded extrusion. Rendering, initialization, and evaluation protocols were identical to those of RoTA-Bench.

Experimental settingSuccess rate
Objaverse 1.0 real 3D assets54%
Poly Haven real 3D assets56%
Poly Haven contours + sphere-bounded extrusion40%
RoTA-Bench test set17%

The results show a clear performance gradient:

  1. Success on both real 3D asset sets (54% and 56%) is substantially higher than on RoTA-Bench (17%), indicating that common objects with real-world 3D structures are indeed much easier for the model.
  2. Keeping Poly Haven's frontal 2D contours and semantics unchanged while replacing its real 3D structures with sphere-bounded extrusion decreases success from 56% to 40%. Thus, even when the frontal 2D image contains exactly the same familiar object semantics and contour, 2D semantic familiarity alone cannot preserve 3D alignment performance once sphere-bounded extrusion disrupts the real 3D depth structure.
  3. Under the same sphere-bounded extrusion process, vector icons are more difficult than real-object contours (17% vs. 40%), indicating that the morphology of the 2D contours also affects task difficulty. Under the same setting, humans achieve 100% success on RoTA-Bench, showing that the generated geometry is fully solvable for humans and that its difficulty remains within an acceptable range.

We have added these explanations and experimental analyses to Appendix K.8 of the revised manuscript.

Reviewer_6so6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an interactive benchmark for evaluating whether VLMs can convert 2D observations into 3D rotation actions. Given a target silhouette, a model iteratively selects from 24 discrete rotations to align a 3D object, built by extruding a 2D icon, until its projection matches the target. Evaluating frontier VLMs, agent frameworks, and human participants on a 100-task test set, the paper reports a large human–model gap, finds that models favor silhouette overlap over true pose progress, and shows that removing interaction history degrades fine-grained angle selection.

Critical strengths
  1. Design. Reversible initialization guarantees solvability, and the contrast between pose-greedy and IoU-greedy oracles shows that the task cannot be solved by greedily matching silhouettes, so it genuinely requires inferring 3D pose.

  2. Human study. Humans are evaluated under exactly the same observations and action space as the models, and the matched-state analysis offers an informative comparison of how humans and models trade off image-space and pose-space progress.

  3. Diagnostics. Decomposing each action into direction and angle choice localizes model failures more precisely than success rate alone.

Critical weaknesses
  1. Objects. The objects are 2D icons symmetrically thickened under a spherical envelope, always aligned to the same frontal target, so they carry little genuine 3D structure. The claim that real 3D assets such as Objaverse would trigger semantic shortcuts is never tested, and the icons themselves are recognizable objects; evaluation on real 3D assets is needed to support claims about 3D spatial reasoning.

  2. Granularity. Every task is generated from, and solvable by, 30° rotations alone, so finer angles are never necessary. Treating the models' preference for 30° steps as a failure of fine-grained control, one of the paper's main findings, is therefore not justified.

  3. Termination. Most model failures end through the stagnation rule, which terminates episodes without a new best IoU and thus penalizes the very strategy of accepting temporary IoU drops that the paper credits to humans. Results without this rule are needed to show that the reported gap is not driven by it.

  4. Analysis. GPT-6-Astra is a model known for substantially stronger 3D understanding, yet underperforms GPT-5.6-Sol, choosing rotation directions barely above chance while selecting angles more accurately than any other model. This counterintuitive result is left entirely unexplained, raising the question of whether the benchmark measures 3D understanding or artifacts of its design and inference setup.

  5. Positioning. Closely related work on VLM viewpoint and orientation reasoning, such as MindCube [1] and 3DSRBench [2], and on object pose alignment, such as Open6DOR [3], is not cited.

[1] Yin, Baiqiao, et al. "Spatial mental modeling from limited views." ICLR. 2026.
[2] Ma, Wufei, et al. "3DSRBench: A comprehensive 3D spatial reasoning benchmark." CVPR. 2025.
[3] Ding, Yufei, et al. "Open6DOR: Benchmarking open-instructed 6-DoF object rearrangement and a VLM-based approach." IROS. 2024.
Questions for authors

Do any reference paths require rotations finer than 30°, given that all initial poses are generated by 30° rotations?

9 comments

W1. Objects. The objects are 2D icons symmetrically thickened under a spherical envelope, always aligned to the same frontal target, so they carry little genuine 3D structure. The claim that real 3D assets such as Objaverse would trigger semantic shortcuts is never tested, and the icons themselves are recognizable objects; evaluation on real 3D assets is needed to support claims about 3D spatial reasoning.

While objects constructed through sphere-bounded extrusion have limited 3D structure compared with real-world meshes, this design intentionally decouples recognizable 2D icon semantics from real-world 3D shape priors, which cannot directly transfer to sphere-truncated volumes. To directly evaluate model performance on real 3D assets and test how familiar 3D structures affect task difficulty, we selected 100 CC0 real 3D assets from each of Objaverse 1.0 and Poly Haven and conducted additional evaluations with GPT-5.6-Sol. We also reconstructed 100 Poly Haven target silhouettes using the same sphere-bounded extrusion process as RoTA-Bench.

Because real-world objects differ in symmetry, visible structure, and projection characteristics, we cannot directly reuse the initial poses and reference paths of the formal test set. For a fair comparison, we used the same initialization procedure and reference-path search as RoTA-Bench to generate 100 tasks for each setting, matching the reference-path length distribution of the RoTA-Bench test set. The renderer, observation settings, and frozen evaluation protocol were also kept unchanged.

Evaluation settingSuccess rate
RoTA-Bench17/100 (17%)
Objaverse real 3D assets54/100 (54%)
Poly Haven real 3D assets56/100 (56%)
Poly Haven contours + sphere-bounded extrusion40/100 (40%)

As shown in the table, GPT-5.6-Sol achieves substantially higher success rates on both real 3D asset sets than on RoTA-Bench (54% and 56% vs. 17%), indicating that these real-object tasks are easier for the model. Together with similar observations in our early exploratory experiments, this leads us to hypothesize that the model may exploit priors about 3D object structure acquired during training as a semantic shortcut; this mechanism requires further validation.

Keeping the Poly Haven target contours and semantics while replacing their real 3D structures with sphere-bounded extrusion decreases success from 56% to 40%. This suggests that the generated geometry introduces additional difficulty and weakens the usefulness of familiar 3D structure priors. Under the same sphere-bounded extrusion process, the icon-derived RoTA-Bench tasks achieve 17%, compared with 40% for Poly Haven contours, indicating that the 2D source contours also affect task difficulty.

Although the source 2D icons and their semantics may appear in pretraining data, the 3D structures generated by sphere-bounded extrusion are unrelated to the corresponding real-world forms. For example, the T-shirt icon in Figure 11 becomes a thick, sphere-truncated solid rather than the thin structure of real clothing. Even if the model recognizes the icon's semantics in 2D, its pretrained 3D priors for real objects cannot directly transfer to the sphere-bounded 3D structure. Our design therefore aims to reduce reliance on familiar 3D object priors and more directly evaluate the model's ability to reason about 3D space from current observations.

W2. Granularity. Every task is generated from, and solvable by, 30° rotations alone, so finer angles are never necessary. Treating the models' preference for 30° steps as a failure of fine-grained control, one of the paper's main findings, is therefore not justified.

We would like to clarify a misunderstanding regarding the practical role of smaller angles and our finding on fine-grained control. First, although reversing the initialization sequence of 30° rotations theoretically recovers the target pose, smaller angles are essential in practice for refinement. Because 3D rotations are order-dependent (noncommutative), once an exploratory trajectory deviates from the original inverse sequence, coarse 30° rotations can easily overshoot a local alignment near the target. In our early 10-task exploratory experiment, humans struggled to align the projections using only 30° rotations, achieving 0% success; allowing smaller angles increased success to 100%. Figure 4 also shows that humans and most models choose smaller angles as IoU increases and pose error decreases, illustrating the practical role of fine-grained refinement.

The third finding in the Abstract and Introduction, as well as Section 5.3, emphasizes that fine-grained control is sensitive to interaction history. In the complete 100-task ablation, removing history leaves the Effective Direction Rate largely unchanged, but increases the proportion of 30° actions from 59.89% to 92.60%, decreases 1° actions from 13.06% to 0.49%, and weakens the IoU–angle correlation from −0.536 to −0.283. These results show that removing history substantially impairs fine-grained refinement, rather than implying that choosing 30° rotations is itself a control failure.

W3. Termination. Most model failures end through the stagnation rule, which terminates episodes without a new best IoU and thus penalizes the very strategy of accepting temporary IoU drops that the paper credits to humans. Results without this rule are needed to show that the reported gap is not driven by it.

The substantial human–model gap is not driven by the stagnation early-stopping rule, as a large performance gap persists even when this restriction is completely removed. To verify this, we compared GPT-5.6-Sol with and without the 12-step early-stopping rule on the same 100 tasks, retaining the maximum 48-step budget in both settings:

Evaluation group and settingSuccess rateAverage cost / task
GPT-5.6-Sol, 12-step stagnation early stopping17%$1.70
GPT-5.6-Sol, no stagnation early stopping35%$3.62
Human, 12-step stagnation early stopping100%—

Without early stopping, model success increases from 17% to 35%, but remains far below the human success rate of 100% under the same early-stopping rule. This indicates that the human–model gap is not driven by early stopping and that the 12-step window does not prevent humans from temporarily accepting decreases in IoU. Removing early stopping also increases the average cost to 2.13 times the original cost; we therefore retain the rule to control evaluation cost.

Reviewer_9MbkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

RoTA-Bench evaluates interactive 3D spatial reasoning through silhouette alignment using discrete rotations. It provides 674 procedurally generated, solvable tasks and a 100-task test set. Evaluations reveal a substantial human–model gap and examine rotation choices, interaction history, and differences between projection and pose progress.

Critical strengths
  1. Focused interactive task. RoTA-Bench evaluates spatial reasoning through successive rotations and visual feedback, providing a concrete setting for studying how models translate visual observations into actions beyond static question answering.

  2. Verifiable task construction. The automated object-generation pipeline, reversible initialization, and replay-verified reference trajectories support reproducible evaluation and establish task solvability under a clearly specified action interface.

  3. Substantial comparative evidence. The study includes nine VLMs, two agent configurations, human participants, and random and privileged greedy baselines. Task-level uncertainty estimates, input ablations, and action-level analyses strengthen the evidence for a substantial human–model performance gap.

Critical weaknesses
  1. Limited overall contribution. The contribution centers on a narrowly defined projection-alignment benchmark. The human–model gap and descriptive behavioral analyses offer limited new insight beyond existing spatial evaluations. The study also does not demonstrate whether its diagnosis can guide targeted solver improvements under the permitted observations. Consequently, the task’s novelty and resulting insights do not establish a sufficiently compelling standalone contribution.

  2. Insufficient motivation. The task primarily involves visual pose estimation and rotational control, with language serving mainly as an instruction interface. Its relevance to broader VLM capabilities remains insufficiently established. Comparable baselines using geometric reconstruction, pose estimation, or visual servoing are missing, making it difficult to distinguish meaningful spatial-reasoning limitations from difficulty approximating a specialized controller.

  3. Limited diagnostic validity. Task success accepts any pose satisfying the IoU threshold, whereas the proposed diagnostics reward progress toward canonical, symmetry-equivalent poses. The consequences of this mismatch are insufficiently validated. Moreover, the analyses do not adequately separate perception, action interpretation, planning, and control, limiting what the diagnostic differences reveal about the underlying causes of failure.

  4. Unvalidated termination protocol. Episodes stop after twelve rotations without improving the best IoU, although useful geometric progress can temporarily reduce overlap. Without testing alternative stopping thresholds or allowing the full budget, the dependence of reported failure rates on this restriction remains unclear.

  5. Restricted experimental coverage. The core evaluation uses 100 icon-derived tasks with one initialization per object and fixed rendering conditions. Key input ablations cover only one model on a small subset, leaving the stability of the findings across poses, geometry, and model configurations insufficiently established.

Questions for authors

See the weaknesses.

8 comments

W1. Limited overall contribution. The contribution centers on a narrowly defined projection-alignment benchmark. The human–model gap and descriptive behavioral analyses offer limited new insight beyond existing spatial evaluations. The study also does not demonstrate whether its diagnosis can guide targeted solver improvements under the permitted observations. Consequently, the task’s novelty and resulting insights do not establish a sufficiently compelling standalone contribution.

We focus on projection alignment to exclude extraneous physical factors, such as object translation, grasping, and friction, and specifically test whether models can turn 2D observations into 3D understanding that supports precise action.

Our in-depth analysis yields several valuable insights. The substantial human–model performance gap (17% vs. 100%) reveals limitations in models' 3D spatial reasoning and precise control. Our analysis further indicates that models rely more on 2D surface-contour overlap than on the underlying 3D object pose error: at matched states, the model improves 2D overlap while worsening 3D pose alignment in 19.4% of cases, compared with 10.9% for humans. Finally, our ablations reveal the importance of interaction history for guiding step size, or rotation-angle magnitude, in spatial control: after history is removed, 92.60% of actions become 30° rotations in the complete 100-task ablation.

Regarding solver improvements, we have reserved 574 tasks and can generate additional tasks in the future for reinforcement-learning post-training to improve models' 3D perception and control.

Overall, our contributions can be summarized as follows: (1) an interactive benchmark with an automated, solvability-guaranteed task-construction pipeline for evaluating whether VLMs can turn 2D observations into 3D understanding that supports precise action; (2) evaluations of nine VLMs, two agent configurations, and human participants that establish a substantial human–model gap; and (3) action-level diagnostics and input ablations that reveal limitations in selecting rotation directions and determining rotation angles, as well as the importance of interaction history for fine-grained control.

W2. Insufficient motivation. The task primarily involves visual pose estimation and rotational control, with language serving mainly as an instruction interface. Its relevance to broader VLM capabilities remains insufficiently established. Comparable baselines using geometric reconstruction, pose estimation, or visual servoing are missing, making it difficult to distinguish meaningful spatial-reasoning limitations from difficulty approximating a specialized controller.

Language output primarily serves as an instruction interface in this task because our goal is not to measure language ability in isolation. The relevance of this task to broader VLM capabilities is explained in the first paragraph of the Introduction. As VLMs are increasingly used in interactive agents and embodied systems, they must turn 2D visual observations into 3D understanding and produce precise actions. Yet current models still face challenges in reasoning about 3D positions, orientations, and spatial relations. This creates a need for evaluation tools that test whether models can turn 2D observations into 3D understanding that supports precise action.

Regarding the distinction between meaningful limitations in spatial reasoning and the difficulty of approximating a specialized controller, the existing baseline comparison in Section 5.1 and Table 1 provides a clear chain of evidence, without requiring additional geometry-reconstruction, pose-estimation, or visual-servoing algorithms. First, IoU-greedy selects an action greedily based only on the next state's 2D projection IoU. Its success rate is only 39.0%, showing that a controller that greedily optimizes 2D image alignment alone is insufficient to solve the task. Second, Pose-greedy uses the true 3D pose error to select an action with the simplest one-step greedy strategy. By comparison, the strongest-performing model, GPT-5.6-Sol, performs far worse than Pose-greedy in Effective Direction Rate (58.75% vs. 100.0%; only slightly above the random baseline of 53.78%), Best Legal Angle Rate (61.80% vs. 100.0%), and task success (17.0% vs. 100.0%). This shows that once the true 3D pose information is available, even the simplest controller can solve every task, whereas the model cannot determine the correct rotation direction and angle. This indicates that its spatial reasoning is not yet sufficient to accurately perceive an object's 3D geometry and pose. In addition, undergraduate participants without any training or practice also achieve 100.0% success with the same visual observations and discrete action interface (83.91% Effective Direction Rate and 71.82% Best Legal Angle Rate). This further shows that the task is fully solvable and does not require specialized controller training. Therefore, the large gap between the model, the baselines, and humans directly reflects a limitation in the model's spatial reasoning: its ability to infer effective 3D rotation directions and angles from visual observations, rather than difficulty in approximating a complex specialized controller.

W3. Limited diagnostic validity. Task success accepts any pose satisfying the IoU threshold, whereas the proposed diagnostics reward progress toward canonical, symmetry-equivalent poses. The consequences of this mismatch are insufficiently validated. Moreover, the analyses do not adequately separate perception, action interpretation, planning, and control, limiting what the diagnostic differences reveal about the underlying causes of failure.

Thank you for raising this important issue. You are right that we need to examine the relationship between poses satisfying the IoU threshold and final 3D pose error. We therefore separately computed final pose errors for all successful runs:

Evaluation groupSuccessful runsMean pose error (°)Maximum (°)Second highest (°)Error ≤10°
Human4004.44179.94179.72394/400
GPT-5.6-Sol1711.95178.405.2316/17
ReAct162.063.413.1816/16
Gemini 3.6 Flash1020.23179.944.479/10
GPT-6-Astra101.743.632.5010/10
GPT-5.6-Terra82.566.143.948/8
GPT-5.6-Luna42.164.121.594/4
Magentic22.073.021.122/2
Total4674.85179.94179.94459/467

All statistics include every successful run in each evaluation group. Human includes all 400 successful runs across four passes of the 100-task test set.

Of the 467 successful runs, 459 (98.3%) have final pose errors no greater than 10°. The mean final pose error across all 467 successful runs is 4.85°, while the remaining 459 runs excluding the eight near-180° cases have a mean error of only 1.81°, indicating that successful silhouette alignment corresponds to highly accurate 3D pose alignment in the vast majority of cases.

For the eight cases with pose errors close to 180°, we compared seven target projections with their own 180°-rotated versions and compared the broken-heart target projection with its left–right reflection (corresponding to an out-of-plane 180° rotation about the vertical axis). Using the 320 × 320 evaluation masks, the resulting IoUs were 0.938 and 0.953 for the two model cases, and 0.953, 0.946, 0.920, 0.946, 0.871, and 0.953 for the six human cases, respectively. Seven comparisons exceed the success threshold of 0.90. Although the broken-heart reflection falls slightly below the threshold, its actual terminal projection achieves an IoU of 0.9005 with the target. Comparing the target and terminal projections shows that their main outer contours and most of their projected areas overlap substantially, with differences concentrated in a small number of patterns. Thus, although the patterns are not fully aligned, the main contours account for most of the area, allowing the overall IoU to reach the success threshold. The model and human examples are provided in the model figure and the human figure, respectively.

Thus, IoU reflects 3D pose alignment in the vast majority of cases, while these eight examples show that the 0.90 threshold can still admit a small amount of projection ambiguity. Future work will use a higher IoU threshold or incorporate 3D pose error for more precise alignment assessment.

Regarding the distinction among perception, action interpretation, planning, and control, the corresponding sections of the paper provide the following core findings:

  • Perception (Section 5.4): When selecting rotation angles, the model relies more heavily on 2D projection overlap, whereas humans use both projection overlap and 3D pose differences, with a slightly stronger association for the latter.
  • Action interpretation (Section 5.2): Decomposing rotations into direction and angle shows that the model's Effective Direction Rate is far below that of humans (58.75% vs. 83.91%) and only slightly above uniform random sampling (53.78%).
  • Planning (Sections 5.1 and 5.4): Greedily maximizing immediate projection overlap is insufficient to solve the task, achieving only 39.0% success. At matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%).
  • Control (Section 5.3): Fine-grained control is highly sensitive to interaction history. In the complete 100-task ablation, without history, 92.60% of the model's actions remain at the coarse 30° step size, indicating the importance of interaction history for rotation-angle selection.

Comments on the paper

1 comment

Dear reviewers,

As the rebuttal period is coming to an end, I kindly remind you to check the authors' responses. If you have not yet responded, please indicate whether your concerns have been adequately addressed. Many thanks!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.