- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RoTA-Bench, which evaluates vision-language models’ interactive spatial control capabilities by requiring them to rotate 3D objects until their 2D projections match target silhouettes. The authors automatically construct 674 tasks and evaluate nine models, two agent configurations, and human participants on a 100-task test set. The experiments reveal a substantial human–model performance gap and investigate model failure modes and the role of interaction history through greedy baselines, action-level analyses, and input ablations.
Critical strengths
-
The task is well motivated and combines 2D visual observations, 3D rotation actions, and interactive feedback in a clear, controlled setting. It provides a useful benchmark for evaluating current models’ interactive spatial control capabilities.
-
The automated construction pipeline preserves target silhouettes and guarantees task solvability through reversible initialization. This supports reproducible evaluation and provides a practical basis for expanding the dataset.
-
The experiments cover multiple models, human participants, greedy baselines, and input ablations. The analyses of rotation direction, angle selection, and interaction history help characterize model behavior and identify specific failure modes.
Critical weaknesses
-
The interaction is restricted to discrete rotations of a single object about a fixed center. While this helps control experimental variables, it does not cover translation, joint position and orientation adjustment, or multi-object spatial relations. The applicability of the conclusions to more general interactive spatial manipulation therefore remains limited and should be clarified.
-
Task success is defined by 2D projection IoU, but high IoU does not necessarily imply alignment with the target 3D pose. Table 9 reports final pose error averaged across successful and failed tasks, without separately examining its distribution among successful tasks. It therefore remains unclear how closely successful silhouette alignment corresponds to 3D pose alignment, limiting the interpretation of model spatial reasoning behavior.
-
The authors generate 3D objects from 2D vector icons to reduce reliance on priors about familiar 3D shapes. However, the source icons and their associated semantics may also have been encountered during pretraining, and the effectiveness of this design choice lacks direct validation. The lower performance on icon-derived objects reported in the appendix does not distinguish the effects of familiarity from additional difficulty introduced by the generated geometries.
-
Although the automated pipeline generates 674 tasks, the main evaluation uses only 100, the input ablations use only 30, and each object is evaluated from a single initial pose. The paper does not sufficiently justify the choice of evaluation scale.
The paper is good overall, and I would be willing to raise my score if the authors satisfactorily address the questions and concerns above.
Questions for authors
See Weaknesses.
9 comments
W2. Task success is defined by 2D projection IoU, but high IoU does not necessarily imply alignment with the target 3D pose. Table 9 reports final pose error averaged across successful and failed tasks, without separately examining its distribution among successful tasks. It therefore remains unclear how closely successful silhouette alignment corresponds to 3D pose alignment, limiting the interpretation of model spatial reasoning behavior.
Thank you for raising this important issue. Indeed, high 2D projection IoU does not guarantee exact alignment with the target 3D pose. We therefore separately analyzed the distribution of final pose errors across all successful runs:
| Evaluation group | Successful runs | Mean pose error (°) | Maximum (°) | Second highest (°) | Error ≤10° |
|---|---|---|---|---|---|
| Human | 400 | 4.44 | 179.94 | 179.72 | 394/400 |
| GPT-5.6-Sol | 17 | 11.95 | 178.40 | 5.23 | 16/17 |
| ReAct | 16 | 2.06 | 3.41 | 3.18 | 16/16 |
| Gemini 3.6 Flash | 10 | 20.23 | 179.94 | 4.47 | 9/10 |
| GPT-6-Astra | 10 | 1.74 | 3.63 | 2.50 | 10/10 |
| GPT-5.6-Terra | 8 | 2.56 | 6.14 | 3.94 | 8/8 |
| GPT-5.6-Luna | 4 | 2.16 | 4.12 | 1.59 | 4/4 |
| Magentic | 2 | 2.07 | 3.02 | 1.12 | 2/2 |
| Total | 467 | 4.85 | 179.94 | 179.94 | 459/467 |
All statistics include every successful run in each evaluation group. Human includes all 400 successful runs across four passes of the 100-task test set.
Of the 467 successful runs, 459 (98.3%) have final pose errors no greater than 10°. The mean final pose error across all 467 successful runs is 4.85°, while the remaining 459 runs excluding the eight near-180° cases have a mean error of only 1.81°, indicating that successful silhouette alignment corresponds to highly accurate 3D pose alignment in the vast majority of cases.
For the eight cases with pose errors close to 180°, we compared seven target projections with their own 180°-rotated versions and compared the broken-heart target projection with its left–right reflection (corresponding to an out-of-plane 180° rotation about the vertical axis). Using the 320 × 320 evaluation masks, the resulting IoUs were 0.938 and 0.953 for the two model cases, and 0.953, 0.946, 0.920, 0.946, 0.871, and 0.953 for the six human cases, respectively. Seven comparisons exceed the success threshold of 0.90. Although the broken-heart reflection falls slightly below the threshold, its actual terminal projection achieves an IoU of 0.9005 with the target. Comparing the target and terminal projections shows that their main outer contours and most of their projected areas overlap substantially, with differences concentrated in a small number of patterns. Thus, although the patterns are not fully aligned, the main contours account for most of the area, allowing the overall IoU to reach the success threshold. The model and human examples are provided in the model figure and the human figure, respectively.
Therefore, IoU faithfully reflects 3D pose alignment in the vast majority of cases and does not affect our interpretation of model spatial reasoning behavior. However, these eight cases show that the 0.90 threshold can still admit a small amount of ambiguity. We have added this distributional analysis to Appendix K.9 of the revised manuscript. Future work will use a higher IoU threshold or incorporate 3D pose error for more precise alignment assessment.
W3. The authors generate 3D objects from 2D vector icons to reduce reliance on priors about familiar 3D shapes. However, the source icons and their associated semantics may also have been encountered during pretraining, and the effectiveness of this design choice lacks direct validation. The lower performance on icon-derived objects reported in the appendix does not distinguish the effects of familiarity from additional difficulty introduced by the generated geometries.
Our central conclusion is that sphere-bounded extrusion directly breaks the association between 2D icon semantics and real-world 3D depth priors. Because we cannot independently erase a black-box model's pretrained memory without changing the 3D geometry, weakening familiar 3D priors and introducing synthetic geometric difficulty are naturally coupled aspects of the geometry-generation process.
First, although the source 2D icons and their semantics may appear in pretraining data, the 3D structures generated through sphere-bounded extrusion are unrelated to the corresponding real-world forms. For example, the T-shirt icon in Figure 11 becomes a thick, sphere-truncated solid after extrusion, rather than the thin structure of real clothing. Even if the model recognizes the icon's semantics in 2D, its pretrained 3D priors for real objects cannot transfer to the sphere-bounded 3D structure.
Second, regarding the distinction between familiarity and synthetic geometric difficulty, blocking reliance on familiar 3D shape priors requires disrupting the corresponding 3D structure. This necessarily introduces additional difficulty from the generated geometry. These factors are coupled by construction, so we do not attribute the performance decrease to a single isolated factor.
To further investigate the effects of 3D shape priors and 2D contour sources and address your concerns, we conducted three additional controlled evaluations with GPT-5.6-Sol, each comprising 100 tasks with matched reference-path length distributions. Two sets were constructed by randomly sampling real 3D assets from Objaverse 1.0 and Poly Haven, respectively; the third used 2D contours extracted from Poly Haven objects and reconstructed through sphere-bounded extrusion. Rendering, initialization, and evaluation protocols were identical to those of RoTA-Bench.
| Experimental setting | Success rate |
|---|---|
| Objaverse 1.0 real 3D assets | 54% |
| Poly Haven real 3D assets | 56% |
| Poly Haven contours + sphere-bounded extrusion | 40% |
| RoTA-Bench test set | 17% |
The results show a clear performance gradient:
- Success on both real 3D asset sets (54% and 56%) is substantially higher than on RoTA-Bench (17%), indicating that common objects with real-world 3D structures are indeed much easier for the model.
- Keeping Poly Haven's frontal 2D contours and semantics unchanged while replacing its real 3D structures with sphere-bounded extrusion decreases success from 56% to 40%. Thus, even when the frontal 2D image contains exactly the same familiar object semantics and contour, 2D semantic familiarity alone cannot preserve 3D alignment performance once sphere-bounded extrusion disrupts the real 3D depth structure.
- Under the same sphere-bounded extrusion process, vector icons are more difficult than real-object contours (17% vs. 40%), indicating that the morphology of the 2D contours also affects task difficulty. Under the same setting, humans achieve 100% success on RoTA-Bench, showing that the generated geometry is fully solvable for humans and that its difficulty remains within an acceptable range.
We have added these explanations and experimental analyses to Appendix K.8 of the revised manuscript.
W1. The interaction is restricted to discrete rotations of a single object about a fixed center. While this helps control experimental variables, it does not cover translation, joint position and orientation adjustment, or multi-object spatial relations. The applicability of the conclusions to more general interactive spatial manipulation therefore remains limited and should be clarified.
We also considered this limitation regarding more general interactive spatial manipulation and included a similar discussion in Section 5.5 (Limitations and Future Work) of the original manuscript: "Second, each task features only a single target under discrete rotation actions, making the interaction dynamics relatively simplified compared to continuous physical environments... In addition, the benchmark can be extended by introducing continuous rotation control, external occlusions, and multi-object alignment or assembly tasks to further heighten the spatial reasoning challenge." To address your concern more explicitly, we have also added the following clarification to the Motivation and Design Rationale subsection (Section 3.1) in the revised manuscript to clearly define the scope of our conclusions and outline future extensions: