acceptodds
Withdrawn from ICLR 2027

Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much. We introduce OR-Bench, a fine-grained benchmark for object-rotation reasoning with eight tasks covering rotation detection, rotation magnitude estimation, and multi-view rotation reasoning. Across 12 VLMs, the gap is stark: the strongest models approach 100% accuracy on detection, yet even coarse magnitude estimation is near chance. When asked for exact angles, models place 91.8–100% of their predictions on just 0°, 90°, and 180°, a failure we term canonical-angle collapse. This collapse persists even without visual input. Representation probing shows that missing information is only part of the explanation. Although rotation information becomes less recoverable at finer granularity, substantial coarse-grained information remains, and a simple linear probe outperforms the models' generated answers. This suggests that VLMs underuse rotation information they already encode. We therefore propose RotationCue, a lightweight decoder that recovers coarse rotation information from the VLM's own frozen representations and feeds it back to the model as intermediate textual context. Across three VLMs, RotationCue improves every model–task combination on OR-Bench, raising macro-average accuracy by 7.9–12.6 points while preserving general capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Benchmarks, Spatial Reasoning, Representation Probing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s6tyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how well VLMs reason about an object's rotation about its vertical axis between views. It introduces OR-Bench: 2,109 examples built from 73 ABO turntable objects, covering eight tasks from rotation detection, through coarse and exact angle estimation, to multi-view use such as view matching, rotation comparison, and post-rotation view prediction. Across 12 VLMs, detection is near ceiling while magnitude estimation is near chance. Exact-angle predictions collapse onto 0°/90°/180°, and this happens even without images.

Linear probes on vision-encoder features (|f1 − f2|) and on the pre-answer hidden state recover more coarse rotation information than the model's own answers. The authors read this as "encoded but underused."

They then propose RotationCue:

  • a small decoder on frozen vision-encoder layer-8 features of both images;
  • trained on external data (ALOI, ModelNet40 multi-view);
  • predicts same-object, rotation-present, and a 3-way angle range;
  • these predictions are inserted as text into the prompt.

RotationCue improves all 24 model–task combinations (Macro-8 +7.9–12.6), beats a LoRA Rotation-SFT baseline, and preserves general benchmarks.

Critical strengths
  1. The benchmark is cleanly constructed. Turntable frames give noise-free angle labels, symmetric objects are removed, candidates come from the same object, and the shared image pairs across Coarse/Exact tasks allow direct comparison across output granularities.
  2. The canonical-angle collapse analysis is a nice diagnostic. Removing visual tokens shows a strong textual prior, and the precision/recall breakdown in Sec. 3.2 shows the models only reliably separate zero from nonzero rotation.
  3. The probing is done with care: object-disjoint cross-validation, a random-label control (Appendix E.6), probes trained on external ALOI data (Appendix E.5), and a direct probe vs. LM-head comparison on the same hidden state.
  4. The RotationCue analysis has good controls: shuffled cues, oracle cues, a self-generated cue in the same format, and CoT baselines (Table 3), plus a check that general benchmarks are preserved (Table 4). The method is cheap (0.23 GPU-hours) and leaves the VLM unchanged.
Critical weaknesses
  1. RotationCue mostly supplies the answer for some tasks, so Macro-8 overstates the improvement. The three cue fields map directly onto benchmark tasks:

    • rotation_present is the answer to Rotation Detection;
    • angle_range uses exactly the Coarse Angle bins;
    • angle_range largely determines 90° Threshold.

    These three tasks account for the biggest gains (+4.7, +25.4, +15.0 mean ∆). In effect, a supervised rotation classifier is trained and its prediction is placed in the prompt. The key missing comparison is the decoder's own accuracy used directly as the answer on these tasks. If the decoder alone matches or beats "VLM + cue," the VLM adds nothing there. The gains on tasks not directly answered by the cue (View Matching +1.8, Exact Open +5.6, Exact MCQ +8.5) should be the headline evidence. They are real but much smaller, and with about 243 items per task each has a 95% CI of roughly ±6 points.

  2. The "underused information" conclusion is not supported by the probe-vs-model comparison. A probe trained with in-distribution supervision beating a zero-shot generated answer is expected for almost any representation and label, so it doesn't show that the model "underuses" what it encodes. In the vision-encoder analysis, the two images are encoded independently and the relation (absdiff) is computed by the probe, not the model. The hidden-state comparison (Fig. 5) is fairer, but the probe is still fit to these exact label bins while the LM head must answer a free-form prompt. Supporting "underuse" would require showing the information is available but not acted on without supervision. For example, few-shot prompting or a small in-context calibration could close the gap, or a probe trained on unrelated rotation data could transfer while the LM head fails. Appendix E.5 (ALOI-trained probes) is closer to this and deserves to be in the main text.

  3. RotationCue doesn't depend on the VLM's own representations, and the obvious external alternatives are missing. The paper stresses that all cue information is "decoded from its own frozen representations." But the decoder is trained with rotation labels on external data, so the information comes from that supervision applied to generic visual features. The natural baselines are:

    • the same decoder on a different backbone, e.g., DINOv2 or CLIP features;
    • an off-the-shelf orientation estimator (e.g., OrientAnything) whose predicted relative azimuth is binned into the same three-field cue.

    If these work as well or better, the "own representations" framing is unnecessary and the method reduces to tool-augmented prompting with a rotation estimator. The layer choice (L8) was also guided by probing on OR-Bench itself (Sec. 5.1), which is a mild use of test data for a design decision.

  4. The benchmark's scope is narrow relative to its claims.

    • All examples come from 73 ABO product turntables: clean renders, a single elevation, a fixed camera, and only vertical-axis rotation.
    • There is no evaluation on real photographs, cluttered scenes, or rotations around other axes, so it's unclear whether the findings hold beyond this setting. A held-out real-image test set (e.g., CO3D or Objectron with known poses) would strengthen the conclusions considerably.
    • Exact Angle (Open) uses exact match after normalization, which is very harsh for a continuous quantity. Humans reach only 56.5% on it. A tolerance-based metric (e.g., within ±15°) or mean angular error would be more informative.
    • The "=90°" class in 90° Threshold also interacts with the models' default answer of 90°, which complicates its interpretation.
  5. The Rotation-SFT comparison may be under-tuned. Rotation-SFT uses 42 GPU-hours of LoRA on the language side with a frozen vision encoder, improves mainly Coarse Angle, and degrades general benchmarks by 4–8 points. That degradation is large for LoRA, which suggests the recipe was not tuned for retention, e.g., no data mixing with general instruction data. A fairer comparison would mix rotation data with general SFT data, or fine-tune only on cue-style outputs. Also, the comparison changes two things at once: Rotation-SFT changes the model, while RotationCue adds an external classifier.

  6. Missing related work and minor issues.

    • Prior work on mental rotation and multi-view orientation reasoning in VLMs, beyond SpinBench and LRR-Bench, should be discussed, for example MindCube-style multi-view benchmarks and orientation-estimation models such as OrientAnything.
    • The human study (5 participants, a stratified subset of 628 items) does not report inter-rater agreement.
    • Results are single-run with no confidence intervals. Given about 243 items per task, several reported differences are within noise, e.g., some Table 3 entries and the general-benchmark deltas in Table 4.
Questions for authors
  1. What is the decoder's own accuracy on Rotation Detection, Coarse Angle, and 90° Threshold if its prediction is used directly as the answer? How does "VLM + cue" compare?
  2. How does RotationCue perform when the same decoder is trained on DINOv2/CLIP features, or when the cue comes from an off-the-shelf orientation estimator?
  3. Can you report macro gains restricted to tasks whose answer is not directly encoded in the cue fields?
  4. How was layer L8 chosen? Does the choice change if it is selected only on external validation data?
  5. Do the findings hold on real-image data with pose labels (e.g., Objectron or CO3D), or for rotations around non-vertical axes?
Reviewer_fZv6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OR-Bench to diagnose object-rotation understanding under controlled azimuth changes. VLMs detect rotation more reliably than they estimate its magnitude, and exact-angle responses often collapse to canonical angles. Probing suggests that finer angular information is less recoverable, while available coarse information is underused. RotationCue decodes structured cues from frozen visual representations using external supervision and feeds them back as text. Experiments report improvements across evaluated rotation tasks.

Critical strengths
  • The paper combines controlled evaluation with representation probing to investigate VLM rotation failures. OR-Bench evaluates rotation-magnitude estimation at different angular granularities. Layerwise probes recover rotation-magnitude labels less accurately from final vision-encoder features than from selected intermediate layers. This suggests reduced accessibility of rotation information as a possible contributor to estimation failures.
  • RotationCue offers a practical way to make recoverable rotation information available to the VLM during answer generation. An externally supervised decoder predicts structured rotation states from intermediate vision-encoder features and feeds them back as textual context while keeping the base VLM frozen.
Critical weaknesses
  • The probing results suggest that fine-grained rotation information is less accessible in final vision-encoder features than in selected intermediate layers, raising a concern about the visual evidence available to language-side training. If relevant angular distinctions are not preserved at the vision–language interface, downstream language-side training would lack the visual evidence needed to learn those distinctions reliably. RotationCue supplies additional evidence from intermediate layers, but compresses it into coarse textual states while leaving the native visual pathway unchanged. Its improvements therefore leave open whether this textual interface provides sufficient information for finer angular discrimination. The exact-angle gains and reduced canonical-angle prediction rate are compatible with narrowing the answer space through coarse cues; these aggregate results do not by themselves establish better discrimination between angles within the same range. The external evaluation primarily supports capability preservation under the stated input construction and routing policy, leaving the scope of broader spatial reasoning benefits insufficiently characterized.
  • Coarse Angle and 90° Threshold already establish substantial difficulty with rotation-magnitude estimation, so low exact-angle accuracy alone provides limited additional explanation of this failure. In my assessment, the main conceptual advance therefore appears concentrated in this representation-level diagnosis, while the finer-grained evaluation primarily sharpens the characterization of an already evident magnitude-estimation failure.
Questions for authors

Given that the relation decoder already recovers useful rotation cues, what motivates converting them into coarse textual states instead of feeding its continuous hidden features directly to the LLM through a learned adapter? Have the authors compared this feature-level interface, or direct use of intermediate vision-encoder features, with RotationCue to assess whether textual discretization limits fine-angle estimation?

Reviewer_r3EWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OR-bench and highlights VLMs' ability to estimate how far an object has rotated. The authors evaluate 12 VLMs on OR-Bench, an eight-task benchmark covering rotation detection, angle estimation, and multi-view rotation reasoning. Most VLMs easily detect whether an object has rotated. However, on the Coarse Angle task, GPT-5.4 and Qwen3-VL-8B achieve accuracies of 53.1% and 40.7%, compared with a chance level of 33.3%. Also, responses on Exact Angle concentrate on three canonical angles (0°, 90°, and 180°) across Qwen3-VL-8B, Qwen2.5-VL-7B, and InternVL3.5-8B. The authors report linear probing and output-head probing results showing that rotation information becomes less recoverable at finer angular granularities but is more recoverable on Coarse Angle, where the gap between probe performance and the models' own answer performance is largest. Motivated by these findings, the authors argue that useful information is exist but under-used by the models on Coarse Angle. They therefore introduce RotationCue, which uses a lightweight external decoder to recover coarse rotation information from the VLM's own frozen representations and inserts it as input. Across the three models, RotationCue improves performance on every OR-Bench task, raising macro-average accuracy by 7.9~12.6 percentage points while preserving performance on the VSR-ZS, CV-Bench, and SpinBench benchmarks.

Critical strengths
  1. The paper provides a benchmark for evaluating VLMs' understanding of object rotation. OR-Bench includes eight tasks covering rotation detection, rotation magnitude estimation, and reasoning about rotation across multiple views. It provides a resource for studying how VLMs perceive and reason about object rotation and for evaluating improvements in future work.

  2. The paper identifies an interesting failure pattern and demonstrates performance improvements. Canonical-angle collapse, where Exact Angle predictions concentrate on 0°, 90°, and 180°, is an interesting observation that reveals a specific failure pattern beyond low accuracy. RotationCue also improves performance on every OR-Bench task across three VLMs, increasing Macro-8 by 7.9~12.6 percentage points while preserving performance on external benchmarks.

Critical weaknesses
  1. In the probing experiment, there could be another possible explanation, but the analysis does not separate out this effect. Zero-rotation questions use identical frames, so their absdiff representation is exactly zero. A probe could therefore achieve high accuracy on the Rotation Detection task by separating zero from nonzero feature differences, without distinguishing different rotation magnitudes. This could be one possible reason for the probe's higher performance on Rotation Detection and lower performance on Exact Angle. Also, the gains from language-side fine-tuning while keeping the vision encoder and projector frozen might suggest that useful rotation information remains in the encoder's output representations and the model can learn to use it, which is relevant to interpreting the low linear-probe recoverability at later layers.

  2. The added benefit of RotationCue over SFT is difficult to assess. Rotation-SFT also substantially outperforms RotationCue on Coarse Angle (From Table18, 85.2% vs. 66.7%), but the comparison is limited to coarse rotation supervision and a fixed question-answering format. It therefore leaves open whether more varied task formats or finer angle supervision could make SFT more effective across tasks beyond Coarse Angle task.

Questions for authors
  1. The paper explains that rotation is common and existing VLMs struggle with estimating it, but it is difficult to understand from the current discussion why rotation magnitude estimation is an important capability to evaluate specifically in VLMs. What additional knowledge is gained by evaluating VLMs' ability to estimate 'how far' an object has rotated?

  2. Could the authors provide the human evaluation instructions, specify the order of the MCQ and Open tasks and whether participants encountered the same image pairs repeatedly during the human evaluation, and report the distribution of human angle responses in Exact Angle (Open)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.