- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how well VLMs reason about an object's rotation about its vertical axis between views. It introduces OR-Bench: 2,109 examples built from 73 ABO turntable objects, covering eight tasks from rotation detection, through coarse and exact angle estimation, to multi-view use such as view matching, rotation comparison, and post-rotation view prediction. Across 12 VLMs, detection is near ceiling while magnitude estimation is near chance. Exact-angle predictions collapse onto 0°/90°/180°, and this happens even without images.
Linear probes on vision-encoder features (|f1 − f2|) and on the pre-answer hidden state recover more coarse rotation information than the model's own answers. The authors read this as "encoded but underused."
They then propose RotationCue:
- a small decoder on frozen vision-encoder layer-8 features of both images;
- trained on external data (ALOI, ModelNet40 multi-view);
- predicts same-object, rotation-present, and a 3-way angle range;
- these predictions are inserted as text into the prompt.
RotationCue improves all 24 model–task combinations (Macro-8 +7.9–12.6), beats a LoRA Rotation-SFT baseline, and preserves general benchmarks.
Critical strengths
- The benchmark is cleanly constructed. Turntable frames give noise-free angle labels, symmetric objects are removed, candidates come from the same object, and the shared image pairs across Coarse/Exact tasks allow direct comparison across output granularities.
- The canonical-angle collapse analysis is a nice diagnostic. Removing visual tokens shows a strong textual prior, and the precision/recall breakdown in Sec. 3.2 shows the models only reliably separate zero from nonzero rotation.
- The probing is done with care: object-disjoint cross-validation, a random-label control (Appendix E.6), probes trained on external ALOI data (Appendix E.5), and a direct probe vs. LM-head comparison on the same hidden state.
- The RotationCue analysis has good controls: shuffled cues, oracle cues, a self-generated cue in the same format, and CoT baselines (Table 3), plus a check that general benchmarks are preserved (Table 4). The method is cheap (0.23 GPU-hours) and leaves the VLM unchanged.
Critical weaknesses
-
RotationCue mostly supplies the answer for some tasks, so Macro-8 overstates the improvement. The three cue fields map directly onto benchmark tasks:
- rotation_present is the answer to Rotation Detection;
- angle_range uses exactly the Coarse Angle bins;
- angle_range largely determines 90° Threshold.
These three tasks account for the biggest gains (+4.7, +25.4, +15.0 mean ∆). In effect, a supervised rotation classifier is trained and its prediction is placed in the prompt. The key missing comparison is the decoder's own accuracy used directly as the answer on these tasks. If the decoder alone matches or beats "VLM + cue," the VLM adds nothing there. The gains on tasks not directly answered by the cue (View Matching +1.8, Exact Open +5.6, Exact MCQ +8.5) should be the headline evidence. They are real but much smaller, and with about 243 items per task each has a 95% CI of roughly ±6 points.
-
The "underused information" conclusion is not supported by the probe-vs-model comparison. A probe trained with in-distribution supervision beating a zero-shot generated answer is expected for almost any representation and label, so it doesn't show that the model "underuses" what it encodes. In the vision-encoder analysis, the two images are encoded independently and the relation (absdiff) is computed by the probe, not the model. The hidden-state comparison (Fig. 5) is fairer, but the probe is still fit to these exact label bins while the LM head must answer a free-form prompt. Supporting "underuse" would require showing the information is available but not acted on without supervision. For example, few-shot prompting or a small in-context calibration could close the gap, or a probe trained on unrelated rotation data could transfer while the LM head fails. Appendix E.5 (ALOI-trained probes) is closer to this and deserves to be in the main text.
-
RotationCue doesn't depend on the VLM's own representations, and the obvious external alternatives are missing. The paper stresses that all cue information is "decoded from its own frozen representations." But the decoder is trained with rotation labels on external data, so the information comes from that supervision applied to generic visual features. The natural baselines are:
- the same decoder on a different backbone, e.g., DINOv2 or CLIP features;
- an off-the-shelf orientation estimator (e.g., OrientAnything) whose predicted relative azimuth is binned into the same three-field cue.
If these work as well or better, the "own representations" framing is unnecessary and the method reduces to tool-augmented prompting with a rotation estimator. The layer choice (L8) was also guided by probing on OR-Bench itself (Sec. 5.1), which is a mild use of test data for a design decision.
-
The benchmark's scope is narrow relative to its claims.
- All examples come from 73 ABO product turntables: clean renders, a single elevation, a fixed camera, and only vertical-axis rotation.
- There is no evaluation on real photographs, cluttered scenes, or rotations around other axes, so it's unclear whether the findings hold beyond this setting. A held-out real-image test set (e.g., CO3D or Objectron with known poses) would strengthen the conclusions considerably.
- Exact Angle (Open) uses exact match after normalization, which is very harsh for a continuous quantity. Humans reach only 56.5% on it. A tolerance-based metric (e.g., within ±15°) or mean angular error would be more informative.
- The "=90°" class in 90° Threshold also interacts with the models' default answer of 90°, which complicates its interpretation.
-
The Rotation-SFT comparison may be under-tuned. Rotation-SFT uses 42 GPU-hours of LoRA on the language side with a frozen vision encoder, improves mainly Coarse Angle, and degrades general benchmarks by 4–8 points. That degradation is large for LoRA, which suggests the recipe was not tuned for retention, e.g., no data mixing with general instruction data. A fairer comparison would mix rotation data with general SFT data, or fine-tune only on cue-style outputs. Also, the comparison changes two things at once: Rotation-SFT changes the model, while RotationCue adds an external classifier.
-
Missing related work and minor issues.
- Prior work on mental rotation and multi-view orientation reasoning in VLMs, beyond SpinBench and LRR-Bench, should be discussed, for example MindCube-style multi-view benchmarks and orientation-estimation models such as OrientAnything.
- The human study (5 participants, a stratified subset of 628 items) does not report inter-rater agreement.
- Results are single-run with no confidence intervals. Given about 243 items per task, several reported differences are within noise, e.g., some Table 3 entries and the general-benchmark deltas in Table 4.
Questions for authors
- What is the decoder's own accuracy on Rotation Detection, Coarse Angle, and 90° Threshold if its prediction is used directly as the answer? How does "VLM + cue" compare?
- How does RotationCue perform when the same decoder is trained on DINOv2/CLIP features, or when the cue comes from an off-the-shelf orientation estimator?
- Can you report macro gains restricted to tasks whose answer is not directly encoded in the cue fields?
- How was layer L8 chosen? Does the choice change if it is selected only on external validation data?
- Do the findings hold on real-image data with pose labels (e.g., Objectron or CO3D), or for rotations around non-vertical axes?