- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets perspective-taking questions for VLMs, such as "from person X's viewpoint, what is on their left." LeRF trains Qwen3.5-4B/9B to first decide whether a reference frame is needed. If so, the model outputs a tool call with the reference entity and four 2D points: the projected origin and the front, left, and up axis endpoints. A deterministic renderer draws these axes on the image, and the model then reasons over the rendered image with the numeric coordinates hidden. Training has two stages:
- SFT (LoRA): frame labels derived from ImageNet3D, Omni6DPose, and BEDLAM poses through first-order projection, plus about 20% "no tool call" examples.
- GRPO with a correctness-only reward: on MultihopSpatial and SpatialReasoner-RL, with frame tokens excluded from the policy gradient.
LeRF-9B improves over Qwen3.5-9B on OmniSpatial-PT Allo/Hypo (+6.9 / +11.1), ViewSpatial person-perspective tasks, and 3DSRBench orientation. It loses about 6 points on egocentric questions. On EMDB/Objectron, its projected-frame estimation is far better than the base model and roughly comparable to OrientAnything-V2.
Critical strengths
- The idea is intuitive and well motivated. Making the requested viewpoint explicit as a drawn coordinate frame turns implicit mental rotation into something visible. The Fig. 1 pilot with OrientAnything-V2 overlays is a nice motivating experiment.
- Predicting a 2D-projected frame instead of full 3D geometry is a pragmatic choice that avoids the error-prone reconstruction used by SpatialReasoner and APC. The out-of-plane symbol handling (Appendix A) is careful.
- Gains on the viewpoint-dependent splits are consistent for both model sizes. LeRF-9B is competitive with or better than the proprietary models on OmniSpatial Hypo and ViewSpatial P-Obj (Table 1).
- The paper evaluates frame estimation directly on EMDB and Objectron (Table 2), not only downstream QA, and runs useful diagnostics: tool-call selectivity (Fig. 3), frame perturbation (Fig. 4), and the render on/off ablation (Table 3). The failure analysis (Appendix D.2) is honest.
Critical weaknesses
-
The paper's own ablations suggest the rendered frame contributes little. The central claim is that rendering the frame onto the image is what helps. But Table 3 shows that with SFT initialization, training and evaluating without rendering gives 57.18% overall vs. 57.97% with rendering, a 0.8-point gap. Allo is actually higher without rendering (54.63 vs. 54.04). Most of the gain over the base model (52.76%) therefore comes from frame-prediction SFT, acting as an auxiliary task, plus RL, not from the visual overlay.
The perturbation test points the same way. Rotating the rendered frame by 90° swaps the meaning of front and left, which should be catastrophic for left/right questions if the model were reading the axes. Yet overall accuracy drops only 57.97 → 54.05, and Allo barely moves (Fig. 4). Rendering does help hypothetical questions (Hypo 49.4 → 55.7), but that is a narrower claim than the title and contributions make. The paper should reframe accordingly. It should also add a cleaner perturbation (e.g., a 180° flip, or swapping the front and left labels) with per-split results.
-
The most direct baseline is missing. The motivating experiment is Qwen3.5-9B with an OrientAnything-V2 frame overlay (Fig. 1), and Appendix B.4 runs exactly this pipeline on 415 OmniSpatial cases. Yet Table 1 does not include it. It is the natural comparison for a learned frame: the same overlay from an external estimator, both zero-shot and with the same RL stage. Table 2 shows OrientAnything-V2 is comparable or better on objects (Objectron All@30: 83.2 vs. 72.5). Without this baseline, it is unclear whether learning the frame inside the VLM is better than calling an off-the-shelf orientation model, which is the main design argument of the paper. Related work on orientation-aware VLMs built on OrientAnything (e.g., SoFar, 2025) should also be discussed.
-
Hypothetical-viewpoint frames are never directly supervised. The SFT frame labels come from pose datasets, so they are always anchored to a visible entity with a known pose. In RL, frame-prediction tokens are excluded from the policy gradient. So the frame for an imagined observer ("if you're seated on the sofa…") never receives a training signal, even though Hypo is where the paper reports the largest gain and the clearest benefit from rendering. How does the model produce these frames, and how accurate are they? An evaluation of hypothetical-frame quality, even a small manually annotated set, is needed to support the claim that LeRF "establishes" frames for imagined viewpoints. It is also unclear whether the Call/NoCall decision token is included in the RL gradient.
Questions for authors
- How does Qwen3.5-9B with OrientAnything-V2 frame overlays perform on Table 1 benchmarks, both zero-shot and after the same RL stage?
- How are frames for hypothetical observers produced, given that neither SFT nor RL supervises them? Can you report their accuracy on a small annotated set?
- What happens under a 180° frame flip or a front/left label swap, broken down by Ego/Allo/Hypo? Why does a 90 rotation leave Allo nearly unchanged?
- Why does Ego accuracy drop by about 6 points when the tool is almost never called on Ego questions? How does LeRF do on a general VLM benchmark compared with the base model?
11 comments
Response to Reviewer NBwD [2/5]
W1-perturbation, Q3: What happens under a 180° frame flip or a front/left label swap, broken down by Ego/Allo/Hypo? Why does a 90° rotation leave Allo nearly unchanged?
Thank you for this insightful question. We further analyzed the model's behavior under 90° rotations and conducted an additional experiment with a 180° left-axis flip.
Why does 90° rotation leave Allo nearly unchanged? By inspecting the model's chain-of-thought (CoT) traces, we found that the model can sometimes detect and correct obvious inconsistencies in the rotated coordinate frame. For example, when the up-axis points horizontally, the model explicitly notes: "The blue arrow (Up) points left. This is confusing." It then attempts to reconstruct the correct coordinate orientation before proceeding with spatial reasoning. This self-correction helps explain why certain 90° rotations cause only limited degradation on Allo. We also include representative CoT examples in Appendix C.5, highlighted in .
Additional experiment: 180° left-axis flip. To further examine the model's dependence on the rendered coordinate frame, we conducted an additional experiment in which the left axis is reversed by 180°, with all other settings unchanged. The results are shown below (accuracy in %).
| Setting | Overall | Ego | Allo | Hypo |
|---|---|---|---|---|
| Original frame | 57.97 | 74.31 | 54.04 | 55.66 |
| 180° left-axis flip | 45.53 | 74.90 | 38.30 | 42.17 |
| Δ (90° all-axes rotation) | −3.92 | -0.98 | −3.83 | −7.95 |
| Δ (180° left-axis flip) | −12.44 | +0.59 | −15.74 | −13.49 |
The 180° flip leads to a 12.44 percentage-point drop in overall accuracy, with particularly substantial degradation on Allo (−15.74 pp) and Hypo (−13.49 pp). In contrast, Ego remains largely unaffected (+0.59 pp). This distinction is consistent with the nature of these tasks: Allo and Hypo require reasoning about reference-dependent coordinate systems, making them more sensitive to incorrect frame orientations, whereas Ego primarily involves reasoning from the observer's own perspective.
Overall, these findings suggest that relatively minor degradation to certain 90° rotations can arise from explicit error detection and self-correction. The substantial degradation under the 180° flip further supports the importance of correctly oriented coordinate frames for LeRF's spatial reasoning.
Response to Reviewer NBwD [3/5]
W2, Q1: How does Qwen3.5-9B with OrientAnything-V2 frame overlays perform on Table 1 benchmarks, both zero-shot and after the same RL stage? Related work on orientation-aware VLMs built on OrientAnything (e.g., SoFar, 2025) should also be discussed.
Thank you for pointing out this important missing baseline. We conducted additional experiments comparing LeRF against Qwen3.5-9B (thinking) with OrientAnything-V2 (OA-V2) overlays, both zero-shot and after RL fine-tuning, on all three benchmarks in Table 1. We also evaluated SoFar (Qi et al., 2025) as an additional orientation-aware baseline.
Experimental setup. For the OA-V2 baselines, we use GPT-5.6-Terra to identify the reference entity and its bounding box, estimate its orientation using OA-V2, and render the corresponding coordinate frame with axis descriptions. Ego questions retain the original image, while Allo/Hypo questions receive the rendered frame when requested by the model in the first turn. We evaluate (1) zero-shot Qwen3.5-9B with OA-V2 overlays, and (2) Qwen3.5-9B fine-tuned with RL using OA-V2-annotated training data. During RL, Ego samples use the original image, while Allo/Hypo samples use OA-V2 overlays. The answer process is single-turn without the tool. We additionally adapt SoFar using GPT-4o and Qwen3.5-9B Thinking for the evaluated benchmarks.
Additional experimental results (%):
| Method | Ori | M-Obj | P-Obj | P-Rel | Ego | Allo | Hypo |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | 48.17 | 48.66 | 56.23 | 65.51 | 80.20 | 47.13 | 44.58 |
| + OA-V2 (Zero-shot) | 50.75 | 49.36 | 60.16 | 68.75 | 80.28 | 51.70 | 52.77 |
| + OA-V2 (RL) | 47.37 | 48.98 | 59.77 | 66.93 | 78.18 | 55.29 | 50.12 |
| SoFar[1] | 41.41 | 48.61 | 48.80 | 61.05 | 77.45 | 45.96 | 40.96 |
| LeRF-9B (Ours) | 53.76 | 50.29 | 61.91 | 74.23 | 74.31 | 54.04 | 55.66 |
Ori and M-Obj denote the subtasks of 3DSRBench; P-Obj and P-Rel denote those of ViewSpatial-Bench; Ego, Allo, and Hypo denote those of OmniSpatial-PT.
Analysis. External OA-V2 overlays substantially improve zero-shot perspective taking, validating our motivation. Nevertheless, LeRF outperforms this baseline on all six non-Ego metrics. Compared with OA-V2 + RL, LeRF achieves higher accuracy on five of six non-Ego metrics. Although OA-V2 + RL slightly outperforms LeRF on Allo (55.29% vs. 54.04%), this 1.25 pp advantage comes at the cost of an additional external orientation estimator, which requires loading an additional model with a 5.05 GB checkpoint, as well as introducing inference latency in grounding and cropping the reference entity in advance for OrientAnything model. In contrast, LeRF predicts reference frames within the VLM and requires only a lightweight deterministic renderer, avoiding external perception models. LeRF also consistently outperforms the adapted SoFar baseline across all six non-Ego metrics.
Conclusion. Although OA-V2 achieves higher standalone object-orientation accuracy on Objectron (paper Table 2), this advantage does not consistently translate into better downstream perspective-taking performance. Our results suggest that learning to construct question-conditioned reference frames and reason with their rendered cues is more effective and generalizable overall than relying on externally estimated orientations, while eliminating the need for external perception models at inference time.
Response to Reviewer NBwD [1/5]
We are grateful to the reviewer for the extensive review. We address your questions point by point below.
Thanks for pointing this out. We agree that "indispensable" was an overly strong characterization, and have revised the Introduction accordingly to clarify that rendering provides additional visual guidance rather than being strictly necessary for perspective-taking reasoning.
We would like to clarify the distinct contributions of these components.
(1) SFT alone does not improve downstream reasoning. We additionally evaluate the model after SFT but before RL:
SFT alone reduces overall accuracy from 52.76% to 40.86%, indicating that frame-prediction supervision by itself does not translate into improved downstream QA performance. Subsequent RL is important for leveraging the learned reference-frame knowledge in perspective-taking reasoning.
(2) Rendering provides consistent additional benefits across benchmarks. We conduct additional experiments on 3DSRBench and ViewSpatial-Bench, comparing models trained and evaluated consistently with or without rendering:
Rendering in both RL and Eval consistently improves performance by 2.45–5.25 pp across all four tasks. On OmniSpatial-PT, while Allo slightly favors the no-rendering variant (54.63% vs. 54.04%), rendering substantially improves Hypo from 49.40% to 55.66% (+6.26 pp).
(3) The model actively uses rendered visual cues. As shown in our answer of Q3, we conduct an additional experiment that reverses the rendered left axis by 180° at evaluation, keeping the trained model and all other settings unchanged. The 180° flip causes a 12.44 pp overall drop, with substantial degradation on Allo (−15.74 pp) and Hypo (−13.49 pp), while Ego remains largely unaffected.
This, along with Fig. 4 in our paper, provides direct evidence that LeRF's predictions depend on the rendered directional cues, rather than relying exclusively on internal spatial knowledge acquired through SFT and RL.
Overall, these results suggest complementary roles: SFT provides reference-frame prediction capabilities, RL improves downstream reasoning, and explicit rendering offers additional visual guidance. Our contribution lies in learning to construct and reason with reference frames, with rendering providing consistent additional benefits across multiple benchmarks.