acceptodds
Under review as a conference paper at ICLR 2027

LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

Abstract

Perspective taking is a fundamental component of spatial intelligence, requiring models interpret spatial relations from a specified viewpoint, such as that of another entity or an imagined observer. Despite the increasing spatial reasoning capabilities of Vision-Language Models (VLMs), they still struggle with perspective taking, often defaulting to the camera viewpoint when a query requires reasoning from a different perspective. We introduce Learning Reference Coordinate Frames for Perspective Taking (LeRF), a framework that trains VLMs to construct and use explicit reference frames for viewpoint-dependent reasoning. Given an image and a query, LeRF decides whether a coordinate frame is necessary. If so, it grounds the reference entity and predicts the frame's origin and entity-centered reference frame. A lightweight renderer overlays the frame onto the image, enabling subsequent reasoning over these visual cues without external perception models or explicit 3D reconstruction. To learn this process, we first perform supervised fine-tuning to teach selective tool invocation and reference coordinate frame prediction, followed by reinforcement learning on spatial VQA pairs to improve frame-guided reasoning. Across diverse perspective-taking benchmarks, LeRF consistently improves over its backbone and achieves strong performance against existing open-source methods. Further evaluations also show improved reference-frame grounding and orientation estimation, supporting the effectiveness of learned reference frames for viewpoint-dependent reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Perspective Taking, Vision Language Models, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NBwDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets perspective-taking questions for VLMs, such as "from person X's viewpoint, what is on their left." LeRF trains Qwen3.5-4B/9B to first decide whether a reference frame is needed. If so, the model outputs a tool call with the reference entity and four 2D points: the projected origin and the front, left, and up axis endpoints. A deterministic renderer draws these axes on the image, and the model then reasons over the rendered image with the numeric coordinates hidden. Training has two stages:

  • SFT (LoRA): frame labels derived from ImageNet3D, Omni6DPose, and BEDLAM poses through first-order projection, plus about 20% "no tool call" examples.
  • GRPO with a correctness-only reward: on MultihopSpatial and SpatialReasoner-RL, with frame tokens excluded from the policy gradient.

LeRF-9B improves over Qwen3.5-9B on OmniSpatial-PT Allo/Hypo (+6.9 / +11.1), ViewSpatial person-perspective tasks, and 3DSRBench orientation. It loses about 6 points on egocentric questions. On EMDB/Objectron, its projected-frame estimation is far better than the base model and roughly comparable to OrientAnything-V2.

Critical strengths
  1. The idea is intuitive and well motivated. Making the requested viewpoint explicit as a drawn coordinate frame turns implicit mental rotation into something visible. The Fig. 1 pilot with OrientAnything-V2 overlays is a nice motivating experiment.
  2. Predicting a 2D-projected frame instead of full 3D geometry is a pragmatic choice that avoids the error-prone reconstruction used by SpatialReasoner and APC. The out-of-plane symbol handling (Appendix A) is careful.
  3. Gains on the viewpoint-dependent splits are consistent for both model sizes. LeRF-9B is competitive with or better than the proprietary models on OmniSpatial Hypo and ViewSpatial P-Obj (Table 1).
  4. The paper evaluates frame estimation directly on EMDB and Objectron (Table 2), not only downstream QA, and runs useful diagnostics: tool-call selectivity (Fig. 3), frame perturbation (Fig. 4), and the render on/off ablation (Table 3). The failure analysis (Appendix D.2) is honest.
Critical weaknesses
  1. The paper's own ablations suggest the rendered frame contributes little. The central claim is that rendering the frame onto the image is what helps. But Table 3 shows that with SFT initialization, training and evaluating without rendering gives 57.18% overall vs. 57.97% with rendering, a 0.8-point gap. Allo is actually higher without rendering (54.63 vs. 54.04). Most of the gain over the base model (52.76%) therefore comes from frame-prediction SFT, acting as an auxiliary task, plus RL, not from the visual overlay.

    The perturbation test points the same way. Rotating the rendered frame by 90° swaps the meaning of front and left, which should be catastrophic for left/right questions if the model were reading the axes. Yet overall accuracy drops only 57.97 → 54.05, and Allo barely moves (Fig. 4). Rendering does help hypothetical questions (Hypo 49.4 → 55.7), but that is a narrower claim than the title and contributions make. The paper should reframe accordingly. It should also add a cleaner perturbation (e.g., a 180° flip, or swapping the front and left labels) with per-split results.

  2. The most direct baseline is missing. The motivating experiment is Qwen3.5-9B with an OrientAnything-V2 frame overlay (Fig. 1), and Appendix B.4 runs exactly this pipeline on 415 OmniSpatial cases. Yet Table 1 does not include it. It is the natural comparison for a learned frame: the same overlay from an external estimator, both zero-shot and with the same RL stage. Table 2 shows OrientAnything-V2 is comparable or better on objects (Objectron All@30: 83.2 vs. 72.5). Without this baseline, it is unclear whether learning the frame inside the VLM is better than calling an off-the-shelf orientation model, which is the main design argument of the paper. Related work on orientation-aware VLMs built on OrientAnything (e.g., SoFar, 2025) should also be discussed.

  3. Hypothetical-viewpoint frames are never directly supervised. The SFT frame labels come from pose datasets, so they are always anchored to a visible entity with a known pose. In RL, frame-prediction tokens are excluded from the policy gradient. So the frame for an imagined observer ("if you're seated on the sofa…") never receives a training signal, even though Hypo is where the paper reports the largest gain and the clearest benefit from rendering. How does the model produce these frames, and how accurate are they? An evaluation of hypothetical-frame quality, even a small manually annotated set, is needed to support the claim that LeRF "establishes" frames for imagined viewpoints. It is also unclear whether the Call/NoCall decision token is included in the RL gradient.

Questions for authors
  1. How does Qwen3.5-9B with OrientAnything-V2 frame overlays perform on Table 1 benchmarks, both zero-shot and after the same RL stage?
  2. How are frames for hypothetical observers produced, given that neither SFT nor RL supervises them? Can you report their accuracy on a small annotated set?
  3. What happens under a 180° frame flip or a front/left label swap, broken down by Ego/Allo/Hypo? Why does a 90 rotation leave Allo nearly unchanged?
  4. Why does Ego accuracy drop by about 6 points when the tool is almost never called on Ego questions? How does LeRF do on a general VLM benchmark compared with the base model?
11 comments
AuthorsOpenReview ↗

Response to Reviewer NBwD [1/5]

We are grateful to the reviewer for the extensive review. We address your questions point by point below.

W1-rendered frame contribution: The paper's own ablations suggest the rendered frame contributes little. Most of the gain over the base model (52.76%) therefore comes from frame-prediction SFT, acting as an auxiliary task, plus RL, not from the visual overlay.

Thanks for pointing this out. We agree that "indispensable" was an overly strong characterization, and have revised the Introduction accordingly to clarify that rendering provides additional visual guidance rather than being strictly necessary for perspective-taking reasoning.

We would like to clarify the distinct contributions of these components.

(1) SFT alone does not improve downstream reasoning. We additionally evaluate the model after SFT but before RL:

ModelEgoAlloHypoOverall
Qwen3.5-9B (Base)80.2047.1344.5852.76
After SFT69.8034.4734.2240.86
After SFT + RL (LeRF-9B)74.3154.0455.6657.97

SFT alone reduces overall accuracy from 52.76% to 40.86%, indicating that frame-prediction supervision by itself does not translate into improved downstream QA performance. Subsequent RL is important for leveraging the learned reference-frame knowledge in perspective-taking reasoning.

(2) Rendering provides consistent additional benefits across benchmarks. We conduct additional experiments on 3DSRBench and ViewSpatial-Bench, comparing models trained and evaluated consistently with or without rendering:

BenchmarkTaskRL & Eval: No RenderRL & Eval: RenderΔ (pp)
3DSRBenchOri50.8353.76+2.93
3DSRBenchM-Obj47.8450.29+2.45
ViewSpatial-BenchP-Obj56.6661.91+5.25
ViewSpatial-BenchP-Rel70.9674.23+3.27

Rendering in both RL and Eval consistently improves performance by 2.45–5.25 pp across all four tasks. On OmniSpatial-PT, while Allo slightly favors the no-rendering variant (54.63% vs. 54.04%), rendering substantially improves Hypo from 49.40% to 55.66% (+6.26 pp).

(3) The model actively uses rendered visual cues. As shown in our answer of Q3, we conduct an additional experiment that reverses the rendered left axis by 180° at evaluation, keeping the trained model and all other settings unchanged. The 180° flip causes a 12.44 pp overall drop, with substantial degradation on Allo (−15.74 pp) and Hypo (−13.49 pp), while Ego remains largely unaffected.

This, along with Fig. 4 in our paper, provides direct evidence that LeRF's predictions depend on the rendered directional cues, rather than relying exclusively on internal spatial knowledge acquired through SFT and RL.

Overall, these results suggest complementary roles: SFT provides reference-frame prediction capabilities, RL improves downstream reasoning, and explicit rendering offers additional visual guidance. Our contribution lies in learning to construct and reason with reference frames, with rendering providing consistent additional benefits across multiple benchmarks.

AuthorsOpenReview ↗

Response to Reviewer NBwD [2/5]

W1-perturbation, Q3: What happens under a 180° frame flip or a front/left label swap, broken down by Ego/Allo/Hypo? Why does a 90° rotation leave Allo nearly unchanged?

Thank you for this insightful question. We further analyzed the model's behavior under 90° rotations and conducted an additional experiment with a 180° left-axis flip.

Why does 90° rotation leave Allo nearly unchanged? By inspecting the model's chain-of-thought (CoT) traces, we found that the model can sometimes detect and correct obvious inconsistencies in the rotated coordinate frame. For example, when the up-axis points horizontally, the model explicitly notes: "The blue arrow (Up) points left. This is confusing." It then attempts to reconstruct the correct coordinate orientation before proceeding with spatial reasoning. This self-correction helps explain why certain 90° rotations cause only limited degradation on Allo. We also include representative CoT examples in Appendix C.5, highlighted in blue\textcolor{blue}{\text{blue}}.

Additional experiment: 180° left-axis flip. To further examine the model's dependence on the rendered coordinate frame, we conducted an additional experiment in which the left axis is reversed by 180°, with all other settings unchanged. The results are shown below (accuracy in %).

SettingOverallEgoAlloHypo
Original frame57.9774.3154.0455.66
180° left-axis flip45.5374.9038.3042.17
Δ (90° all-axes rotation)−3.92-0.98−3.83−7.95
Δ (180° left-axis flip)−12.44+0.59−15.74−13.49

The 180° flip leads to a 12.44 percentage-point drop in overall accuracy, with particularly substantial degradation on Allo (−15.74 pp) and Hypo (−13.49 pp). In contrast, Ego remains largely unaffected (+0.59 pp). This distinction is consistent with the nature of these tasks: Allo and Hypo require reasoning about reference-dependent coordinate systems, making them more sensitive to incorrect frame orientations, whereas Ego primarily involves reasoning from the observer's own perspective.

Overall, these findings suggest that relatively minor degradation to certain 90° rotations can arise from explicit error detection and self-correction. The substantial degradation under the 180° flip further supports the importance of correctly oriented coordinate frames for LeRF's spatial reasoning.

AuthorsOpenReview ↗

Response to Reviewer NBwD [3/5]

W2, Q1: How does Qwen3.5-9B with OrientAnything-V2 frame overlays perform on Table 1 benchmarks, both zero-shot and after the same RL stage? Related work on orientation-aware VLMs built on OrientAnything (e.g., SoFar, 2025) should also be discussed.

Thank you for pointing out this important missing baseline. We conducted additional experiments comparing LeRF against Qwen3.5-9B (thinking) with OrientAnything-V2 (OA-V2) overlays, both zero-shot and after RL fine-tuning, on all three benchmarks in Table 1. We also evaluated SoFar (Qi et al., 2025) as an additional orientation-aware baseline.

Experimental setup. For the OA-V2 baselines, we use GPT-5.6-Terra to identify the reference entity and its bounding box, estimate its orientation using OA-V2, and render the corresponding coordinate frame with axis descriptions. Ego questions retain the original image, while Allo/Hypo questions receive the rendered frame when requested by the model in the first turn. We evaluate (1) zero-shot Qwen3.5-9B with OA-V2 overlays, and (2) Qwen3.5-9B fine-tuned with RL using OA-V2-annotated training data. During RL, Ego samples use the original image, while Allo/Hypo samples use OA-V2 overlays. The answer process is single-turn without the tool. We additionally adapt SoFar using GPT-4o and Qwen3.5-9B Thinking for the evaluated benchmarks.

Additional experimental results (%):

MethodOriM-ObjP-ObjP-RelEgoAlloHypo
Qwen3.5-9B48.1748.6656.2365.5180.2047.1344.58
+ OA-V2 (Zero-shot)50.7549.3660.1668.7580.2851.7052.77
+ OA-V2 (RL)47.3748.9859.7766.9378.1855.2950.12
SoFar[1]41.4148.6148.8061.0577.4545.9640.96
LeRF-9B (Ours)53.7650.2961.9174.2374.3154.0455.66

Ori and M-Obj denote the subtasks of 3DSRBench; P-Obj and P-Rel denote those of ViewSpatial-Bench; Ego, Allo, and Hypo denote those of OmniSpatial-PT.

Analysis. External OA-V2 overlays substantially improve zero-shot perspective taking, validating our motivation. Nevertheless, LeRF outperforms this baseline on all six non-Ego metrics. Compared with OA-V2 + RL, LeRF achieves higher accuracy on five of six non-Ego metrics. Although OA-V2 + RL slightly outperforms LeRF on Allo (55.29% vs. 54.04%), this 1.25 pp advantage comes at the cost of an additional external orientation estimator, which requires loading an additional model with a 5.05 GB checkpoint, as well as introducing inference latency in grounding and cropping the reference entity in advance for OrientAnything model. In contrast, LeRF predicts reference frames within the VLM and requires only a lightweight deterministic renderer, avoiding external perception models. LeRF also consistently outperforms the adapted SoFar baseline across all six non-Ego metrics.

Conclusion. Although OA-V2 achieves higher standalone object-orientation accuracy on Objectron (paper Table 2), this advantage does not consistently translate into better downstream perspective-taking performance. Our results suggest that learning to construct question-conditioned reference frames and reason with their rendered cues is more effective and generalizable overall than relying on externally estimated orientations, while eliminating the need for external perception models at inference time.

Reviewer_5MjoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LeRF addresses spatial questions that require an object’s or person’s viewpoint rather than the camera’s. A VLM decides whether to use a reference frame, then locates the entity and predicts a projected origin and front, left, and up axes. A fixed renderer overlays these axes on the image, which the model uses to answer without seeing the coordinates. Training combines supervised frame prediction with GRPO based on answer correctness. LeRF-9B improves every non-camera-view subtask over Qwen3.5-9B; on OmniSpatial Hypo, accuracy rises from 44.58% to 55.66%. It also improves projected-axis accuracy. Ablations examine SFT initialization and the use of rendering during training and evaluation.

Critical strengths
  • Consistent same-backbone gains. LeRF-9B improves all six non-camera-view subtasks over Qwen3.5-9B, including gains of 11.08 points on Hypo and 8.72 on P-Rel (Table 1). The case therefore does not rely on weak specialist baselines.

  • Diagnostics show real use of the frame. Projected-axis accuracy on EMDB rises from 1.11% to 43.12% (Table 2), and SFT initialization beats direct RL at equal steps (Table 3). Rotating the rendered frame by 90° costs 7.95 points on Hypo.

  • The paper is well-presented.

Critical weaknesses
  • Rendering is not shown to be indispensable. Although the paper calls rendering an indispensable anchor (p.2), training and evaluating without rendering reach 57.18 versus 57.97 with rendering (Table 3). This 0.79-point gap does not support the claim.

  • The rendering effect is mixed. Adding rendering only at evaluation raises Hypo from 49.40 to 55.42 but lowers overall accuracy from 57.18 to 56.93 (Table 3). The rotation test shows sensitivity to the frame, not that it is necessary.

  • The training data cannot be audited. The final SFT size, the splits per source, and any overlap check between training and test images are not reported. App. B.3 lists the categories but not the coordinate mappings described in §3.4.

  • Scoring and baselines are unclear. The paper does not say how boxed answers are matched to labels, or whether evaluation and RL share a scorer (Table 4). The APC outputs in Fig. 7 look off-topic or anomalous, so that baseline needs checking.

  • Validation data. Table 4 lists a rollout temperature for validation, but the RL validation set is never described. Please state its source, its size, and how it was used during training.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5Mjo [1/2]

We are grateful to the reviewer for the extensive review. We address your questions point by point below.

W1: Rendering is not shown to be indispensable. Although the paper calls rendering an indispensable anchor (p.2), training and evaluating without rendering reach 57.18 versus 57.97 with rendering (Table 3). This 0.79-point gap does not support the claim.

Thank you for pointing this out! We partially agree that the 0.79 pp improvement on OmniSpatial-PT alone is insufficient to establish rendering as strictly indispensable, and we have revised this wording in the Introduction to avoid overstating the claim.

(1) Clarification of the rendering ablation in Table 3. As shown in Table 3, for the same RL-with-rendering checkpoint, disabling rendering at evaluation reduces overall accuracy from 57.97% to 55.15%, with larger drops on Allo (54.04% → 50.37%) and Hypo (55.66% → 50.60%). This controlled comparison isolates the effect of rendering at inference and demonstrates that the learned policy benefits from the rendered reference frames. In contrast, the reviewer-highlighted comparison (57.18% vs. 57.97%) involves models trained under different rendering conditions, where the no-rendering model has adapted to reasoning without rendered frames.

(2) Additional experiments on broader benchmarks. To further investigate whether the benefit of rendering generalizes beyond OmniSpatial-PT, we conducted additional experiments on 3DSRBench and ViewSpatial-Bench, comparing models trained and evaluated consistently with or without rendering:

BenchmarkTaskRL & Eval: No RenderRL & Eval: RenderΔ (pp)
3DSRBenchOri50.8353.76+2.93
3DSRBenchM-Obj47.8450.29+2.45
ViewSpatial-BenchP-Obj56.6661.91+5.25
ViewSpatial-BenchP-Rel70.9674.23+3.27

Rendering in both RL and Eval consistently improves performance by 2.45–5.25 pp across all four tasks.

Overall, while reference-frame supervision enables perspective-taking reasoning even without explicit rendering, our controlled ablations and additional benchmark results demonstrate that explicit rendering provides consistent benefits and yields the best overall performance across the evaluated benchmarks.

W2: The rendering effect is mixed. Adding rendering only at evaluation raises Hypo from 49.40 to 55.42 but lowers overall accuracy from 57.18 to 56.93 (Table 3). The rotation test shows sensitivity to the frame, not that it is necessary.

Thank you for pointing this out.

(1) Clarification of the rendering ablation. The reviewer-highlighted comparison (57.18% → 56.93%) adds rendering only at evaluation to a model trained without rendering during RL. Since the model was not trained to utilize rendered visual cues, this comparison does not isolate the benefit of rendering for a rendering-trained policy. As shown in Table 3, for the same RL-with-rendering checkpoint, disabling rendering at evaluation reduces overall accuracy from 57.97% to 55.15%, with larger drops on Allo (54.04% → 50.37%) and Hypo (55.66% → 50.60%). This demonstrates that the learned policy benefits from rendered reference frames.

(2) Additional experiments on broader benchmarks. To further validate the effectiveness of rendering, we conducted additional experiments on 3DSRBench and ViewSpatial-Bench, comparing models trained and evaluated consistently with or without rendering. As reported in our response to W1, rendering consistently improves performance by 2.45–5.25 pp across all four tasks, providing further evidence of its benefits beyond OmniSpatial-PT.

Regarding the rotation test, we agree that it establishes sensitivity to rendered frames rather than their strict necessity. Together with the controlled ablations and additional benchmarks, however, it supports the conclusion that the model actively utilizes rendered reference frames for spatial reasoning.

Overall, while frame-supervised training enables effective perspective-taking even without explicit rendering, our additional experiments demonstrate that rendering provides consistent performance gains across multiple benchmarks and achieves the best overall results among the evaluated configurations.

AuthorsOpenReview ↗

Response to Reviewer 5Mjo [2/2]

W3: The training data cannot be audited. The final SFT size, the splits per source, and any overlap check between training and test images are not reported. App. B.3 lists the categories but not the coordinate mappings described in §3.4.

We thank the reviewer for pointing this out.

SFT size and splits. The final SFT set contains 162,249 training samples from 63,867 unique images: BEDLAM 70,654, ImageNet3D 13,467 and Omni6DPose-SOPE 78,128. Of these, 129,799 are tool-call samples and 32,450 (20.0%) are no-tool-call samples. ImageNet3D is upsampled 2× during training (175,716 samples per epoch, 2 epochs). A held-out validation split of 9,450 samples is image-disjoint from training. LeRF-4B and LeRF-9B use the identical set.

Train/test overlap. We compared every SFT and RL training image against every evaluation image, using exact byte hashes, perceptual hashes (64-bit dHash, Hamming ≤ 4) with manual inspection of all candidates, and source-ID matching for COCO-based benchmarks.

  • SFT data overlaps with OmniSpatial-PT and ViewSpatial-Bench in 0 images, and with 3DSRBench in 1 image (an image indexed by both ImageNet and COCO).
  • For RL, OmniSpatial-PT and ViewSpatial-Bench (COCO val2017) have no overlap. MultihopSpatial and 3DSRBench both draw about 20% of images from COCO train2017: 235 of the 1,210 3DSRBench Orientation/Multi-Object images also appear in our RL set, with different questions.

Coordinate mappings.

  • Omni6DPose-SOPE: a signed-axis table for every adopted category.
  • ImageNet3D: its shared category-level canonical frame.
  • BEDLAM: the SMPL-X body frame derived from joint positions.

W4: Scoring and baselines are unclear. The paper does not say how boxed answers are matched to labels, or whether evaluation and RL share a scorer (Table 4). The APC outputs in Fig. 7 look off-topic or anomalous, so that baseline needs checking.

Scoring protocals:

RL training and evaluation use different scoring procedures.

During RL training, we use a strict, deterministic rule-based scorer. The final answer must be enclosed in \boxed{}. We then use regular expressions to match the extracted answer against the ground truth, accepting three formats: (1) option letter only, (2) option letter with option text, and (3) option text only. Matching is case-insensitive.

During evaluation, we follow VLMEvalKit's multiple-choice evaluation protocol, which first attempts rule-based answer extraction and falls back to GPT-based choice extraction when heuristic matching fails.

APC outputs in Fig. 7:

We clarify that Fig. 7 presents selected excerpts rather than complete reasoning traces, which may make some APC outputs appear disconnected from the original questions.

Specifically, APC constructs a coarse 3D scene abstraction and represents objects as color-coded cubes in its visual prompting stage. Thus, "green cube" refers to an object in the abstracted scene, rather than an unrelated hallucination. APC also includes a response-mapping step that converts answers from the abstract representation back to the original question, explaining references to previous responses. The phrase "L/R dichotomy" simply refers to the left/right answer choices.

W5: Validation data. Table 4 lists a rollout temperature for validation, but the RL validation set is never described. Please state its source, its size, and how it was used during training.

Thank you for pointing out this missing detail.

Validation data. We use a held-out validation set of 620 questions from two sources: (1) 500 questions randomly sampled from MultihopSpatial-RL-Val (319 ego / 181 exo); and (2) 120 questions from SpatialReasoner (70 ego / 50 exo), obtained by randomly partitioning its official training data into training and validation subsets. We ensure no image overlap between the RL training and validation sets.

Usage during RL. Validation is performed with one rollout per question at temperature 0.6 (Table 4). We use the same reward function as training to monitor accuracy, tool-call rate, no-tool routing rate, and truncation rate. The validation set is mainly used for monitoring the GRPO training.


Hope our explanation and experiments can address your concerns. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you!

Best regards,

All Authors

Thanks for the detailed response. The new 3DSRBench and ViewSpatial-Bench results show clear gains from rendering when training and evaluation settings match, and with the softer wording in the introduction, this resolves my main concern. The data statistics, scoring protocol, and validation set are now clear too. Please include these results in the final version and tone down the claims about rendering. I'm raising my score to weak acceptance.

Thank you very much for your response! We are very glad that your main concerns have been addressed, and we sincerely appreciate your decision to raise the score.

Your insightful questions and suggestions have been truly valuable in improving our work. We will include all the new results in our final version and tone down our claims about rendering.

Best regards,

All Authors

Reviewer_QTB4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LeRF, a framework that trains VLMs to construct and use explicit reference frames for viewpoint-dependent reasoning. The authors curate a dataset with reference-frame supervision and use SFT to achieve reference-entity grounding, projected frame prediction, and selective tool invocation. RL training on spatial VQA data enables the model to reason using reference frames rendered and overlaid onto the input image. Experiments against baselines and the ablation study demonstrate the effectiveness and advantages of the proposed method.

Critical strengths
  1. The paper clearly motivates the perspective-taking problem and presents a coherent rationale for using explicit reference frames. The experimental results and diagnostic analyses provide supporting evidence for the proposed approach.

  2. The paper presents a practical data-curation pipeline that converts existing object and human pose annotations into projected reference-frame supervision for training VLMs.

Critical weaknesses
  1. Performance on the OmniSpatial-PT Ego subset decreases after training, from 80.20% to 74.31% for the 9B model. Although the authors acknowledge this trade-off, the underlying causes are not sufficiently analyzed.

  2. The reasoning depends heavily on the predicted frame: rotating the rendered frame by 90° causes a large accuracy drop (Fig. 4). Yet there is no analysis of how often the reference entity or the frame orientation is predicted wrongly, or what happens with occluded, ambiguous, or multiple candidate entities.

Minor:

  • The extra inference cost (the tool-call decision, rendering, and re-encoding the overlaid image) is not reported, so the efficiency advantage over multi-stage 3D pipelines claimed in the introduction is not quantified.
Questions for authors
  1. Could the authors explain the performance drop on the Ego subset? Reporting Ego accuracy after SFT and after RL would help identify which training stage contributes to this degradation.
  2. How often is the reference entity or the frame orientation predicted incorrectly, and how does accuracy change in those cases?
  3. What is the inference latency and token overhead of LeRF compared with the backbone and the agentic 3D baseline?
3 comments
AuthorsOpenReview ↗

Response to Reviewer QTB4 [1/2]

We sincerely thank the reviewer for the constructive comments. We address your questions point by point below.

W1, Q1: Could the authors explain the performance drop on the Ego subset? Reporting Ego accuracy after SFT and after RL would help identify which training stage contributes to this degradation.

Thank you for this important question. We conducted additional stage-wise evaluations and error analysis to investigate the Ego performance drop.

(1) The Ego degradation primarily originates from SFT. The following table reports stage-wise performance on OmniSpatial-PT and coordinate-frame estimation benchmarks.

ModelEgoAlloHypoOverallEMDB All@15°OmniNOCS-Objectron All@15°
Qwen3.5-9B80.2047.1344.5852.761.110.52
After SFT69.8034.4734.2240.8643.5250.32
After SFT + RL (LeRF-9B)74.3154.0455.6657.9743.1249.58

SFT substantially improves coordinate-frame estimation, raising All@15° from 1.11% to 43.52% on EMDB and from 0.52% to 50.32% on Objectron. However, this comes with a decline in VQA performance, including a 10.40 pp drop on Ego. Subsequent RL preserves the learned frame estimation capabilities while substantially improving reasoning performance, recovering 4.51 pp on Ego and improving Allo and Hypo by 19.57 and 21.44 pp, respectively. These results indicate that the Ego degradation primarily arises during SFT, while RL partially mitigates it and improves perspective-taking performance without substantially compromising the frame estimation capabilities acquired during SFT.

(2) Error analysis suggests a regression in general visual reasoning. We did thorough inspection of reasoning traces, which reveals errors mostly in object counting, spatial relationship judgments, and action understanding, rather than perspective-taking. We include representative failure cases in the Appendix C.4 and are highlighted with blue\textcolor{blue}{\text{blue}}.

Overall, the Ego degradation largely emerges during SFT and is partially recovered by RL. Our analysis suggests that the remaining drop is associated with visual reasoning failures rather than the proposed coordinate-frame rendering mechanism.

AuthorsOpenReview ↗

Response to Reviewer QTB4 [2/2]

W2, Q2: There is no analysis of how often the reference entity or the frame orientation is predicted wrongly, or what happens with occluded, ambiguous, or multiple candidate entities. How often is the reference entity or the frame orientation predicted incorrectly, and how does accuracy change in those cases?

Thank you for highlighting the need for a more detailed error analysis. We conducted additional human evaluation and analyzed how reference-frame prediction errors affect downstream reasoning.

(1) Human evaluation of reference-frame correctness. We randomly sampled 74 OmniSpatial-PT Allo cases where LeRF-9B invoked the frame-rendering tool. Three independent, non-author annotators (an undergraduate student, a graduate student, and a professor) independently assessed reference-entity grounding and projected axis orientations, using a 45° angular tolerance.

CriterionUndergraduateGraduateProfessor
Reference Entity66/74 (89.2%)66/74 (89.2%)66/74 (89.2%)
Front (F)64/74 (86.5%)64/74 (86.5%)65/74 (87.8%)
Left (L)63/74 (85.1%)64/74 (86.5%)62/74 (83.8%)
Up (U)68/74 (91.9%)69/74 (93.2%)67/74 (90.5%)
All Axes (F/L/U)60/74 (81.1%)61/74 (82.4%)60/74 (81.1%)

(2) Error breakdown and downstream accuracy. We further analyzed VQA accuracy conditioned on reference-entity and frame correctness, and categorized the 74 cases into challenging scenarios using GPT-5.6-sol-high annotation.

Scenario#CasesOverall Acc.Entity ✓, Frame ✓ (#Cases / Acc.)Entity ✓, Frame ✗ (#Cases / Acc.)Entity ✗ (#Cases / Acc.)
Occluded3847.4%30 / 53.3%4 / 0.0%4 / 50.0%
Ambiguous1127.3%6 / 33.3%3 / 0.0%2 / 50.0%
Multiple candidate entities3946.2%29 / 51.7%5 / 20.0%5 / 40.0%
None of the above1384.6%9 / 100%3 / 33.3%1 / 100%
Overall7452.7%54 / 61.1%12 / 16.7%8 / 50.0%

Each cell reports the number of cases and corresponding VQA accuracy. Scenario categories are not mutually exclusive.

The reference entity is correctly identified in 89.2% (66/74) of cases, among which 81.8% (54/66) have correctly oriented frames. Across challenging scenarios, VQA accuracy is 47.4% for occluded cases, 27.3% for ambiguous cases, and 46.2% for cases with multiple candidate entities, compared with 84.6% for cases without these challenges. Importantly, when the reference entity is correct, VQA accuracy decreases from 61.1% to 16.7% with an incorrectly oriented frame, further supporting that the model actively utilizes the rendered coordinate axes for spatial reasoning.

Overall, these findings complement the reference-frame estimation results in paper Table 2, where LeRF-9B substantially outperforms its backbone on both EMDB and OmniNOCS-Objectron. Together, they demonstrate LeRF's ability to predict meaningful reference frames while identifying challenging visual scenarios and frame prediction errors as important factors affecting downstream reasoning.

Weakness-Minor, Q3: What is the inference latency and token overhead of LeRF compared with the backbone and the agentic 3D baseline?

Thank you for raising this question. We have already provided a detailed inference efficiency comparison in Appendix C.2 (Table 5). All models are evaluated under single-request inference on the same RTX PRO 6000 GPU.

Compared with the Qwen3.5-9B Thinking backbone, LeRF-9B reduces average inference latency from 77.0s to 51.5s (33.1% reduction) and average output tokens from 6,603 to 3,757 (43.1% reduction). Compared with the agentic 3D baseline APC-Qwen3.5-9B, LeRF achieves 7.1× lower latency (51.5s vs. 367.5s) and 7.0× fewer output tokens (3,757 vs. 26,140).

These results indicate that LeRF's explicit reference-frame rendering does not introduce prohibitive inference overhead. Instead, LeRF achieves substantially lower end-to-end latency and token consumption than both the thinking backbone and the agentic 3D baseline.


Hope our explanation and experiments can address your concerns. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you!

Best regards,

All Authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer QTB4,

As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.

We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!

Best regards,

All Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary: Additional Experiments, Analyses, and Clarifications

Dear Reviewers and ACs,

We sincerely thank all reviewers for their constructive feedback. We have addressed the concerns, conducted additional experiments, and revised the manuscript, with changes highlighted in blue\textcolor{blue}{\text{blue}}.

Summary

We are encouraged that reviewers recognize the following strengths:

  • Well-Motivated, Intuitive Idea: clear motivation and coherent rationale for explicit reference frames (QTB4); an intuitive idea supported by the Fig. 1 pilot (NBwD).
  • Consistent Gains: same-backbone gains on all six non-Ego subtasks (5Mjo); gains for both model sizes, competitive with or better than proprietary models on Hypo and P-Obj (NBwD).
  • Practical Design: a practical frame-supervision pipeline (QTB4); projected 2D frames instead of error-prone 3D reconstruction (NBwD).
  • Diagnostics and Presentation: diagnostics showing real use of the frame (5Mjo); direct frame-estimation evaluation and honest failure analysis (NBwD); clear presentation (all reviewers).

Additional Experiments

  • Rendering Effectiveness (5Mjo, NBwD): On 3DSRBench and ViewSpatial-Bench, training and evaluating with rendering improves all four tasks by 2.45–5.25 pp. This complements Table 3, where disabling rendering for the same checkpoint reduces OmniSpatial-PT accuracy from 57.97% to 55.15%.
  • Orientation-Aware Baselines (NBwD): LeRF outperforms OrientAnything-V2 overlays on all six non-Ego metrics zero-shot and on five of six after RL, and SoFar on all six, without any external orientation model at inference.
  • Human Evaluation of Frames (QTB4, NBwD): Three non-author annotators evaluated 74 tool-invoking cases each for Allo and Hypo (45° tolerance). Reference entities are correct in 89.2% (Allo) and 78.4% (Hypo) of cases, and all three axes in 81.5% and 75.2%. We also analyze downstream accuracy under occlusion, ambiguity, and multiple candidates.
  • 180° Left-Axis Flip (NBwD): Overall accuracy drops by 12.44 pp (Allo −15.74, Hypo −13.49), further showing that the model uses the rendered directional cues.
  • Training Stages and General Ability (QTB4, NBwD): The Ego drop mainly arises from SFT and is partially recovered by RL. On MMMU, MMBench, and MMStar, LeRF stays within −0.40 to +0.33 pp of the backbone.

Key Clarifications

  • Roles of SFT, RL, and Rendering (5Mjo, NBwD): SFT learns frame prediction, RL improves reasoning, and rendering adds visual guidance. The Introduction no longer calls rendering indispensable.
  • Hypothetical Frames (NBwD): They arise from combining pretrained grounding priors with the SFT-learned pose-to-axes mapping.
  • Reproducibility (5Mjo): Full SFT composition (162,249 samples), validation splits, train/test overlap checks, and coordinate mappings.
  • Scoring, Baselines, and Protocols (5Mjo, NBwD): RL and evaluation scorers, APC outputs in Fig. 7, the RL validation set, and gradient masking of tool-call and frame tokens.
  • Efficiency (QTB4): 33.1% lower latency than the thinking backbone and 7.1× lower than the agentic 3D baseline.

Best regards,
All Authors

AuthorsOpenReview ↗

Rebuttal Summary for AC

Dear Area Chair,

Following our rebuttal and subsequent discussions, all three reviewers now rate the submission at 3 (Weak Acceptance). Both 5Mjo and NBwD raised their scores to 3: 5Mjo explicitly confirmed that their main concern was resolved, while NBwD acknowledged that most concerns had been resolved. QTB4, who initially rated the paper at 3, did not provide a follow-up assessment.

To facilitate your assessment, we summarize below how each concern was addressed and whether its resolution was acknowledged by the reviewers. We group overlapping concerns by topic to avoid repetition. The table below organizes all concerns by topic and summarizes their resolution status. Several of QTB4's concerns overlap with those evaluated by NBwD, while others were addressed through additional analyses specific to QTB4. All contents in the table can be directly grounded to original comments.

Issue / ThemeReviewersResponse (Details in Rebuttal Summary)Final Status
1. Rendering Contribution & Perturbations5Mjo (W1, W2), NBwD (W1, Q3)Added rendering ablations across two additional benchmarks, a 180° axis-flip experiment, 90° rotation analysis, and revised claims.Resolved. Explicitly confirmed by both 5Mjo and NBwD.
2. External Orientation BaselinesNBwD (W2/Q1)Added OrientAnything-V2 (zero-shot and RL-trained) and SoFar comparisons.Resolved. Explicitly confirmed by NBwD, with minor caveats about performance margins and Ego accuracy.
3. Hypothetical Frame PredictionNBwD (W3/Q2)Added human evaluation of hypothetical frames and clarified how they arise without direct supervision, including RL gradient masking.Mostly resolved. Explicitly confirmed by NBwD; the explanation of the underlying mechanism remains a hypothesis.
4. Frame/Entity Prediction Errors & Downstream ImpactQTB4 (W2/Q2)Added human evaluation of reference-entity and frame correctness, plus downstream accuracy conditioned on prediction errors and challenging scenarios (occlusion, ambiguity, multiple candidates).No response from QTB4. We believe our new analysis directly answers QTB4's questions and adressed the concerns to a large degree, though QTB4 did not reply yet.
5. Ego Performance Drop & General CapabilityQTB4 (W1/Q1), NBwD (Q4)Added stage-wise SFT/RL analysis, general VLM benchmarks, and qualitative failure analysis.Partially resolved. NBwD accepted the experimental findings while identifying the remaining causal explanation as a minor limitation. This also addresses the overlapping questions from QTB4, without their explicit confirmation.
6. Data Audit, Scoring & Validation5Mjo (W3, W4, W5)Provided SFT statistics, overlap checks, coordinate mappings, scorer/APC clarifications, and validation details.Clarifications acknowledged. 5Mjo explicitly confirmed that data statistics, scoring, and validation are now clear.
7. Inference EfficiencyQTB4 (Minor/Q3)Directed the reviewer to latency and token-overhead comparisons already reported in Appendix C.2.Addressed. Quantified in the manuscript; no follow-up from QTB4.

In summary, both reviewers who engaged in the discussion raised their scores after acknowledging that their main concerns had been addressed. For QTB4, we provided targeted analyses covering all three questions, including a dedicated evaluation of how reference-frame prediction errors affect downstream reasoning. Although QTB4 did not respond, we believe these additions address the concerns raised in their review.

We provide a detailed summary of all additional experiments and clarifications in comment "Rebuttal Summary: Additional Experiments, Analyses, and Clarifications" below. All additional experiments and analyses are included in the revised manuscript. We also incorporate the remaining minor refinements discussed with the reviewers in the final version.

We appreciate your time and consideration in reviewing our work!

Best regards,
All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.