acceptodds
Under review as a conference paper at ICLR 2027

VLMs Control the View: Learning to Plan Camera Movements through Self-Exploration

Abstract

Can VLMs predict how camera movements change the view, and learn to choose those movements to find a target view? We study this capability as view planning, separating tracking a supplied path from composing one through interaction. We introduce ViewSuite, a benchmark with three tasks that pair supplied-path understanding and interactive camera control in real 3D scenes. Across 13 VLMs, the strongest models exceed 70% on short-distance tracking tasks but reach at most 21.3% on interactive view planning. Learning Interactive View Planning is difficult when the base policy rarely succeeds: Qwen2.5-VL-7B begins at 2.5% success, and direct PPO and GRPO reach only 3.2% and 4.9% under the tested recipes. Our key observation is that every exploration trajectory, successful or not, records valid transitions between views. View Graph Distillation accumulates these transitions across episodes and connects them at nearby camera poses. It samples paths whose final views become new targets, while the recorded actions and intermediate observations provide demonstrations for those targets. The same graph supplies auxiliary view-understanding supervision. Alternating this distillation with reinforcement learning improves Qwen2.5-VL-7B's IVP success from 2.5% to 47.7% on held-out scenes. The trained checkpoint is also a better starting point for related view-understanding tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Multimodal Language Models, View Planning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vCDuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies view understanding and interactive camera planning in vision-language models. It introduces ViewSuite, a benchmark comprising Path-to-View, View-to-Path, and Interactive View Planning tasks constructed from shared initial–target view pairs. The evaluations reveal a substantial performance gap between interpreting supplied paths and planning camera movements through interaction. To improve learning from a low-success initial policy, the authors propose View Graph Distillation, which organizes self-exploration experience into a persistent graph and constructs supervision through goal relabeling and cross-trajectory path sampling. Alternating supervised fine-tuning with reinforcement learning improves Qwen2.5-VL-7B’s IVP success from 2.5% to 47.7% on held-out scenes.

Critical strengths

1.The benchmark and diagnostic analyses provide useful insights. Constructing three complementary tasks from shared view pairs provides a useful framework for studying supplied-path understanding and interactive camera control. The analyses of viewpoint distance, interaction budgets, reasoning settings, and failure types help characterize the specific difficulties current VLMs face.

2.The training framework effectively reuses exploration experience. Turning unsuccessful exploration into valid supervision substantially improves learning over direct reinforcement learning. Although this idea is related to hindsight relabeling, its application to the learning bottleneck identified in this paper is well motivated and effective. The inclusion of a trajectory-level relabeling baseline is particularly helpful for distinguishing the benefits of reusing failed trajectories from the additional contribution of the graph.

Critical weaknesses

1.Appendix A.3 shows that benchmark construction already produces executable action sequences between initial and target views. These could be used for SFT followed by RL, potentially addressing the low-success initialization problem without self-exploration. Without this baseline, the additional benefit of the proposed pipeline over simpler supervised training remains unclear.

2.Removing rotation snapping reduces the trained model's success rate from 47.8% to 19.6%. This suggests that the trained policy is more sensitive to changes in camera execution rules, which deserves further investigation.

3.The graph-based approach improves IVP success from 41.7% to 47.7% over trajectory-level relabeling. Since the comparison uses single training runs, repeated experiments would help establish the consistency of this improvement.

4.The final model shows lower P2V and V2P accuracy than the base model, despite the auxiliary view-understanding supervision. Further analysis of this performance drop would be helpful.

Questions for authors

1.Is the improvement over trajectory-level relabeling consistent across random seeds? Are any repeated-run results available to support this comparison?

2.Why does removing rotation snapping cause such a large performance drop? Could the authors provide a brief analysis to help explain it?

3.Appendix A.3 indicates that benchmark construction already produces executable action sequences connecting the initial and target views. Could the authors compare their method with a simple SFT-plus-RL baseline using these existing training-split paths as demonstrations? This would help establish the additional benefit of self-exploration and graph construction over readily available supervision.

4.Does the ScanNet camera environment enforce collision constraints with walls and objects, or can the camera move freely through scene geometry? Please clarify whether any special restrictions or handling are applied when a camera movement would result in a collision.

4 comments
AuthorsOpenReview ↗

Rebuttal-1

We thank Reviewer vCDu for recognizing the insights from our benchmark and diagnostic analyses, as well as for noting that our training framework and trajectory-level relabeling baseline effectively isolate the benefits of reusing exploration experience. We address your specific questions and concerns below.

[W1, Q3] Training on Benchmark-Construction Solver Paths vs. Self-Exploration

We thank the reviewer for this suggestion and ran the requested baseline: SFT on the training-split solver paths from Appendix A.3, followed by the same RL stage as our method, on Qwen2.5-VL-7B.

As shown in the table below, while the Oracle SFT baseline's performance plateaus around 0.30, whereas our proposed method reaches 0.47.

MethodSuccess Rate
Oracle SFT + RL (Benchmark Solver Paths)0.300.30
Ours (Self-Exploration)0.460.46
  • Why Oracle SFT falls short: The construction solver generates single open-loop paths by accessing ground-truth target poses along fixed axes. Imitating these paths does not teach the policy how to recover once the camera drifts. By contrast, our scene graph is built from states the learner actually visits, providing crucial state-recovery supervision. Our method needs only the success signal that RL already uses, and turns the agent's own exploration into demonstrations.
  • Generalizability: Training directly on solver trajectories risks fitting the procedural artifacts of benchmark construction, whereas our graph-based pipeline enables effective learning without external oracle trajectories.

[W2, Q2] Performance Drop without Rotation Snapping

The drop mainly reflects a train-test mismatch in execution rules, and our policy is still the strongest model under No-Snap. Our policy was trained only with the default interface, which snaps rotations to multiples of 30°. Removing snapping at test time changes how every rotation executes, so the policy meets camera poses it never saw during training. The zero-shot models were never adapted to either rule, which is why their drop is smaller.

ModelDefaultNo-SnapChange
Gemini 3.1 Pro21.315.7-5.6
GPT-5.416.613.0-3.6
Qwen2.5-VL-7B (untrained base)2.5n/an/a
Ours (from Qwen2.5-VL-7B)47.819.6-28.2

Without training, Qwen2.5-VL-7B reaches 2.5%, far below our 19.6% without snapping.

  • Unconstrained rotation is harder for every model, especially on Long targets (Gemini 17.4 to 9.6, GPT-5.4 7.5 to 5.2, ours 36.9 to 9.3, on par with Gemini). Without snapping, small angular errors are no longer corrected after each action. We hypothesize that they accumulate over long paths and move the camera off the 30° grid on which the target views lie.
  • The sensitivity is specific to execution rules. Under a change in appearance (3DGS re-rendering of the same scenes),our policy keeps 36.0% IVP success (47.7% under the default renderer), still above every zero-shot model.
AuthorsOpenReview ↗

Rebuttal-2

[W3, Q1] Consistency of Gains over Trajectory-Level Relabeling

The gain is consistent. We retrained both variants from scratch under an updated environment configuration (mesh rendering, ground-aligned camera actions, strict success thresholds) on the same 530 test pairs. Because the configuration changed, we compare the gap within each round.

Split (n)Run 1Run 2Pooled [95% CI]
Short (187)+2.1-1.1+0.5 [-4.9, +5.9]
Long (343)+8.1+7.3+7.7 [+3.8, +11.6]
All (530)+6.0+4.3+5.2 [+2.0, +8.3]

Δ = IVP success of view graph (ours) minus trajectory-level relabeling

Three observations show that this gap is not run-to-run noise:

  • It replicates. Across rounds, the gap changes by only 1.7 points overall and 0.8 points on Long targets, less than the sampling error of a single comparison on this test set. On Long targets, the graph variant alone solves 59 test episodes and relabeling alone solves 34 (paired McNemar p = 0.012).
  • It grows with target distance. The two variants are level on nearby targets, which a single trajectory already reaches, and the graph leads on distant ones:
View distance d< 22 to 33 to 44 to 5≥ 5
Δ (pts), re-run0.0-1.9+3.8+4.7+12.8 (p = 0.009)

[W4] P2V and V2P Accuracy of the Final Checkpoint

The drop is answer-format collapse in a checkpoint trained without auxiliary supervision, not weaker view understanding. The reported 6.0 / 20.9 come from a checkpoint trained on IVP alone; the paper wrongly attributes them to the full model, and we will correct this. A P2V score far below the 25% chance level already signals a format failure.

New analysis: format audit of the test rollouts. In this IVP-only checkpoint, about four-fifths of P2V replies and over a third of V2P replies are camera-action strings that the answer(letter) parser rejects (almost none for the base model). On replies that do give an option, accuracy is comparable to the base model.

Auxiliary supervision prevents the drift. In the paper's ablation, distilling planning alone halves single-turn accuracy, while adding the auxiliary view tasks keeps it at the base level:

Distilled supervisionP2VV2P
Untrained base30.027.1
Planning only14.011.1
+ Auxiliary view tasks (ours)28.029.0

The full model, trained with auxiliary supervision through final RL in our re-run (W3), scores 27.0 / 33.0 against 28.5 / 27.9 for the base model, with only 26 and 25 of 530 replies off-format.


[Q4] Collision Handling

The environment does not enforce collisions, by design, and re-scoring with a collision rule leaves our conclusions unchanged. ViewSuite studies viewpoint planning with a free virtual camera rather than embodied navigation. Poses therefore update deterministically, and no restriction is applied when a movement passes through scene geometry. When building training data, the view graph discards renders that are mostly empty (void fraction > 0.7), such as views from inside geometry.

New analysis: collision re-scoring. We replayed every IVP test episode, checked each translation step against the ScanNet scene mesh, and counted an episode as failed if the camera collided with the scene before submitting.

IVP, All (%)Default successFailed if any collision before submittingEpisodes with at least one collision
Ours47.744.513.0
Gemini 3.1 Pro21.317.040.8
GPT-5.16.25.337.2
Qwen2.5-VL-7B base2.52.337.9

Success drops by at most 4.3 points for every model and the ranking is unchanged. Our lead over Gemini 3.1 Pro even grows slightly, from 26.4 to 27.5 points. Our policy also collides far less often: in 13.0% of episodes, compared with about 40% for the other models, including its own base model. A 5 cm collision radius gives the same trend.

We thank Reviewer vCDu again and hope these clarifications address the concerns. We are happy to provide further analysis during the discussion period.

Dear Reviewer vCDu,

Thank you again for your thoughtful reviews. As the discussion period is ending soon, we wanted to kindly check whether our responses and the added results have addressed your concerns. If they have, we would be grateful if you could consider updating your assessment. We are also happy to clarify anything further.

Best, The Authors

Dear Reviewer vCDu,

Thank you again for your detailed review. Before the discussion closes, we would like to briefly summarize how we addressed your concerns. In short, our main conclusions hold across every suggested additional experiments, and the variation between runs is small. If these results address your concerns, we would be grateful if you would consider updating your score.

Your concernWhat we did
W1 / Q3: Comparison with SFT + RL on the benchmark-construction pathsWe ran this baseline. Oracle SFT + RL reaches 30% success, while our method reaches 46% without using any expert paths.
W2 / Q2: Drop without rotation snappingThe drop reflects a train-test shift: the policy was trained only with snapped rotations and is specialized to these dynamics. Even under this shift, it remains the strongest evaluated model (19.6% vs. 15.7% for Gemini 3.1 Pro). The revised PDF states this in Section 5, Appendix B.4 and the limitations.
W3 / Q1: Consistency of the gainThe gain is robust. An independent training round reproduces it on Long targets (+8.1 and +7.3 points), and the 0.8-point difference between the two rounds is smaller than the sampling error of the test set. Pooled over both rounds, the gain is +7.7 points (95% CI +3.8 to +11.6) and is significant in a paired test (p = 0.012).
W4: Lower P2V / V2P after trainingThe drop comes from answer-format collapse, not from weaker view understanding. With auxiliary supervision, the final model stays at the base level (27.0 / 33.0 vs. 28.5 / 27.9).
Q4: Collision handlingWhen any collision counts as a failure, success changes only slightly and the ranking of models stays the same. Our policy also collides much less often, in 13.0% of episodes compared with about 40% for the other models.

Please let us know if you have any further concern, we are happy to clarify anything further.

Best, The Authors

Reviewer_XPptOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VIEWSUITE to study supplied-path understanding and interactive view planning. For IVP, a VLM controls the camera using target and current views, a top-down reference, and environment-provided poses. View Graph Distillation merges nearby viewpoints from self-exploration and turns sampled graph paths into action demonstrations and auxiliary supervision. Alternating distillation with RL raises Qwen2.5-VL-7B success from 2.5% to 47.7% on held-out scenes.

Critical strengths
  1. VIEWSUITE moves spatial understanding from passive observation to active viewpoint control. This is a useful problem. Evaluating supplied-path understanding alongside interactive planning also provides a useful benchmark for studying VLM spatial capabilities.
  2. Using unsuccessful exploration to construct supervision is a reasonable idea. It addresses the difficulty of training a weak initial policy with direct RL.
Critical weaknesses
  1. I see the main contribution as data construction and the training procedure. The graph combines exploration experience for goal relabeling and alternating training. The approach works, but the methodological advance over existing experience-reuse methods needs a clearer explanation.
  2. I am less sure how well the trained policy transfers. The held-out scene results are convincing, but the AI2-THOR model is trained separately, and changing the action rules causes a substantial performance drop. These results make a stronger case for the training recipe than for a broadly transferable view-planning policy.
Questions for authors
  1. Building on the existing no-graph control, compare the full training cost alongside performance to see whether the graph-based gains justify the additional data-processing and training overhead.
  2. I would like to see direct transfer of the same trained policy to a new environment, or results with limited adaptation.
  3. P2V/V2P use multiple-choice answers, while IVP requires interaction and pose reporting. A control with more closely matched answer formats can clarify how much of the observed gap comes from planning rather than response-format difficulty.
3 comments
AuthorsOpenReview ↗

Rebuttal-1

We thank Reviewer XPpt for recognizing that ViewSuite moves spatial understanding from passive observation to active viewpoint control, and that turning unsuccessful exploration into supervision is a reasonable way to train a weak initial policy that direct RL struggles with. We address your specific questions and concerns point by point below.

[W1] Relation to Existing Experience-Reuse Methods

Our method combines different episodes into supervision during training, and the policy needs no graph at test time. This separates it from the two closest lines of work:

MethodWhere experience is combinedGraph needed at test time
Hindsight relabeling (HER)Within one episodeNo graph
Graph-based planning (SPTM, SoRB)Across episodesYes, as a planner
OursAcross episodesNo: the policy acts from images

The view graph merges nearby viewpoints from different episodes, so a demonstration can follow one episode and continue along an edge found in another. Its endpoints become a new initial-target pair, and the policy is distilled from these demonstrations. Our ablations isolate each ingredient (IVP success, %):

Training variantLongAll
Direct PPO (no reuse)1.23.2
Success-only distillation2.06.0
Random-action graph6.113.0
Trajectory-level relabeling (HER-style, no graph)29.241.7
Ours (view graph)37.347.7

Reusing failed episodes accounts for most of the gain over direct RL, and the graph adds a further gain concentrated on Long targets (+8.1). We will make this positioning explicit in the method and related-work sections.

[W2, Q2] Dimensions of Generalization: Recipe Generality vs. Policy Transfer

The same trained checkpoint transfers without adaptation to new scenes and a new renderer, and with limited adaptation to a new benchmark; it is weaker under new execution rules.

Direct transfer, no adaptation (IVP success, %):

Shift from trainingOursBest zero-shot model (Gemini 3.1 Pro)Base
New scenes (held-out ScanNet)47.721.32.5
New renderer (3DGS instead of point clouds)36.023.2n/a
New execution rule (no rotation snapping)19.615.7n/a

Limited adaptation (both models fine-tuned with the same 401-step GRPO on each task; accuracy, %):

TaskInitialized from oursInitialized from base
New benchmark (MindCube)66.256.3
P2V57.345.1
V2P52.844.8
  • New renderer. The policy was trained only on point-cloud renders with holes and blur. On photorealistic 3DGS renders of the same test pairs, it keeps 36.0% success (47.7% under the default renderer) with no adaptation, 12.8 points above the best zero-shot model.
  • New benchmark with limited adaptation. MindCube uses its own scenes and images. Starting from the same accuracy as the base model, our checkpoint reaches 66.2% after identical short GRPO, against 56.3% from the base model. The same holds for P2V and V2P.
  • New execution rule. Removing rotation snapping is the hardest shift: the policy drops to 19.6%, though it remains the best evaluated model.

The recipe also transfers when retrained: on AI2-THOR it raises Qwen2.5-VL-7B from 5.6% to 61.1%, and on ViewSuite it raises InternVL3-8B from 0.6% to 20.2%. We will state this distinction between policy transfer and recipe generality explicitly. Direct transfer to an unseen simulator is a natural next step, which we expect to require training across several environments. We also note that the training method is one of our contributions; the benchmark and its main finding, that current VLMs recognize supplied paths far better than they plan them, do not depend on how far a single trained policy transfers.

AuthorsOpenReview ↗

Rebuttal-2

[Q1] Training Cost of the Graph

The graph adds no training stage, and the comparison already holds the training budget fixed. The graph and no-graph runs use the same exploration schedule, the same number of demonstrations per round, the same distillation settings and auxiliary tasks, and the same total RL steps. They differ only in whether demonstration pairs are sampled within individual trajectories or from the accumulated graph.

Building the graph uses only pose distances and lightweight image filters on views already rendered during RL rollouts; after three rounds the graph has 66,492 nodes over 193 scenes. For this overhead, the graph raises IVP success from 41.7% to 47.7% overall and from 29.2% to 37.3% on Long targets.

[Q3] Planning versus Answer Format

Once the answer format is matched, most of the gap remains, and it grows with how much of the path the model must plan. We ran two new format-matched controls with Gemini 3.1 Pro and GPT-5.1 on our ScanNet test pairs, one from each side of the gap.

Removing pose reporting from IVP (No-Submit). An episode counts as successful as soon as the camera reaches the target region, so no pose report is needed.

All (%)P2V / V2P (multiple choice)IVP without pose reportIVP
Gemini 3.1 Pro48.9 / 49.432.821.3
GPT-5.144.0 / 40.012.8 (new)6.2

Removing pose reporting closes about 40% of the gap for Gemini 3.1 Pro and 18% for GPT-5.1. The remaining 60% and 82% persist even when the model only has to reach the target.

Making the supplied-path task open-ended (Path-Fill, new). Given the V2P inputs and a ground-truth action sequence with part of it masked, the model writes the missing actions in the IVP action format. It succeeds if the completed sequence reaches the target. The answer format is fixed; only the amount of path to plan changes.

All (%), 150 pairsV2P (multiple choice)30% masked60% maskedWhole path masked
Gemini 3.1 Pro51.332.718.010.7
GPT-5.138.027.37.35.3
Random answer25.014.13.82.1

With the format held fixed, success falls by 22 points for both models from 30% masked to the whole path. This planning effect is larger than the full drop from multiple choice to the 30% setting (18.6 and 10.7 points), which bounds the format effect. Planning the whole path without feedback is even harder than interactive IVP (10.7 vs. 21.3; 5.3 vs. 6.2), so the difficulty lies in multi-step planning, and interaction helps.

Dear Reviewer XPpt,

Thank you again for your thoughtful reviews. As the discussion period is ending soon, we wanted to kindly check whether our responses and the added results have addressed your concerns. If they have, we would be grateful if you could consider updating your assessment. We are also happy to clarify anything further.

Best, The Authors

Reviewer_4S6yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether VLMs can reason about how camera movements change an image and, more ambitiously, actively control a camera to find a specified target viewpoint within a 3D scene. In doing so they make two main contributions:

  1. ViewSuite, a benchmark evaluating the spatial reasoning and interactive viewpoint planning capabilities of 13 VLMs.
  2. View Graph Distillation, a training method that converts exploratory camera movements into supervised demonstrations, including when the original exploration fails to reach its target.

The central result shows that, using their training procedure, Qwen2.5-VL-7B improves from 2.5% to 47.7% success on interactive viewpoint planning in held-out ScanNet scenes. This substantially exceeds the strongest zero-shot proprietary model.

The paper's key insight is that even unsuccessful exploration provides useful supervision: an agent may fail to reach its intended target, but it nevertheless knows how its actions moved it between the viewpoints it actually visited. To me this is quite a non-obvious and interesting finding.

Critical strengths

The strongest aspect is the simplicity and effectiveness of the training insight. By changing the goal associated with an unsuccessful trajectory, the authors extract meaningful supervision from almost every exploration episode. Graph recombination then extends this to paths spanning multiple rollouts.

The benchmark is also interesting. Comparing path interpretation against interactive path construction on the same viewpoint pairs is a sensible way to expose limitations of existing VLMs. Evaluating 13 models provides a useful breadth of evidence.

The experiments go beyond demonstrating one strong result. The trajectory-relabel baseline is particularly important, since it separates the value of hindsight relabelling from the additional contribution of graph construction.

Critical weaknesses

The improvement is highly dependent on the action interface. The trained policy uses 30 degree rotational increments with explicit rounding of Euler angles after each action. Removing this rotation snapping at test time reduces success from 47.8% to 19.6%. This suggests that a substantial part of the learned capability is specialised to the benchmark's discrete camera dynamics. It remains unclear how well the technique would transfer to continuous camera control or different action discretisations. There is also an appearance-domain sensitivity: changing from the default point-cloud renderer to 3D Gaussian Splatting reduces the trained model's success from 47.7% to 36.0%.

I wonder whether the task may be closer to visual search than true 6-DoF spatial reasoning. Because the model receives its exact current pose after every action, it can solve the problem by finding a visually matching viewpoint and reporting the environment-provided coordinates. 98% of the trained model's successful episodes include an observation within the target-pose tolerance. That does not make the task trivial (locating the correct viewpoint is itself challenging). However, it limits what the results establish about the model's ability to infer 3D transformations, maintain a spatial representation, or perform accurate 6-DoF pose estimation.

Questions for authors

The substantial reduction in performance when rotation snapping is disabled (47.8% to 19.6%) suggests strong dependence on the particular camera-control interface. Could the authors elaborate on what causes this sensitivity? In particular, do they believe the learned policy develops generalisable spatial reasoning capabilities, or is it primarily learning strategies specialised to the discrete action dynamics? Have they investigated training or fine-tuning with alternative discretisations?

The paper reports that 98% of successful episodes include an observation already within the target-pose tolerance, while the environment provides the exact pose of each observed viewpoint. To what extent do the authors believe the method improves 3D spatial reasoning, as opposed to learning an effective visual search strategy? Could the authors provide evidence that the trained model can estimate a target pose accurately without first reaching its vicinity, or otherwise clarify the scope of the capability being learned?

Despite including auxiliary view-understanding supervision, the final trained model performs substantially worse on P2V and V2P. Could the authors clarify whether this degradation is primarily attributable to output-format failures rather than reduced spatial reasoning ability? For example, what accuracy would be obtained if predictions were evaluated independently of response formatting, where possible?

4 comments
AuthorsOpenReview ↗

Rebuttal-1

We thank Reviewer 4S6y for finding our graph-based training insightful and our benchmark interesting, as well as for raising valuable questions that allow us to further explain our motivation, findings, and insights.

Before addressing each point in detail, we would like to briefly clarify the motivation and scope of our work. We mainly want to investigate whether current VLMs are capable of view planning and design a set of probing experiments to find their capability boundaries. Within this context, our graph-based method is designed to serve as a strong baseline for this benchmark, specifically addressing the cold-start failure of direct RL. Therefore, we don't want to expect a single RL training recipe alone to fundamentally solve the spatial reasoning and generalization challenges of current VLMs. Achieving that will naturally require joint efforts across pretraining, data scale, environment diversity, and model capacity, which we also examine in our paper and view as valuable follow-up directions enabled by our benchmark. We address your specific questions and concerns point by point below.

[W1, Q1] Rotation snapping, Rendering, Generalization

For the performance drop under the No-Snap setting, we believe it mainly comes from (1) the inherent difficulty of the No-snap setting and (2) the train-test mismatch problem. We first want to mention that the No-Snapping setting is much harder than the Snapping setting for all models, not just our trained policy. As shown in the paper, every evaluated model performs worse without snapping, including the strongest zero-shot models such as Gemini 3.1 Pro and GPT-5.4. Removing snapping relaxes output constraints and gives the model more freedom to control the camera, which places a higher demand on precise spatial control. In our experiments, we see models might issue off-angle rotations that twists the camera into an weird pose and fails to recover in time, so errors accumulate across steps until the episode fails.

In addition to the harder task dynamics, we think the common train-test mismatch problem is another main reason. Our policy was trained only with discrete rotation snapping in a single environment setup with a 7B model, so evaluating it zero-shot without snapping introduces a clear distribution shift. Similarly, when switching to 3D Gaussian Splatting (3DGS) rendering, the drop comes from the same kind of train and test mismatch. Testing zero-shot performance on an unseen renderer without adaptation was one of the exact questions we wanted to study, which is why we included and analyzed these experiments in the paper.

Regarding whether the learned policy develops generalizable spatial reasoning or only specialized strategies, our results suggest that the model does, to some extent, learn generalizable capabilities, as it still outperforms all other evaluated baselines in both settings despite these drops. Our downstream and cross-environment experiments point in the same direction. Under identical downstream fine-tuning, our checkpoint performed better than the base model on P2V, V2P, and MindCube. And more importantly, our training recipe itself generalizes well across different environments and models.

We also explored alternative discretizations and continuous camera control (where the model outputs both the action type and its continuous value) in our early experiments. On discrete control, we think the step size is mainly a choice of control granularity. A step that is too large makes it hard for the model to align the camera precisely with the target pose, whereas a step that is too small makes the horizon very long and costs much more training time and tokens. Changing the rotation and step size don't affect the overall learning trend. For continuous action experiments, we found that giving current models this extra freedom exposed the limits of their capabilities across model scales. Frontier models struggle with fine-grained multi-step continuous control, while smaller base models often fail at basic instruction following for the continuous action format, producing too few valid actions to build a useful exploration graph. We will describe these explorations in the revised appendix.

To summarize our view on W1 and Q1, we believe the learned policy does develop generalizable spatial reasoning skills, while broader generalization is currently limited by our single training setup and 7B model size. We see addressing this through more diverse environments, larger-scale data, and larger models as a valuable follow-up direction, and we will make this clearer in our Limitations and Future Work.

[W2, Q2] Visual Search, Spatial Reasoning, and Distant Pose Estimation

Concerning visual search versus 6-DoF spatial reasoning, we agree that most of our successful episodes include an observation within the target-pose tolerance. However, we believe that effective visual search in 3D is an essential part of spatial reasoning rather than separate from it, and in fact most current models cannot even perform such 3D visual search well. To bring the camera to a matching view, the agent still has to decide which 6-DoF translations and rotations move it toward the target, which visual matching alone cannot achieve.

Several findings in our paper support this view. The successful episodes of zero-shot models follow the exact same pattern, with 96.5% of Gemini 3.1 Pro's and 94.1% of GPT-5.4 Pro's successes passing a near-target view, yet they reach at most 21.3% overall accuracy. A greedy DINOv2 similarity search with two-step lookahead reaches only 9.1%. Moreover, 89.5% of our model's remaining failures involve position error, and IVP success drops mainly with position distance whereas P2V and V2P drop mainly with rotation distance. When success is credited from visited poses alone without requiring a submitted pose (No-Submit, Table 14), our model reaches the target region in 57.5% of episodes, clearly outperforming Gemini 3.1 Pro at 32.8% and GPT-5.4 at 26.0%. Together, these results show that the main challenge lies in planning movements through 3D space to reach the target region rather than simply recognizing the view once reached.

Regarding whether a model can estimate the target pose without first reaching its vicinity, we shared the exact same question as the reviewer when designing our probing experiments. Our finding is that neither current frontier models nor our trained policy can reliably estimate a target pose without approaching it first. We believe this happens for two main reasons. On the capability side, estimating a distant pose requires building an implicit 3D mental model of the scene during exploration and localizing the target view within it, and current VLMs lack this strong 3D prior. On the optimization side, training with a task-completion reward alone naturally encourages the policy to take the most reliable shortcut, which is to actively search for the target and move closer to it, just as humans naturally do. We view this observation as an interesting probing finding of our paper and a valuable follow-up direction, which could be explored through specialized RL reward designs (such as penalties on step count or distance traveled) or synthesizing such SFT data. We will make both this finding and our capability scope clearer in the revision.

[Q3] P2V/V2P Drop after Training

We thank the reviewer for this careful observation. Re-examining these numbers, we found that the 6.0% and 20.9% scores and our downstream initialization scores reflect different settings that our current text presents in a confusing way, and we will correct this presentation in the revision. Specifically, the drop to 6.0% and 20.9% comes from training on IVP trajectories without auxiliary view-understanding supervision, where the low scores are dominated by output format collapse. Because that training optimizes exclusively for multi-turn camera control, the model frequently responds to single-turn multiple-choice prompts with raw IVP action strings instead, which cannot be parsed as answer(x) options and are counted as zero, pulling P2V down to 6.0% (well below the 25% random-guessing floor). Re-examining the test rollouts confirms this directly: the vast majority of its P2V responses and over a third of its V2P responses are unparsed camera-action strings (compared with almost none for the base model), whereas on the subset of responses that do provide a valid option, its accuracy remains on par with the base model.

By contrast, when our auxiliary view-understanding supervision is included during training, this format collapse is prevented and the model maintains valid multiple-choice formatting. As shown in our ablations and downstream evaluations, incorporating the auxiliary supervision restores single-turn P2V and V2P performance to a level comparable to the base model prior to downstream fine-tuning, and further enables the checkpoint to achieve substantially higher P2V and V2P accuracy than the base model under an identical downstream training schedule. We will revise the main text and appendix to clearly separate the results with and without auxiliary supervision and report the share of invalid action-format responses.

We thank Reviewer 4S6y again and hope these clarifications address the concerns. We are happy to provide further analysis during the discussion period.

Hi authors. Thanks for your detailed response to my review. I take your point about framing and narrowing slightly would fit my concerns. The abstract and the claims about spatial reasoning do perhaps slightly oversell so could perhaps be slightly re-framed to pitch more as a probing study with a baseline attached. Regarding the rebuttal responses, visual search vs 3D reasoning was well answered and so was the question about P2V/V2P drop. On rotation snapping, I still slightly question whether the results support the claims. The paper itself says the trained policy shows a larger drop than either zero-shot model. Calling it a train-test mismatch mostly restates my concern: the policy is specialised to the discrete dynamics. But this is really only a minor concern. I was positive about the paper before the rebuttal and am slightly more positive now. So my current view is still weak accept but leaning to strong accept.

Hi Reviewer 4S6y,

Thank you very much for your positive assessment of our work. We are glad that our responses could help address most of your concerns, and are greatly encouraged that you are now moving towards a stronger accept.

We appreciate your suggestion on framing, which is well aligned with our original motivation. Accordingly, we revised the paper to reflect this more clearly and to scope our claims more precisely, while extending the discussion of generalization in Section 5 and in Section 7. As promised in our earlier response, the appendix now also includes our discussion of alternative action discretizations and continuous control. The updated PDF is now available. Thank you again for your careful and constructive feedback, which has been truly helpful in improving our paper.

Comments on the paper

2 comments

Dear reviewers,

Thanks for your continued efforts. The authors have provided a rebuttal to the initial reviews. Please review it carefully, along with all other reviews and the corresponding author responses, and consider whether you have any further questions for the authors or whether the rebuttal has addressed your concerns. There are only a few days remaining before the discussion period closes, so please use this time to seek any further clarifications or raise any additional questions about this paper.

Best,

AC

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for recognizing the contributions of this work:

  • A benchmark for active viewpoint control: Reviewers XPpt and 4S6y found that ViewSuite moves spatial understanding from passive observation to active viewpoint control, and Reviewer vCDu noted that its three complementary tasks on shared view pairs provide a useful framework.
  • Informative diagnostic analyses: Reviewer vCDu highlighted that the analyses of viewpoint distance, interaction budgets, reasoning settings and failure types help characterize the difficulties current VLMs face.
  • Learning from failed exploration: All three reviewers appreciated turning unsuccessful exploration into supervision for a weak initial policy that direct RL struggles to train.

Discussion with reviewers. After the discussion, Reviewer 4S6y wrote that they were "slightly more positive now" and that their current view is "weak accept but leaning to strong accept". They described rotation snapping as "really only a minor concern", and found our answers on the P2V/V2P drop and on visual search versus 3D reasoning well answered.

We addressed the concerns with additional experiments, analyses and manuscript revisions. Across all of these checks, the main conclusions of the paper remain robust:

  • Rotation snapping (vCDu, 4S6y): After our response, Reviewer 4S6y described this as "really only a minor concern". Even without snapping, our policy has the highest overall success of the evaluated models, and the revised paper states directly that the policy is specialized to the discrete dynamics it was trained on (Section 5, Appendix B.4 and the limitations).
  • P2V/V2P accuracy after training (vCDu, 4S6y): Reviewer 4S6y found this question well answered. The drop comes from answer-format collapse in a checkpoint trained without auxiliary tasks; with auxiliary supervision, the final model stays at the level of the base model (Section 5 and Appendix C.8).
  • Planning versus answer format (XPpt, 4S6y): Two format-matched controls show that the planning gap persists when the answer formats are matched and grows with how much of the path a model has to plan (Section 3 and Appendix B.5).
  • Consistency of the graph's gain (vCDu): An independent training round reproduces the gain on Long targets (+8.1 and +7.3 points). The difference between the two rounds is smaller than the sampling error of the test set, and the pooled 95% confidence interval excludes zero (Appendix C.10).
  • Expert demonstrations (vCDu): SFT followed by RL on the benchmark-construction paths reaches 30% validation success, while our method reaches 46% without using any expert paths (Appendix C.11).
  • Collision handling (vCDu): When collisions count as failures, success changes only slightly and the ranking of models stays the same. Our policy also collides far less often than the other models (Section 2.2 and Appendix B.6).
  • Method positioning and training cost (XPpt): Our method combines experience across episodes during training, unlike within-episode relabeling, and needs no graph at test time, unlike graph-based planners. The graph and no-graph variants use matched training settings (Related Work and Appendix C.5).
  • Visual search versus 3D reasoning (4S6y): Frontier models show the same near-target pattern yet reach at most 21.3% success, and image-similarity search reaches only 9.1%, so the main challenge lies in planning movements through 3D space. Reviewer 4S6y found this question well answered.
  • Transfer of the trained policy (XPpt): Without adaptation, the policy generalizes to held-out scenes and to a new renderer. After the same short fine-tuning, it also outperforms the base model on MindCube, P2V and V2P (Appendix D.3).

We believe these results substantially strengthen the empirical evidence and address the main concerns raised in the reviews. We would welcome any final feedback before the discussion closes. We sincerely appreciate your continued feedback of our work.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.