acceptodds
Under review as a conference paper at ICLR 2027

Knowing Where to Go: Generalizing Mobile Manipulation to Unseen Environments from a Single Video

Abstract

Vision-language-action policies have achieved impressive mobile manipulation capabilities, but remain brittle in unfamiliar multi-room environments. We find that much of this brittleness stems from failures of spatial generalization: strong policies often resort to reactive exploration when task-relevant locations move outside familiar layouts. Oracle spatial guidance largely prevents this degradation. We study how spatial context from a prior environment walkthrough video should be represented to guide mobile manipulation policies under environment shift. We find that strong in-distribution spatial prediction can be a poor predictor of out-of-distribution transfer, while models that explicitly encode 3D structure generalize more robustly to new robot environments. Building on this, we develop a hierarchical policy that uses a single environment video to provide spatially grounded guidance to a VLA. We find that navigation performance of mobile manipulation policies is heavily bottlenecked by spatial understanding, and that our strongest policy more than doubles the placement rate of its flat counterpart ( vs. ) in our hardest setup. For more information, see https://know-where-to-go.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action models, mobile manipulation, spatial reasoning, robot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_widmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies spatial generalization in mobile manipulation using a prior environment walkthrough video. A high-level VLM predicts subtasks and robot-relative navigation hints, which are executed by a low-level π0.5 policy. The authors compare implicit spatial learning, geometric supervision, geometry-feature fusion, and a Qwen-3D-style representation. On controlled multi-room extensions of RoboCasa365, the estimated-geometry Q3D system improves Unseen-Hard placement rate from 10.9% to 24.8%. Additional analyses examine room selection, dependence on tour content, and sensitivity to geometry quality.

Critical strengths
  1. The navigation improvement is good, and supported by behavioral evidence. Beyond placement rate, estimated-geometry Q3D improves correct first-room selection from 41.6% to 73.5%, conditional on leaving the kitchen, and reduces decoy-room entry from 63.0% to 25.5% (Table 9). Shared executors among tour-conditioned planners and identical evaluation states strengthen this comparison.
  2. The diagnostic experiments are informative. Tour replacement, target masking, adaptation-strategy comparisons, and geometry-source ablations provide useful evidence about what the models exploit and where the pipeline fails. The finding that target masking barely changes bearing accuracy is particularly valuable for interpreting spatial benchmark performance.
Critical weaknesses
  1. The contribution of explicit 3D encoding is unclear. Q3D receives extra geometry and localization inputs, and the training settings also differ. The results show that the full pipeline works, but do not isolate the benefit of the 3D representation.
  2. The generalization setting is narrow. The tours highlight the destination and skip the decoy room, while the test layouts modify familiar kitchens. It is unclear whether the gains hold with tours that cover all rooms without highlighting the target.
Questions for authors
  1. What geometry and odometry information does each planner receive? Does Q3D retain its advantage with matched inputs and training budgets?
  2. Do the gains hold when tours include the decoy room and do not highlight the destination?
  3. Please report full-task success, per-task results, and paired uncertainty, especially for Q3D versus V3R.
4 comments
AuthorsOpenReview ↗

Reviewer widm responses

Thank you for the careful review and for highlighting the navigation improvements and informative diagnostic experiments. We have added tour ablations, full-task and statistical reporting, and an evaluation with a second executor.

W1/Q1: What localization information does Q3D receive, and does the comparison establish an explicit-3D advantage?

Yes, the first-frame-only variant in Table 19 (originally table 10) still uses simulator odometry after its initial geometry estimate. It removes subsequent visual relocalization, rather than removing odometry. We have clarified this in Appendix D.2. Estimated-geometry Q3D uses Depth Anything 3 for tour depth and camera poses, together with camera-to-base calibration and simulator odometry for registration. The GT variant receives ground-truth tour geometry. Alternative planners do not receive equivalent registered geometric inputs.

We agree that this prevents attributing the full advantage to explicit 3D encoding alone. We revised the claims to describe the performance of the complete Q3D recipe. As you note, fully finetuned Q3D does not outperform fully finetuned CP on Hard bearing prediction, Table 2 therefore demonstrates that adaptation strategy matters, rather than establishing an architecture-only advantage.

The empirical transfer finding remains: the recipe with stronger held-out midtraining-domain performance need not transfer better to robot environments. For example, CP achieves 70.5% bearing accuracy on held-out MolmoSpaces houses versus 46.7% for Q3D, while their ordering reverses on RoboCasa Unseen-Hard (43.1% versus 56.5%). The common backbone and training-data mix, followed by shared downstream optimization and executor, make this recipe-level comparison informative. They do not substitute for a matched-geometry and matched-tuning study, which we now identify as a remaining limitation (Section 7.1).

W2/Q2: Do improvements persist when tours include the decoy room without task-informed view selection?

Thanks for the great question. We added tour ablations using XR-1 as the shared executor. One removes explicit dining-table framing and another extends the Unseen-Hard tours to include the decoy room. The original two-room and new three-room tours contain 48 and 72 stored frames, respectively, but both provide 16 selected views to the planner. Comparisons use the same executor and original-tour baseline.

We report these results in Appendix E.4, Tables 16–17. We also clarify that the benchmark tests controlled room-layout changes while retaining training object splits. These shifts are deliberately structured to probe whether the policy knows where to go when familiar spatial relationships change. The first-room and decoy-entry results you highlighted provide direct evidence about this capability, beyond the aggregate placement metric. We have made this specific generalization claim clearer without extending it to arbitrary unseen environments.

Q3: Full-task success, per-task results, and paired uncertainty, particularly Q3D versus V3R.

We added the requested full-task success rates and 95% confidence intervals (Tables 9–10), per-task results (Tables 13–15), and paired permutation tests accounting for shared layouts (Table 22).

For the original π₀.₅ evaluation, the Unseen-Hard results are:

MetricEstimated-geometry Q3DV3RHolm-adjusted p
Placement rate24.8%18.6%0.180
Full-task success4.0%1.5%1.000

The point estimates favor Q3D, but these tests do not establish a statistically significant difference.

To test whether the planner comparison extends beyond π₀.₅, we additionally evaluated the same VLM planners with XR-1, trained on the same RoboCasa/MolmoSpaces data mixture and shared across planners:

Metric, Unseen-HardEstimated-geometry Q3DV3RHolm-adjusted p
Placement rate49.6%23.4%0.014
Full-task success18.5%5.0%0.018

This second-executor experiment provides stronger evidence that the spatial guidance supplied by the Q3D pipeline improves downstream performance. The improvement appears in both placement and full-task completion, addressing whether the navigation gains translate into completing the manipulation objective (Appendix E.2, Tables 11–12).

These additions support a stronger system-level result while preserving the distinction between that result and the architectural attribution discussed above. Thank you for prompting the additional analyses. We would like to ask whether our responses addressed your concerns in full, and we are happy to continue this discussion through the rebuttal.

Dear reviewer, as the rebuttal period draws to a close, we wanted to ask whether our responses and additional experiments have addressed your concerns. If so, would you consider raising your score in light of the new results? We are happy to clarify any remaining questions. Thank you again for your time and constructive feedback!

Thanks for your rebuttal and your additional experiment. As you acknowledge W1, please update your submission. Other experiments mostly convince me. I remain my rating as a positive score.

Dear reviewer, thank you very much for your response and for acknowledging that our supplemental experiments have addressed your criticisms. We've uploaded a new version of the draft which fully acknowledges W1. The abstract now attributes the transfer result to the full Qwen3D recipe, and Section 7.1 explains why our comparisons leave the independent effect of explicit 3D encoding open. Appendix D.2 details the geometry and localization inputs available to each planner. If the revision fully addresses your concerns with the paper, we'd greatly appreciate if you consider increasing your rating.

Thanks again for taking time to review and help improve our work.

Reviewer_YpyROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical policy that uses a single environment video to provide spatially grounded guidance to a VLA. Its strongest contribution is the finding that strong in-distribution spatial prediction does not always lead to good transfer. The experimental results show that the explicit 3D planner improves navigation and placement rates in unseen multi-room layouts. However, the evaluation is limited to simulation and uses task-informed tours, leaving performance on more general environment videos unclear.

Critical strengths
  1. Knowing where to go in an unseen environment is a common challenge in mobile manipulation, and it’s intuitive to use a single environment video to provide spatial guidance to a VLA.

  2. The paper compares different spatial models as planner and training choices and shows that strong in-distribution spatial prediction does not always lead to good transfer.

Critical weaknesses
  1. The paper does not clearly explain how the 16 tour frames are selected during inference.
  2. The tours are task-informed, so performance with task-independent videos remains unclear.
  3. The experiments are all on simulation, leaving the performance on real-world scenarios and the generalization ability of the method unclear.
Questions for authors
  1. The inference path is a little bit confusing, as it is unclear how the 16 tour frames are selected, whether they remain fixed throughout an episode, and when the planner predicts the next subtask.
  2. A comparison between task-informed and task-independent tours would help clarify how video collection affects performance.
3 comments
AuthorsOpenReview ↗

Reviewer YpyR Responses

Thank you for the constructive review and for highlighting the relevance of the problem and the finding that strong in-distribution spatial prediction need not imply strong transfer. We have clarified the inference procedure and added experiments addressing tour collection.

W1/Q1: How are the 16 frames selected, do they change during execution, and when does replanning occur?

For the original 48-frame tours, we uniformly subsample 16 frames once before execution. These frames remain fixed throughout the episode. At each planner invocation, the model receives this fixed tour context together with the current egocentric observations and task instruction, and predicts the next subtask, including a polar navigation hint for navigation subtasks.

The planner is queried every 16 control timesteps, and the executor produces the corresponding 16-step action chunk. Thus, the spatial context remains fixed, while current observations and subtask predictions are updated throughout execution. We added these details to Appendix D.3.

W2/Q2: How does performance change with less task-informed tours?

We added two evaluations using the same VLM planners and a shared XR-1 executor. First, we removed the explicit requirement to frame the dining table while preserving scene coverage. Second, we extended the Unseen-Hard tours to cover the decoy room. Both protocols retain a 16-view planner input budget, although the stored tours contain 48 frames for two rooms and 72 for three rooms. We find that removing the table framing requirement leads to mixed changes in performance across planners and suites, leading to a tiny 0.64% gain in placement rate overall and indicating that this factor did not have a significant effect on the results. When adding the decoy room to the tour, we observe a consistent but small performance drop averaging out to 2.21% placement rate, indicating that omitting the decoy room led to a moderate improvement but that even with the decoy room included the Q3D planner still leads to a consistent performance improvement over the flat policy.

These comparisons appear in Appendix E.4, Tables 16–17. They directly test the effect of relaxing the original tour protocol, and we now distinguish these coverage-controlled tours from arbitrary videos that may omit relevant parts of the environment.

W3: What does the simulation-only evaluation establish about generalization?

Our experiments establish transfer under controlled simulated layout changes. The evaluation moves task-relevant destinations outside the training kitchen configuration and, in Unseen-Hard, introduces a competing decoy room. This tests whether a policy can use prior spatial context to reach a relevant location beyond its current field of view.

The gains are supported by behavior as well as placement: estimated-geometry Q3D increases correct first-room choice from 41.6% to 73.5%, conditional on leaving the kitchen, and reduces decoy-room entry from 63.0% to 25.5%. Simulation allows these comparisons to use identical initial states and shared executors, and permits controlled interventions on tour content and geometry. We agree that these results do not establish real-world transfer. In particular, the estimated-geometry system still uses simulator odometry, thus deployment with noisy localization and real robot execution remains untested. We have clarified this limitation (Section 7.1) and scoped our conclusions to the evaluated setting.We also expanded the evaluation to a second executor, XR-1. In Unseen-Hard, estimated-geometry Q3D achieves 49.6% placement and 18.5% full-task success, compared with 23.4% and 5.0% for V3R, with Holm-adjusted paired-test p-values of 0.014 and 0.018. This shows that the downstream benefit extends beyond the original π₀.₅ executor and includes full-task completion, while real-world validation remains future work.

Thank you for helping us make the deployment assumptions and the scope of the generalization results clearer. We would like to ask whether our responses addressed your concerns in full, and we are happy to continue this discussion through the rebuttal.

Dear reviewer, as the rebuttal period draws to a close, we wanted to ask whether our responses and additional experiments have addressed your concerns. If so, would you consider raising your score in light of the new results? We are happy to clarify any remaining questions. Thank you again for your time and constructive feedback!

Most of my concerns are addressed. I will maintain the score.

Reviewer_1jzbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies mobile manipulation when familiar task destinations move into previously unseen rooms. It introduces a planner–executor system: a spatial VLM uses a prerecorded environment tour and current observations to predict subtasks and polar navigation hints, which a π₀.₅ executor follows. The authors compare several spatial learning recipes and evaluate ten RoboCasa365 tasks in edited multi-room layouts.

In the hardest setting, the Q3D planner with estimated geometry achieves 24.8% placement rate, compared with 10.9% for flat π₀.₅; ground-truth tour geometry increases this to 32.5% (Table 5). Estimated-geometry Q3D also chooses the correct first room in 73.5% of episodes that leave the kitchen, versus 41.6% for the flat policy (Table 9).

Critical strengths

The paper tests whether spatial learning transfers beyond its training distribution. Models that score higher on held-out spatial questions do not necessarily perform better in a different simulator or when guiding a robot. This raises a representation-learning question: which training objectives and inputs produce capabilities that remain useful under distribution shift? Evaluating the same recipes in another simulator and through robot behavior highlights the differences that the original evaluation could miss (Tables 1 and 5.)

The geometry ablations help separate representation quality from input quality. Comparing estimated and ground-truth geometry measures how reconstruction errors affect downstream performance. This helps distinguish difficulty using spatial information from errors in the spatial information itself (Table 10).

Critical weaknesses

The architectural comparison remains confounded. Table 1 gives Q3D ground-truth geometry unavailable to other models. Its recipe also changes attention, token pooling, learning rate, regularization, and adaptation strategy. Table 2 addresses one part of this problem but does not establish that explicit 3D encoding itself explains superior transfer (Appendices A and B.3).

The benchmark supports a narrower generalization claim than the framing suggests. Test layouts modify familiar training kitchens, retain training object/style splits, and provide privileged, task-informed tours that visit the destination while skipping the decoy room. The estimated-geometry system additionally uses simulator odometry. These are valid experimental assumptions, but their contribution is not separated from general spatial understanding (Appendices C.2–C.3, D.2).

The reported comparisons support the narrower conclusion that the complete Q3D-based system performs better on this benchmark. They do not cleanly isolate explicit 3D representation as the cause: Q3D receives different geometric information, uses different attention and optimization, and its estimated-geometry deployment includes online registration using simulator odometry (Appendices A, B.3, D.2).

These concerns are addressable by narrowing the claims and clarifying the protocol. Retaining the stronger architectural conclusion would require additional controlled evidence. The interpretation of the object-masking probe also needs qualification (§5.3.1).

Questions for authors

Which gains remain when alternative planners receive the same estimated depth, camera registration, and odometry? A matched-geometry comparison would materially strengthen the architectural conclusion. Otherwise, please frame the result as a comparison of complete recipes.

How much does the advantage depend on tours deliberately visiting the destination and excluding the decoy room? Results with task-independent tours covering all rooms would establish whether the system generalizes beyond this informative tour-selection protocol.

Please provide paired uncertainty estimates for Table 5 and per-task results, preferably accounting for shared layouts. These would help assess the smaller planner differences and whether the aggregate advantage is consistent across tasks.

3 comments
AuthorsOpenReview ↗

Reviewer 1jzb responses 1/2

Thank you for the detailed and constructive review. We appreciate your recognition that the cross-domain and robot evaluations reveal differences that spatial prediction benchmarks alone can miss. Your comments prompted new tour ablations, a second-executor evaluation, and clearer architectural claims.

W1/Q1: Does the comparison isolate explicit 3D encoding, and what happens with matched geometric inputs?

We agree that the comparison supports the complete Q3D recipe more directly than explicit 3D encoding in isolation. We revised section 5.3 and added commentary about this to a new limitations section accordingly. Q3D differs in geometric inputs, representation and optimization, Table 2 addresses adaptation strategy but does not eliminate all these differences.

The comparison nevertheless controls several important factors: the midtrained models use the same Qwen3-VL-4B backbone and training-data mix, and the downstream tour-conditioned planners use the same subtask-finetuning optimization recipe and share a single executor. Under this setup, the spatial evaluation reveals a substantial reversal: Q3D scores 46.7% versus CP’s 70.5% on held-out MolmoSpaces bearing questions, yet 56.5% versus 43.1% on RoboCasa Unseen-Hard. Thus, stronger performance within the midtraining domain does not reliably identify the recipe that transfers best.

Regarding input information, estimated-geometry Q3D uses Depth Anything 3 to reconstruct tour depth and poses, together with camera-to-base calibration and simulator odometry for registration. The privileged variant receives ground-truth tour geometry. Alternative planners do not receive equivalent registered geometry, so we cannot establish which gains would remain under the requested matched-input comparison. We now make this distinction explicit rather than attributing the difference solely to 3D encoding.

W2/Q2: Do gains depend on tours visiting the destination and excluding the decoy room?

Thanks for the great question. We added two tour ablations with XR-1 as the shared executor:

  1. Removing the requirement to explicitly frame the dining table while retaining scene coverage.
  2. Extending Unseen-Hard tours to include the decoy room.

The original two-room and new three-room tours contain 48 and 72 stored frames, respectively, but both supply 16 selected views to the planner. Each comparison uses the same executor, isolating the effect of the tour protocol. When we remove the requirement to frame the table, the results change minimally, leading for example to an average drop in placement rate of 0.15% for the unseen-hard setting. When we add the decoy room to the tour in addition to removing the framing requirement, the change is more directionally consistent across planners but still leads to a small drop of 2.21% across planners. These results indicate that the task-informed nature of the tours we sample does not substantively affect our results. These results appear in Appendix E.4, Tables 16–17.

We also clarify the generalization being tested (Section 7.1). The benchmark retains training object splits and constructs new room configurations from training kitchens. This deliberately tests whether policies can find familiar task destinations when their spatial relationship to the kitchen changes, including when a competing room is present. The behavioral results show that this shift is important: estimated-geometry Q3D increases correct first-room choice from 41.6% to 73.5%, conditional on leaving the kitchen, and reduces decoy-room entry from 63.0% to 25.5%. We present these as evidence for spatial transfer under controlled layout changes, without extending the claim to arbitrary unseen homes.

AuthorsOpenReview ↗

Reviewer 1jzb responses 2/2

Q3: Paired uncertainty and per-task results.

We added placement and full-task success rates with 95% confidence intervals (Tables 9–10), per-task results (Tables 13–15), and paired permutation tests accounting for shared layouts (Table 22).

For the original π₀.₅ Unseen-Hard evaluation, estimated-geometry Q3D improves placement over the flat policy by 13.9 percentage points, with a Holm-adjusted p-value of 0.059. Against V3R, placement increases from 18.6% to 24.8%, but the adjusted p-value is 0.180; the full-task success comparison has an adjusted p-value of 1.000. We therefore distinguish these numerical improvements from statistically established differences.

We additionally tested whether the planner advantage extends to a second executor:

MetricEstimated-geometry Q3DV3RHolm-adjusted p
Placement rate49.6%23.4%0.014
Full-task success18.5%5.0%0.018

These results provide stronger evidence for the Q3D pipeline’s downstream advantage and show that it extends to completed tasks, rather than only partial placement progress (Appendix E.2, Tables 11–12).

As mentioned above we add per-task results in Tables 13-15. For example we see that performance improvement from the flat policy to the Q3D planner for the Unseen-hard setting is across many tasks rather than driven by a small set of tasks with large improvements.

W3: Interpretation of the object-masking probe.

We agree and have qualified Section 5.3.1. The result establishes that masking the target’s visible pixels has little effect on bearing accuracy under this intervention. It is consistent with reliance on unmasked context, such as supporting furniture, but does not uniquely distinguish room-level localization from contextual inference of the object’s location.

We are thankful for your feedback, which was helpful in strengthening the empirical analysis. We would like to ask whether our responses addressed your concerns in full, and we are happy to continue this discussion through the rebuttal.

Dear reviewer, as the rebuttal period draws to a close, we wanted to ask whether our responses and additional experiments have addressed your concerns. If so, would you consider raising your score in light of the new results? We are happy to clarify any remaining questions. Thank you again for your time and constructive feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you to all reviewers for your time with the paper

We thank the reviewers for their constructive feedback and for highlighting the informative transfer study, navigation improvements, and diagnostic experiments. In response, we have strengthened the evaluation and clarified the scope of our conclusions:

  • Second executor and full-task completion. With XR-1, estimated-geometry Q3D achieves 49.6% placement and 18.5% full-task success in Unseen-Hard, compared with 23.4% and 5.0% for V3R. The Holm-adjusted paired-test p-values are 0.014 and 0.018, respectively.
  • Tour-protocol ablations. We added evaluations removing the required dining-table view and extending tours to include the decoy room, while keeping the planner’s input budget fixed at 16 views. We find that removing the table framing requirement leads to a small and mixed change in performance, indicating that this factor did not have a significant effect on the results. When adding the decoy room to the tour, we observe a small performance drop, indicating that omitting the decoy room led to a moderate improvement but that even with the decoy room included Q3D planner still leads to a consistent performance improvement over the flat policy.
  • More complete reporting. We added full-task success, per-task results, confidence intervals and paired tests accounting for shared layouts.
  • Clearer interpretation and implementation details. We clarify frame selection, replanning, geometric inputs, simulator odometry and distinguish evidence for the complete Q3D recipe from attribution to explicit 3D encoding alone.
  • We added a limitations section discussing several limitations in our results and openings for future work (Section 7.1).

Together, these additions strengthen the central finding: spatial prediction performance within the training domain can be a poor guide to downstream transfer, while effective spatial guidance improves navigation and manipulation under the evaluated layout shifts.

We address each reviewer’s questions below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.