acceptodds
Under review as a conference paper at ICLR 2027

Nav4Manip: From Navigation to Dexterous Mobile Manipulation through Co-Evolution

Abstract

Dexterous mobile manipulation is essential for robots to perform useful tasks in complex, unstructured environments. However, existing systems face two challenges. First, navigation remains too coarse for downstream manipulation, which often requires a precise base position and approach orientation rather than merely reaching a meter-scale goal region. Second, navigation and manipulation are commonly trained in isolation, lacking an interactive feedback loop that enables the two capabilities to co-evolve. These limitations prevent current systems from reliably solving practical, long-horizon dexterous mobile-manipulation tasks. To address them, we introduce Nav4Manip, a bidirectionally coupled co-evolution system for long-horizon dexterous mobile manipulation. It comprises two complementary architectures: Nav4Manip-N learns long-horizon navigation and manipulation-aware stopping, while Nav4Manip-M post-trains a world-action model (WAM) to provide the downstream dexterous-manipulation interface. We train Nav4Manip-N in three stages: multi-task co-tuning learns navigation behavior and manipulation-aware affordance prediction; rejection sampling fine-tuning (RFT) rejects low-quality policy-generated paths and fine-tunes the model on retained manipulation-successful trajectories; and reinforcement learning from AI feedback (RLAIF) uses execution-grounded counterfactual preferences to further improve navigation decisions. To support this framework, we establish the Nav-N-Manip benchmark and construct Nav4Manip-Pairs from benchmark rollouts, providing large-scale paired navigation–manipulation trajectories and rollout-derived affordance supervision. Extensive experiments on established vision-language navigation (VLN) benchmarks and Nav-N-Manip demonstrate state-of-the-art performance, while real-world evaluations show that Nav4Manip completes diverse dexterous mobile-manipulation tasks. Project website: https://nav4manip-anonymous.github.io/Nav4Manip.

Primary area:
applications to robotics, autonomy, planning
Keywords:
navigation, manipulation, mobile manipulation, vision-language navigation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AQR4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing VLN methods does not provide a stable enough robot base pose for downstream manipulation. This paper proposes a co-evolution system with a navigation model called NAV4MANIP-N and a manipulation model called NAV4MANIP-M. The navigation and manipulation policy are first trained with a combined dataset generated within this paper. The manipulation policy is then fixed while the navigation policy is finetuned with RFT and RLAIF to further improve the end-to-end performance. The method is evaluated on the built benchmark and in real-wolrd, and demonstrates strong end-to-end performance on navigation-manipulation tasks.

Critical strengths
  1. The problem is well-motivated.
  2. The method extends navigation-manipulation policy to navigation-dexterous-manipulation policy with multi-fingers end-effector. And the reported results are promising.
Critical weaknesses
  1. This method highly relies on simulator. Both two finetuning process rely on resetting the environment to specific state and obtain different trajectories.
  2. The paper emphasize on bidirectional training. However, the manipulation policy is fixed after the first stage training. Only the navigation policy is improved with a frozen manipulation policy.
Questions for authors
  1. The presentation is bad. Figure 2 and figure 3 are purely AI-generated without any further polishment. Please at least make sure the letters in each word are equally spacing. The names for some modules are not consistent, such as Nav-N-Manip/NAV-N-MANIP.
  2. The experiments section lacks of lots of details. The benchmark details, the training and evaluation setting are all very coarse.
2 comments

Thank you for your time and helpful feedback on our method, figures, and experimental details. We have revised the manuscript and conducted additional experiments to address your comments below.

Q1: "1. This method highly relies on simulator."

A1: Following standard VLN training paradigms (Uni-NaVid [1], NavFoM [2], InternVLA-N1 [3]), training in simulation enables safe, large-scale exploration with strong foundational generalization. To ensure our method does not rely on a single simulator, we implemented three designs:

  1. Cross-simulator mix-training: We jointly train across large-scale heterogeneous datasets collected from multiple simulators and scene engines, combining Habitat-based navigation data (R2R, RxR, HM3D ObjectNav) with our coupled MuJoCo physics-verified rollouts and NVIDIA Isaac Sim path-traced renderings (EmbodiedGen v2 and VLNVerse scenes).
  2. Domain randomization: During trajectory rollouts and rendering, we apply extensive randomization over house layouts, lighting conditions, and camera viewpoints (height, pitch, and field of view).
  3. Real-to-Sim modeling: We reconstruct real-world environments to align visual appearance, geometry, and obstacle interactions with physical deployment. Video visualizations of our reconstructed Real-to-Sim dataset are shown at https://nav4manip.github.io/.

Real-world deployment demos are also shown at https://nav4manip.github.io/. All real-world policies were trained strictly in simulation without overfitting to the real test environment, achieving high end-to-end success rates on physical hardware.


Q2: "Both two finetuning process rely on resetting the environment to specific state and obtain different trajectories."

A2: We apologize for the inaccurate phrasing in our original manuscript (specifically "We then reset the simulator to the same task initialization..." in Section 3.5, as well as the descriptions in Section 3.4 and Algorithm 1), which caused this confusion. In our actual implementation, neither RFT nor RLAIF requires resetting the environment to specific states to obtain different trajectories. Both stages rely purely on clean on-policy trajectories without state resets. Compared to SFT on static offline data, training and RL fine-tuning on reset-free on-policy data allow the policy to build an intrinsic understanding of the environment without state priors, addressing counterfactual inference and distribution shift more effectively. We have corrected these descriptions in Sections 3.4, 3.5, and Algorithm 1 of the revised manuscript.


Q3: "2. The paper emphasize on bidirectional training. However, the manipulation policy is fixed after the first stage training. Only the navigation policy is improved with a frozen manipulation policy."

A3: To complete the bidirectional loop, after VLN training we ran inference with the trained VLN policy in both simulation and real-world scenes, used the resulting navigation endpoints to re-collect WAM training data, and conducted a second round of post-training on the WAM:

Training SettingSim SR@E2E (100 trials)Real SR@E2E (30 trials)
Original WAM30%17%
Fine-tuned WAM52%30%

As shown in the table and the simulation/real-world videos at https://nav4manip.github.io/, fine-tuning the WAM on rollouts from the updated VLN endpoints increases end-to-end success rates from 30% to 52% in simulation and from 17% to 30% in the real world, completing the bidirectional training loop.


Q4: "1. The presentation is bad. Figure 2 and figure 3 are purely AI-generated without any further polishment. Please at least make sure the letters in each word are equally spacing. The names for some modules are not consistent, such as Nav-N-Manip/NAV-N-MANIP."

A4: We have redrawn and replaced Figure 2 and Figure 3 to fix the character spacing issues, and standardized all module names and notations across the main text in the revised manuscript.


Q5: "2. The experiments section lacks of lots of details. The benchmark details, the training and evaluation setting are all very coarse."

A5: We have added explicit pointers in Section 4.1 and the Reproducibility Statement, and expanded the appendix with full experimental details:

  • Training setups and hyperparameters: Appendices B.1–B.4 (Table 5) for Nav4Manip-N and Appendix G.3 for Nav4Manip-M.
  • Simulation benchmark settings and scene splits: Appendices D.1–D.6.
  • Evaluation settings and deployment protocols: Appendices C.1–C.3, Appendices F.1–F.2, and Appendices G.1, G.2, and G.4.

References

[1] Zhang et al., "Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks," RSS, 2025.

[2] Zhang et al., "Embodied Navigation Foundation Model," ICLR, 2026.

[3] InternVLA-N1 Team, "InternVLA-N1: An Open Dual-System Navigation Foundation Model with Learned Latent Plans," CoRR, 2025.

Thank you so much for your reply and new experiments. The reply clarifies my misunderstanding and improves the presentation. I would like to increase the score.

Reviewer_qjfPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Nav4Manip, a framework for integrating navigation and manipulation through co-evolution. It consists of a post-trained WAM for manipulation and a fine-tuned VLM for navigation. After training the WAM, the authors fine-tune the navigation VLM through three stages: Multi-Task Co-Tuning, Rejection Sampling Fine-Tuning (RFT), and Reinforcement Learning from AI Feedback (RLAIF). The authors also introduce the Nav-N-Manip benchmark and Nav4Manip-Pairs dataset. The proposed method achieves strong performance on navigation and manipulation tasks.

Critical strengths
  • Important research problem. The paper addresses the gap between coarse navigation success and downstream manipulation feasibility, an important limitation of traditional VLN.
  • Valuable dataset contribution. Nav4Manip-Pairs provides paired navigation–manipulation trajectories with manipulation-affordance supervision.
  • Extensive experiments. The authors evaluate their approach on multiple benchmarks and provide comprehensive ablation studies.
Critical weaknesses
  • Questionable co-evolution claim. The manipulation WAM remains frozen after post-training, while only the navigation VLM is further fine-tuned. Although manipulation feedback improves navigation, the paper does not demonstrate how navigation improvements lead to updates of the manipulation policy. The authors should better justify the term co-evolution.
  • Insufficient validation of the three-stage pipeline. As shown in Table 3, RFT provides only a marginal improvement in end-to-end success rate (31.01% to 31.79%). A Co-Tuning + RLAIF baseline without RFT is needed to demonstrate the necessity of all three stages.
  • Potential mismatch between offline affordance labels and WAM capabilities. The affordance labels are generated by a Dexonomy-based verifier, whose success rates may differ from those of the learned WAM. The authors should compare both at the same candidate base poses and report the correlation between offline affordance scores and actual WAM success rates.
  • Limited generalization across manipulation policies. The framework is primarily evaluated using a single post-trained WAM. It remains unclear whether the learned navigation policy can transfer to other manipulation models. Cross-WAM evaluation would strengthen the generalization claims.
Questions for authors

See weakness.

2 comments

Thank you for your thoughtful review and for suggesting concrete experiments to further test our pipeline. We have conducted additional closed-loop, ablation, correlation, and cross-policy experiments to address each of your points below.

Q1: "Questionable co-evolution claim. The manipulation WAM remains frozen after post-training, while only the navigation VLM is further fine-tuned. ... The authors should better justify the term co-evolution."

A1: To validate the closed-loop co-evolution between navigation and manipulation, we ran inference with the trained VLN policy in both simulation and real-world scenes. Using the actual stopping poses reached by the VLN policy as the initial base distribution, we re-collected manipulation training data and post-trained the WAM. We evaluated this closed-loop WAM on Nav-N-Manip in both simulation and the real world:

Training SettingSim SR@E2E (100 trials)Real SR@E2E (30 trials)
Original WAM30%17%
Fine-tuned WAM52%30%

Post-training the WAM on rollouts from actual VLN endpoints adapts the manipulation policy to upstream navigation stopping characteristics, improving end-to-end success rates from 30% to 52% in simulation and from 17% to 30% in the real world. Videos of both simulation and real-world experiments are available at https://nav4manip.github.io/.


Q2: "Insufficient validation of the three-stage pipeline. As shown in Table 3, RFT provides only a marginal improvement in end-to-end success rate (31.01% to 31.79%). A Co-Tuning + RLAIF baseline without RFT is needed to demonstrate the necessity of all three stages."

A2: We added the Co-Tuning (MTC) + RLAIF baseline (skipping the RFT stage) and compared it against our original settings:

MethodSR@3[email protected][email protected][email protected]NESR@ManipSR@E2E
MTC80.277.169.852.71.8139.3431.01
MTC + RLAIF84.580.078.564.21.4651.1643.02
MTC + RFT + RLAIF89.584.180.471.71.1556.8752.13

Skipping RFT (MTC + RLAIF) yields end-to-end success rate of 43.02%, lower than the full three-stage pipeline (52.13%). By filtering high-quality trajectories, RFT provides a stable behavioral prior that prevents performance degradation caused by overly large exploration spaces during direct RLAIF fine-tuning.


Q3: "Potential mismatch between offline affordance labels and WAM capabilities. ... The authors should compare both at the same candidate base poses and report the correlation between offline affordance scores and actual WAM success rates."

A3: Because the WAM is trained on expert trajectories provided by the Dexonomy-based oracle manipulation policy, their capabilities are inherently aligned. We evaluated both the Dexonomy oracle and the learned WAM across repeated trials at the same candidate navigation endpoints in simulation:

Candidate EndpointOracle (Dexonomy) SRLearned WAM SR
A100% (20/20)90% (18/20)
B55% (11/20)45% (9/20)
C0% (0/20)0% (0/20)
D0% (0/20)0% (0/20)

The Pearson correlation coefficient between the offline Dexonomy affordance scores and actual WAM grasp success rates is r=0.9987r = 0.9987 (Spearman rank correlation ρ=1.0\rho = 1.0), showing that the offline labels match the actual capabilities of the learned policy. Visualizations of these trials are shown at https://nav4manip.github.io/.


Q4: "Limited generalization across manipulation policies. ... Cross-WAM evaluation would strengthen the generalization claims."

A4: In the same real-world shelf scene, we evaluated 3 independently trained WAMs—each targeting a different task and shelf level—directly from the stopping poses produced by our navigation policy:

Navigation MethodReal Task 1 SR@E2E (10 trials)Real Task 2 SR@E2E (10 trials)Real Task 3 SR@E2E (10 trials)
UniNaVid [1]0%0%0%
Ours30%20%40%

The stopping poses produced by our navigation policy achieve consistent grasp success across all 3 independently trained WAMs, indicating that the navigation model learns generalizable manipulation-aware spatial priors across distinct manipulation policies. Video recordings of this experiment are available at https://nav4manip.github.io/.


References

[1] Zhang et al., "Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks," RSS, 2025.

Thanks for the reply and my concern has been addressed. But I decided to keep my current score.

Reviewer_yd4JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

NAV4MANIP studies the gap between reaching a navigation goal and stopping at a base pose that enables downstream manipulation. Its contributions include Nav4Manip-Pairs, the NAV-N-MANIP benchmark, and a navigation–manipulation system trained through multi-task co-tuning, rejection-sampling fine-tuning, and execution-validated agentic refinement. The final stage distills accepted corrections into the navigator using supervised learning; the manipulation policy remains fixed during feedback-based navigation training. Experiments cover R2R, RxR, VLNVerse, and simulated navigation-to-grasp tasks, supplemented by training and architecture ablations and real-robot demonstrations. Manipulation performance is measured as average single-attempt success over 16 independently reset trials. The principal contribution is a practical data and training framework for improving the handoff between long-horizon navigation and manipulation.

Critical strengths
  1. Important problem and meaningful system-level contribution.The paper addresses an important gap between conventional navigation success and manipulation-ready stopping. Rather than optimizing only for proximity to the target, it explicitly considers whether the terminal base pose supports downstream manipulation. The proposed Nav4Manip-Pairs dataset, manipulation-aware affordance prediction, and execution-feedback training form a coherent framework for addressing this problem. I find this contribution valuable because it connects long-horizon navigation decisions with actual manipulation outcomes and provides a concrete approach to improving the handoff between these two capabilities.
  2. Convincing experimental results and comprehensive real-world validation.The experimental results provide convincing evidence of the effectiveness of the overall framework.
Critical weaknesses

The main unresolved methodological issue concerns the filtering used to construct the offline affordance labels. The manuscript leaves an important boundary of that filtering insufficiently specified, but this does not establish that legitimate failures were excluded or that endpoint rankings are biased. I therefore treat this as a clarification request below, rather than as a demonstrated failure of the method.

Questions for authors

What exactly causes rejection before an affordance evaluation cell is created?

Appendix D.2 and Eq. 34 make clear how failures within recorded evaluation cells contribute to the score. The remaining ambiguity concerns rejection before cell creation. Can an initial object configuration that is physically valid and belongs to the intended task distribution be omitted because the current dock admits no feasible manipulation target? Please distinguish invalid initializations, rejected action candidates within a task, and entire valid initializations for which the dock permits no feasible action.

If the last category can be excluded, how is it accounted for when comparing candidate docks? The frequency of such exclusions and their effect on the selected endpoints would help determine whether the scores remain comparable across docks.

1 comment

Thank you for your careful reading of Appendix D.2 and for framing the evaluation-cell filtering boundary as a clear, constructive question. We address each part of your inquiry below.

Q1: "What exactly causes rejection before an affordance evaluation cell is created? ... Please distinguish invalid initializations, rejected action candidates within a task, and entire valid initializations for which the dock permits no feasible action."

A1: To directly answer your question: only physically invalid object initializations (Category 1) can be rejected before cell creation; a valid object initialization is never omitted even if the current dock admits no feasible manipulation target (Category 3).
Specifically, the three categories you highlighted are distinguished and handled as follows:

  • Case 1 (Invalid initializations): When constructing the simulation scenes, we enforce physical validity constraints and collision checks on initial object placements and randomization. Configurations with physical penetration or out-of-bound placement are prevented during scene generation and never enter the evaluation pipeline.
  • Case 2 (Rejected action candidates within a task — feasible plan, failed physics verification): For valid object configurations, if the planner generates a trajectory that subsequently fails physics verification (e.g., due to grasp instability), the attempt creates a record in the Affordance Evaluation Cell, is scored as 00, and is counted in the denominator of Eq. (34).
  • Case 3 (Valid initializations where the dock permits no feasible action): If the object configuration is valid but the candidate dock is too far or poorly positioned for the planner to find any feasible trajectory, the attempt also creates a record in the Affordance Evaluation Cell, is scored as 00, and is fully included in the denominator of Eq. (34).

Q2: "If the last category can be excluded, how is it accounted for when comparing candidate docks? The frequency of such exclusions and their effect on the selected endpoints would help determine whether the scores remain comparable across docks."

A2: Because Category 3 cases are not excluded prior to cell creation, all candidate docks for a given task are evaluated over the exact same set of valid initial object configurations. To provide quantitative transparency on how frequently each outcome occurs, we audited all candidate dock evaluations across our training set:

  • Case 1 (Invalid initializations): 0%
  • Case 2 (Planning succeeded, physics verification failed): 19.72%
  • Case 3 (No feasible plan due to poor or distant dock pose): 67.69%
  • Successful execution: 12.59%

As these statistics show, Category 3 accounts for the majority of failed trials (67.69%), and including them in the denominator is essential for properly penalizing unsuitable docks. We have also added visual examples of Category 2 and Category 3 failures at https://nav4manip.github.io/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.