acceptodds
Under review as a conference paper at ICLR 2027

NaviVLA: Socially-aware Robot Navigation Vision-Language Action Model

Abstract

Socially-aware robot navigation remains meaningful yet challenging, as robots must jointly optimize goal reaching, collision avoidance, human comfort, and implicit social norms under uncertain environmental dynamics. Existing methods often depend on simplified 2D geometric states, while recent many VLA models are not directly designed for socially compliant navigation. We propose NaviVLA, an RL fine-tuned social navigation VLA planner equipped with a flow matching action expert. NaviVLA integrates BEV maps, multi-sensor input, and world model powered language context to perform social navigation reasoning. Moreover, to support scalable and realistic training, we further construct an IsaacSim-based social navigation benchmark with diverse pedestrian dynamics simulators, standardized baselines, and multiple metrics. Its crowd module integrates both rule-based models and diffusion-powered simulators trained on real-world crowd trajectory datasets. Additionally, we introduce a RLHF-based social norm encoder to align the policy with respect to human preferences. Experiments in simulation and a physical robot show that NaviVLA improves navigation success, safety, and social compliance, establishing a new VLA-based framework for socially-aware robot navigation. The project website is accessible at: https://sites.google.com/view/NaviVLA

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Social Robot Navigation, RL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JmFXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors present NAVIVLA, a framework for social/crowd navigation via VLA and RL finetuning. Specifically, they adopt a Qwen3.5-9B as the backbone, a flow matching module as the action generator, and performed two-stage RL finetuning. The first stage of RL finetuning targets at transforming general models to function in the context of crowd navigation, following the finetuning strategy of Pi 0.6, and the second stage uses LoRA and DPO, targeting directly at the metrics regarding social norms. The effectiveness of the pipeline is evaluated in an end-to-end manner, using the metrics like success rate (for safety), social score (for social awareness), uncomfortable rate (for social awareness), and navigation time (for efficiency). The authors also present a new crowd navigation simulator using Isaac Sim.

Critical strengths

The paper set up a new simulator for crowd navigation using Isaac Sim, which could be potentially beneficial for enabling more photorealistic and sensor-integrated simulations. The methodology used in this paper is also quite complicated and combined modern techniques. The training details are provided in the appendix including the prompts.

Critical weaknesses

While the paper presents a framework that takes multi-modality data as input, the task the authors present is still crowd-only navigation (Figure 5) without complex buildings or static objects as background, which makes the problem relatively well solved in existing work. In the same level of complexity of about 20 humans with dense interactions, MPC-based, CrowdNav++ (arXiv:2203.01821), and GenSafeNav (arXiv:2508.05634) could potentially reach 90%+ in success rate, but the authors did not include these methods in the evaluation table. The authors also need to show the semantic expressiveness of their method brought by the 9B model rather than only using common metrics, otherwise it does not make a lot of sense to use a 9B model for this task while a simple RL policy with millions of parameters could easily solve it.

Also, while the authors claim they are using Isaac Sim to enable a new powerful benchmark, the experiments shown in this paper do not show the benefits it brings. Although in Figure 2 and 3, the authors show the scenarios of real-life and photorealistic rendering, the actual setup used in this paper is quite simple (Figure 5). The VLA policy trained in human-only simulations can hardly generalize to real-world deployment. The real-world validation shown in Figure 4 is also weak with interactions with a single human.

Overall, if the authors want to validate the effectiveness of NAVIVLA, I genuinely suggest developing and validating the algorithms in real-life and cluttered environment to prove the worth brought by the large model and extra computation overhead, and include comprehensive baselines to perform fair and persuasive evaluations.

Questions for authors
  1. For figure 3, are the pictures in "Social Navigation Scenario" and "Camera Images" rendered from simulations?
  2. Different robots have different kinematics, how are the actions generated by the flow matching action expert converted to the actual executing of robots? Since in B.2.1 the generated action is defined as a velocity command [vx, vy, ωz], in simulations, does this command applied directly to different robots or applied as a target for the controllers to follow?
  3. After fine-tuning the 9B model, how can we evaluate the expressiveness of the Qwen backbone to rule out the possibility that the model has simply overfitted to the crowd navigation task and lost its general capabilities?
Reviewer_3PCLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces NaviVLA, a vision-language-action model for socially aware robot navigation. The model combines multi-camera observations, bird’s-eye-view maps, and textual context generated by a world model, using reinforcement learning and a flow-matching action expert to produce navigation commands. The authors also introduce an Isaac Sim benchmark with multiple crowd-generation and pedestrian-motion models, and a social norm encoder trained using human preferences. Evaluation is conducted in simulation and on a physical robot, with reported improvements in navigation success, safety, and social compliance.

Critical strengths

I could not confidently assess the technical strengths because of the presentation and formulation issues.

Critical weaknesses

The manuscript requires substantial rewriting before it can be assessed reliably. Citation-formatting problems repeatedly interrupt the prose and make reading and comprehension considerably harder. For example, L124–125 introduces “Dec-POMDP Åström (1965)” without expanding the acronym or integrating the citation grammatically. The term should first be introduced as “Decentralized Partially Observable Markov Decision Process (Dec-POMDP),” followed by an appropriately formatted citation and a clear explanation of the formulation.

Section 3.1 is particularly difficult to follow and contains inconsistent notation. At L130, a pedestrian action is described as the robot action. At L135–136, the low-level output uses conventional joint-velocity notation but is described as joint-pose commands. At L125–127, the tuple introduces Ω\Omega, but its role and conditioning are not clearly explained. These issues obscure the meaning of the formulation and prevent a reliable assessment of the methodology. In its current form, I recommend rejection.

Finally, while detailed supplementary materials can be valuable for documenting experiments and implementation, the submitted PDF contains 70 pages, including supplementary material, and I have not read all of it. My assessment is based on the main paper and the supplementary sections I consulted. I encourage the authors to organize the additional material around information directly relevant to the contribution, with clear references from the main text. Essential definitions and methodological explanations should be self-contained in the main paper so that readers can understand and assess the work efficiently.

Questions for authors

I have no additional questions. My concerns are detailed in the weaknesses above.

Reviewer_YQJ8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies socially aware navigation for a quadruped robot in dynamic crowds and proposes NaviVLA, a VLA used as the high-level planner. Its inputs are four RGB-D views, a BEV map, and structured text from a social-navigation world model, including the robot state, goal, and pedestrian positions and velocities. Qwen3.5-9B with LoRA encodes these inputs, followed by learnable-query pooling and a "flow matching action expert" that outputs a body-frame velocity command [vx, vy, ωz] for a pretrained Spot locomotion policy. Starting from a deterministic action, the expert applies K Euler refinement steps; the result is the mean of a Gaussian policy. Training uses PPO, first with a frozen backbone and then with LoRA, followed by online DPO. Preference pairs are labeled automatically using uncomfortable rate, social violations, and social score. The paper also introduces an IsaacSim benchmark with Circle/Random/Corridor/Group and data-driven Human-Diffusion crowds, ORCA/Social-Force/diffusion pedestrian models, and 12 baselines, of which the 8 learning-based ones are retrained with the same PPO configuration. All trainable methods are trained only on Random+ORCA and ETH. NaviVLA reports the highest average social score (75.36 vs. 52.75 for NaviSTAR), with SR 0.790 (0.809 without DPO). The real-robot section shows one qualitative example.

Critical strengths

S1. A useful simulated benchmark for social navigation. It combines a quadruped robot, cameras, ray-casting and contact sensors in IsaacSim physics with five crowd generators—including a diffusion crowd model trained on real trajectory data—and three pedestrian motion models. The authors also compare 12 classical and learning-based planners under one protocol. Since much social-navigation work still uses simplified 2D simulators, this platform would be useful to the community if released.

S2. A practical recipe for on-policy RL of a large VLA in closed-loop crowds. The appendix describes the split between rollout and training machines, the frozen-to-LoRA schedule, and a comparison of training strategies in a grid world (Table 6). These details show that PPO training of a 9B VLA in IsaacSim is feasible. On the Human-Diffusion average, NaviVLA also outperforms all baselines (SR 0.906, SC 93.80).

Critical weaknesses

W1. The reported results do not yet establish the claimed safety and social-compliance gains. The tables report SR, SC, NL, and NT, but not collision rate, uncomfortable rate, or minimum human distance. On rule-based crowds, NaviVLA has lower average SR than NaviDIFF (0.674 vs. 0.710), while its paths are shorter. Without the safety and comfort results and SC weights k1–k4, I cannot assess this trade-off. The real-robot evidence is also limited to one qualitative example, so the corresponding claims should be supported quantitatively or narrowed.

W2. The contribution of the VLA inputs and preference alignment remains unclear. The paper does not clearly explain whether pedestrian states in the world-model text come directly from the simulator or are estimated from sensors, including how visibility and occlusion are handled. Information beyond the camera range could come from RayCaster sensing or the BEV map, so range alone is not evidence of privileged access. The main issue is the lack of modality ablations showing what images and BEV add beyond structured states; a VLA/VLM navigation baseline would also help. DPO uses automatic labels derived from UR and SC. This is a reasonable training signal, but the +8.08 SC gain does not by itself establish alignment with human preferences. I would like the authors to clarify that claim and what DPO adds beyond using the same social criteria in PPO rewards.

Questions for authors

Q1 (W1): Could you report collision rate, UR, minimum human distance, and timeout rate for all methods in Table 1, together with k1–k4? Did all baselines use the same PPO hyperparameters, or were they tuned separately? Why do the MLP and Transformer baselines fall below A*?

Q2 (W2): Are the pedestrian positions and velocities in the world-model text taken directly from the simulator or estimated from sensors? How are visibility, range limits, and occlusion handled? What are the results with text only and with images/BEV only? How does the action expert differ from standard flow matching in its training objective and noise sampling, and how does it compare with a plain MLP head?

Q3 (W1, W2): How many preference pairs, what β, and how many update steps are used for DPO in IsaacSim? Were any preferences provided by human annotators, and is there an independent assessment of human preference alignment? Could the authors explain, through available comparisons or analysis, what DPO adds beyond using the same social criteria in PPO rewards?

Q4 (W1): What onboard compute is used for the real robot, and what is the per-step inference latency of the 9B model? How are pedestrian positions and velocities obtained in the real world? Could you report success and collision counts over a small number of repeated trials? If not, I suggest describing the real-robot result in the abstract as a qualitative demonstration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.