acceptodds
Under review as a conference paper at ICLR 2027

SocioWM: Social-Conditioned World Model for Social Context Vision-and-Language Navigation

Abstract

Conventional vision-and-language navigation (VLN) enables embodied agents to navigate static scenes by following natural-language instructions. Existing Social-VLN works further introduce moving pedestrians into navigation scenarios, but primarily model them as dynamic obstacles or generic visual entities, overlooking the social context that influences pedestrian behavior, including personal attributes and human–robot interactions. Consequently, existing methods may fail to account for attribute-dependent differences among pedestrians and their interaction responses to robot actions, leading to navigation behavior that prioritizes goal progress while neglecting social norms. To address this limitation, we formulate Social Context Vision-and-Language Navigation (Social Context-VLN), a new VLN setting that incorporates pedestrian motion and social context into navigation decision-making. To support this setting, we construct , the first benchmark for Social Context-VLN. It explicitly annotates Pedestrian Attribute, including age and gender, and Human–Robot Interaction as social context, and contains more than 35K language-guided navigation episodes involving 1.4K pedestrians. To validate the effectiveness of these structured annotations, we further introduce , a socially conditioned world model that predicts the social consequences of candidate routes and selectively modulates candidate world states for action selection. Experiments on show that improves the social success rate (SR) from 18.27% to 27.07%, while reducing collisions and improving personal-space compliance. Evaluation of existing VLN methods further shows that strong conventional navigation performance does not necessarily imply socially compliant behavior. Real‑world experiments verify our method transfers to unseen indoor environments and real‑world pedestrian interactions.Ours project page:https://anonymous.4open.science/r/iclr2027anony

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-and-language navigation, embodied AI, social navigation, human-robot interaction, world models, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BFvDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Social Context-VLN, a VLN setting in which an agent must reach an instructed goal among moving pedestrians while taking into account pedestrian attributes (age, gender) and human–robot interaction (HRI) responses (passing, slowing down, stopping, yielding). To support this setting, the authors build SocialBench-VLN in Habitat: 35K language-guided episodes (33,659 train / 1,839 validation, scene-disjoint) with 8–16 pedestrians per scene (1.4K pedestrians in total). Each pedestrian's appearance, motion parameters, latent personality, and predefined interaction behaviors are generated from its assigned age and gender. Per-step attributes, HRI states, and future motion are recorded as privileged training annotations.

They also propose SocioWM, an add-on to the SC²-WM candidate-conditioned world model. Route-Conditioned Social Reasoning (RCSR) encodes age, gender, interaction, and motion tokens from RGB-D. It interacts bidirectionally with each candidate route to produce a candidate-specific "Social Future Latent", and a gated residual injects the resulting correction into each reconstructed candidate state before scoring. Training adds collision, interaction, motion, and attribute (KL) losses computed on the simulator labels.

On the validation split, SocioWM raises SR (goal reached without collision) from 18.27% to 27.07% over SC²-WM and gives the best ISR, PSC, and Coll. among seven VLN baselines. The paper also reports module, annotation, and loss-weight ablations, an inference-time attribute-masking analysis on two scenes, and a qualitative real-robot sequence on a Unitree G1.

Critical strengths

1. A benchmark with structured, per-step pedestrian annotations, and a useful diagnostic finding. SocialBench-VLN pairs language-guided episodes with time-aligned pedestrian attributes, HRI responses, semantic destinations, and future motion. It uses scene-disjoint splits and formally defined metrics (Eq. 13). Table 2 shows that seven recent VLN agents, including strong video-LLM agents, come within 0.6 m of a pedestrian in 52–59% of episodes. In interaction-triggered episodes they reach the goal collision-free only 5–9% of the time. A resource with privileged social labels would be useful for work on socially aware VLN if it is released as promised. This is the main reason I see value in the submission.

2. A modular design with informative ablations. SocioWM adds social reasoning to an existing candidate-conditioned world model without modifying the backbone, and it needs only RGB-D and the instruction at inference. Its design choices are ablated. Removing the gate lowers SR to 20.74, and making the route–social interaction unidirectional lowers it to 24.25. The "w/o All Structured Sup." control (SR 19.34) shows openly how much of the gain comes from supervision rather than architecture. Appendix B documents the full data-generation chain, which makes the benchmark's assumptions inspectable.

Critical weaknesses

1. The central claim is only partly supported. The claim is that conditioning on social context (age, gender, HRI) is what makes navigation socially compliant. The evidence supports the value of interaction supervision inside this simulator. It does not yet support the role of age and gender, and it does not show that the effect holds beyond the simulator's own rules.

(a) Training and evaluation share the same hand-designed behavior rules. Every simulator benchmark relies on generative rules, so the simulator itself is not the problem. The problem is that the model is supervised with the rules' own parameters and tested only under the same rules. Age and gender set walking speed, turning, and reaction distance, and they shape the sampling of a latent personality. The personality then selects a predefined reaction, triggered when the robot enters an attribute-scaled 2 m region (Appendix B.1, B.3). Training and validation episodes differ only in scenes, so the current results cannot separate learning social regularities from learning the generator. Two additions would address this:

  • a rule-shift evaluation split, for example held-out personality types, a different attribute-to-personality mapping, or rescaled reaction distances;
  • a grounding check for the generator: compare the simulated walking speeds, reaction distances, and yielding behavior per age group with published measurements, and cite the sources used to set the mapping.

(b) Attribution of the gain. The full architecture trained without structured supervision reaches SR 19.34, against 18.27 for SC²-WM (Tables 3–4), so roughly 7.7 of the 8.8-point gain comes from the privileged labels. Table 5 splits this further. With λ = 0, which drops the interaction and attribute losses but keeps the collision and motion losses, SR is 23.12. About half of the 7.7 points therefore comes from collision and motion supervision, which are standard auxiliary signals in social navigation. The interaction and attribute losses add about 4 points, mostly through interaction. Age and gender, which motivate the paper, add about 1.1 points (27.07 vs. 25.94 without them).

Aggregate SR is probably the wrong place to look for an attribute effect, because most episodes may not depend on who the pedestrian is. The paper could make the case directly:

  • report results on the interaction episodes where attributes change pedestrian behavior most (for example elderly or child pedestrians, or cautious vs. aggressive personalities), and per age and gender group;
  • report how accurately the encoder recovers age and gender at inference;
  • align the metrics with the generator. PSC and Coll. use fixed 1.0 m and 0.6 m thresholds for every pedestrian, while the simulator scales reaction distance by attribute. An attribute-aware personal-space metric would test the behavior the paper motivates.

This weakness decides whether the paper contributes a new insight (social context matters for VLN) or an auxiliary-supervision recipe that works in a simulator built around those same labels.

2. The comparisons and the transfer evidence weaken the headline conclusions.

(a) Baseline comparability. The paper never describes how the baselines were trained on SocialBench-VLN, while SocioWM receives about 100K in-domain rollouts with privileged labels (Sec. 4.1). Table 3's "w/o RCSR" row matches the SC²-WM row to two decimals, so it is unclear whether any baseline saw the benchmark's training split. Please state the protocol for each baseline (zero-shot or fine-tuned, and on what data). Two comparisons would isolate the contribution of RCSR:

  • SC²-WM fine-tuned with the same collision, motion, and interaction losses on plain heads;
  • a social-navigation baseline with future-aware auxiliary tasks, for example Falcon from Gong et al. (2025), "From Cognition to Precognition", which is in the paper's own reference list.

(b) No evaluation outside the authors' simulator. SocioWM is not tested on HA-VLN or DPed-VLN. Even a zero-shot evaluation on one of them would show whether the learned social reasoning transfers to pedestrian behavior the authors did not design. DPed-VLN, the closest concurrent benchmark, appears only in Table 1; a paragraph comparing the two benchmarks would help readers. The "first benchmark" claim rests on Table 1's Attr. and Inter. columns, which are by construction the paper's own annotation types. Stating it as "the first VLN benchmark with structured pedestrian-attribute and HRI annotations" would be accurate.

(c) Real-world claim. The abstract and conclusion say that real-world experiments "verify" transfer, but the evidence is one qualitative sequence (Fig. 4, bottom) and a hardware description (App. D). A small quantitative study would support the claim, for example a fixed set of routes with a crossing pedestrian, reporting success and minimum robot–pedestrian distance for SocioWM and SC²-WM. Without it, "demonstrate" is more accurate than "verify".

Additional, lower-weight issues (not decisive for my rating):

  • The claim that strong conventional navigation does not imply social compliance is supported by Coll. and PSC. For example, AwareVLN has the second-highest SR among the baselines but the worst Coll. (59.05%), while SC²-WM has a low SR but the best baseline Coll. (52.31%) and PSC (88.60%). It is not supported by ISR, which rises with SR across all seven baselines because it applies the same collision-free-goal indicator to a subset of episodes. Stating the claim in terms of Coll. and PSC would make it stronger.
  • Fig. 2(c) reports "3.5K+" episodes, while the text reports 35K.
  • The Introduction states SR rises from 19% to 27%; Table 2 and the abstract give 18.27%.
  • SR is called the "social success rate" in the abstract but a "conventional navigation metric" in the Table 2 caption, although Eq. 13 includes the collision term. One consistent label would avoid confusion.
  • Fig. 3 contains typos ("SocialWM", "Socinl Context modlulation").
  • L_nav, the collision labels used by L_col, and the "enhanced SAP head" are not defined. A short description in the appendix would make the method self-contained.
  • The leave-one-attribute-out analysis in App. C is shown for two scenes. Aggregating ΔP over all interaction episodes would turn it from an illustration into evidence.
Questions for authors

My score depends mostly on Q1–Q3. If SocioWM keeps a clear margin over a matched SC²-WM + auxiliary-loss baseline and a social-navigation baseline, I would be more confident in acceptance. If the baselines were evaluated zero-shot and the matched control closes most of the gap, I would lower my score.

Q1 (clarification and requested experiment). Baseline protocol. Were NaVid, UniNaVid, NaVILA, StreamVLN, AwareVLN, InternNav, and SC²-WM trained or fine-tuned on the SocialBench-VLN training split, and with how many rollouts? Could you add SC²-WM fine-tuned with the same collision, motion, and interaction losses but without RCSR, and one social-navigation baseline?

Q2 (requested experiment). Robustness to changed behavior rules. How does SocioWM perform when the behavior rules change between training and testing? Examples: held-out personality types, a different attribute-to-personality mapping, rescaled reaction distances, or zero-shot evaluation on HA-VLN or DPed-VLN.

Q3 (requested analysis). Where attributes matter. On interaction episodes with the pedestrian groups whose behavior differs most (for example elderly or child pedestrians, or cautious vs. aggressive personalities), how do SocioWM, the w/o Attribute variant, and SC²-WM compare? How accurately are age and gender recovered at inference? Does the agent keep more distance from pedestrians whose simulated reaction distance is larger?

Q4 (clarification). ISR denominator. E_inter appears to depend on the agent's own trajectory, since interactions are triggered by the robot. If so, ISR is computed over different episode sets for different methods. Could you report |E_inter| per method, or compute ISR on a fixed set of episodes in which the reference path triggers an interaction?

Q5 (clarification). Real-world experiments. How many runs and environments are behind the real-world claim, and how did SC²-WM behave in the same setting? Were consent, institutional review, and safety procedures in place for the participants?

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer BFvD(1/4)

We sincerely appreciate your constructive and insightful comments. We address your concerns point by point below.

Q1 & W2(a): baseline protocol and auxiliary‑supervision controls

Q1, original. Were NaVid, UniNaVid, NaVILA, StreamVLN, AwareVLN, InternNav, and SC²‑WM trained or fine‑tuned on the SocialBench‑VLN training split, and with how many rollouts? Could you add SC²‑WM fine‑tuned with the same collision, motion, and interaction losses but without RCSR, and one social‑navigation baseline?

(1.1) In‑domain fine‑tuning and zero‑shot comparisons. Thank you for your question. The original evaluation fine‑tuned selected baselines; SocioWM retains an advantage after in‑domain adaptation, reaching27.07SR versus 25.82 for fine‑tuned InternNav and 22.62 for fine‑tuned SC²‑WM. The expanded comparison now includes NaVid, UniNaVid, NaVILA, StreamVLN, InternNav, and SC²‑WM on the same SocialBench‑VLN training split and evaluation protocol. NaVid is included in both the zero‑shot and in‑domain columns. We agree that in‑domain AwareVLN fine‑tuning would be valuable. Its training recipe requires reasoning‑text supervision absent from SocialBench‑VLN, so the current annotations do not directly support that protocol. We report its zero‑shot results alongside the expanded in‑domain comparisons below, while a matched AwareVLN fine‑tuning evaluation remains future work.

Table 2: Zero‑shot and fine‑tuning (FT) results on SocialBench‑VLN.

MethodZero‑shotFT
SR ↑SPL ↑PSC ↑Coll. ↓ISR ↑SR ↑SPL ↑PSC ↑Coll. ↓ISR ↑
NaVid16.8612.1587.7355.215.14 (57/1109)21.6416.1087.9553.406.60 (74/1121)
UniNaVid17.5612.8685.5654.875.52 (61/1105)22.2916.5088.1051.117.00 (78/1114)
NaVILA20.3418.9285.7755.366.05 (69/1140)24.6920.4588.4051.507.50 (85/1133)
StreamVLN17.7313.8785.0258.245.24 (61/1164)21.1518.5886.2556.886.27 (72/1148)
AwareVLN22.9519.6486.9959.056.44 (72/1118)----------
InternNav23.4420.1185.4256.019.14 (100/1094)25.8222.2887.0253.948.72 (97/1112)
SC²‑WM18.2713.6688.6052.315.60 (62/1107)22.6216.9088.9051.226.70 (75/1119)
SocioWM----------27.0720.7890.3547.2511.20 (126/1125)

(1.2) Training budget, matched supervision, and the added baseline control. Thank you for your constructive suggestion. SocioWM maintains higher success and social compliance than ordinary fine‑tuning, matched auxiliary supervision, and the trajectory‑prediction control. In particular, its SR exceeds Aux./Attr. heads by 1.97 points, supporting the contribution of candidate‑conditioned social modeling beyond the additional labels alone. Compared with the trajectory‑prediction control, SocioWM improves SR by 5.65 points and reduces Coll. by 2.40 points. The SC²‑WM‑based comparisons use SocioWM's training settings and 40K + 60K global‑rollout budget. NaVid, UniNaVid, NaVILA, StreamVLN, and InternNav use two‑epoch LoRA fine‑tuning. These baselines reuse the provided SocialBench‑VLN offline training episodes without generating additional online rollouts during fine‑tuning, following hyper‑parameters from their public implementations. Aux. heads receive collision, motion, and interaction supervision without RCSR; Aux./Attr. heads additionally receive age and gender supervision. These architectural comparisons use single trained checkpoints. The Trajectron++ control retains the candidate planner and reconstructs pedestrian histories from past RGB‑D detections and tracks using depth and robot pose.

ConfigurationSRPSCColl.
SC²‑WM original18.2788.6052.31
SC²‑WM ordinary FT22.6288.9051.22
SC²‑WM + Aux. heads24.8089.6049.00
SC²‑WM + Aux./Attr. heads25.1089.8048.60
SC²‑WM + Trajectron++21.4289.6249.65
Full SocioWM27.0790.3547.25

We also agree that independent social‑navigation comparisons are valuable. InternNav, already included in our comparison, has demonstrated strong social‑navigation performance on the MP3D‑based Social‑VLN benchmark in its associated paper (Ground Slow, Move Fast, Section 4 and Table 3), making it a relevant VLN reference with social‑navigation capabilities. Falcon is a point‑goal method and requires an additional language‑grounding interface for this VLN task. The shared‑planner Trajectron++ control further addresses the anticipatory‑modeling aspect of your suggestion; a matched standalone Falcon comparison remains future work.

AuthorsOpenReview ↗

Official Response to Reviewer BFvD(2/4)

Q2 & W1(a) & W2(b): changed behavior rules and external transfer

Q2, original. How does SocioWM perform when the behavior rules change between training and testing? Examples: held‑out personality types, a different attribute‑to‑personality mapping, rescaled reaction distances, or zero‑shot evaluation on HA‑VLN or DPed‑VLN. Related weakness excerpts. “a grounding check for the generator”; “Even a zero‑shot evaluation on one of them would show whether the learned social reasoning transfers to pedestrian behavior the authors did not design.”

Thank you for your question.

(2.1) Robustness to changed appearance–behavior associations. Thank you for your constructive suggestion. In this experiment, we freeze all trained model checkpoints without further fine‑tuning. We shuffle and reassign pedestrian appearance assets together with their corresponding age‑gender labels, while strictly keeping the original sampled pedestrian behavioral configurations and total appearance counts unchanged. After this reassignment procedure, SocioWM retains a 0.83‑point SR advantage (25.83 versus 25.00), together with higher PSC and lower Coll. This supports a benefit that is not wholly dependent on the original simulator‑defined appearance–behavior pairing.

Association diagnostics.

AssociationStatisticOriginalReassigned
Age–linear speedη2\eta^20.8270.003
Age–angular speedη2\eta^20.8440.004
Age–reaction distanceη2\eta^20.1300.003
Age–personalityCramér's VV0.1370.025

For the navigation results below, Full uses one trained checkpoint; w/o Attribute reports five‑seed means with 95% confidence‑interval half‑widths.

ConditionMethodSRPSCColl.
Originalw/o Attribute25.75±0.3225.75\pm0.3289.90±0.2389.90\pm0.2348.80±0.3748.80\pm0.37
OriginalFull27.0790.3547.25
Reassignedw/o Attribute25.00±0.3725.00\pm0.3789.54±0.2689.54\pm0.2649.35±0.4349.35\pm0.43
ReassignedFull25.8389.8348.46

(2.2) Robustness to reaction‑distance scaling. Thank you for your constructive suggestion. For this robustness test, all model checkpoints remain frozen. We only scale the built‑in pedestrian reaction‑distance parameter by factors α={0.8,1.0,1.2}\alpha = \{0.8,1.0,1.2\} (‑20 %, original, +20 %). All validation episodes (1,839 total) are evaluated with fixed, unmodified metric thresholds. For ISRcommon\mathrm{ISR}_{\mathrm{common}}, we select 600 shared episodes where every model configuration triggers interactions under all three scaling factors.

Under these modified pedestrian interaction rules, SocioWM maintains higher SR, PSC, and common‑subset ISR and lower Coll. than both controls at all three scales. Reducing or increasing reaction distances by 20% gives SR gains of 4.68 and 4.41 points over ordinary fine‑tuned SC²‑WM, and 1.09 and 1.32 points over the five‑seed w/o Attribute means.

SocioWM under reaction‑distance scaling.

α\alphaSR ↑PSC ↑Coll. ↓ISRcommon↑\mathrm{ISR}_{\mathrm{common}}\uparrow
0.825.2389.8648.789.33
1.027.0790.3547.2511.00
1.226.4390.2247.7410.67

(2.3) Transfer to external human‑aware navigation benchmarks. Thank you for this valuable suggestion. Compared with SC²‑WM ordinarily fine‑tuned on SocialBench‑VLN, SocioWM improves SR by 3.15 points on DPed‑VLN and 4.32 points on HA‑VLN‑CE, while reducing human collision rates by 6.84 and 8.38 points, respectively. These results support transfer of the learned social representations across benchmarks.

Both checkpoints remain frozen after SocialBench‑VLN training, with no target‑domain training or checkpoint selection. We evaluate the validation‑unseen splits of DPed‑VLN (level‑1 global instructions) and HA‑VLN‑CE (continuous environments), using matched episodes, observations, and action interfaces within each benchmark. For context, the benchmark reports give zero‑shot SRs of 21.61% for NaVILA on DPed‑VLN and 15.00% for BEVBert on HA‑VLN‑CE. Our evaluations are summarized below.

DPed‑VLN: zero‑shot transfer, validation‑unseen.

MethodSR ↑SPL ↑STL ↑PSC ↑HCR ↓
SC²‑WM (SocialBench‑FT)21.1718.4220.0592.1838.65
SocioWM24.3221.3523.0694.0831.81

HA‑VLN‑CE: zero‑shot transfer, validation‑unseen.

MethodNE ↓TCR ↓CR ↓SR ↑
SC²‑WM (SocialBench‑FT)6.436.8459.1214.46
SocioWM5.964.6850.7418.78
AuthorsOpenReview ↗

Official Response to Reviewer BFvD(3/4)

Q3 & W1(b): where attributes matter and how much they contribute

Q3, original. On interaction episodes with the pedestrian groups whose behavior differs most (for example elderly or child pedestrians, or cautious vs. aggressive personalities), how do SocioWM, the w/o Attribute variant, and SC²‑WM compare? How accurately are age and gender recovered at inference? Does the agent keep more distance from pedestrians whose simulated reaction distance is larger? W1(b), excerpt. “Aggregate SR is probably the wrong place to look for an attribute effect, because most episodes may not depend on who the pedestrian is.”

(3.1) Navigation outcomes and behavior by pedestrian group. Thank you for your question. We construct controlled evaluations built upon 50 matched base navigation tasks, generating 200 total configurations covering four pedestrian‑age conditions. We compare three models: original SC²‑WM, the w/o Attribute ablation variant, and full SocioWM. Attribute supervision provides larger benefits in the child and older‑adult conditions: relative to w/o Attribute, SocioWM improves SR by 6 percentage points and reduces Coll. by 6 points in each group. Across all configurations, SR increases from 25.0 to 29.0 and Coll. decreases from 46.0 to 42.0. Full‑validation analysis also shows greater clearance, higher PSC, and more yielding across the reported age and gender groups. Instructions, robot start‑goal positions, base navigation route and encounter location stay fixed; only pedestrian appearance and behavioral parameters vary across age conditions. Each age condition contains 50 configurations; the aggregate pools all 200. For gender subgroups, SocioWM obtains median minimum clearance of 0.92 m (female) and 0.86 m (male), alongside PSC values of 97.4 % and 96.9 %, demonstrating consistent improvements for both subgroups.

Selected controlled age‑conditioned navigation results.

Age groupSC²‑WM SR ↑w/o Attr. SR ↑SocioWM SR ↑SC²‑WM Coll. ↓w/o Attr. Coll. ↓SocioWM Coll. ↓
Child16.020.026.054.048.042.0
Older adult18.022.028.056.050.044.0
All four age groups19.025.029.052.046.042.0

(3.2) Age and gender recovery at inference. On visible simulated pedestrians across the full validation split, the existing prediction heads achieve age accuracy/Macro‑F1 of 84.6%/83.2% and gender accuracy/Macro‑F1 of 91.3%/91.3%. This confirms that the learned representations recover usable attribute information at inference.

(3.3) Separation for pedestrians with larger reaction distances. To examine agent responses, we group pedestrian‑instance observations within validation rollouts by each pedestrian’s configured reaction‑distance into low / middle / high tertiles. A single rollout may contain pedestrians falling into multiple tertiles. Failed episodes with valid pedestrian observations are kept for statistics, and we keep the unified 1.0 m PSC evaluation threshold (reaction‑distance itself defines interaction‑trigger conditions). SocioWM achieves median episode‑level minimum distances of 0.84/0.94/1.10 m across tertiles, compared with 0.82/0.88/0.93 m for w/o Attribute and 0.74/0.77/0.79 m for SC²‑WM. Its growing separation across increasing pedestrian reaction‑distance levels demonstrates responsiveness to pedestrian contextual properties.

(3.4) Contributions of the different supervision signals. Removing attribute, motion, or interaction supervision reduces performance, supporting their complementary roles. The attribute‑specific tests characterize one component of a broader contribution: SocialBench‑VLN supplies structured context supervision, and SocioWM integrates that context into route‑conditioned navigation. In the following table, all six ablated variants report the mean ±\pm 95% confidence‑interval half‑width across five training seeds; Full Supervision reports a single‑run result.

Annotation ablation.

SupervisionSR ↑PSC ↑Coll. ↓
w/o Age26.20±0.2926.20 \pm 0.2990.10±0.1990.10 \pm 0.1948.30±0.3448.30 \pm 0.34
w/o Gender26.30±0.2626.30 \pm 0.2690.15±0.1790.15 \pm 0.1748.20±0.3048.20 \pm 0.30
w/o Attribute25.75±0.3225.75 \pm 0.3289.90±0.2389.90 \pm 0.2348.80±0.3748.80 \pm 0.37
w/o Interaction24.54±0.3824.54 \pm 0.3889.42±0.2589.42 \pm 0.2549.76±0.4149.76 \pm 0.41
w/o Motion24.08±0.4324.08 \pm 0.4389.12±0.2489.12 \pm 0.2450.29±0.4650.29 \pm 0.46
w/o All Structured Sup.19.41±0.4719.41 \pm 0.4788.68±0.2888.68 \pm 0.2851.83±0.4451.83 \pm 0.44
Full Supervision27.0790.3547.25
AuthorsOpenReview ↗

Official Response to Reviewer BFvD(4/4)

Q4 & M1: ISR denominators and interpretation

Q4, original. EinterE_{\mathrm{inter}} appears to depend on the agent's own trajectory, since interactions are triggered by the robot. If so, ISR is computed over different episode sets for different methods. Could you report ∣Einter∣|E_{\mathrm{inter}}| per method, or compute ISR on a fixed set of episodes in which the reference path triggers an interaction? M1, excerpt. Stating the claim in terms of Coll. and PSC would make it stronger.

(4.1) Method‑specific counts. Thank you for your question. Response (1.1) lists each configuration's ISR with successful/triggered interaction counts: 11.20% (126/1125) for SocioWM versus 6.70% (75/1119) for fine‑tuned SC²‑WM.

(4.2) Common interaction subset. SocioWM retains the highest ISR at 11.00% (88/800), versus 8.63% (69/800) for zero‑shot InternNav, 8.38% (67/800) for fine‑tuned InternNav, and 6.38% (51/800) for fine‑tuned SC²‑WM. We use the same 800 episodes from the intersection of 12 configurations' interaction sets, selected by triggering, not success. SocioWM therefore retains its advantage with identical tasks and denominators.

(4.3) Social compliance (M1). Thank you for your suggestion. We will discuss compliance using PSC and Coll. in the final manuscript: SocioWM achieves 90.35 PSC and 47.25 Coll., versus 87.02 and 53.94 for fine‑tuned InternNav. ISR separately measures collision‑free success on interaction episodes.

Q5 & W2(c): real‑robot evidence and participant safeguards

Q5, original. How many runs and environments are behind the real‑world claim, and how did SC²‑WM behave in the same setting? Were consent, institutional review, and safety procedures in place for the participants? W2(c), summary. Quantitative real‑robot success and minimum‑distance comparisons would strengthen the transfer evidence.

(5.1) Environments and trials. Thank you for your question. We test 20 matched tasks in each of three environments (home, office, corridor). Three methods each execute 60 trials per condition, with/without pedestrians: 360 runs in total. Three adult participants provide one to three pedestrians per trial, covering head‑on, crossing, and same‑direction encounters.

(5.2) Results. With pedestrians, SocioWM achieves 50.00% SR, versus 25.00% for zero‑shot and 33.33% for fine‑tuned SC²‑WM. Without pedestrians, the respective SRs are 70.00%, 65.00%, and 68.33%. The 16.67‑point gain over the fine‑tuned baseline with pedestrians supports social‑navigation transfer. Median per‑trial minimum distances are 1.05/0.65/0.70 m in the same method order, using all 60 pedestrian‑present trials per method, including failures. Success requires timely goal completion without collision or intervention. We will include these results and crossing examples in the final manuscript.

(5.3) Participant protection. All three adults provided informed consent, and the study received institutional ethics approval. Emergency stops at measured distances of 0.5 m or less terminate the trial and count as failures; distances recorded up to termination remain included. We will document these procedures in the final manuscript.

M2–M5: Writing, terminology, and figure corrections

Thank you for your feedback. In the final manuscript, we will correct the dataset count to 35K+, the SR improvement to 18.27%–27.07%, and the figure labels to SocioWM and Social Context Modulation, and consistently define SR as collision‑free goal completion.

M6: Loss definitions and enhanced SAP

Thank you for your suggestion. LnavL_{\mathrm{nav}} combines candidate/STOP‑action and visual‑prediction supervision. LcolL_{\mathrm{col}} uses positive labels when same‑floor robot–pedestrian distance falls below 0.6 m during candidate rollout; reactions align to each candidate's trigger time. Enhanced SAP scores candidates from socially conditioned states. We will make these definitions explicit in the final manuscript.

M7: Full‑validation masking analysis

Summary. Aggregate representation masking over all interaction episodes rather than relying on two cases.

Thank you for your suggestion. Across all 1,839 validation episodes (1,125 interaction, 714 noninteraction), interaction masking produces the largest changes: mean ΔP\Delta P is 8.72 versus 2.08 points, and preferred‑action changes occur at 22.76% versus 6.42%. Age/gender masking yields interaction‑group ΔP\Delta P of 3.18/1.26 points. We compare masked and full outputs at the same states without executing masked actions. ΔP\Delta P is the signed change in the full policy's preferred‑action probability; we average within episodes, then equally across episodes. These full‑validation results support the decision relevance of social representations; we will include them in the final manuscript.

Reviewer_ipdCOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper argues that vision-and-language navigation with pedestrians usually treats people as moving obstacles. It introduces Social Context-VLN and the SocialBench-VLN, which built in Habitat by adding 8-16 pedestrains to indoor-guided episodes, for more than 35k episodes and 1.4k pedestrains. SocioWM extends the SC2-WM with route-conditioned social reasoning that predicts a social correction for each candidate route and injects it through a learned gate. Training uses navigation, collision, motion, interaction, and attribute supervision, while deployment receives only instructions and RGB-D observations. The Evaluation includes existing navigation agents, module and supervision ablations, and loss weight sensitivity on 1839 validation episodes. The paper then reports an increase in collision free goal success from 18.27% to 27.07% and shows qualitative navigation with a humanoid robot.

Critical strengths
  1. The paper targets a real gap between VLN with pedestrains and socially aware navigation, ans the SocialBench-VLN makes the gap measureable, The robot must complete a language instruction while anticipating that approaching a person may trigger slowing, stopping, or yielding. Structured motion and interaction annotations, together with the separate navigation and safety outcomes, provide a controlled resource for studying this dependence and gap.

2 The method uses social annotations only as the training signals and conditions them on candidate routes, A pedestrian may pose different risks depending on where the robot intends to move, so considering that information before selecting a route really well motivated. Deployment does not require privileged pedestrian labels. Table 4 keeps the complete architecture but removes structured supervision, reducing success from 27.07% to 19.34%, so the gain over SC2-WM comes from the auxiliary supervision.

Critical weaknesses
  1. The evidence does not show that the value of demographic context matter beyond the simulator's own rules, The setting and benchmark are built around the age and gender as the social context. In SocialBench-VLN, these attributes are authored causes. Each age and gender combinations has its own avatar appearacne and scales walking speed, turning, reaction distance, and the sampling of a latent personality that selects predefined reations. A model can therefore benefit from recognizing a simulated attribute because the generator has made that attribute predictive of behavior. Holding out scenes does not test whether this relationship transfers to a different population. Table 4 supports an effect within this setting, with success falling from 27.07% to 25.94% when age and gender supervision are removed. With 1839 episodes, the standard error of an SR near 27% is about 1.0 point, so the attribute effect is within the noise of a single run. The evidence supports collision, motion, and interaction supervision in a simulated crowd. It does not support the broader claim that attribute aware social context improves navigation.

  2. The evidence does not directly validate the predicted consequences of alternative routes, SocioWM claims to anticipate how each candidate route will affect pedestrain interactions. Both Sections 3.3 and 4.1 describe the collision and interaction prediction objectives and candidate route replay, but these do not explain the prediction horizon or how targets reflect pedestrain reactions to each alternative. This matters because Appendix B.3 makes those reactions depend on the robot’s approach. That means, the future observed along one route may not describe another? Also about the comparision, Section 4.1 states that all methods share the candidate action space, but the NaVid, UniNaVid, and StreamVLN might produce low level actions? ISR is averaged over episodes in which an interaction was triggered, and the triggering depends on each agent’s own behavior, so ISR in paper might just compares different subsets across methods. Whatsmore, the Abstract states that real-world experiments verify transfer to unseen environments and real pedestrian interactions. However in the paper it only shows one qualitative sequence and hardware details, with no number of trials, outcomes, or check that attribute inference works on real people.

Questions for authors

Here is just some of my questions for paper in detail, the author can choose to answer some of them or not.

  1. Appendix B.1 states that each age and gender combination sets the avatar appearance, scales walking speed, turning, and reaction distance, and shifts the distribution over six latent personalities that select the predefined reactions. Can the paper report these weights and personality probabilities, the number of age groups and avatar appearances, and what “1.4K distinct pedestrians” counts?
  2. I am curious, how are avatar appearances shared across training and validation scenes? Does the attribute supervision still help with unseen appearances for the same attributes, or when appearance and behavior associations are weakened while preserving the overall motion distribution?
  3. As social context matters by pedestrian group, Section 4.1 uses a common 1.0 m threshold for PSC, so the aggregate score does not explain how the policy responds to different pedestrian contexts. Can existing rollouts report some more metric scores like minimum clearance, PSC, yielding behavior, and goal completion by age group for SocioWM and SC²-WM, and with group counts and interaction exposure?
  4. How does the safe candidate-route replay the construct collision and interaction targets for each alternative route, including the prediction horizon and pedestrian reactions? Does the replay use simulator collision information to select training routes, if so, and how do its targets relate to recorded future motion?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer ipdC(1/3)

We sincerely appreciate your constructive and insightful comments. We address your concerns point by point below.

Q1 & W1: Generator specification and the meaning of 1.4K

(1.1) Generator weights and personality sampling. Age sets base motion ranges; age and gender jointly set the avatar category and the sampling distribution over six personalities. The six personalities scale linear speed, turning speed and reaction distance as follows, and each age–gender cell samples them with the probabilities listed in the second table.

MultiplierNeutralCooperativeCautiousExploratoryAggressiveDistracted
Linear speed1.000.950.900.951.100.95
Turning speed1.001.000.951.101.050.90
Reaction distance1.001.151.251.050.850.80
Age / genderLinear speedTurning speedReaction dist.NeutralCoop.CautiousExplor.Aggr.Distr.
Child / male[1.20,1.40][1.10,1.30][0.90,1.10]201713231710
Child / female[1.20,1.40][1.10,1.30][0.90,1.10]201317271310
Young / male[1.10,1.30][1.10,1.30][0.90,1.10]201317221810
Young / female[1.10,1.30][1.10,1.30][0.90,1.10]201713182210
Middle / male[0.90,1.10][0.90,1.10][0.90,1.10]202718111410
Middle / female[0.90,1.10][0.90,1.10][0.90,1.10]20232291610
Older / male[0.70,0.90][0.70,0.90][1.00,1.20]20232791110
Older / female[0.70,0.90][0.70,0.90][1.00,1.20]20272311910

Probabilities are percentages and sum to 100 for each cell. Reaction distance is the controller's interaction-trigger parameter; the same weights and probabilities are used for all scenes without per-scene reconditioning.

(1.2) Age groups and avatar counts. There are four age groups and two gender categories, with 14 avatar assets: one per gender for children and two per gender in each other age group.

(1.3) Meaning of the 1.4K count. 1.4K distinct pedestrians means over 1,400 scene-specific pedestrian route templates across 90 scenes, with 8–16 routes per scene. Reusing a route with different appearances or attributes does not create another route template.

Q2 & W1: Shared appearances, changed mappings, and uncertainty

Answer. Thank you for this important suggestion. SocialBench-VLN studies structured social cues in controlled simulation, with age and gender as representative attributes. SocioWM's core is social-context modeling conditioned on candidate routes.

(2.1) Shared and unseen appearances. The current scene-disjoint splits share 14 avatar assets. The reassignment experiment below addresses the mapping aspect of your question; we will extend the appearance evaluation with new avatars reserved exclusively for validation.

(2.2) Weakened appearance–behavior associations.

To test the mapping aspect directly, we run a reassignment experiment. All models are frozen without retraining: appearances are reassigned together with their labels on the same 1,839 tasks, preserving behavioral configurations and appearance counts, and results average over three fixed reassignment manifests. Both association measures below are unitless.

Association diagnostics.

AssociationStatisticOriginalReassigned
Age–linear speedη2\eta^20.8270.003
Age–angular speedη2\eta^20.8440.004
Age–reaction distanceη2\eta^20.1300.003
Age–personalityCramér's VV0.1370.025

For the mapping results below, w/o Attribute reports five-seed means with 95% Student-tt confidence-interval half-widths; Full uses one fixed checkpoint.

ConditionMethodSR (%)PSC (%)Coll. (%)
Originalw/o Attribute25.75±0.3225.75\pm0.3289.90±0.2389.90\pm0.2348.80±0.3748.80\pm0.37
OriginalFull27.0790.3547.25
Reassignedw/o Attribute25.00±0.3725.00\pm0.3789.54±0.2689.54\pm0.2649.35±0.4349.35\pm0.43
ReassignedFull25.8389.8348.46

After reassignment, SocioWM achieves 25.83 SR versus 25.00 for w/o Attribute, with higher PSC and lower Coll., retaining its benefit under weakened associations.

AuthorsOpenReview ↗

Official Response to Reviewer ipdC(2/3)

(2.3) Training‑seed variability. All three attribute ablations underperform Full. Together with the mapping test, these ablations validate the attribute pathway’s contribution. SocialBench‑VLN provides motion and interaction annotations integrated via candidate‑conditioned social modeling. All six ablated variants report mean ±\pm 95% t‑confidence‑interval half‑width over five training seeds; Full Supervision uses a single run.

Annotation ablation.

SupervisionSR↑PSC↑Coll.↓
w/o Age26.20±0.2926.20 \pm 0.2990.10±0.1990.10 \pm 0.1948.30±0.3448.30 \pm 0.34
w/o Gender26.30±0.2626.30 \pm 0.2690.15±0.1790.15 \pm 0.1748.20±0.3048.20 \pm 0.30
w/o Attribute25.75±0.3225.75 \pm 0.3289.90±0.2389.90 \pm 0.2348.80±0.3748.80 \pm 0.37
w/o Interaction24.54±0.3824.54 \pm 0.3889.42±0.2589.42 \pm 0.2549.76±0.4149.76 \pm 0.41
w/o Motion24.08±0.4324.08 \pm 0.4389.12±0.2489.12 \pm 0.2450.29±0.4650.29 \pm 0.46
w/o All Structured Sup.19.41±0.4719.41 \pm 0.4788.68±0.2888.68 \pm 0.2851.83±0.4451.83 \pm 0.44
Full Supervision27.0790.3547.25

Q3 & W1: Where attribute‑conditioned behavior is observed

Q3, original: As social context matters by pedestrian group, Section 4.1 uses a common 1.0 m threshold for PSC, so the aggregate score does not explain how the policy responds to different pedestrian contexts. Can existing rollouts report some more metric scores like minimum clearance, PSC, yielding behavior, and goal completion by age group for SocioWM and SC²‑WM, and with group counts and interaction exposure?

W1, relevant original excerpt: “Aggregate SR is probably the wrong place to look for an attribute effect, because most episodes may not depend on who the pedestrian is.”

Answer. Thank you for this suggestion. We analyze social‑context effects on navigation performance and pedestrian‑aware behavior.

(3.1) Group behavior and interaction exposure. Across all 1,839 validation rollouts, we compare SocioWM and original SC²‑WM. Key statistics for child / older‑adult subgroups: ‑ Child: episode/pedestrian exposure 1050/1800, interaction episodes 394. SocioWM median min‑dist 0.90 m vs SC²‑WM 0.72 m; yielding rate 46% vs 32%. ‑ Older‑adult: episode/pedestrian exposure 990/1730, interaction episodes 369. SocioWM median min‑dist 0.98 m vs SC²‑WM 0.74 m; yielding rate 52% vs 35%. Young‑ and middle‑age groups show intermediate improvements in clearance, PSC and yielding. We apply the unified 1.0 m PSC threshold.

(3.2) Goal completion by age group. In a controlled evaluation built upon 50 matched base navigation tasks expanded to 200 total configurations covering four age conditions: SocioWM aggregate SR = 29.0 %, w/o‑Attribute = 25.0 %, original SC²‑WM = 19.0 %. For child SR: SocioWM = 26 %, SC²‑WM = 16 %; for older‑adult SR: SocioWM = 28 %, SC²‑WM = 18 %.

(3.3) Separation for pedestrians with larger reaction distances. We group pedestrian‑instance observations within rollouts by each pedestrian’s configured reaction‑distance into low / middle / high tertiles. Failed episodes with valid pedestrian observations are kept under fixed 1.0 m PSC threshold (reaction‑distance defines interaction‑trigger conditions).

SocioWM achieves median episode‑level minimum distances of 0.84/0.94/1.10 m across tertiles, compared with 0.82/0.88/0.93 m for w/o Attribute and 0.74/0.77/0.79 m for SC²‑WM. Its growing separation across rising pedestrian reaction‑distance demonstrates responsiveness to pedestrian contextual properties.

Q4 & W2: Candidate‑conditioned targets and replay

Q4, original: How does the safe candidate‑route replay the construct collision and interaction targets for each alternative route, including the prediction horizon and pedestrian reactions? Does the replay use simulator collision information to select training routes, if so, and how do its targets relate to recorded future motion?

W2, relevant original excerpt: Both Sections 3.3 and 4.1 describe the collision and interaction prediction objectives and candidate route replay, but these do not explain the prediction horizon or how targets reflect pedestrian reactions to each alternative.

Answer. Thank you for your question.

(4.1) Prediction horizon and candidate‑specific reactions. Candidate targets recompute each online decision step over a 2.0 s simulated horizon. For non‑responding pedestrians, non‑interaction motion runs until candidate‑specific trigger; stored reaction segments align to trigger time. Same‑floor distance <0.6 m yields positive collision targets.

(4.2) Collision information and recorded future motion. Future collision ground truth only provides supervision labels and losses: it never prunes, reranks online candidates or overrides actions. Targets reflect route‑specific reactions; recorded Future Motion supervises motion prediction separately. Evaluation uses RGB‑D + language inputs without future trajectories, collision or interaction labels.

AuthorsOpenReview ↗

Official Response to Reviewer ipdC(3/3)

W2: Common execution interface versus internal action representation

W2, original excerpt: Also about the comparison, Section 4.1 states that all methods share the candidate action space, but the NaVid, UniNaVid, and StreamVLN might produce low‑level actions?

Answer. Thank you for your valuable feedback. Our intended point is that all methods ultimately execute the simulator's common atomic actions, such as moving forward and turning. Direct‑action methods predict these actions directly, while candidate‑based methods first select a waypoint or route and convert that decision into atomic actions through the same execution interface. We will correct the wording in the final manuscript: the shared interface concerns final action execution, while each method retains its own internal action representation.

W2: Policy‑dependent ISR denominators

W2, original excerpt: ISR is averaged over episodes in which an interaction was triggered, and the triggering depends on each agent's own behavior, so ISR in paper might just compares different subsets across methods.

Answer. Thank you for your question.

(W2(b).1) Policy‑specific counts and complete results. All metric values are percentages. ISR parentheses show collision‑free successful interaction episodes / interaction‑triggered episodes, so the denominator of each method is visible. FT denotes fine‑tuning on SocialBench‑VLN, and dashes indicate unevaluated settings.

MethodZero‑shotFT
SR ↑SPL ↑PSC ↑Coll. ↓ISR ↑SR ↑SPL ↑PSC ↑Coll. ↓ISR ↑
NaVid16.8612.1587.7355.215.14 (57/1109)21.6416.1087.9553.406.60 (74/1121)
UniNaVid17.5612.8685.5654.875.52 (61/1105)22.2916.5088.1051.117.00 (78/1114)
NaVILA20.3418.9285.7755.366.05 (69/1140)24.6920.4588.4051.507.50 (85/1133)
StreamVLN17.7313.8785.0258.245.24 (61/1164)21.1518.5886.2556.886.27 (72/1148)
AwareVLN22.9519.6486.9959.056.44 (72/1118)----------
InternNav23.4420.1185.4256.019.14 (100/1094)25.8222.2887.0253.948.72 (97/1112)
SC²‑WM18.2713.6688.6052.315.60 (62/1107)22.6216.9088.9051.226.70 (75/1119)
SocioWM----------27.0720.7890.3547.2511.20 (126/1125)

(W2(b).2) Common‑episode validation. We further evaluate over the fixed intersection set of 800 interaction‑triggered episodes shared across 12 configurations (selection by interaction trigger, not success). On this identical‑denominator set, SocioWM achieves the highest ISR of 11.00 % (88/800). For reference: InternNav zero‑shot = 8.63 % (69/800), InternNav FT = 8.38 % (67/800), SC²‑WM FT = 6.38 % (51/800). This confirms the ISR advantage is not driven by method‑specific interaction‑trigger episode subsets.

W2(c): Real‑robot evidence and real‑person attribute‑related observations

W2(c), excerpt: Abstract states real‑world experiments verify transfer to unseen environments and real pedestrian interactions. However the paper only shows one qualitative sequence and hardware details; no trial counts, outcomes, or check that attribute inference works on real people.

Answer. Thank you for your question. We evaluate three methods across three physical environments: home, office, corridor. There are 20 matched navigation tasks per environment. Each method runs 60 trials under pedestrian‑present and 60 trials under pedestrian‑absent conditions, giving 360 total runs. Three adult human participants act as pedestrians, covering head‑on, crossing, and same‑direction encounters.

With pedestrians present: SocioWM SR = 50.00 %, zero‑shot SC²‑WM = 25.00 %, fine‑tuned SC²‑WM = 33.33 %. Without pedestrians, SRs are 70.00 %, 65.00 %, 68.33 %. The +16.67 % gain over fine‑tuned baseline demonstrates social‑navigation transfer to real‑world settings. In all 60 pedestrian‑present trials (including failures), median per‑trial minimum robot‑pedestrian distances: SocioWM 1.05 m; zero‑shot SC²‑WM 0.65 m; fine‑tuned SC²‑WM 0.70 m.

Qualitative observations from real‑participant runs: in head‑on encounters SocioWM initiates avoidance earlier for one male participant than for a female participant; in a crossing‑type encounter SocioWM completes navigation successfully while zero‑shot SC²‑WM fails. These give preliminary qualitative evidence of context‑aware responses toward real‑world pedestrians. Full quantitative statistics, these example sequences and our ethics safeguards will be documented in the manuscript.

Reviewer_gULnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors formulate Social Context Vision-and-Language Navigation (Social Context-VLN), a VLN setting that incorporates pedestrian motion and social context, namely Pedestrian Attribute (age and gender) and Human–Robot Interaction, into navigation decision-making. They construct SocialBench-VLN in Habitat, with more than 35K language-guided episodes involving 1.4K pedestrians and structured annotations of attributes, interactions, and future motion. They also propose SocioWM, a socially conditioned world model that predicts a candidate-specific Social Future Latent for each route and injects it into candidate world states through gated modulation.

Critical strengths
  1. A useful new problem setting. Existing Social-VLN work treats pedestrians as moving obstacles. The paper argues that pedestrian behavior and pedestrian responses to the robot should shape navigation decisions, and builds a benchmark with language instructions, dynamic pedestrians, and structured interaction annotations.

  2. A simple, well-motivated method with clear ablations. SocioWM predicts the social consequences of each candidate route and adds them to the candidate world state through a learned gate, using only RGB-D and language at inference

Critical weaknesses
  1. Age and gender add little and their effect is built into the simulator (1) Circular signal. In SocialBench-VLN, age and gender are hand-coded by the authors to set walking speed, turning speed, reaction distance, and the sampling of a latent personality that decides interaction behavior (Appendix B.1). The model therefore learns rules that the authors programmed into the simulator, not real relationships between age, gender, and how people behave. Gains on this benchmark do not show that these attributes help in the real world. (2) Effect within noise. In Table 4, removing age, gender, or both changes SR by only 0.8 to 1.1 points (27.07 to 25.94-26.31) and ISR by 0.2 to 0.7 points, from a single run on 1,839 episodes. This is likely within run-to-run variance, while motion and interaction supervision account for most of the gain. The claim that attributes "play distinct and complementary roles" is not supported. (3) Ethical and practical concerns. Inferring age and gender from appearance at deployment is error-prone and can encode stereotypes, as the ethics statement itself acknowledges. Given the small benefit, the cost and risk of these attributes are not justified. (4) Incomplete pedestrian attributes. Besides age and gender, some other components such as BMI should also be discussed.

  2. Missing comparisons with related social navigation benchmarks and methods (1) Benchmarks. Table 1 lists closely related benchmarks with language and dynamic pedestrians (HA-VLN, AdaVLN, DPed-VLN) and social-navigation benchmarks (Social-HM3D, HabiCrowd), but SocioWM is evaluated only on the authors' own benchmark. Results on at least one existing benchmark are needed to show the method is not tuned to the authors' simulator. (2) Methods. The baselines are mostly general VLN agents and SC2-WM, which do not model pedestrians. The paper does not compare against social-navigation methods that anticipate human behavior, such as Falcon (Gong et al., ICRA 2025) or the recent SocialNav foundation model (Chen et al., CVPR 2026), nor against a trajectory predictor such as Trajectron++ (Salzmann et al., ECCV 2020) combined with the same candidate planner. It also does not evaluate on the published human-aware VLN benchmark HA-VLN (Li et al., NeurIPS 2024). It is also unclear whether the 7B baselines were trained or fine-tuned on SocialBench-VLN or evaluated zero-shot, while SocioWM is trained on it with privileged supervision. Without this, the comparison in Table 2 is not fair.

Gong, Z., Hu, T., Qiu, R., & Liang, J. (2025). From Cognition to Precognition: A Future-Aware Framework for Social Navigation. In IEEE International Conference on Robotics and Automation (ICRA), pp. 9122–9129. Chen, Z., Guo, Y., Chu, Z., Luo, M., Shen, Y., Sun, M., Hu, J., Xie, S., Kuan, Y., Shi, P., Gu, Z., Liu, L., Han, H., Wu, X., Xu, M., & Zhang, Y. (2026). SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28796–28806. (Oral) Li, H., Li, M., Cheng, Z.-Q., Dong, Y., Zhou, Y., He, J.-Y., Dai, Q., Mitamura, T., & Hauptmann, A. G. (2024). Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. Salzmann, T., Ivanovic, B., Chakravarty, P., & Pavone, M. (2020). Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data. In European Conference on Computer Vision (ECCV), pp. 683–700.

Questions for authors
  1. In SocialBench-VLN, age and gender are hand-coded to set walking speed, reaction distance and personality (Appendix B.1). What evidence shows that modeling these attributes helps beyond reproducing the simulator's own rules? Can you report the attribute ablations in Table 4 over multiple seeds with confidence intervals, given that removing them changes SR by only about 1 point?
  2. Which method is "AwareVLN" in Table 2? It is cited as Gong et al. (2025), the Falcon social-navigation framework (ICRA 2025), but listed as a 7B model.
  3. Were the 7B baselines in Table 2 trained or fine-tuned on SocialBench-VLN, or evaluated zero-shot? SocioWM is trained on the benchmark with privileged supervision, so the comparison is fair only if the baselines see the same data.
  4. Can you compare against social-navigation methods that anticipate human behavior, such as Falcon (ICRA 2025) or SocialNav (CVPR 2026), and against a trajectory predictor such as Trajectron++ (ECCV 2020) combined with the same candidate planner?
  5. Can you evaluate SocioWM on an existing human-aware VLN benchmark, such as HA-VLN (NeurIPS 2024), to show the method is not tuned to the authors' own simulator?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer gULn(1/3)

We sincerely appreciate your constructive and insightful comments. We address your concerns point by point below.

Q1 & W1(a,b): Attribute validity, simulator rules, and variability

In SocialBench-VLN, age and gender are hand-coded to set walking speed, reaction distance and personality (Appendix B.1). What evidence shows that modeling these attributes helps beyond reproducing the simulator's own rules? Can you report the attribute ablations in Table 4 over multiple seeds with confidence intervals, given that removing them changes SR by only about 1 point?

Weakness 1: “Gains on this benchmark do not show that these attributes help in the real world.” “The claim that attributes ‘play distinct and complementary roles’ is not supported.”

Answer. Thank you for this helpful suggestion. SocialBench-VLN enables controlled study of structured social cues, with age and gender as representative attributes. For the reported navigation results, ±\pm denotes five‑seed means with 95% confidence‑interval half‑widths for the attribute ablations; Full uses one trained checkpoint.

(1.1) Evidence beyond the original simulator mappings.

To test whether the benefit survives once the simulator's own age–behavior mappings are weakened, we run a reassignment experiment. All checkpoints are frozen and no retraining is performed: we reassign appearances together with their labels on the same 1,839 episodes, strictly preserving sampled behavioral configurations and appearance counts, and we average over three fixed reassignment manifests. Both association measures reported below are unitless.

The diagnostics confirm that this reassignment substantially weakens the simulator's age–behavior associations.

Association diagnostics.

AssociationStatisticOriginalReassigned
Age–linear speedη2\eta^20.8270.003
Age–angular speedη2\eta^20.8440.004
Age–reaction distanceη2\eta^20.1300.003
Age–personalityCramér's VV0.1370.025

Under this weakened mapping, SocioWM retains 25.83 SR versus 25.00 for w/o Attribute, alongside higher PSC and fewer collisions. The retained benefit shows that its advantage does not depend entirely on the original appearance–behavior bindings.

The corresponding navigation results are shown below.

ConditionMethodSRPSCColl.
Originalw/o Attribute25.75±0.3225.75 \pm 0.3289.90±0.2389.90 \pm 0.2348.80±0.3748.80 \pm 0.37
OriginalSocioWM27.0790.3547.25
Reassignedw/o Attribute25.00±0.3725.00 \pm 0.3789.54±0.2689.54 \pm 0.2649.35±0.4349.35 \pm 0.43
ReassignedSocioWM25.8389.8348.46

For the reaction‑distance scaling robustness test we again freeze all checkpoints and modify only pedestrian reaction‑distance parameters, scaling by factors α={0.8,1.2}\alpha=\{0.8,1.2\} while keeping every other setting and all metric thresholds unchanged. SocioWM achieves SR of 25.23 (α=0.8\alpha=0.8) and 26.43 (α=1.2\alpha=1.2), outperforming ordinary fine‑tuned SC²‑WM (20.55 / 22.02) and w/o‑Attribute variant (24.14 / 25.11). SocioWM retains higher SR and PSC and lower Coll. than both controls at both scales. This test uses existing avatar assets; unseen‑appearance evaluation is planned for future work.

(1.2) Multi‑seed ablations and confidence intervals. All three ablation means remain below Full in SR, with the largest decrease when both attributes are removed. Together with the motion and interaction ablations, this supports complementary structured social cues in SocioWM.

Table 4 compares the three attribute ablations under the reporting convention stated above. All metrics are percentages.

VariantSR ↑PSC ↑Coll. ↓
w/o Age26.20±0.2926.20\pm0.2990.10±0.1990.10\pm0.1948.30±0.3448.30\pm0.34
w/o Gender26.30±0.2626.30\pm0.2690.15±0.1790.15\pm0.1748.20±0.3048.20\pm0.30
w/o Attribute25.75±0.3225.75\pm0.3289.90±0.2389.90\pm0.2348.80±0.3748.80\pm0.37
Full, single run27.0790.3547.25

Removing motion or interaction supervision yields five‑seed mean SRs of 24.08 and 24.54, respectively, versus 27.07 for Full. We will include these supplementary results in the final manuscript.

Q2: AwareVLN identity and erroneous citation

Which method is “AwareVLN” in Table 2? It is cited as Gong et al. (2025), the Falcon social‑navigation framework (ICRA 2025), but listed as a 7B model.

Answer. Thank you for your valuable feedback. We evaluated the 8B model from AwareVLN: Reasoning with Self‑awareness for Vision‑Language Navigation. We will correct its citation and model‑size entry in the final manuscript.

AuthorsOpenReview ↗

Official Response to Reviewer gULn(2/3)

Q3 & W2(b): In‑domain training and baseline comparability

Were the 7B baselines in Table 2 trained or fine‑tuned on SocialBench‑VLN, or evaluated zero‑shot? SocioWM is trained on the benchmark with privileged supervision, so the comparison is fair only if the baselines see the same data.

Answer. (3.1) Zero‑shot versus in‑domain evaluation. Thank you for your question. The original evaluation fine‑tuned only selected baselines. In the expanded in‑domain comparison, SocioWM achieves the highest SR, ISR, and PSC and the lowest collision rate, exceeding ordinary SC²‑WM fine‑tuning by 4.45 SR points.

Revised Table 2 separates zero‑shot and in‑domain results on the same SocialBench‑VLN training split. SC²‑WM‑based controls share SocioWM's settings and budget; other fine‑tuned baselines use two‑epoch LoRA fine‑tuning.

In‑domain AwareVLN fine‑tuning is valuable yet infeasible: its training recipe requires reasoning‑text labels absent from SocialBench‑VLN. We report its zero‑shot results and address same‑data comparability via other in‑domain baselines and matched‑supervision controls below. Matched AwareVLN fine‑tuning remains future work.

All metrics are percentages. ISR parentheses report collision‑free successful interaction episodes / interaction‑triggered episodes; dash = unevaluated setting.

Table 2: Zero‑shot and fine‑tuning (FT) results on SocialBench‑VLN.

MethodZero‑shotFT
SR ↑SPL ↑PSC ↑Coll. ↓ISR ↑SR ↑SPL ↑PSC ↑Coll. ↓ISR ↑
NaVid16.8612.1587.7355.215.14 (57/1109)21.6416.1087.9553.406.60 (74/1121)
UniNaVid17.5612.8685.5654.875.52 (61/1105)22.2916.5088.1051.117.00 (78/1114)
NaVILA20.3418.9285.7755.366.05 (69/1140)24.6920.4588.4051.507.50 (85/1133)
StreamVLN17.7313.8785.0258.245.24 (61/1164)21.1518.5886.2556.886.27 (72/1148)
AwareVLN22.9519.6486.9959.056.44 (72/1118)–––––
InternNav23.4420.1185.4256.019.14 (100/1094)25.8222.2887.0253.948.72 (97/1112)
SC²‑WM18.2713.6688.6052.315.60 (62/1107)22.6216.9088.9051.226.70 (75/1119)
SocioWM–––––27.0720.7890.3547.2511.20 (126/1125)

(3.2) Controls with matched auxiliary supervision. With matched label types, SocioWM improves SR by 1.97 points over SC²‑WM + Aux./Attr. heads, alongside better social‑compliance metrics. This supports contributions from the social‑modeling architecture beyond auxiliary supervision.

Auxiliary‑head controls omit RCSR, match SocioWM in labels, training split, settings, budget, and update both heads and shared trainable navigation representations. Aux. = collision, motion, interaction supervision; Attr. adds age and gender.

ConfigurationSRPSCColl.
SC²‑WM, ordinary FT22.6288.9051.22
SC²‑WM + Aux. heads24.8089.6049.00
SC²‑WM + Aux./Attr. heads25.1089.8048.60
Full SocioWM27.0790.3547.25

These matched‑supervision comparisons will appear in the final manuscript.

Q4 & W2(b): Social‑navigation and trajectory‑prediction controls

Can you compare against social‑navigation methods that anticipate human behavior, such as Falcon (ICRA 2025) or SocialNav (CVPR 2026), and against a trajectory predictor such as Trajectron++ (ECCV 2020) combined with the same candidate planner?

Answer. Thank you for your constructive suggestion.

(4.1) Standalone social‑navigation methods. Falcon uses point goals, not VLN instructions, and requires language‑interface adaptation. Our shared‑planner Trajectron++ control partially addresses pedestrian anticipation; standalone Falcon and SocialNav comparisons remain future work.

(4.2) Trajectory prediction with the same candidate planner. We replace the social‑modeling branch with Trajectron++ under identical candidate‑planner setup. Histories reconstruct from RGB‑D detection/tracking, depth, robot poses; no test‑time trajectory ground‑truth is used. SC²‑WM denotes ordinary in‑domain FT. SR requires collision‑free goal completion; ISR follows Table 2 convention. SocioWM exceeds Trajectron++ by 5.65 SR and 5.06 ISR points and reduces Coll. by 2.40 points. Compared with ordinary fine‑tuning, Trajectron++ lowers collisions but also SR and SPL, consistent with conservative navigation.

MethodSRSPLISRPSCColl.
SC²‑WM22.6216.906.70 (75/1119)88.9051.22
SC²‑WM + Trajectron++21.4215.836.14 (68/1107)89.6249.65
SocioWM27.0720.7811.20 (126/1125)90.3547.25
AuthorsOpenReview ↗

Official Response to Reviewer gULn(3/3)

Q5 & W2(a): Evaluation beyond the proposed benchmark

Can you evaluate SocioWM on an existing human‑aware VLN benchmark, such as HA‑VLN (NeurIPS 2024), to show the method is not tuned to the authors' own simulator?

Weakness 2: “Results on at least one existing benchmark are needed to show the method is not tuned to the authors' simulator.”

Answer. Thank you for this valuable suggestion.

We evaluate on validation‑unseen splits: DPed‑VLN uses level‑1 global instructions and HA‑VLN‑CE is continuous. All our model checkpoints remain frozen after SocialBench‑VLN training. We perform only zero‑shot transfer, with no target‑domain training or fine‑tuning, hence we report one single set of results for our models. Baseline numbers from prior publications are quoted for context. Under these frozen zero‑shot conditions, SocioWM improves SR by 3.15 / 4.32 points and reduces human collisions by 6.84 / 8.38 points versus SC²‑WM (SocialBench‑FT), supporting cross‑benchmark transfer.

DPed‑VLN: zero‑shot transfer, validation‑unseen.

MethodSR ↑SPL ↑STL ↑PSC ↑HCR ↓
NaVILA21.6120.8421.2690.3440.17
StreamVLN15.7414.0614.4093.3437.24
SC²‑WM (SocialBench‑FT)21.1718.4220.0592.1838.65
SocioWM24.3221.3523.0694.0831.81

HA‑VLN‑CE: zero‑shot transfer, validation‑unseen.

MethodNE ↓TCR ↓CR ↓SR ↑
HA‑VLN‑V7.153.9746.006.00
BEVBert6.105.7256.0015.00
ETPNav7.407.9471.008.00
SC²‑WM (SocialBench‑FT)6.436.8459.1214.46
SocioWM5.964.6850.7418.78

W1(c) & Ethics Concern: Demographic labels and participants

Inferring age and gender from appearance at deployment is error‑prone and can encode stereotypes, as the ethics statement itself acknowledges. Given the small benefit, the cost and risk of these attributes are not justified. This benchmark requires volunteers' age and gender. This may have ethics concerns.

Answer. Thank you for your valuable feedback. The demographic‑related labels within SocialBench‑VLN apply to synthetic simulated avatars only for controlled navigation research; no volunteer human subject data is used to generate benchmark pedestrian annotations. We use common PSC and collision thresholds across groups and report the w/o Attribute ablation as a baseline comparison. In the final manuscript we will explicitly clarify the synthetic origin of these demographic labels, their intended research‑only use case, and potential stereotype‑related risks.

For our real‑robot experiments, the study obtained institutional ethics approval, and informed consent was collected from all three adult human participants. During pedestrian‑present trials, an emergency stop triggers when robot‑pedestrian measured distance drops to 0.5 m or below; any safety intervention terminates a trial and counts it as a failure. All these participant‑protection procedures will be documented in the final manuscript.

W1(d): Additional attributes, including BMI

Besides age and gender, some other components such as BMI should also be discussed.

Answer. Thank you for this insightful suggestion. Our work targets representative social cues, starting with age and gender. Attributes such as BMI and body shape represent valuable future extensions, yet they demand extra annotation effort and also suffer from inherent ambiguity under monocular visual estimation. We leave these finer‑grained physical attributes for future research on social navigation.

AuthorsOpenReview ↗

Reminder: Discussion Phase Closing Soon

We sincerely appreciate your invaluable feedback, which has significantly contributed to the improvement of our work. In response, we have incorporated specific experiments to address your concerns. As the rebuttal period draws to a close, we kindly request your prompt review of our responses to ensure that we have thoroughly addressed all your comments. If our responses have satisfactorily resolved your concerns, we would be deeply grateful for a reconsideration of our score. Should any issues remain, we are more than willing to provide further clarifications to address them comprehensively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.