- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submission introduces NS-VLN, a continuous VLN benchmark for negative spatial constraints, and C²Nav, a constraint-conditioned navigator. NS-VLN is built from R2R-CE and English RxR-CE in Matterport3D and organises prohibitions into three levels: L1 separated, L2 linguistically integrated, and L3 semantically entangled. Each episode pairs a goal-directed instruction with a forbidden region defined by a dilated horizontal object footprint; success requires both reaching the goal and avoiding all forbidden regions over the full trajectory. C²Nav augments ETP-R1 with instruction-conditioned transition costs, counterfactual role calibration, and risk-aware graph planning. Experiments on Validation Unseen show that goal-only GSR masks unsafe success and that C²Nav improves SSR over ETP-R1+.
Critical strengths
The problem is timely and well motivated. The GSR/SSR decomposition is a clean diagnostic, and the L1–L3 taxonomy is a useful operationalisation of linguistic and semantic difficulty. The counterfactual forbidden-allowed pairing is a sensible supervision signal for role sensitivity, and the metric suite—SSR, VR, Unsafe, NDTW, SDTW, MSM—is appropriately safety-aware. The core empirical observation, that goal-reaching metrics can conceal constraint violations, is valuable for the embodied navigation community.
Critical weaknesses
- C²Nav is not yet empirically established. NaVIDA reports higher SSR (36.9 vs. 36.1), and C²Nav regresses on L2 (25.8 vs. 35.6 for ETP-R1 and 31.2 for ETP-R1†), directly challenging the claimed advantage on linguistically integrated constraints.
- The contribution of each component remains unclear. No ablations isolate role-sensitive transition costs, counterfactual calibration, risk-aware planning, or the two-stage schedule; sensitivity to λ, m, β, and noisy-OR is absent.
- Benchmark construction lacks validation. There is no inter-annotator agreement for L2/L3, the LLM rewriting/manual check is unquantified, and L1 uses a permissive similarity threshold (>0.2) with 42.3% retention. The main protocol checks only the agent centerline, ignores y, and fixes δ = 0.3 m; a stricter footprint-based check would strengthen the safety claims.
- Evaluation is narrow and statistically weak: only Validation Unseen, with no test split, seeds, confidence intervals, or significance tests. Baselines differ in training regime, and zero-shot LLM adaptation is under-specified.
- Generalization is limited to English, 72 Matterport3D scenes, and static indoor settings. L3 claims about coreference and memory are not directly probed.
- Minor clarity/reproducibility issues include notation inconsistencies (e.g., f_flue vs. f_true), a caption typo, and missing compute/code/hyperparameter details.
Questions for authors
- Table 4 reports C²Nav L2 SSR of 25.8, below ETP-R1 (35.6) and ETP-R1† (31.2). Since L2 is precisely where linguistic integration should matter, what explains this regression—over-avoidance, degraded instruction parsing after fusion, or optimization instability?
- In Table 3, NaVIDA achieves higher SSR than C²Nav (36.9 vs. 36.1). Is this comparison under identical observation, action, and training protocols? If not, could you report C²Nav with the same backbone and training budget, or clarify why this gap does not undermine the primary safety claim?
- Could you report performance without role-sensitive transition costs, without counterfactual role calibration, without risk-aware graph planning, and with a single-stage schedule? Sensitivity to λ, margin m, β, and the noisy-OR aggregation would also help isolate what drives the gains.
- For L2/L3 human annotations, what is the inter-annotator agreement? How many annotators and items were involved, and how were disagreements resolved? For L1, what is the path-similarity distribution and rejection rate by source corpus?
- Why does the main evaluation check only the agent centerline and ignore the vertical coordinate? Would the ranking in Table 3 change under an agent-footprint sweep or a full 3D violation check? Appendix B reports one controlled pair; extending this to all methods would strengthen the safety claims.
- How were the LLM-based baselines prompted and adapted to NS-VLN? Were they given the full L1–L3 instructions, and were they purely zero-shot or tuned? Details on ETP-R1† training data, schedule, and compute would also clarify comparability.
2 comments
Thank you for the clarifications and additional results. I acknowledge that I misread the L1 violation rate as the L2 safe success rate, and I withdraw my concern about L2 regression. I also acknowledge the component ablations in Table 5. The additional experiments and annotation details substantially address my main concerns. Some limitations remain regarding the geometric assumptions and component attribution. Overall, I view the benchmark contribution positively and will raise my rating from 2 (weak reject) to 3 (weak accept), keeping my confidence at 2.
Dear Reviewer u8qj,
Thank you for your review. The authors have submitted their rebuttal. Given your initial rating of , could you please check whether their response has addressed your concerns? Your prompt feedback would be greatly appreciated, as it would allow the authors to provide further clarification if needed.
Best,
Your AC