acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Obstacle-Conditioned Safety in Household Robot Navigation

Abstract

Safe household navigation requires robots to regulate clearance and speed according to what an obstacle is and how close it is, without unnecessary conservatism. Yet existing evaluations do not test whether semantic safety distinctions are reflected in continuous motion. To that end, we introduce OSMA-Bench, an Obstacle-Conditioned Safe Motion Adaptation Benchmark built on spatially matched scenario sets that differ only in obstacle type. We also propose the Semantic Sensitivity Index (SSI), which measures whether clearance and distance-dependent speed and smoothness follow the caution ordering of obstacles. Evaluating VLM-based and vision-language-action (VLA) policies on the benchmark, we find that neither consistently adapts its motion to obstacle safety, even with explicit safety prompts. Our analysis shows that the policies can correctly distinguish obstacles by safety relevance but fail to reflect this distinction in the motion parameters or actions they produce. For VLM-based policies, which recognize the obstacle and then generate a navigation program, adding an explicit parameter-inference step between these two stages improves adaptation. These findings expose a gap in semantic-to-motion grounding, highlighting the need for caution into appropriately cautious motion.

Primary area:
datasets and benchmarks
Keywords:
Benchmark for robotics safety, context-dependent safety, obstacle-condition safe navigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kGYoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced OSMA-Bench, a RoboCasa simulation benchmark to test whether robot navigation policies adjust their motion to what an obstacle is. The key contribution is a matched design: across varied routes and kitchen layouts, the obstacle's position stays fixed while its type changes across three caution tiers (Living, Fragile and Robust). A new metric, the Semantic Sensitivity Index (SSI), checks whether clearance and distance-adjusted speed and smoothness follow that caution order. The authors show that current VLM- and VLA-based policies don't reliably adapt their motion to obstacle safety, even when the models can tell obstacles apart. Splitting control-value inference from code generation narrows this gap for VLM-based policies.

Critical strengths
  • To my knowledge, this is the first benchmark I could confirm that swaps only the obstacle's identity at a fixed pose, layout and route in navigation and scores graded caution in executed motion.
  • The framing of caution should be selective and not maximal is well-motivated. And research on safety of action-based foundation model is timely.
  • Broad evaluation with numerous policy configurations, tasks, and extra metrics.
Critical weaknesses
  • I'm not convinced the RQ2 improvement comes from separating parameter inference from code generation. In the baseline, the speed setting never actually takes effect. And with RV capped at 80 cm it couldn't for High-tier obstacles anyway. The 3-step method, on the other hand, gets a larger range, a clamp, and has the target obstacle's parameters rewritten directly. The 2-step version, which only separates the steps, improves SSI by just 0.01, so I'd like to see ablations that separate out these factors.
  • Four of the six Fragile obstacles are on tables, and F.6 says their measured clearance stays around 0.6 m no matter what clearance the policy asks for.
  • The novelty is a bit overclaimed and the closest prior work is missing. These include SafeVLA [1], a peer-reviewed household VLA safety benchmark with semantic hazards; Weber et al. [2], which use safety classes with class-specific clearance; and HuNavSim [3], which uses the same clearance, speed, acceleration and jerk features. HazardArena [4] already uses paired safe/unsafe VLA scenarios with matched setups, and AGENTSAFE [5] and INT-ACT [6] already report the "knows but doesn't act" gap. The specific combination OSMA offers is new to my knowledge however. [1] SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning. NeurIPS 2025. https://openreview.net/forum?id=dt940loCBT

[2] Weber et al. Safety Classes Semantic Costmaps from RGBD Sensors for Risk-Aware Robot Navigation. CoDIT 2024. https://doi.org/10.1109/CoDIT62066.2024.10708523

[3] Pérez-Higueras et al. HuNavSim: A ROS 2 Human Navigation Simulator for Benchmarking Human-Aware Robot Navigation. IEEE RA-L, 2023. https://doi.org/10.1109/LRA.2023.3316072

[4] Chen et al. HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models. arXiv:2604.12447, 2026. https://arxiv.org/abs/2604.12447

[5] AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions. arXiv:2506.14697, 2025. https://arxiv.org/abs/2506.14697

[6] From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models (INT-ACT). arXiv:2506.09930, 2025. https://arxiv.org/abs/2506.09930

Questions for authors
  • What are SSI and the tier-pair margins for all models when the four tabletop Fragile obstacles are excluded or moved to the floor? Do the RQ1 rankings and the negative High-minus-Moderate margins still hold?
  • How does the 3-step method do without the step that rewrites the evaluated obstacle's parameters and inserts missing calls?
4 comments
AuthorsOpenReview ↗

[1/2] Rebuttal by Authors

We thank Reviewer kGYo for recognizing the task design, the caution framing, and the breadth of our evaluation. Below, we address your concerns in detail.

[W2] Tabletop placement

W2. Four of the six Fragile obstacles are on tables, and F.6 says their measured clearance stays around 0.6 m no matter what clearance the policy asks for.

We thank the reviewer for highlighting this factor. We agree that the supporting table affects traversable space and measured clearance. Although placing these objects (wine, hot chocolate, a table lamp, and water) on a table is more natural setting, we also evaluate a floor-placement setting to assess the table’s influence. We have updated Tables 14 and 15 to report both settings. Here, we report the mean unconditional SSI for each policy group to ensure a fair comparison on the same tasks. Results for the other task intersections (TSR-I or CFSR-I) are also provided in Table 15.

Policy groupWith tableWithout tableChange
VLA−0.036−0.021+0.015
VoxPoser+0.414+0.356−0.058
PIVOT+0.251+0.299+0.048

The comparison preserves the main qualitative conclusions. For VLA, collision-free success remains low in both settings, while All and TSR-I SSI remain near zero. For VoxPoser, CFSR-I SSI remains positive across all eight models, indicating that its observed caution-tier differentiation persists after table removal. For PIVOT, task success and All and TSR-I SSI show relatively small changes, although CFSR-I SSI increases for most models. Thus, while the supporting table influences individual measurements, the evaluated policies show similar overall SSI trends.

[W3] Novelty and missing prior work

W3. The novelty is a bit overclaimed and the closest prior work is missing. These include SafeVLA [1], a peer-reviewed household VLA safety benchmark with semantic hazards; Weber et al. [2], which use safety classes with class-specific clearance; and HuNavSim [3], which uses the same clearance, speed, acceleration and jerk features. HazardArena [4] already uses paired safe/unsafe VLA scenarios with matched setups, and AGENTSAFE [5] and INT-ACT [6] already report the "knows but doesn't act" gap. The specific combination OSMA offers is new to my knowledge however. [1] SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning. NeurIPS 2025.

We thank the reviewer for highlighting these closely related works. We have added them to Section 2 and revised our novelty claims to better distinguish our contribution. Our contribution lies in an integrated evaluation design that varies obstacle identity at a fixed placement, layout, and route to assess related caution of obstacle's safety implication in executed motion. We have clarified in the revised manuscript (Section 2). Also, we also note in the revised manuscript (Section 4.4, RQ3) that our findings are consistent with those reported in AGENTSAFE and INT-ACT.

AuthorsOpenReview ↗

[2/2] Rebuttal by Authors

[W1, Q2] Ablation of the three-step reasoning

W1. I'm not convinced the RQ2 improvement comes from separating parameter inference from code generation. In the baseline, the speed setting never actually takes effect. And with RV capped at 80 cm it couldn't for High-tier obstacles anyway. The 3-step method, on the other hand, gets a larger range, a clamp, and has the target obstacle's parameters rewritten directly. The 2-step version, which only separates the steps, improves SSI by just 0.01, so I'd like to see ablations that separate out these factors.

Q2. How does the 3-step method do without the step that rewrites the evaluated obstacle's parameters and inserts missing calls?

Thank you for your constructive discussion. We agree that the parameter range and clamp operation are entangled. We first clarify the other points raised and then present the ablation results.

[Clarification of remaining concerns]

  • The inferred speed rarely affected execution in the baseline was the main motivation for extending the two-step approach to three steps Compared with the baseline, three-step approach introduces two changes: (1) separating parameter inference from code generation and (2) accounts for the physical interaction between the clearance radius and the velocity map.
  • Parameter rewriting is the mechanism through which both modular approaches apply estimated values to the final generated code. The approaches separate parameter estimation from code generation, and rewriting applies the estimated parameter values to the code generated in the final step. Rewriting modifies the target obstacle’s parameters only when the generated code references that obstacle for a fair comparison.
  • We verify that missing call insertion was never triggered and affects the results although the appendix text mentions a fallback to insert a missing call.

To avoid confusion, we have clarified them in the revised manuscript.

[Clamp ablation] In three step approach, we separately ablate clamping under both parameter ranges. ✓ denotes the clamp operation. In RV∈[0,80]R_V \in [0,80], the additional "✓ + max clip" setting also restricts the velocity-radius upper bound to 80 cm after clamp. We do not include this setting for RV∈[0,200]R_V \in [0,200], since it already predicts wide range exceeding 80 cm. Here, the results are reported using the intersection of 548 tasks.

  • RV∈[0,80]R_V \in [0,80] cm

    ClampValue SSISSITSR (%)CFSR (%)Vel. (%)
    ✗+0.94+0.4199.799.70.2
    ✓+0.90+0.5798.697.40 →\rightarrow 100
    ✓ + max clip+0.89+0.5199.899.40 →\rightarrow 68.0
  • RV∈[0,200]R_V \in [0,200] cm

    ClampValue SSISSITSR (%)CFSR (%)Vel. (%)
    ✗+0.94+0.6296.595.458.4
    ✓ (paper)+0.93+0.6396.595.759.0 →\rightarrow 100

In RV∈[0,200]R_V \in [0,200] cm, it rarely affects the SSI. For clarity, we have reported the results without clamp operation.

[RVR_V range ablation] For fair comparison, we extends 1-step baseline to extends RV ranges from [0,80][0,80] to [0,200][0,200] without clamping. This analysis uses a task intersection of 539 episodes; the SSI and Value SSI values could differ from those in Table 2.

  • RV∈[0,80]R_V \in [0,80] cm
MethodSSIValue SSIVel. (%)
1-step+0.13+0.290
2-step+0.37+0.940
3-step+0.43+0.940.2
  • RV∈[0,200]R_V \in [0,200] cm
MethodSSIValue SSIVel. (%)
1-step+0.13+0.280
2-step+0.37+0.940
3-step+0.62+0.9458.4

With matched parameter ranges, the three-step method improves SSI over the two-step method by +0.06+0.06 for RV∈[0,80]R_V \in [0,80] and +0.25+0.25 for RV∈[0,200]R_V \in [0,200], while Value SSI remains identical. The larger SSI gain coincides with more frequent application of the inferred velocity (58.4% vs. 0.2%), suggesting that the wider range lets it affect execution more often. We have reported them on Table 22, and updated the Table 2 for RV∈[0,80]R_V \in [0,80] cm, along with the three-step result for RV∈[0,200]R_V \in [0,200] cm in the revised manuscript.

AuthorsOpenReview ↗

A gentle reminder for feedback

Dear Reviewer kGYo,

As the discussion period draws to a close, we would like to thank you again for your thoughtful review. We hope our response has helped clarify the points you raised. If you have any further thoughts or remaining concerns, we would welcome your feedback whenever your schedule allows.

Thank you for your time and consideration.

AuthorsOpenReview ↗

A gentle reminder for feedback

Dear Reviewer kGYo,

With approximately 24 hours remaining in the discussion period, we wanted to follow up on our response to your review. We would greatly appreciate it if you could let us know whether our response addresses your concerns or if there are any points that still need clarification.

Thank you for your time and consideration.

Reviewer_LmUPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a simulation benchmark to evaluate whether navigation policies adapt their clearance, speed, and smoothness to obstacle semantics. Experiments reveal inconsistent adaptation in VLM-based and VLA policies. Another conclusion is that separating motion-parameter inference from execution-code generation improves adaptation for VLM-based policies. The paper also introduces a metric called SSI.

Critical strengths
  • It is nice to frame safety beyond collision avoidance, but it is not novel (nor does the paper claim so)
  • VLM/VLA evaluations are broad. Some analyses of the paper may be useful.
Critical weaknesses
  • I am not sure if the benchmark cleanly measures "semantic caution" because semantics seem to be correlated with geometry, like fragile objects being on the table. It is possible that a model that has "geometric caution" succeed without semantic caution.
  • Although I agree people may care more about evaluating VLMs and VLAs, it would be nice to have some classical planners as baselines as well.
  • The benchmark has five kitchen layouts, and the obstacles are all static. So coverage is limited. There is no physical or human validation either although this is not unacceptable.
  • If my understanding is correct, SSI measures ranking/alignment, and it is not necessarily a measure of safety. Considering the main focus is safety, I am not sure if SSI deserves the focus it gets in the current version of the manuscript.
  • Also, the evaluation uses tasks completed without collision by EVERY model in a group. So scores depend on which models are being evaluated. If someone is comparing A vs B, they may see that A is better. But then comparing A vs B vs C may result in B being better than A. This would be a violation of Luce choice axiom, which is not a gold standard or anything especially for benchmarks, but it is a desirable property that the current evaluation protocol does not satisfy.
Questions for authors

See the weaknesses section.

4 comments
AuthorsOpenReview ↗

[1/2] Rebuttal by Authors

We sincerely thank Reviewer LmUP for the feedback and for noting the breadth of our VLM/VLA evaluation. Below, we address your concerns in detail.

[W4] Relationship Between SSI and Safety

W4. If my understanding is correct, SSI measures ranking/alignment, and it is not necessarily a measure of safety. Considering the main focus is safety, I am not sure if SSI deserves the focus it gets in the current version of the manuscript.

We thank the reviewer for raising this important point. We agree that SSI measures relative ordering rather than absolute safety or risk. However, we examine not only whether the robot avoids collisions with nearby obstacles, but also whether its motion around them reflects their differing safety implications. As illustrated in Figure 1, a robot should adjust its motion to potential harm, passing an infant more cautiously than a trash bin. These differing safety implications necessitate different levels of caution, on which we base our obstacle caution tiers (OCTs), as further supported by the human validation study described below. By assessing whether motion follows this safety-related ordering, SSI provides a safety-oriented measure of motion adaptation, along with collsion metric. We will make this distinction explicit in the revised manuscript to avoid confusion.

[Human validation of the OCT ordering] To provide additional empirical support for this ordering, we conducted a human study in response to the reviewer’s concern. Twenty participants recruited through Prolific [1] completed pairwise comparisons, indicating which of two obstacles in a scene should prompt greater caution from the robot. Overall, 89.1% of responses agreed with the predefined OCT ordering, showing strong alignment with human judgments of relative caution. Fleiss’ κ\kappa was 0.62 and increased to 0.75 in a supplementary sensitivity analysis excluding one participant whose judgments differed from the OCT ordering in 83% of comparisons. The study protocol and detailed results are provided in Appendix B.2.

[1] https://www.prolific.com/

[W1] Semantic caution and tabletop placement

W1. I am not sure if the benchmark cleanly measures "semantic caution" because semantics seem to be correlated with geometry, like fragile objects being on the table. It is possible that a model that has "geometric caution" succeed without semantic caution.

We thank the reviewer for highlighting this factor. We agree that the supporting table affects traversable space and measured clearance. Although placing these objects (wine, hot chocolate, a table lamp, and water) on a table is more natural setting, we also evaluate a floor-placement setting to assess the table’s influence. We have updated Tables 14 and 15 to report both settings. Here, we report the mean unconditional SSI for each policy group to ensure a fair comparison on the same tasks. Results for the other task intersections (TSR-I or CFSR-I) are also provided in Table 15.

Policy groupWith tableWithout tableChange
VLA−0.036−0.021+0.015
VoxPoser+0.414+0.356−0.058
PIVOT+0.251+0.299+0.048

The comparison preserves the main qualitative conclusions. For VLA, collision-free success remains low in both settings, while All and TSR-I SSI remain near zero. For VoxPoser, CFSR-I SSI remains positive across all eight models, indicating that its observed caution-tier differentiation persists after table removal. For PIVOT, task success and All and TSR-I SSI show relatively small changes, although CFSR-I SSI increases for most models. Thus, while the supporting table influences individual measurements, the evaluated policies show similar overall SSI trends.

[W5] The problem of Luce choice axim

W5. Also, the evaluation uses tasks completed without collision by EVERY model in a group. So scores depend on which models are being evaluated. If someone is comparing A vs B, they may see that A is better. But then comparing A vs B vs C may result in B being better than A. This would be a violation of Luce choice axiom, which is not a gold standard or anything especially for benchmarks, but it is a desirable property that the current evaluation protocol does not satisfy.

We thank the reviewer for this observation. We agree that the CFSR intersection depends on which models are in the group, so adding a model can change the evaluation subset and the ranking. For group-independent comparisons, we report unconditioned SSI over all tasks (Appendix C.6, Table 15). The main motivation for reporting CFSR-I SSI in the main table is to distinguish task failures and collisions from motion-level relative safety. This comparison is most informative when all models have CFSR close to 1, so the intersection covers most tasks and is less affected by group-dependent sample selection. We clarified the complementary roles of these measures in the revised manuscript.

AuthorsOpenReview ↗

[2/2] Rebuttal by Authors

[W2] Classical planner baselines

W2. Although I agree people may care more about evaluating VLMs and VLAs, it would be nice to have some classical planners as baselines as well.

We thank the reviewer for this suggestion. We added a rule-based planner that makes no LLM calls and applies a uniform clearance of either 30 or 60 cm across obstacle categories, evaluated with and without the supporting table.

Rule-based baselineTSR (%)CFSR (%)SSI
With table
Uniform clearance: 30 cm100.0100.0+0.067
Uniform clearance: 60 cm99.899.8+0.072
Without table
Uniform clearance: 30 cm100.0100.0+0.072
Uniform clearance: 60 cm100.0100.0+0.050

Both configurations achieve high TSR and CFSR. However, the planner applies the same clearance regardless of obstacle semantics, so it can avoid collisions but cannot adapt its behavior to the semantic safety implications of different obstacles. Consistent with this limitation, the higher-clearance setting achieves perfect CFSR but low SSI, reflecting limited motion differentiation across caution tiers. We have updated on Tables 14 and 15, and added classical planner on Figure 3 (a).

[W3] Benchmark coverage

W3. The benchmark has five kitchen layouts, and the obstacles are all static. So coverage is limited. There is no physical or human validation either although this is not unacceptable.

We thank the reviewer for this comment. We acknowledge that our benchmark is restricted to five kitchen layouts with static obstacles. Nevertheless, we believe the benchmark provides meaningful diversity along two complementary dimensions. First, five layouts and seven routes yield 35 layout–route configurations, varying approach directions and turning requirements. More importantly, our primary contribution lies in the diversity of safety-relevant obstacle semantics, extending beyond the human-aware navigation settings emphasized in prior benchmarks [1–3]. Specifically, we consider 18 obstacle types across three caution tiers, including infants, pets, and fragile objects, to evaluate safety reasoning under different levels of required caution. While these dimensions do not fully address the limitations raised, we believe they provide a valuable foundation for systematically evaluating safety-aware robotic navigation. We have explicitly acknowledged the remaining limitations in conclusion of the revised manuscript.

[1] iGibson Challenge 2021: Interactive and Social Navigation. CVPR 2021 Embodied AI Workshop.
[2] X. Puig et al. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. ICLR 2024.
[3] Y. Dong et al. HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions.

AuthorsOpenReview ↗

A gentle reminder for feedback

Dear Reviewer LmUP,

As the discussion period draws to a close, we would like to thank you again for your thoughtful review. We hope our response has helped clarify the points you raised. If you have any further thoughts or remaining concerns, we would welcome your feedback whenever your schedule allows.

Thank you for your time and consideration.

AuthorsOpenReview ↗

A gentle reminder for feedback

Dear Reviewer LmUP,

With approximately 24 hours remaining in the discussion period, we wanted to follow up on our response to your review. We would greatly appreciate it if you could let us know whether our response addresses your concerns or if there are any points that still need clarification.

Thank you for your time and consideration!

Reviewer_2RJ8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OSMA-Bench, a benchmark for domestic robot navigation designed to investigate whether robots can adjust their motion based on the "safety implications" of obstacles—going beyond simple collision avoidance. The benchmark comprises 625 scenarios, encompassing 18 categories of obstacles, 7 types of routes, and 5 kitchen layouts. Obstacles are classified into three distinct "caution levels": living beings, fragile objects, and robust objects.

The authors propose the Semantic Sensitivity Index (SSI), which evaluates whether a robot’s motion aligns with the prescribed hierarchy of caution levels by analyzing the minimum distance to obstacles, alongside velocity, acceleration, and jerk—all normalized by distance. Experimental evaluations of VLA, VoxPoser, and PIVOT strategies reveal a notable gap between semantic reasoning and actual robot motion. Furthermore, the authors introduce a three-step process—involving parameter inference and code generation—which improves SSI scores in certain experimental settings.

Critical strengths
  • The research problem holds practical significance. Distinguishing between "successful, collision-free navigation" and "motion adjustment in response to objects" helps reveal behavioral nuances that traditional success rate metrics fail to capture.
  • The benchmark features a well-defined experimental structure. By swapping obstacles within fixed layouts and routes—and conducting comparisons across multiple scenarios—the approach offers greater diagnostic value than demonstrations limited to a single setting.
  • The analysis of the control pipeline is highly specific. The authors discovered that the region where velocity adjustments are intended to take effect may not intersect with the actual trajectory, rendering the model’s deceleration parameters ineffective. This insight provides actionable guidance for system implementation and debugging.
  • The appendix discloses significant negative results—covering alternative sample sets, varying visual styles, smaller model variants, execution failures, and the operational limits of linear probes. These materials are instrumental in assessing the robustness and reliability of the study's conclusions.
Critical weaknesses
  • A conflation of semantics and geometry exists. The dimensions of obstacles and the supporting tabletop alter the traversable space; removing objects from the table increases the accuracy of distance-based ranking from approximately 3% to 62%, which undermines the argument that the observed discrepancies stem from semantic factors.
  • Sample selection influences the conclusions. The primary "Safety-Sensitive Instances" (SSI) for the VLA model cover only 6 out of 35 scene units and exhibit significant variance across different random seeds—insufficient evidence to support a consistent, counterintuitive ranking pattern overall.
  • Methodological improvements lack adequate ablation. The three-step approach—simultaneously expanding the parameter range, correcting procedural calls, and incorporating execution constraints—precludes isolating the specific contribution of "reasoning decomposition." Furthermore, the improvement in SSI is accompanied by a decline in collision-free success rates.
  • SSI is not equivalent to actual safety. This metric primarily assesses ordinal ranking rather than absolute risk; moreover, since multiple features share the same distance-based denominator, the metric may redundantly reflect changes in spatial gaps.
  • Evidence for the proposed mechanism is insufficient. Neither the model's performance following the addition of a decoding layer nor the results from obstacle-classification probes confirm that the original VLA model truly understands safety risks; furthermore, differences in execution frequency may also confound the results.
Questions for authors
  • Do the primary SSI results still hold if the semantic appearance or labels of the obstacles are altered while keeping their collision shapes, dimensions, and supporting structures constant? Furthermore, what is the SSI value when no semantic planner is used?
  • Could you report the number of valid scene units and the sample counts for each category across different random seeds, and perform a paired bootstrap or permutation test on these scene units?
  • When using consistent parameter ranges, target object references, and radius constraints, what performance gains—if any—does the three-step reasoning approach retain compared to one-step or two-step methods?
  • In Table 2, "Vel." is described as the proportion of tasks where the velocity constraint is effectively applied, whereas the appendix defines it as the proportion of instances meeting the criteria before the clamping operation. Could you clarify the distinction between the model's predicted compliance rate and the actual effectiveness rate post-execution, and ensure consistent terminology throughout the main text?
  • Could you provide an ablation study on the execution length of the VLA action chunks? Additionally, could you include an analysis—such as a risk-level probe or a causal intervention—applied directly to the input of the action head, without introducing any additional decoder layers?
8 comments
AuthorsOpenReview ↗

[1/6] Rebuttal by Authors

We sincerely thank Reviewer 2RJ8 for the positive comments on our problem setting and benchmark design. We also appreciate the reviewer's recognition of our control-pipeline analysis and negative results.

We group related points and answer them: (1) Relationship between SSI and safety [W4], (2) Obstacle geometry [W1, Q1], (3) Sample size and statistical support [W2, Q2], (4) Ablation of the three-step reasoning [W3, Q3], (5) Evidence for the mechanism [W5, Q5] and (6) other question [Q4].

[W4] Relationship Between SSI and Safety

W4. SSI is not equivalent to actual safety. This metric primarily assesses ordinal ranking rather than absolute risk; moreover, since multiple features share the same distance-based denominator, the metric may redundantly reflect changes in spatial gaps.

We thank the reviewer for raising this important point. We agree that SSI measures relative ordering rather than absolute safety or risk. However, we examine not only whether the robot avoids collisions with nearby obstacles, but also whether its motion around them reflects their differing safety implications. As illustrated in Figure 1, a robot should adjust its motion to potential harm, passing an infant more cautiously than a trash bin. These differing safety implications necessitate different levels of caution, on which we base our obstacle caution tiers (OCTs), as further supported by the human validation study described below. By assessing whether motion follows this safety-related ordering, SSI provides a safety-oriented measure of motion adaptation, along with collsion metric. We will make this distinction explicit in the revised manuscript to avoid confusion.

[Human validation of the OCT ordering] To provide additional empirical support for this ordering, we conducted a human study in response to the reviewer’s concern. Twenty participants recruited through Prolific [1] completed pairwise comparisons, indicating which of two obstacles in a scene should prompt greater caution from the robot. Overall, 89.1% of responses agreed with the predefined OCT ordering, showing strong alignment with human judgments of relative caution. Fleiss’ κ\kappa was 0.62 and increased to 0.75 in a supplementary sensitivity analysis excluding one participant whose judgments differed from the OCT ordering in 83% of comparisons. The study protocol and detailed results are provided in Appendix B.2.

[1] https://www.prolific.com/

[Distinct motion features and their distance dependence] Regarding the shared distance term, we acknowledge that it may introduce correlations among the features. However, each feature captures a distinct aspect of robot motion—clearance, speed, acceleration, or jerk—whose safety implications depend on proximity to obstacles.

[W1, Q1] Obstacle geometry

W1. A conflation of semantics and geometry exists. The dimensions of obstacles and the supporting tabletop alter the traversable space; removing objects from the table increases the accuracy of distance-based ranking from approximately 3% to 62%, which undermines the argument that the observed discrepancies stem from semantic factors.

We thank the reviewer for highlighting this factor. We agree that the supporting table affects traversable space and measured clearance. Although placing these objects (wine, hot chocolate, a table lamp, and water) on a table is more natural setting, we also evaluate a floor-placement setting to assess the table’s influence. We have updated Tables 14 and 15 to report both settings. Here, we report the mean unconditional SSI for each policy group to ensure a fair comparison on the same tasks. Results for the other task intersections (TSR-I or CFSR-I) are also provided in Table 15.

Policy groupWith tableWithout tableChange
VLA−0.036−0.021+0.015
VoxPoser+0.414+0.356−0.058
PIVOT+0.251+0.299+0.048

The comparison preserves the main qualitative conclusions. For VLA, collision-free success remains low in both settings, while All and TSR-I SSI remain near zero. For VoxPoser, CFSR-I SSI remains positive across all eight models, indicating that its observed caution-tier differentiation persists after table removal. For PIVOT, task success and All and TSR-I SSI show relatively small changes, although CFSR-I SSI increases for most models. Thus, while the supporting table influences individual measurements, the evaluated policies show similar overall SSI trends.

AuthorsOpenReview ↗

[2/6] Rebuttal by Authors

Q1. Do the primary SSI results still hold if the semantic appearance or labels of the obstacles are altered while keeping their collision shapes, dimensions, and supporting structures constant?

Thank you for this constructive question. We tested whether the SSI results persist when obstacle appearance or labels change, while keeping collision geometry, dimensions, and supporting structures fixed. We tested two color changes and replaced object names with same-meaning synonyms. Note that label change test is not conducted on VLA since it does not receive object labels as input. For these ablations, we selected six of the 18 obstacle types—two from each caution tier. To keep collision shape and dimensions, we place all obstacle on the floor (not on the table) at the same position and use the same transparent collision shape for every obstacle. Following the paper, the distance is measured surface-to-surface from that shape.

  • VLM · Qwen3.5-9B VoxPoser (194 tasks in the CFSR intersection)
SettingSSICFSR/TSR
Fixed collision shape (reference)+0.29199.5%
Color change 1+0.31799.5%
Color change 2+0.31999.5%
Same-meaning synonym labels 1+0.342100.0%
Same-meaning synonym labels 2+0.221100.0%
  • VLA · XR-1 (120 task)
SettingSSICFSR/TSR
Fixed collision shape (reference)−0.14026.6%
Color change 1−0.09128.5%
Color change 2−0.25530.7%

As shown in the table, the qualitative SSI pattern remains consistent across the tested appearance variations. It has been added to the Appendix E.5.

Q1. ... Furthermore, what is the SSI value when no semantic planner is used?

We evaluate a rule-based planner that makes no LLM calls and applies the same clearance to every obstacle category. We have added it to Figure 3a as a non-semantic planning baseline.

Rule-based baselineTSR (%)CFSR (%)SSI
With table
Uniform clearance: 30 cm100.0100.0+0.067
Uniform clearance: 60 cm99.899.8+0.072
Without table
Uniform clearance: 30 cm100.0100.0+0.072
Uniform clearance: 60 cm100.0100.0+0.050

Both configurations achieve high TSR and CFSR. However, the planner applies the same clearance regardless of obstacle semantics, so it can avoid collisions but cannot adapt its behavior to the semantic safety implications of different obstacles. Consistent with this limitation, the higher-clearance setting achieves perfect CFSR but low SSI, reflecting limited motion differentiation across caution tiers.

[W2, Q2] Sample size and statistical support

W2. Sample selection influences the conclusions. The primary "Safety-Sensitive Instances" (SSI) for the VLA model cover only 6 out of 35 scene units and exhibit significant variance across different random seeds—insufficient evidence to support a consistent, counterintuitive ranking pattern overall.

We thank the reviewer for highlighting this limitation. Frequent collisions leave a small shared subset of collision-free successful episodes for estimating the Semantic Sensitivity Index (SSI), resulting in substantial variability across seeds. We agree that these estimates are insufficient to establish a consistent reverse caution ordering across VLA policies. Accordingly, we emphasize low CFSR as the primary VLA finding and interpret SSI as a tentative diagnostic of motion adaptation within the observed subset. We have clarified this interpretation in the main-text RQ1 discussion and Appendix E.1.

AuthorsOpenReview ↗

[3/6] Rebuttal by Authors

Q2. Could you report the number of valid scene units and the sample counts for each category across different random seeds, and perform a paired bootstrap or permutation test on these scene units?

We thank the reviewer for this suggestion. For VLA policies, SSI is supported by only 6 of the 35 cells in total: seeds 0, 3, and 16 contribute 1, 1, and 4 cells, respectively, with no overlap across seeds. Table 16 reports the task counts by caution tier and seed.

We conducted a cell-level bootstrap on seed 16, since the single contributing cell in each of the other seeds cannot support estimation of between-scene variability. We repeatedly sampled four cells with replacement from the four available cells and recomputed SSI, keeping each cell’s seven motion features together and using the same draws across models. We evaluated all 256 possible ordered samples.

ModelSeed 16 SSICentral 95% bootstrap range
XR-1−0.500[−1.000, 0.000]
π0.5\pi_{0.5}-OpenTau0.000[−1.000, 1.000]
GR00T N1.5−0.071[−0.714, 0.357]
π0.5\pi_{0.5}-Human300−0.071[−0.857, 0.714]

All four models have broad bootstrap ranges that include zero, indicating that the observed cells provide limited support for a consistent reverse caution ordering. These results are restricted to the four contributing cells in seed 16 and do not establish consistency across scenes or seeds. We therefore consider low CFSR as main observation. Higher CFSR would support a more meaningful SSI evaluation. For now, we interpret SSI cautiously.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks

We sincerely thank the reviewers for their constructive feedback. They highlighted our safety framing beyond collision avoidance [2RJ8, LmUP], with an emphasis on selective caution [kGYo]. They also noted the diagnostic value of our experimental design and control-pipeline analysis [2RJ8] and our broad VLM/VLA evaluation [LmUP, kGYo].

We summarize our revisions below:

  • Distinguishing semantics from geometry [2RJ8, LmUP, kGYo]. We tested obstacles placed on the floor (without table) and varied obstacle appearance with fixed collision geometry. The main qualitative findings persisted, although the presence of table affected individual measurements. These results are updated in Tables 14–15 and Table 20, respectively.
  • Validating and clarifying SSI [2RJ8, LmUP]. We clarified that SSI is relative safety metric and complements collision metrics by measuring how executed motion follows the expected ordering of caution across obstacle tiers. An additional human study showed strong agreement with this expected ordering, supporting its use as a reference for evaluating motion adaptation and thereby strengthening the rationale for SSI. We also clarified that unconditional SSI, which is reported in the Table 15, supports group-independent comparisons. The human study is updated in Appendix B.2 and Table 6.
  • Isolating the effects of the the modular approach [2RJ8, kGYo]. We conducted controlled experiments under matched parameter ranges and tested the effect of removing clamping. The modular approach retained its advantage in SSI in both comparisons. These results are updated in Table 2 and 22.
  • Strengthening the VLA analysis [2RJ8]. We reported bootstrap confidence intervals, number of valid scene units and the sample counts for VLA policies’ SSI estimates. We also replaced the decoder-based VLA analysis with direct probing and interventions at the action-head input. The results remain consistent with our main findings. These are updated in Tables 16 and 4, respectively.
  • Clarifying novelty and scope [kGYo, LmUP]. We expanded the related-work discussion and refined our contribution: using matched obstacle comparisons to assess whether executed motion follows the expected caution ordering. We have addressed these points in “Related Work” and “Conclusions” of the revised manuscript.

Beyond the revisions summarized above, we have incorporated additional changes into the manuscript in response to the reviewers’ valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.