- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduced OSMA-Bench, a RoboCasa simulation benchmark to test whether robot navigation policies adjust their motion to what an obstacle is. The key contribution is a matched design: across varied routes and kitchen layouts, the obstacle's position stays fixed while its type changes across three caution tiers (Living, Fragile and Robust). A new metric, the Semantic Sensitivity Index (SSI), checks whether clearance and distance-adjusted speed and smoothness follow that caution order. The authors show that current VLM- and VLA-based policies don't reliably adapt their motion to obstacle safety, even when the models can tell obstacles apart. Splitting control-value inference from code generation narrows this gap for VLM-based policies.
Critical strengths
- To my knowledge, this is the first benchmark I could confirm that swaps only the obstacle's identity at a fixed pose, layout and route in navigation and scores graded caution in executed motion.
- The framing of caution should be selective and not maximal is well-motivated. And research on safety of action-based foundation model is timely.
- Broad evaluation with numerous policy configurations, tasks, and extra metrics.
Critical weaknesses
- I'm not convinced the RQ2 improvement comes from separating parameter inference from code generation. In the baseline, the speed setting never actually takes effect. And with RV capped at 80 cm it couldn't for High-tier obstacles anyway. The 3-step method, on the other hand, gets a larger range, a clamp, and has the target obstacle's parameters rewritten directly. The 2-step version, which only separates the steps, improves SSI by just 0.01, so I'd like to see ablations that separate out these factors.
- Four of the six Fragile obstacles are on tables, and F.6 says their measured clearance stays around 0.6 m no matter what clearance the policy asks for.
- The novelty is a bit overclaimed and the closest prior work is missing. These include SafeVLA [1], a peer-reviewed household VLA safety benchmark with semantic hazards; Weber et al. [2], which use safety classes with class-specific clearance; and HuNavSim [3], which uses the same clearance, speed, acceleration and jerk features. HazardArena [4] already uses paired safe/unsafe VLA scenarios with matched setups, and AGENTSAFE [5] and INT-ACT [6] already report the "knows but doesn't act" gap. The specific combination OSMA offers is new to my knowledge however. [1] SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning. NeurIPS 2025. https://openreview.net/forum?id=dt940loCBT
[2] Weber et al. Safety Classes Semantic Costmaps from RGBD Sensors for Risk-Aware Robot Navigation. CoDIT 2024. https://doi.org/10.1109/CoDIT62066.2024.10708523
[3] Pérez-Higueras et al. HuNavSim: A ROS 2 Human Navigation Simulator for Benchmarking Human-Aware Robot Navigation. IEEE RA-L, 2023. https://doi.org/10.1109/LRA.2023.3316072
[4] Chen et al. HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models. arXiv:2604.12447, 2026. https://arxiv.org/abs/2604.12447
[5] AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions. arXiv:2506.14697, 2025. https://arxiv.org/abs/2506.14697
[6] From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models (INT-ACT). arXiv:2506.09930, 2025. https://arxiv.org/abs/2506.09930
Questions for authors
- What are SSI and the tier-pair margins for all models when the four tabletop Fragile obstacles are excluded or moved to the floor? Do the RQ1 rankings and the negative High-minus-Moderate margins still hold?
- How does the 3-step method do without the step that rewrites the evaluated obstacle's parameters and inserts missing calls?
4 comments
[2/2] Rebuttal by Authors
[W1, Q2] Ablation of the three-step reasoning
W1. I'm not convinced the RQ2 improvement comes from separating parameter inference from code generation. In the baseline, the speed setting never actually takes effect. And with RV capped at 80 cm it couldn't for High-tier obstacles anyway. The 3-step method, on the other hand, gets a larger range, a clamp, and has the target obstacle's parameters rewritten directly. The 2-step version, which only separates the steps, improves SSI by just 0.01, so I'd like to see ablations that separate out these factors.
Q2. How does the 3-step method do without the step that rewrites the evaluated obstacle's parameters and inserts missing calls?
Thank you for your constructive discussion. We agree that the parameter range and clamp operation are entangled. We first clarify the other points raised and then present the ablation results.
[Clarification of remaining concerns]
- The inferred speed rarely affected execution in the baseline was the main motivation for extending the two-step approach to three steps Compared with the baseline, three-step approach introduces two changes: (1) separating parameter inference from code generation and (2) accounts for the physical interaction between the clearance radius and the velocity map.
- Parameter rewriting is the mechanism through which both modular approaches apply estimated values to the final generated code. The approaches separate parameter estimation from code generation, and rewriting applies the estimated parameter values to the code generated in the final step. Rewriting modifies the target obstacle’s parameters only when the generated code references that obstacle for a fair comparison.
- We verify that missing call insertion was never triggered and affects the results although the appendix text mentions a fallback to insert a missing call.
To avoid confusion, we have clarified them in the revised manuscript.
[Clamp ablation] In three step approach, we separately ablate clamping under both parameter ranges. ✓ denotes the clamp operation. In , the additional "✓ + max clip" setting also restricts the velocity-radius upper bound to 80 cm after clamp. We do not include this setting for , since it already predicts wide range exceeding 80 cm. Here, the results are reported using the intersection of 548 tasks.
-
cm
Clamp Value SSI SSI TSR (%) CFSR (%) Vel. (%) ✗ +0.94 +0.41 99.7 99.7 0.2 ✓ +0.90 +0.57 98.6 97.4 0 100 ✓ + max clip +0.89 +0.51 99.8 99.4 0 68.0 -
cm
Clamp Value SSI SSI TSR (%) CFSR (%) Vel. (%) ✗ +0.94 +0.62 96.5 95.4 58.4 ✓ (paper) +0.93 +0.63 96.5 95.7 59.0 100
In cm, it rarely affects the SSI. For clarity, we have reported the results without clamp operation.
[ range ablation] For fair comparison, we extends 1-step baseline to extends RV ranges from to without clamping. This analysis uses a task intersection of 539 episodes; the SSI and Value SSI values could differ from those in Table 2.
- cm
| Method | SSI | Value SSI | Vel. (%) |
|---|---|---|---|
| 1-step | +0.13 | +0.29 | 0 |
| 2-step | +0.37 | +0.94 | 0 |
| 3-step | +0.43 | +0.94 | 0.2 |
- cm
| Method | SSI | Value SSI | Vel. (%) |
|---|---|---|---|
| 1-step | +0.13 | +0.28 | 0 |
| 2-step | +0.37 | +0.94 | 0 |
| 3-step | +0.62 | +0.94 | 58.4 |
With matched parameter ranges, the three-step method improves SSI over the two-step method by for and for , while Value SSI remains identical. The larger SSI gain coincides with more frequent application of the inferred velocity (58.4% vs. 0.2%), suggesting that the wider range lets it affect execution more often. We have reported them on Table 22, and updated the Table 2 for cm, along with the three-step result for cm in the revised manuscript.
A gentle reminder for feedback
Dear Reviewer kGYo,
As the discussion period draws to a close, we would like to thank you again for your thoughtful review. We hope our response has helped clarify the points you raised. If you have any further thoughts or remaining concerns, we would welcome your feedback whenever your schedule allows.
Thank you for your time and consideration.
A gentle reminder for feedback
Dear Reviewer kGYo,
With approximately 24 hours remaining in the discussion period, we wanted to follow up on our response to your review. We would greatly appreciate it if you could let us know whether our response addresses your concerns or if there are any points that still need clarification.
Thank you for your time and consideration.
[1/2] Rebuttal by Authors
We thank Reviewer kGYo for recognizing the task design, the caution framing, and the breadth of our evaluation. Below, we address your concerns in detail.
[W2] Tabletop placement
We thank the reviewer for highlighting this factor. We agree that the supporting table affects traversable space and measured clearance. Although placing these objects (wine, hot chocolate, a table lamp, and water) on a table is more natural setting, we also evaluate a floor-placement setting to assess the table’s influence. We have updated Tables 14 and 15 to report both settings. Here, we report the mean unconditional SSI for each policy group to ensure a fair comparison on the same tasks. Results for the other task intersections (TSR-I or CFSR-I) are also provided in Table 15.
The comparison preserves the main qualitative conclusions. For VLA, collision-free success remains low in both settings, while All and TSR-I SSI remain near zero. For VoxPoser, CFSR-I SSI remains positive across all eight models, indicating that its observed caution-tier differentiation persists after table removal. For PIVOT, task success and All and TSR-I SSI show relatively small changes, although CFSR-I SSI increases for most models. Thus, while the supporting table influences individual measurements, the evaluated policies show similar overall SSI trends.
[W3] Novelty and missing prior work
We thank the reviewer for highlighting these closely related works. We have added them to Section 2 and revised our novelty claims to better distinguish our contribution. Our contribution lies in an integrated evaluation design that varies obstacle identity at a fixed placement, layout, and route to assess related caution of obstacle's safety implication in executed motion. We have clarified in the revised manuscript (Section 2). Also, we also note in the revised manuscript (Section 4.4, RQ3) that our findings are consistent with those reported in AGENTSAFE and INT-ACT.