- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RT-SAFE, a benchmark for evaluating agent safety in real-time embodied environments. Built on SimWorld and Unreal Engine 5, it provides 36 navigation routes across five city maps, allowing the environment to evolve during both inference and action execution while separately recording Passive and Active Collisions. Evaluating eight VLMs, the authors find that task completion remains similar between Static and Real-time evaluation, while collision counts increase by 12.3× and Safe Success Rate (SafeSR) drops from 19.8% to 0.7%. The paper also investigates reasoning effort and demonstrates the benchmark's potential for Offline RL training.
Critical strengths
-
RT-SAFE is a a practical benchmark for real-time embodied safety. It incorporates measured inference latency into a continuously evolving simulation and separately records safety events during inference and action execution, enabling a more comprehensive evaluation of real-time safety.
-
It has many systematic experimental analysis. The paper evaluates eight VLMs and investigates the effects of environment difficulty, execution mode, reasoning effort, and navigation behavior on safety. It reveals a gap between task completion and safety, while also demonstrating the benchmark's potential for Offline RL training.
Critical weaknesses
-
There are confounding factors in the Static versus Real-time comparison. Section 3.3 changes both simulator timing and prompt instructions. The Real-time prompt explicitly emphasizes continuous environment evolution and encourages conservative actions, potentially affecting action selection and decision frequency. This makes it difficult to attribute the observed safety differences solely to the real-time execution mechanism.
-
Pedestrian behavior modeling and collision counting may affect evaluation reliability. Approximately 94% of collision counts in the Hard Real-time setting are Human Contacts, but the paper does not clarify whether scripted pedestrians can autonomously avoid the agent. Moreover, Appendix A.3 indicates that sustained contacts may generate repeated collision counts without general deduplication.
-
The framework fails to adequately validate safe task feasibility under the restricted action space. RT-SAFE provides only 16 predefined movement, turning, and waiting actions, potentially limiting obstacle avoidance in dynamic environments. Without an Oracle Planner or Expert Policy under the same action space and real-time constraints, it remains unclear whether the extremely low SafeSR primarily reflects poor decision-making or limitations in task feasibility.
Questions for authors
Q1. Could the authors provide a controlled comparison using identical prompts while changing only simulator timing, to disentangle the effects of real-time execution and prompt instructions?
Q2. Do pedestrians autonomously avoid or react to the agent? Could the authors report deduplicated collision statistics based on independent contact events and analyze the sensitivity of the results to pedestrian behavior and counting rules?
Q3. Could the authors evaluate an Oracle Planner or Expert Policy with privileged simulator state under the same action space and decision-latency constraints, and report its SafeSR in the Hard Real-time setting?
Q4. Response latency varies substantially across VLMs, potentially confounding cross-model safety comparisons. Could the authors use matched decision delays or controlled artificial delays for the same model to better separate the effects of inference latency and decision quality?
3 comments
Thank you again for your detailed review. With around one day remaining in the rebuttal period, we would appreciate any follow up on whether our responses and additional experiments address your concerns.
We have further updated §5 of our general response: with the zero safety event trajectories validated on all 36 hard tasks at 0s and 1s delays, and UE-validated trajectories with zero collisions for all 36 easy tasks at the fix 3s delay, using the same 16 actions. This matches the difficulty and decision delay used in our RL experiments.
If you have any questions or concerns, we would be happy to address them during the remaining discussion period.
The additional experiments address most of my original soundness concerns. The restricted action space remains a limitation, but I no longer view it as sufficient to outweigh the paper’s contributions. I therefore raise my score from 2 to 3.
Thank you for your detailed review and for recognizing RT-SAFE’s practical value and systematic analyses. We address your concerns below, and additional experiments in our main response. W1 / Q1: Identical prompt comparison. Please see §2 of our main response. With the same neutral prompts, mean collisions increase from 4.06 in static to 18.22 in real time, and the shift toward shorter actions. These results strengthen our main finding that real time execution affects both safety and action selection, beyond differences in prompt instructions. W2 / Q2: Pedestrian behavior and collision counting. Please see §§1 and 4 of our main response. Pedestrians can make local adjustments, but they do not anticipate the agent’s future actions or compensate for its inference delay. RT-SAFE therefore evaluates whether the agent can navigate safely while its surroundings continue to evolve, without assuming that other actors will accommodate its reasoning time. Repeated contacts with the same actor are deduplicated within each recording window. When counting only whether an episode contains any collision, the proportion increases from 60.4% in static to 99.3% in real time. This shows that the safety gap extends across episodes, rather than arising solely from repeated collision counts in a few episodes.
W3 / Q3: Safe task feasibility. Please see §5 of our main response. Privileged trajectory search validated successful trajectories with zero recorded safety events using the same action space at both 0 second and 1 second decision delays for all tasks. These results demonstrate safe task feasibility, including under nonzero latency.
Q4: Reasoning quality versus inference latency. Please see §3 of our main response. At controlled delays of 0, 3, and 5 seconds, longer delays lower mean safety scores across all three effort settings, while higher effort does not consistently improve safety at matched delays. These results reinforce our central finding that more reasoning alone does not ensure safer behavior: reasoning effort and decision frequency must be considered together with the exposure time they have.