- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PRESTO, a safety layer that combines multimodal language model reasoning with high-order control barrier functions. The model identifies objects, assesses their safety relevance, and assigns semantic priorities that guide constraint relaxation when modifying nominal control inputs. Validation includes driving and manipulation case studies, integration with existing planners on Dev10 and Bench2Drive, hardware demonstrations, and an ablation of semantic prioritization.
Critical strengths
The paper addresses a meaningful limitation of purely geometric safety constraints: treating obstacles uniformly can cause unnecessary conservatism. Using semantic priorities to guide constraint relaxation offers a useful connection between visual-language understanding and control. The experiments provide evidence of practical utility across driving and manipulation. Table 3 shows collision reductions when PRESTO is added to existing planners, while Table 4 isolates the benefit of semantic prioritization with the bounding-box source and CBF formulation fixed within each comparison.
Critical weaknesses
- The evidence establishing the method’s comparative benefits remains limited. Tables 1 and 2 contain only 12 driving and 10 manipulation scenarios, with insufficient detail on repeated trials and statistical uncertainty. The driving comparisons also omit recent vision-language driving methods, leaving PRESTO’s additional value for language-informed planners unclear. Although Table 3 provides broader evaluation, it does not fully resolve these questions.
- Robustness to dynamic obstacles and environmental variation is not sufficiently validated. Although Appendix A.1 describes constant-velocity prediction and Sections 4.2–4.3 mention moving pedestrians and vehicles, the dynamic-obstacle settings and their coverage in Tables 1 and 2 are not clearly documented. The paper also lacks a controlled evaluation of motion prediction and sensitivity to MLLM query frequency and obstacle speed. A controlled evaluation along these dimensions would help establish whether the safety improvements remain reliable as obstacle information becomes outdated. In addition, the qualitative examples and aggregate results do not establish robustness under different weather and lighting conditions, such as nighttime.
Overall, the approach is promising, but these gaps limit my confidence in the empirical conclusions. I would consider increasing my score if the authors address these concerns with convincing clarification and targeted evidence.
Questions for authors
- Comparative evidence: Could the authors provide a matched-protocol comparison with a representative recent vision-language driving method, such as SimLingo[1] or AutoVLA[2], either directly or with and without PRESTO?
- Environmental coverage: Could the authors clarify the composition of the test environments, particularly the weather and lighting conditions, and provide additional experiments under other conditions, such as nighttime or rain, to assess the method’s robustness?
- Dynamic sensitivity: Which case studies contain moving obstacles, and does simulation continue during MLLM inference? A targeted experiment varying query frequency and obstacle speed, while accounting for actual inference latency, would help establish when the safety improvements remain reliable.
[1] Renz et al. SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment. CVPR, 2025.
[2] Zhou et al. AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning. NeurIPS, 2025.
3 comments
Rebuttal by Authors (pt. 2)
Environmental coverage: Could the authors clarify the composition of the test environments, particularly the weather and lighting conditions, and provide additional experiments under other conditions, such as nighttime or rain, to assess the method’s robustness?
We thank the reviewer for raising this point. Our original case studies use the default weather settings in CARLA Town10HD. Separately, our Bench2Drive evaluation in Table 3 includes varying weather and lighting conditions. The Bench2Drive evaluation contains 27 weather configurations, including nighttime and rain, with their distribution shown in Figure 8 of [1].
To further assess robustness to environmental conditions, we extended four of our case-study scenarios to different lighting (sunset and nighttime) and weather (fog and rain) conditions, with three seeds per scenario. As shown in Table 14, PRESTO maintains a 0% collision rate and 100% route completion across all tested conditions. We also provide qualitative results in Figure 12. Under fog, the MLLM occasionally misidentifies objects, such as describing hay bales as “round objects,” while still correctly classifying them as collision-critical. Under rain, water droplets introduce local blur and partial occlusion in the camera images.
We have added these experimental results, implementation details, and additional analyses to Appendix A.3.4 ("Lighting and Weather").
Dynamic sensitivity: Which case studies contain moving obstacles, and does simulation continue during MLLM inference? A targeted experiment varying query frequency and obstacle speed, while accounting for actual inference latency, would help establish when the safety improvements remain reliable.
We thank the reviewer for raising these questions. Our original case studies consist of static obstacles, while the Bench2Drive evaluation includes moving traffic. Both use a synchronous simulation configuration, consistent with prior Bench2Drive evaluations [2]. In this setting, the simulation is paused and waits for MLLM inference and to complete. To assess the effect of inference latency in dynamic scenes, we added two experiments with moving vehicles under asynchronous execution.
First, we evaluated a controlled stopping scenario in which a lead vehicle brakes in front of the ego-vehicle. We varied the initial bumper-to-bumper distance and ego-vehicle speed from 10 to 30 km/h to examine how inference latency affects collision avoidance (Fig. 10). We compared MLLM perception, with a measured latency of 0.83 ± 0.07 s, against oracle perception with an latency of 0.04 s. The results quantify the additional stopping distance associated with MLLM inference latency, showing that higher speeds require larger initial gaps to avoid collisions. This suggests that the current implementation is more suitable for lower-speed operation.
Second, we evaluated PRESTO online and asynchronously in a dynamic cut-in scenario (Fig. 11). In this scenario, a sedan merges into the ego-vehicle’s lane and brakes abruptly. In this online setting, a new MLLM query is triggered whenever the previous inference completes, with a mean latency of 0.94 s. Across three seeds, the ego-vehicle completes the route without collision and maintains a minimum distance of 0.48 m from the sedan that cut-in front.
Together, these experiments characterize the operating conditions under which inference latency affects collision avoidance and demonstrate asynchronous execution in a dynamic driving scenario. We have included more analyses and experimental setup details in Appendix A.3.3.
References:
[1] Jia, X., Yang, Z., Li, Q., Zhang, Z. and Yan, J., 2024. Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.
[2] Renz, K., Chen, L., Arani, E. and Sinavski, O., 2025. SimLingo: Vision-only closed-loop autonomous driving with language-action alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.11993–12003.
Thank you to the authors for the detailed rebuttal and extensive additional experiments, which have substantially addressed my previous concerns. In particular, the additional comparisons, robustness evaluations under varying environmental conditions, and asynchronous experiments with dynamic obstacles provide stronger empirical support for the proposed approach.
These results further demonstrate the potential of combining semantic understanding with control-theoretic safety constraints, enabling the method to leverage semantic reasoning while providing meaningful safety benefits. I also agree with the authors that inference latency currently limits the method's applicability, particularly in higher-speed scenarios. Further reducing inference latency would improve its robustness and practical applicability in dynamic environments.
Overall, I believe the authors have adequately addressed my major concerns through their thorough responses and additional experiments. I have therefore decided to increase my rating.
Thank you for your suggestions and constructive feedback!! Rebuttal by Authors [pt. 1]
We thank Reviewer CfYy for the suggestions on strengthening our empirical evaluation. In response, we added a matched-protocol comparison with SimLingo, repeated the case studies over four seeds with mean and standard deviation, and added randomized bird's-eye-view experiments, dynamic CARLA experiments, and lighting and weather experiments. We address each point below.
We thank the reviewer for this suggestion. We evaluated SimLingo under the same scenarios and four random seeds as our method. We also implemented AutoVLA, but its publicly released checkpoint is finetuned for NuPlan rather than CARLA, and it did not generalize successfully to our evaluation scenarios. We therefore exclude its result from Table 1.
We report the SimLingo results in Table 1. SimLingo achieves a collision rate of 54.17 ± 8.33% and route completion of 67.86 ± 5.03%, slightly outperforming TransFuser++ , but performing substantially worse than PRESTO with a collision rate of 6.25±4.17 and route completion of 97.92±2.42. We further examined SimLingo's language outputs and observed two failure modes. First, SimLingo was able to correctly identify the hazard but it still led to collisions. In the opening door scenarios, SimLingo recognizes the hazards and suggests changing lanes, but its planned maneuvers vary across frames, resulting in collisions in 3/8 opening-door runs. Second, SimLingo struggles with unfamiliar objects. It describes trash bags as "the black car ahead" and fails to identify the shopping cart, resulting in collisions in all evaluated runs of these scenarios. These results suggests that semantic reasoning alone does not guarantee safe control
We thank the reviewer for raising this concern. We have repeated the comparisons in Tables 1 and 2 across four random seeds and updated the results to report mean and standard deviation. Beyond these case studies, our original manuscript also includes a Bench2Drive evaluation over 220 driving scenarios (Table 3).
To further strengthen the evaluation, we added additional experiments in bird's-eye-view space to isolate the contributions of the safety hierarchy and priority ranking. We generated 100 scenarios across five random seeds (500 scenarios in total), varying obstacle positions and safety priorities. In scenarios where a collision-free path exists (Table 7), PRESTO achieves zero critical collisions across all 500 trials (95% upper confidence bound: 0.6%) with 96.6% route completion. In comparison, treating all constraints as soft results in a 12.8% critical collision rate, while treating all constraints as hard reduces route completion to 69.6%. We have included the experiment results and setup details in Appendix A3.2.
We additionally introduced two dynamic driving experiments in CARLA with MLLM perception. In the controlled stopping experiment, we varied the initial distance to the lead vehicle and the ego-vehicle speed across 97 configurations, each evaluated over three seeds (Fig. 10). We also evaluated an asynchronous cut-in scenario across three seeds (Fig. 11). Both experiments examine collision avoidance with moving obstacles while accounting for measured inference latency. We have included the experiment results and setup details in Appendix A3.3.