- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the robustness of reasoning-enabled driving VLAs. The authors propose ReasonBreak, an evaluation framework that treats reasoning and trajectory as two output surfaces. Reasoning is decomposed by an LLM into object, relation, implication, and planning fields to measure semantic deviation, alongside structural failures, including slowdown and DoS. Besides, trajectory degradation, reasoning–trajectory coupling, and downstream safety are evaluated separately. Using Best-of-N surface-form perturbations in open-loop evaluation and per-step random perturbations in closed-loop simulation, the paper reports high attack success rates, weak reasoning–trajectory coupling, and degraded safety metrics.
Critical strengths
- Reasoning-enabled driving VLAs are an important direction in industry. The paper evaluates reasoning and trajectory as two separate but interacting output surfaces, which is practically relevant for real-time driving. The finding of weak reasoning–trajectory coupling also offers useful insight into the faithfulness of reasoning in driving VLAs.
- The paper covers both open-loop and closed-loop evaluation, provides sensitivity analyses on the query budget, several thresholds, window length, perturbation strength, and input length, and validates the semantic evaluator with multiple LLM judges and a human study. The paper also honestly reports negative and mixed results and provides a simple mitigation baseline.
Critical weaknesses
- Improved Readability: [1] The format of all references in the main paper is inappropriate, except for Sec 4.2 Open-Loop (line 312). Might use \citep{} instead of \cite{}. [2] In Sec 2.1, a potential typo is meaningless '/' in 'black-box/visual attacks' (line 140). [3] In Sec 3.2.2 Open-loop, for definition of ASR (line 236), is only limited to , so how to decide which has two conditions? And in Sec 3.2.1 Reasoning Evaluation, has been noted as 'reasoning target' (line 209), but here is defined as target objective (line 238), so are these two the same meaning, and what is the relation between and if is because there is no definition of for trajectory evaluation beforehand in Sec 3.2.1. And the use of subscripts and superscripts for is highly inconsistent, that is, the definitions indicated by the indices for differ across Sec 3.2.1 Reasoning Evaluation/Trajectory Evaluation and Open-loop here. [4] In Sec 3.2.2 Closed-loop, there is a definition of 'excess deviation' (line 242), so is it different from perturbation-induced excess trajectory error (line 222) in Sec 3.2.1 Trajectory Evaluation, whose meaning is similar? And here is a conflict that denotes either reasoning or trajectory output (line 245), but has been noted as the trajectory output or ego state (line 221) in Sec 3.2.1 Trajectory Evaluation. [5] In Sec 4.2, and are used to represent perturbed input (line 320 and line 326), however, are noted as benign input (line 175) in Sec 3.1, so should the variants of be used here? [6] In Sec 5.2, the referred Table 3, Table 4, Figure 3, and Figure 4 are after the 9th page, which is outside the main paper scope, but it seems like that they do not belong to any section in Appendix. [7] In Appendix C.1, the decoding configurations are outside the paper column (line 1412 and line 1420), and in Appendix C.4, the closed vocabulary is under the same condition (line 1527).
- More Illustration: [1] In Sec 3.2.2 Closed-loop, there is no detailed information for the function , that is, the concrete caculation process. [2] In Sec 4, there is no detailed information on perturbation injection, except for 'capitalization changes, word scrambling, and character-level noise' (line 297). There should be more illustration of how to achieve the perturbed textual input, what these perturbed texts look like, and how these perturbations might influence each type of reasoning surface and trajectory surface, such as slowdown. [3] In Sec 5.2, there is only a brief conclusion about the weak correlation between reasoning and trajectory. It would be better to have an analysis and explanation of this phenomenon, as it has been noted 'contrasts with the intended role of reasoning in reasoning-enabled VLA systems' (line 416). [4] There is no sensitivity analysis for the semantic-deviation threshold , a related experiment is needed to illustrate this parameter setting. For the analysis of window size in Appendix B.4, the temporal consistency requirement is introduced here (line 1385), but in Sec 3.2.2 Closed-loop, the window size is noted as , so please illustrate the precise definition of 'window size' or the relation between and .
Questions for authors
Detailed in Weakness.