- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies MLLMs as online guides to a frozen driving planner. It introduces ActionGuide-Bench with four tasks: evidence grounding, consequence prediction, trajectory verification, and corrective guidance. Verification filters planner proposals, while corrective guidance supplies an alternative when none are accepted. The authors also construct simulation-based training data and fine-tune an MLLM. Experiments show improved reasoning accuracy on real-world driving scenes and better closed-loop performance in simulation.
Critical strengths
- The evaluation setting is well chosen. Because the planner stays frozen, any change in driving behavior can be credited to the MLLM guide instead of to a retrained motion policy, and the effect is measured in closed loop. Driving QA benchmarks can’t answer that question, and it’s the main reason I lean toward the paper’s value.
- simulation-only training lifts overall accuracy on real-world scenes from 41.41% to 53.45%. I found the gap between behavior-level and geometry-sensitive tasks the most informative part of the paper, since it hints at what simulation data can and cannot teach.
Critical weaknesses
-
The methodological novelty appears incremental relative to the VLM-based trajectory evaluation and planner-guidance approaches discussed in Section 6 (VLMPlanner, DriveCritic), making convincing experimental attribution especially important.
-
My main concern is attribution. The paper does not separate verification from corrective guidance, and it does not show which training tasks drive the closed-loop gain. Table 2 makes this pressing. The pretrained guide alone raises the at-fault Scene Score from 43.66 to 47.92, and fine-tuning adds 1.58 on top. Most of the improvement therefore comes from plugging in an off-the-shelf MLLM, and the part attributable to the paper’s training data is small. That 1.58 comes from one base planner, 131 curated scenarios, and a single seed, with no error bars. I can’t tell whether it is outside run-to-run noise. If it isn’t, the case for the fine-tuning data gets much weaker.
-
I’m not convinced every benchmark task requires scene reasoning. In Corrective Guidance, 71.5% of answers are “straight” (Fig. 6), so a model could do well by picking the straightest trajectory without looking at the scene. Shuffling option order and colors doesn’t fix this. The capability–utility correlation has a related problem: it rests on seven models with no stated inclusion criteria, so one or two points could be carrying the trend.establish that the benchmark reliably tracks downstream usefulness.
Questions for authors
- Why leave Qwen2.5-VL-7B out of the correlation analysis? I wonder how much the result changes with all eight guides included, or with ActionGuide removed.
- I couldn't tell whether verification or corrective guidance is doing most of the work. This also makes the training result hard to interpret: does supervision on evidence grounding and consequence prediction help closed-loop performance, beyond training verification and correction alone? I’d consider these ablations necessary for the paper’s main claim.
- How reliable is the improvement over the pretrained guide? A paired confidence interval across scenes would help. Has it held up in repeated rollouts or with another planner?
- For Corrective Guidance, I would first check a baseline that ignores the scene and just picks the straightest candidate. Given the answer imbalance, it is not clear how much scene understanding is needed to score well. Does the improvement remain when directions are weighted equally?
5 comments
Response to Reviewer X2aN (Part 2)
W3 & Q4: Direction-balanced Evaluation
We first report direction-balanced accuracy, showing that ActionGuide’s improvement persists. The capability–utility correlation remains positive with the revised balanced metric.
1. Per-direction and direction-balanced evaluation.
- Direction-balanced results. We report per-direction accuracy and direction-balanced accuracy, the unweighted mean across all six directions. ActionGuide achieves 34.25%, compared with 17.26% for Cosmos-Reason2-8B. For comparison, a policy that always selects the straightest direction without scene understanding would obtain 16.67% under this metric. These results show that ActionGuide’s advantage is not attributable to the direction bias.
- Direction distribution. The benchmark’s directional proportions were informed by the ground-truth trajectories in our closed-loop evaluation scenarios, where 73.7% of reference steps correspond to straight motion. We’ll further include direction-balanced accuracy as an important metric in revision.
| GT direction | Qwen3-VL-8B | Cosmos-Reason2-8B | ActionGuide |
|---|---|---|---|
| Straight | 13.5 | 18.9 | 32.6 |
| Light left | 13.2 | 18.9 | 34.0 |
| Light right | 10.9 | 7.8 | 26.6 |
| Hard left | 10.5 | 10.5 | 47.4 |
| Hard right | 24.3 | 18.9 | 45.9 |
| Stop | 28.6 | 28.6 | 19.0 |
| Overall accuracy | 14.57 | 18.28 | 32.45 |
| Direction-balanced accuracy | 16.85 | 17.26 | 34.25 |
2. Capability–utility correlation with direction balancing.
We recompute the correlation metrics by replacing CG accuracy with its direction-balanced counterpart. Across all eight evaluated MLLMs, the positive association with closed-loop Scene Score persists (Pearson r = 0.866; Spearman ρ = 0.738). Thus, the observed association is not specific to the original direction-weighting scheme. We thank the reviewer for motivating this robustness check and we’ll update the corresponding figure in revision.
Thanks for the additional experiments. The ablations address my main concern about attribution, especially the contribution of consequence-prediction supervision beyond training verification and correction alone. The methodological novelty remains modest, but the additional evidence makes the empirical contribution convincing enough for weak acceptance. I am raising my rating from 2 to 3 and would ask that these results, including the uncertainty estimate, be incorporated into the paper.
Response to Reviewer X2aN (Part 1)
Thank you for the detailed and insightful feedback. We appreciate your recognition of simulation-to-real transfer, behavior-level analysis and clear presentation. Below, we carefully address each of your concerns with additional experiments, analysis and clarifications.
W1: Paper Novelty
In this paper, we propose a unified evaluation-and-learning framework that connects action-grounded reasoning capabilities to their closed-loop driving utility. Compared with previous work, we clarify the novelty lies in:
Q1: Correlation Robustness
We thank the reviewer for the suggestion and have expanded the analysis to all eight evaluated guides. The positive association between action-grounded benchmark performance and closed-loop utility persists (Pearson r = 0.865; Spearman ρ = 0.738). We will update the corresponding figure in the revision.
W2 & Q2: Attribution of performance gains
We conduct two sets of ablations to (1) disentangle the complementary contributions of Trajectory Verification (TV) and Corrective Guidance (CG), and (2) evaluate the importance of Evidence Grounding (EG) and Consequence Prediction (CP) supervision.
1. Disentangle contributions for TV and CG.
We compare TV only, TV with a simple fallback, and TV with CG under the same guidance interface. Adding CG improves the At-fault Scene Score over TV-only by 2.58 points for Cosmos and 1.48 points for ActionGuide, demonstrating the complementary benefits of verification and correction. In contrast, the simple fallback (going straight) consistently underperforms TV-only across all three MLLMs, highlighting the importance of context-aware corrective decisions.
The logged-future reference, which makes decisions by evaluating candidates using recorded future trajectories of surrounding agents, further reaches 59.11, illustrating the potential benefits of more accurate trajectory assessment.
2. Training data ablation for EG and CP.
Adding explicit Consequence Prediction supervision improves the score to 48.75 (+3.51), while incorporating all four tasks achieves the best performance of 49.50 (+4.26). Training only on Trajectory Verification and Corrective Guidance yields less generalizable results. These results highlight the importance of grounding driving decisions in scene evidence and anticipated action consequences.
W2 & Q3: Improvement Reliability
ActionGuide consistently achieves positive gains over Cosmos-Reason2-8B across three evaluation runs (+1.58, +1.74, and +1.49 Scene Score).
1. Paired scenario-level analysis.
We compute a 95% paired bootstrap confidence interval using 10,000 resamples of the 131 scenarios, preserving the Cosmos–ActionGuide pairing within each resample. The original run yields an observed gain of +1.58 points, with a 95% CI of [−0.87, +4.14], showing positive point estimate.
2. Repeated-rollout consistency.
Two additional runs with different seeds show positive gains (+1.74 and +1.49 points), confirming a consistent trend across all three runs.