acceptodds
Under review as a conference paper at ICLR 2027

ActionGuide: Learning Transferable Action-Grounded Reasoning for Closed-Loop Driving

Abstract

We introduce **ActionGuide-Bench**, a unified evaluation framework for connecting action-grounded reasoning in Multimodal Large Language Models (MLLMs) to closed-loop driving utility. Moving beyond isolated open-loop judgments, ActionGuide-Bench evaluates leading MLLMs at two complementary levels: an action-grounded evaluation comprising Evidence Grounding, Consequence Prediction, Trajectory Verification, and Corrective Guidance, and a closed-loop evaluation measuring how these capabilities influence planner decisions during execution. ActionGuide-Bench casts the MLLM as an online guide to a frozen base planner, verifying proposed trajectories and providing corrective guidance when the available behaviors are unsuitable, thereby exploiting the utility of MLLM judgments without retraining the underlying motion policy. To strengthen these capabilities, we further curate **ActionGuide-Train**, which connects decision-critical contexts with alternative ego behaviors, execution outcomes, and corrective actions through rule-based synthesis and planner rollouts. Comprehensive evaluation reveals three findings: (1) **Effective Guidance from MLLMs:** pretrained open-source MLLMs can already improve closed-loop driving as online guides, despite limited action-grounded reasoning; (2) **Transferable Behavior Knowledge:** behavior-level driving knowledge learned from simulation transfers effectively to real-world scenes; and (3) **Capability-Utility Alignment:** stronger and more balanced action-grounded competence is closely associated with better closed-loop utility. Together, these results position action-grounded reasoning as a key interface between multimodal understanding and concrete action control, translating visual-semantic knowledge into effective closed-loop behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal LLM, Autonomous Driving, Close-loop Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X2aNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies MLLMs as online guides to a frozen driving planner. It introduces ActionGuide-Bench with four tasks: evidence grounding, consequence prediction, trajectory verification, and corrective guidance. Verification filters planner proposals, while corrective guidance supplies an alternative when none are accepted. The authors also construct simulation-based training data and fine-tune an MLLM. Experiments show improved reasoning accuracy on real-world driving scenes and better closed-loop performance in simulation.

Critical strengths
  1. The evaluation setting is well chosen. Because the planner stays frozen, any change in driving behavior can be credited to the MLLM guide instead of to a retrained motion policy, and the effect is measured in closed loop. Driving QA benchmarks can’t answer that question, and it’s the main reason I lean toward the paper’s value.
  2. simulation-only training lifts overall accuracy on real-world scenes from 41.41% to 53.45%. I found the gap between behavior-level and geometry-sensitive tasks the most informative part of the paper, since it hints at what simulation data can and cannot teach.
Critical weaknesses
  1. The methodological novelty appears incremental relative to the VLM-based trajectory evaluation and planner-guidance approaches discussed in Section 6 (VLMPlanner, DriveCritic), making convincing experimental attribution especially important.

  2. My main concern is attribution. The paper does not separate verification from corrective guidance, and it does not show which training tasks drive the closed-loop gain. Table 2 makes this pressing. The pretrained guide alone raises the at-fault Scene Score from 43.66 to 47.92, and fine-tuning adds 1.58 on top. Most of the improvement therefore comes from plugging in an off-the-shelf MLLM, and the part attributable to the paper’s training data is small. That 1.58 comes from one base planner, 131 curated scenarios, and a single seed, with no error bars. I can’t tell whether it is outside run-to-run noise. If it isn’t, the case for the fine-tuning data gets much weaker.

  3. I’m not convinced every benchmark task requires scene reasoning. In Corrective Guidance, 71.5% of answers are “straight” (Fig. 6), so a model could do well by picking the straightest trajectory without looking at the scene. Shuffling option order and colors doesn’t fix this. The capability–utility correlation has a related problem: it rests on seven models with no stated inclusion criteria, so one or two points could be carrying the trend.establish that the benchmark reliably tracks downstream usefulness.

Questions for authors
  1. Why leave Qwen2.5-VL-7B out of the correlation analysis? I wonder how much the result changes with all eight guides included, or with ActionGuide removed.
  2. I couldn't tell whether verification or corrective guidance is doing most of the work. This also makes the training result hard to interpret: does supervision on evidence grounding and consequence prediction help closed-loop performance, beyond training verification and correction alone? I’d consider these ablations necessary for the paper’s main claim.
  3. How reliable is the improvement over the pretrained guide? A paired confidence interval across scenes would help. Has it held up in repeated rollouts or with another planner?
  4. For Corrective Guidance, I would first check a baseline that ignores the scene and just picks the straightest candidate. Given the answer imbalance, it is not clear how much scene understanding is needed to score well. Does the improvement remain when directions are weighted equally?
5 comments
AuthorsOpenReview ↗

Response to Reviewer X2aN (Part 1)

Thank you for the detailed and insightful feedback. We appreciate your recognition of simulation-to-real transfer, behavior-level analysis and clear presentation. Below, we carefully address each of your concerns with additional experiments, analysis and clarifications.


W1: Paper Novelty

In this paper, we propose a unified evaluation-and-learning framework that connects action-grounded reasoning capabilities to their closed-loop driving utility. Compared with previous work, we clarify the novelty lies in:

  1. Action-grounded capability evaluation. We focus on reasoning about concrete candidate actions and their anticipated consequences to guide driving decisions. We evaluate whether MLLMs can identify action-relevant evidence, predict consequences, verify trajectory acceptability, and select corrective alternatives. The final target is to study whether these judgments improve closed-loop behavior. This structured reasoning-to-action interface connects capability diagnosis with intervention utility, going beyond aggregate planning scores or trajectory-preference accuracy.
  2. Unified diagnosis-to-improvement framework. We connect behavior- and capability-level in-depth diagnosis with simulation-based data construction and fine-tuning, providing scene–action–outcome supervision aligned with the evaluated capabilities. This pipeline enables us to identify weaknesses, develop relevant supervision, and assess improvements in both action-grounded reasoning and closed-loop driving. The observed gains, together with our training-task ablations, support the training effectiveness in strengthening reasoning for downstream driving guidance.

Q1: Correlation Robustness

We thank the reviewer for the suggestion and have expanded the analysis to all eight evaluated guides. The positive association between action-grounded benchmark performance and closed-loop utility persists (Pearson r = 0.865; Spearman ρ = 0.738). We will update the corresponding figure in the revision.


W2 & Q2: Attribution of performance gains

We conduct two sets of ablations to (1) disentangle the complementary contributions of Trajectory Verification (TV) and Corrective Guidance (CG), and (2) evaluate the importance of Evidence Grounding (EG) and Consequence Prediction (CP) supervision.

1. Disentangle contributions for TV and CG.

  • We compare TV only, TV with a simple fallback, and TV with CG under the same guidance interface. Adding CG improves the At-fault Scene Score over TV-only by 2.58 points for Cosmos and 1.48 points for ActionGuide, demonstrating the complementary benefits of verification and correction. In contrast, the simple fallback (going straight) consistently underperforms TV-only across all three MLLMs, highlighting the importance of context-aware corrective decisions.

  • The logged-future reference, which makes decisions by evaluating candidates using recorded future trajectories of surrounding agents, further reaches 59.11, illustrating the potential benefits of more accurate trajectory assessment.

GuideTV onlyTV + simple fallbackTV + CG
Logged-future Reference57.7356.9759.11
MLLM Guidance
Qwen3-VL-8B44.3227.5544.35
Cosmos-Reason2-8B45.3436.4147.92
ActionGuide (Ours)48.0238.5749.50

2. Training data ablation for EG and CP.

Adding explicit Consequence Prediction supervision improves the score to 48.75 (+3.51), while incorporating all four tasks achieves the best performance of 49.50 (+4.26). Training only on Trajectory Verification and Corrective Guidance yields less generalizable results. These results highlight the importance of grounding driving decisions in scene evidence and anticipated action consequences.

EGCPTVCGScene Score ↑Δ
——✓✓45.24—
✓—✓✓46.59+1.35
—✓✓✓48.75+3.51
✓✓✓✓49.50+4.26

W2 & Q3: Improvement Reliability

ActionGuide consistently achieves positive gains over Cosmos-Reason2-8B across three evaluation runs (+1.58, +1.74, and +1.49 Scene Score).

1. Paired scenario-level analysis.

We compute a 95% paired bootstrap confidence interval using 10,000 resamples of the 131 scenarios, preserving the Cosmos–ActionGuide pairing within each resample. The original run yields an observed gain of +1.58 points, with a 95% CI of [−0.87, +4.14], showing positive point estimate.

2. Repeated-rollout consistency.

Two additional runs with different seeds show positive gains (+1.74 and +1.49 points), confirming a consistent trend across all three runs.

RunCosmosActionGuideGain
Original47.9249.50+1.58
Repeat 147.6849.42+1.74
Repeat 248.3449.83+1.49
AuthorsOpenReview ↗

Response to Reviewer X2aN (Part 2)

W3 & Q4: Direction-balanced Evaluation

We first report direction-balanced accuracy, showing that ActionGuide’s improvement persists. The capability–utility correlation remains positive with the revised balanced metric.

1. Per-direction and direction-balanced evaluation.

  • Direction-balanced results. We report per-direction accuracy and direction-balanced accuracy, the unweighted mean across all six directions. ActionGuide achieves 34.25%, compared with 17.26% for Cosmos-Reason2-8B. For comparison, a policy that always selects the straightest direction without scene understanding would obtain 16.67% under this metric. These results show that ActionGuide’s advantage is not attributable to the direction bias.
  • Direction distribution. The benchmark’s directional proportions were informed by the ground-truth trajectories in our closed-loop evaluation scenarios, where 73.7% of reference steps correspond to straight motion. We’ll further include direction-balanced accuracy as an important metric in revision.
GT directionQwen3-VL-8BCosmos-Reason2-8BActionGuide
Straight13.518.932.6
Light left13.218.934.0
Light right10.97.826.6
Hard left10.510.547.4
Hard right24.318.945.9
Stop28.628.619.0
Overall accuracy14.5718.2832.45
Direction-balanced accuracy16.8517.2634.25

2. Capability–utility correlation with direction balancing.

We recompute the correlation metrics by replacing CG accuracy with its direction-balanced counterpart. Across all eight evaluated MLLMs, the positive association with closed-loop Scene Score persists (Pearson r = 0.866; Spearman ρ = 0.738). Thus, the observed association is not specific to the original direction-weighting scheme. We thank the reviewer for motivating this robustness check and we’ll update the corresponding figure in revision.

Thanks for the additional experiments. The ablations address my main concern about attribution, especially the contribution of consequence-prediction supervision beyond training verification and correction alone. The methodological novelty remains modest, but the additional evidence makes the empirical contribution convincing enough for weak acceptance. I am raising my rating from 2 to 3 and would ask that these results, including the uncertainty estimate, be incorporated into the paper.

Reviewer_fqA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for evaluating whether action-grounded reasoning in MLLMs can improve closed-loop autonomous driving. The benchmark contains four tasks: Evidence Grounding, Consequence Prediction, Trajectory Verification, and Corrective Guidance. These tasks are formulated as multiple-choice questions. In addition to this open-loop evaluation, the authors integrate an MLLM as an online guide to a frozen driving planner, where the MLLM verifies planner proposals and provides corrective guidance when no proposal is considered suitable.

The authors additionally introduce ActionGuide-Train, a CARLA-generated training dataset containing 16,8k samples. Fine-tuning Cosmos-Reason2-8B on this data improves the average action-grounded accuracy from 41.41% to 53.45%. Despite being trained in simulation, the model is evaluated on real-world scenes from nuScenes, Waymo and Argoverse 2, which the authors use to investigate sim-to-real transfer.

For closed-loop validation, the authors evaluate MLLM guidance on selected AlpaSim scenarios using DrivoR as the frozen base planner. ActionGuide improves the at-fault Scene Score from 43.66 to 49.50 and provides modest but noticeable improvements in collision and off-road metrics. The authors further analyze the relationship between action-grounded benchmark accuracy and closed-loop performance.

Critical strengths

For me the strongest contribution is the attempt to go beyond isolated driving VQA and ask whether MLLM reasoning actually improves another planner during execution. Keeping the base planner frozen gives a useful experimental setup for studying the additional utility coming from MLLM guidance.

ActionGuide is trained using simulation data but evaluated on real-world scenes for the action-grounded benchmark. The improvement from 41.41% to 53.45% over the pretrained Cosmos-Reason2-8B is meaningful, and I found the observation that more semantic/behavioral knowledge appears to transfer better than geometry-sensitive reasoning interesting.

Critical weaknesses

Multiple-choice evaluation makes it difficult to know exactly what capability is being measured. All four benchmark tasks ultimately reduce the model output to choosing among predefined alternatives. While the authors make a good effort to remove ambiguity and presentation shortcuts, I am still concerned that part of the measured performance may come from answer elimination, distractor characteristics, or biases introduced by the question construction rather than the intended reasoning capability.

I would find this much more convincing with an ablation using alternative distractors, shuffled/rephrased choices, or an open-ended evaluation on at least a representative subset (with LLM-as-a-judge for example). Without this, it is difficult to separate actual action-grounded reasoning from performance specific to the MCQ formulation.

The benchmark decomposition identifies broad weaknesses such as Evidence Grounding or Consequence Prediction, but I am not convinced that it provides enough introspection to determine why a model fails or what aspect of the model/training should be improved. For a benchmark positioned around understanding action-grounded capability, I think this diagnostic value is quite important.

Also, the practical closed-loop claim would be stronger with runtime measurements. The MLLM operates at 2 Hz in the evaluated system, but the paper does not report whether the complete MLLM inference/guidance pipeline actually satisfies this online requirement.

Questions for authors
  1. All four ActionGuide-Bench tasks are formulated as MCQ. Have you evaluated how sensitive the reported results are to this formulation, e.g., by changing/rephrasing distractors, varying the number of answer choices, or evaluating open-ended responses on a subset? How can we distinguish improvements in the intended action-grounded capability from improvements in exploiting the structure of the answer space? This would materially affect my confidence in the benchmark scores.

  2. A key purpose of a benchmark is, in my opinion, not only to rank models but also to identify concrete model weaknesses that can guide future research. Beyond separating performance into Evidence Grounding, Consequence Prediction, Trajectory Verification and Corrective Guidance, what introspection does ActionGuide-Bench provide into why a model fails? Can the benchmark results be used to identify a concrete deficiency in a model/training setup and verify that targeting this deficiency improves the corresponding capability?

  3. Since the proposed MLLM is used as an online guide, what is the measured end-to-end inference latency of the complete guidance pipeline? The experiments perform MLLM selections at 2 Hz, but can the evaluated models actually satisfy this latency requirement on the hardware used for evaluation, including visual input processing and trajectory verification/guidance? This could change how I interpret the closed-loop contribution.

  4. The related-work discussion seems to miss Drive4C, which appears relevant here. Drive4C similarly combines capability-specific question answering with closed-loop/instruction-based driving evaluation and explicitly aims to identify model limitations. I think discussing the distinction to this setup would help clarify the novelty and diagnostic positioning of ActionGuide-Bench.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fqA9 (Part 1)

We thank the reviewer for the constructive feedback and recognition of our closed-loop evaluation of MLLM guidance, meaningful sim-to-real improvements, and clear presentation. We carefully address each of your concerns below with additional experiments, analysis and clarifications.


W1 & Q1: Robustness to the Answer-space Structure

ActionGuide's improvements persist across alternatives, suggesting that the gains reflect stronger action-grounded reasoning rather than reliance on answer-space shortcuts.

Following the reviewer's suggestions, we conduct robustness evaluations on Evidence Grounding and Consequence Prediction, where answer wording and distractor construction can be varied. All variants follow the original benchmark's answer-validity and ambiguity-control criteria:

  • Rephrasing: Semantically paraphrase all four options, constructing two independent variants. We verify semantic equivalence using an LLM judge and balance option lengths to reduce superficial cues.
  • Distractor replacement: Retain the correct answer while resampling distractors under the original mutual-exclusivity constraints, replacing approximately half of the original distractors.
  • Six choices: Add two distractors to the original four-choice questions.
  • Open-ended: Remove all answer choices and evaluate free-form responses on 150 stratified samples per task using GPT-OSS-120B judge with predefined criteria.

Evidence Grounding (Accuracy, %)

ModelOriginalRephrasing (×2)ReplacementSix ChoicesOpen-ended
Qwen3-VL-8B26.728.1 / 33.629.918.87.3
Cosmos-Reason2-8B44.941.5 / 41.247.427.412.7
ActionGuide (Ours)54.653.8 / 51.454.844.426.0

Consequence Prediction (Accuracy, %)

ModelOriginalRephrasing (×2)ReplacementSix ChoicesOpen-ended
Qwen3-VL-8B42.732.8 / 40.639.726.724.0
Cosmos-Reason2-8B42.14208 / 38.943.035.732.7
ActionGuide (Ours)65.865.9 / 64.863.757.344.7
AuthorsOpenReview ↗

Response to Reviewer fqA9 (Part 2)

W2 & Q2: Diagnostic Value and Targeted Improvement

ActionGuide-Bench provides actionable diagnostic insights by identifying behavior-specific weaknesses, evaluating capability-targeted supervision, and examining potential improvement direction through simulator-grounded reference guidance. We provide additional analyses along these dimensions.

1. Behavior-level Diagnosis

We align offline reasoning and closed-loop evaluation through shared driving skills, enabling a diagnosis–targeting–improvement analysis that connects action-grounded capabilities (Tab.1) with downstream driving performance (Tab.2).

  • Behavioral-skill decomposition. Rather than treating the 131 AlpaSim scenarios as isolated cases, we characterize their recurring behavioral requirements: road geometry, vehicle interactions, traffic-rule compliance, vulnerable road-user interactions, etc. A single scenario may require multiple skills, such as avoiding collisions while remaining within road boundaries.
  • Diagnosis–targeting–improvement. As shown in the table below, behavior-specific evaluation identifies concrete weaknesses that can guide targeted supervision. For example, pretrained Cosmos-Reason2-8B achieves only 25.4% offline accuracy on vehicle interactions (Col. 1). Our simulation-based data engine (Sec. 3) provides 3,548 related training examples (Col. 3). After fine-tuning on the full training mixture, ActionGuide improves offline accuracy to 61.7% (Col. 2), accompanied by a +5.1-point gain in mean closed-loop Scene Score for the corresponding scenarios (Col. 5). This analysis connects concrete behavioral deficiencies with relevant reasoning and evaluates improvement at both levels, providing more actionable guidance for model development.
Driving skillCosmos offline accuracy (%)ActionGuide offline accuracy (%)Related training examplesClosed-loop scenariosMean Scene Score gain
Road geometry57.361.12,62039+3.9
Vehicle interactions25.461.73,54842+5.1
Traffic rules and stopping38.776.82,93041+1.1
Vulnerable road-user interactions55.6100.095015+1.7

2. Capability-guided diagnosis

Complementing the behavior-level diagnosis, our four-task decomposition identifies which reasoning capabilities should be strengthened to improve downstream driving guidance. We conduct training ablations to evaluate the contribution of each supervision, as shown in the table below.

Adding explicit Consequence Prediction supervision improves the score to 48.75 (+3.51), while incorporating all four tasks achieves the best performance of 49.50 (+4.26). Training only on Trajectory Verification and Corrective Guidance yields less generalizable results. These results highlight the importance of grounding driving decisions in scene evidence and anticipated action consequences.

EGCPTVCGScene Score ↑Δ
——✓✓45.24—
✓—✓✓46.59+1.35
—✓✓✓48.75+3.51
✓✓✓✓49.50+4.26

3. Further Analysis: Reference-based Diagnosis

We further identify potential directions for improving MLLM guidance by evaluating two high-performing simulator-grounded reference judges:

  • Current-state reference (Scene Score: 55.24): Evaluates candidate trajectories using simulator-derived agent positions and velocities with constant-velocity extrapolation.
  • Logged-future reference (Scene Score: 59.11): Evaluates candidate safety using the recorded future trajectories of surrounding agents.

These results highlight two promising directions for future improvement: (1) stronger metric spatial grounding, enabling more accurate estimation of surrounding agents' positions, velocities, and spatial relationships from visual observations; and (2) improved action-consequence reasoning, enabling more reliable assessment of interactions between candidate ego trajectories and surrounding agents' anticipated motions.

AuthorsOpenReview ↗

Response to Reviewer fqA9 (Part 3)

W3 & Q3: Online Guidance Latency

We measure inference latency across different model sizes using 50 guidance decisions sampled from 10 scenarios, with vLLM 0.22.1 running each model on a single NVIDIA RTX PRO 6000 GPU (TP=1). The measured latencies are 0.24–0.41 s (with 4B), 0.30–0.61 s (with 8B), and 0.74–2.20 s (with 32B). Using 8B models approaching the 0.5 s budget for 2 Hz operation.

We leverage the base planner's trajectory ranking to verify the highest-priority candidate first, evaluating additional candidates only when necessary and preserving the original decision logic. For achieving practical real-time guidance, distilling action-grounded capabilities into smaller models (under 8B) is a promising direction.

Thanks for the suggestion and we'll include latency as an important metric in the proposed benchmark.


Q4: More Related Work - Drive4C

We thank the reviewer for highlighting Drive4C and will clarify differences with our work in the related work.

  1. Action-grounded capability evaluation. Drive4C evaluates scenario understanding and the ability to translate language instructions into driving actions. Our focus is reasoning about concrete candidate actions and their anticipated consequences to guide the final decision. Rather than requiring the MLLM to generate vehicle controls, we evaluate whether it can identify action-relevant evidence, assess candidate behaviors, and guide verification and corrective choices. This provides a structured, interpretable reasoning-to-action framework designed to leverage multimodal understanding for driving decisions.

  2. Unified diagnosis-to-improvement framework. We further connect behavior- and capability-level diagnosis with simulation-based data construction of scene–action–outcome supervision and fine-tuning. Improvements in action-grounded evaluation and closed-loop performance, together with our training-task ablations, support the training effectiveness of this pipeline for strengthening reasoning with downstream driving utility.

Reviewer_sHEWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether MLLMs can improve closed-loop driving by guiding a frozen planner. The authors introduce ActionGuide-Bench, simulation-generated training data, and a fine-tuned model that verifies candidate trajectories and provides corrective actions. Evaluation on real-world driving questions and 131 AlpaSim scenarios reports improved action-grounded accuracy and closed-loop performance, supporting the utility of MLLM guidance and the transfer of simulation-trained knowledge to real-world offline judgments.

Critical strengths
  1. The paper connects four action-grounded reasoning tasks with closed-loop driving evaluation (Fig. 1; Tab. 1, 2). This makes the benchmark consequential: it assesses whether MLLM judgments improve driving behavior, beyond answering isolated questions correctly.

  2. The frozen-planner setup provides a controlled and practically useful integration framework. Keeping DrivoR fixed while varying the MLLM guide helps establish the guide’s incremental value, and the reported improvements demonstrate benefits without retraining the underlying planner (Tab. 2).

  3. CARLA-generated supervision improves performance on real-world driving questions from 41.41% to 53.45% (Tab. 1). This provides meaningful evidence of cross-domain transfer and supports the value of the proposed training data beyond its source simulator.

Critical weaknesses
  1. The experiments do not separate the contributions of trajectory verification, corrective guidance, and simple fallback behavior (Tab. 2). Without these controls, it remains unclear how much of the closed-loop improvement comes from stronger reasoning, limiting support for the paper’s central interpretation.

  2. The Corrective Guidance evaluation is difficult to interpret: 71.5% of correct directions are straight, and answers are derived from logged future trajectories (Fig. 6). Without direction-aware baselines or balanced metrics, the reported accuracy provides limited evidence of reliable corrective decision-making.

  3. Closed-loop validation uses one base planner and 131 curated scenarios with a fixed seed. The improvement over the pretrained Cosmos guide is 1.58 Scene Score points, without uncertainty estimates (Tab. 2). This leaves the robustness and generality of the additional training benefit insufficiently established.

Questions for authors
  1. In the reported closed-loop runs, how often does the guide filter native trajectories versus invoke corrective fallback? Do existing controls or rollout logs help distinguish the contributions of verification and fallback behavior (Tab. 2)?

  2. Given that 71.5% of Corrective Guidance answers correspond to straight motion, how should the reported accuracy be interpreted? Can existing predictions provide per-direction results, and could you clarify how the logged future determines the unique correct answer (Fig. 6)?

  3. How consistent is the 1.58pp improvement over the pretrained Cosmos guide across scenarios? Are repeated-run results or paired confidence intervals from the existing rollouts available to assess its reliability (Tab. 2)?

6 comments
AuthorsOpenReview ↗

Response to Reviewer sHEW (Part 1)

Thank you very much for the constructive and encouraging feedback. We appreciate your recognition of the capability–utility connection, meaningful cross-domain transfer, and value of our training data. We carefully address each of your concerns below with additional ablations, analyses, and clarifications.


W1 & Q1: Ablating Contributions of Verification and Correction

We disentangle the contributions of Trajectory Verification (TV), Corrective Guidance (CG), and simple fallback through controlled ablations, and quantify their intervention frequencies using rollout statistics.

1. Controlled Ablations.

  • We compare TV only, TV with a simple fallback, and TV with CG under the same guidance interface. Adding CG improves the At-fault Scene Score over TV-only by 2.58 points for Cosmos and 1.48 points for ActionGuide, demonstrating the complementary benefits of verification and correction. In contrast, the simple fallback (going straight) consistently underperforms TV-only across all three MLLMs, highlighting the importance of context-aware corrective decisions.
  • The logged-future reference, which makes decisions by evaluating candidates using recorded future trajectories of surrounding agents, further reaches 59.11, illustrating the potential benefits of more accurate trajectory assessment.
GuideTV onlyTV + simple fallbackTV + CG
Logged-future Reference57.7356.9759.11
MLLM Guidance
Qwen3-VL-8B44.3227.5544.35
Cosmos-Reason2-8B45.3436.4147.92
ActionGuide (Ours)48.0238.5749.50

2. Frequency of TV and CG.

We categorize each decision by its trajectory choice, quantifying how often guidance accept the original top-1 trajectory, selects an alternative native candidate, or executes a corrective candidate. The table reveals the MLLM guidance actively modifies the planner's decisions rather than simply accepting its original proposals.

GuideDecisionsNative Top-1Another CandidateCorrective Candidate
Logged-future Reference5,02569.227.83.0
MLLM Guidance
Qwen3-VL-8B5,19592.26.61.2
Cosmos-Reason2-8B5,33189.48.52.1
ActionGuide (Ours)5,35886.710.82.5

W2 & Q2: Direction-balanced Evaluation and Answer Uniqueness

We first report direction-balanced accuracy, showing that ActionGuide’s improvement persists. We also clarify how mapping and filtering yield a unique correct answer based on the logged future trajectory.

1. Per-direction and Direction-balanced Evaluation.

  • Direction-balanced results. We report per-direction accuracy and direction-balanced accuracy, the unweighted mean across all six directions. ActionGuide achieves 34.25%, compared with 17.26% for Cosmos-Reason2-8B, demonstrating that its advantage persists and is not confined to direction bias.
  • Direction distribution. The benchmark’s directional proportions were informed by the ground-truth trajectories in our closed-loop evaluation scenarios, where 73.7% of reference steps correspond to straight motion. We will further include direction-balanced accuracy alongside overall accuracy in the revision.
GT directionQwen3-VL-8BCosmos-Reason2-8BActionGuide
Straight13.518.932.6
Light left13.218.934.0
Light right10.97.826.6
Hard left10.510.547.4
Hard right24.318.945.9
Stop28.628.619.0
Overall Accuracy14.5718.2832.45
Direction-balanced Accuracy16.8517.2634.25

2. Ensuring a Unique Correct Answer.

Each retained sample has a single reference answer, determined by mapping the validated ego trajectory to one of six directional classes:

  • Stop: Assigned first when ego speed remains at or below 0.5 m/s for at least 1 continuous second within the future window.
  • Directional mapping: For non-stop samples, we assign the nearest reference angle among 0∘,±10∘,±25∘0^\circ,\pm10^\circ,\pm25^\circ based on the endpoint displacement direction in the current ego frame.
  • Ambiguity filtering: We discard samples near the class boundaries at ±5∘\pm5^\circ and ±17.5∘\pm17.5^\circ. The matched template must additionally agree with the logged trajectory within 5∘5^\circ in endpoint direction and 25% in displacement magnitude. Otherwise, the sample is excluded.

These rules yield a unique reference answer corresponding to the direction that best reflects the human driver’s intent as expressed in the logged trajectory.

AuthorsOpenReview ↗

Response to Reviewer sHEW (Part 2)

W3 & Q3: Improvement Reliability

ActionGuide consistently achieves positive gains over Cosmos-Reason2-8B across three evaluation runs (+1.58, +1.74, and +1.49 Scene Score), showing reliable performance gains.

1. Paired scenario-level analysis. We compute a 95% paired bootstrap confidence interval using 10,000 resamples of the 131 scenarios, preserving the Cosmos–ActionGuide pairing within each resample. The original run yields an observed gain of +1.58 points, with a 95% CI of [−0.87, +4.14], showing positive point estimate.

2. Repeated-rollout consistency. Two additional runs with different seeds show positive gains (+1.74 and +1.49 points), confirming a consistent trend across all three runs.

Evaluation runCosmosActionGuideGain
Original47.9249.50+1.58
Repeat 147.6849.42+1.74
Repeat 248.3449.83+1.49
AuthorsOpenReview ↗

Revisions in Response to Reviewer sHEW

Thank you again for your constructive and encouraging feedback. Below, we outline the revisions we made in response to your thoughtful comments.

1. Contribution of Verification and Correction (W1 & Q1)

  • Ablated verification, fallback, and correction (Section 4.4, Table 3).
  • Reported trajectory-choice frequencies (Appendix C.3, Table 11).

2. Direction Balance and Answer Uniqueness (W2 & Q2)

  • Reported per-direction and direction-balanced accuracy (Appendix C.4, Table 12).
  • Used direction-balanced accuracy in the overall score (Table 1).
  • Clarified ground-truth curation and ambiguity filtering (Appendix A.1.3, paragraph 2).

3. Improvement Reliability (W3 & Q3)

  • Reported gains across three runs and a paired confidence interval (Appendix C.5, Table 13).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers & Summary of Revisions

We sincerely thank all reviewers for your thoughtful feedback, which has helped us strengthen the clarity, rigor, and depth of our paper. We are encouraged that reviewers recognize:

  1. Novel evaluation perspective: Explicitly connecting action-grounded reasoning with closed-loop driving utility (sHEW, fqA9).
  2. Well-designed practical evaluation framework: Systematically assessing the value of MLLM guidance with the controlled planner (sHEW, fqA9, X2aN).
  3. Effective training & meaningful findings: Empirically demonstrating that behavior-level supervision generated in simulation effectively improves real-world driving decisions (sHEW, fqA9, X2aN).
  4. Clear presentation: Writing is well organized and easy to follow (sHEW, fqA9, X2aN).

Overall, this work connects action-grounded capability with closed-loop driving utility through a well-designed evaluation framework, diagnosis-guided training, and detailed analyses, linking MLLMs’ rich visual and linguistic prior knowledge to concrete decisions in real-world driving scenarios.


We have carefully responsed to each reviewer’s concerns through detailed clarifications, supporting experiments, and further analyses. The revised paper incorporates the updates summarized below, primarily in Section 4.4 for ablation studies and Appendices C.2–C.7 for additional experiments and analyses:

  1. Clarification and explanation: clarifying our relationship to prior work (fqA9: Q4; X2aN: W1) and explaining reference-answer uniqueness and ambiguity filtering (sHEW: Q2).

    • Revisions: Section 6; Appendix A.1.3, paragraph 2.
  2. Expanded ablations: separating the contributions of verification and correction (sHEW: W1/Q1; X2aN: W2/Q2), evaluating complementary task supervision (fqA9: W2/Q2; X2aN: W2/Q2), and reporting guidance intervention frequencies (sHEW: Q1).

    • Revisions: Section 4.4, Table 3; Table 4; Appendix C.3, Table 11.
  3. Strengthened evaluation robustness and reliability: testing alternative answer spaces (fqA9: W1/Q1), introducing direction-balanced accuracy (sHEW: W2/Q2; X2aN: W3/Q4), and assessing closed-loop gains through repeated runs and paired bootstrap analysis (sHEW: W3/Q3; X2aN: W2/Q3).

    • Revisions: Appendix C.2, Tables 9–10; Table 1, Appendix C.4, Table 12; Appendix C.5, Table 13.
  4. Deepened diagnostic and correlation analyses: identifying improvement directions through driving-skill and reference-based diagnoses (fqA9: W2/Q2), and refining correlation analysis (X2aN: W3/Q1).

    • Revisions: Section 4.4, Table 5, Appendix C.6; Figure 4

We genuinely appreciate the thoughtful engagement throughout the process, and hope the improvements made during the rebuttal period will support your final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.