- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MACA, a slow-fast framework that integrates VLM semantic intentions into existing end-to-end autonomous-driving planners. Two-stage Sequential Reasoning-to-Decision Training (SRDT) produces compact intentions, while Relative Advantage Preference Optimization (RAPO) compares an augmented planner with a frozen E2E reference during training. Evaluations on nuScenes and NAVSIM report improvements across DiffusionDrive, SparseDrive, MomAD, and GoalFlow; for DiffusionDrive, average nuScenes L2 decreases from 0.57 to 0.51 m.
Critical strengths
- A meaningful problem formulation targets ineffective VLM-to-E2E semantic transfer. Figure 1 illustrates weak improvements and very small fusion gradients under naive injection. Such observations provide a specific motivation for optimization-oriented fusion.
- A lightweight integration strategy supports multiple existing planners. Feature reuse, compact SRDT intentions, and additive conditioning avoid extensive alterations to native planning modules (Sec. 3). This choice makes the approach practically attractive for adapting heterogeneous E2E systems.
- Reported gains cover multiple architectures and two driving benchmarks. DiffusionDrive improves nuScenes average L2 from 0.57 m to 0.51 m and collision rate from 0.08% to 0.05% (Table 1). Additional PDMS gains on NAVSIM for DiffusionDrive and GoalFlow strengthen the empirical evidence (Table 2).
Critical weaknesses
- Direct comparisons with closely related slow-fast and VLM–E2E systems are missing. Tables 1–2 emphasize conventional E2E baselines, whereas relevant cited approaches such as AdaDrive, SOLVE, VLMPlanner, and Senna have no matched quantitative comparisons. This omission makes it difficult to assess advantages over existing semantic-fusion frameworks.
- RAPO improvements are not clearly separated from generic trajectory-loss reweighting. Table 3 shows that RAPO alone, without semantic fusion, lowers DiffusionDrive average L2 from 0.57 m to 0.53 m, compared with 0.51 m for complete MACA. Simple weighted regression or hard-example baselines are needed to isolate the claimed preference-optimization benefit.
- Training and inference use different semantic inputs without sufficient robustness checks. Section 4 describes ground-truth semantics during RAPO training but VLM-predicted intentions at inference. Uncertain or incorrect intentions may alter driving decisions, yet this mismatch receives little direct analysis.
- Real-time applicability and reactive driving performance remain inadequately validated. Only CLIP text-encoding latency of roughly 4 ms is provided, rather than complete 3B-VLM inference and slow-fast update costs. Open-loop nuScenes and non-reactive NAVSIM scores cannot fully demonstrate interactive closed-loop safety.
- Collision-rate reporting is inconsistent and some steering scenarios regress. MACA–MomAD reports 0.08% average collision rate in Table 1 versus 0.11% in Appendix Table 9, and its left-turn collision rate rises from 0.20% to 0.32%. An explanation of metric aggregation and targeted safety analysis is needed to support broad stability claims.
- Statistical robustness is insufficiently demonstrated. Although the authors repeat training three times, all runs use the same default random seed, without reporting variance or confidence intervals (Appendix F). Given the relatively small NAVSIM PDMS improvements of 0.7–0.8 points (Table 2), experiments with different random seeds are necessary to establish the reliability and statistical significance of the reported gains.
Questions for authors
- Can the authors provide aligned comparisons with AdaDrive, VLMPlanner, or SOLVE, or explain why these are infeasible?
- How does RAPO compare with sample-weighted L2, baseline-normalized regression, and hard-example mining under matched budgets?
- What exactly are the ground-truth semantics used during training, and how does performance change with predicted or perturbed VLM intentions?
- What are the complete VLM-plus-planner latency and reasoning cadence? Please also reconcile the MACA–MomAD collision rates in Tables 1 and 9.
9 comments
Response to Reviewer eqAG (2/6)
W2 & Q2: RAPO improvements are not clearly separated from generic trajectory-loss reweighting.
R2: Under the tested settings, RAPO outperforms the three alternative strategies. To distinguish RAPO from generic trajectory reweighting, we disable semantic fusion and replace the RAPO auxiliary term with sample-weighted L2, Baseline-normalized Regression, or hard-example mining, while retaining the original diffusion regression loss over all samples. All runs use the same training budget, auxiliary loss coefficient, activation epoch, and warm-up schedule.
Sample-weighted L2 uses the current training-step trajectory errors as weights, with gradients detached and weights normalized to a mean of one. Baseline-normalized Regression instead uses precomputed training-set errors from the original planner, with the same normalization. Hard-example mining applies the auxiliary loss only to the 50% of samples with the highest current trajectory errors in each batch.
The three alternatives achieve average L2 values of 0.63 m, 0.67 m, and 0.62 m, respectively, compared with 0.53 m for RAPO-only. These direct implementations do not yield improvements under the tested configuration; their weighting and sample-selection settings warrant further investigation. RAPO uses a dynamic relative advantage that captures whether the current prediction improves or deteriorates relative to a frozen reference model. Thank you for suggesting these comparisons.
Table 3: Comparison of RAPO with generic trajectory optimization strategies on nuScenes. Semantic fusion is off. Lower is better.
| Optimization Strategy | L2 1s | L2 2s | L2 3s | L2 Avg. | Col. 1s | Col. 2s | Col. 3s | Col. Avg. |
|---|---|---|---|---|---|---|---|---|
| RAPO-only | 0.26 | 0.50 | 0.82 | 0.53 | 0.03 | 0.05 | 0.15 | 0.08 |
| Sample-weighted L2 | 0.30 | 0.60 | 1.00 | 0.63 | 0.00 | 0.09 | 0.34 | 0.14 |
| Baseline-normalized Regression | 0.31 | 0.64 | 1.05 | 0.67 | 0.00 | 0.16 | 0.43 | 0.20 |
| Hard-example Mining | 0.30 | 0.59 | 0.98 | 0.62 | 0.00 | 0.07 | 0.24 | 0.10 |
Response to Reviewer eqAG (3/6)
W3 & Q3: Training and inference use different semantic inputs without sufficient robustness checks.
R3: Training uses the reference text, inference uses the VLM prediction for the current scene, and we added an intervention with incorrect semantics. The ground-truth semantics used in training are the reference texts from OmniDrive. The first SRDT stage uses multi-turn dialogue covering the scene description, perception questions, the key factors, and the final driving decision. The second stage rewrites the same samples as a single-turn question and merges the decision with its key factors into one compact sentence. RAPO training takes this reference text as the semantic input. At inference, the input is the VLM prediction for the current scene. With VLM-predicted semantics for the current scene, MACA achieves 0.51 m and a collision rate of 0.05%.
We keep the weights fixed and replace only the semantic input at inference. As the replacement ratio grows, planning performance declines. Specifically, we replace the current scene's VLM-generated textual answers with those from other scenes. At full replacement, average L2 rises from 0.51 m to 0.58 m and the collision rate from 0.05% to 0.10%. Scale-matched Gaussian noise degrades the result to 0.94 m and 0.29%. The intermediate ratios are reported in the table. Naive injection in Table 12 stays nearly unchanged when its features are swapped for random noise. The predicted semantics can be used for planning, and the quality of that semantic input affects the planning result.
Table 4: Semantic replacement at inference on MACA. The original input is the VLM prediction for the current scene. Lower is better.
| Semantic Input | Replacement (%) | Avg. L2 (m)↓ | Col. Rate (%)↓ |
|---|---|---|---|
| Original | 0 | 0.51 | 0.05 |
| Cross-scene | 10 | 0.52 | 0.07 |
| Cross-scene | 25 | 0.53 | 0.08 |
| Cross-scene | 50 | 0.55 | 0.09 |
| Cross-scene | 100 | 0.58 | 0.10 |
| Gaussian noise | 100 | 0.94 | 0.29 |
Response to Reviewer eqAG (1/6)
We thank the reviewer for the careful reading and for recognizing the problem of ineffective VLM-to-E2E semantic transfer, the practical value of the lightweight integration, and the gains across planners on nuScenes and NAVSIM. We address each point below and will incorporate the clarifications into the revised manuscript.
W1 & Q1: Direct comparisons with closely related slow-fast and VLM–E2E systems are missing.
R1: We added a comparison of the guidance interfaces and cross-architecture validation of related dual-system methods (Table 1). MACA jointly encodes the driving decision and its scene-specific rationale. This guidance is integrated through a lightweight additive interface with RAPO, while continuous trajectory generation remains with the E2E planner. We validate this design on SparseDrive, DiffusionDrive, MomAD, and GoalFlow.
Table 1: Guidance interfaces and cross-architecture validation of dual-system methods.
The first four columns describe the guidance passed to the planner, rather than the VLM's reasoning capabilities. Explicit decision-and-rationale guidance jointly encodes the generated decision and its scene-specific reason; hidden-state fusion directly uses LLM hidden representations. Cross-family E2E validation refers to reported adaptation across distinct full E2E architecture families, rather than variants within one family or standalone motion planners. A cross in this column indicates that such validation was not reported, not that adaptation is impossible.
On the experimental comparison, we further add the reported nuScenes open-loop results of Senna, FASIONAD, and NetRoller. AdaDrive is evaluated on LangAuto, and VLMPlanner is evaluated on nuPlan. Their published scores therefore cannot be placed next to our nuScenes results. SOLVE is evaluated on nuScenes, but it uses ego state and historical trajectories to retrieve candidate trajectories, and it includes velocity and acceleration in trajectory refinement. Our nuScenes setting omits ego state. BEV-Planner shows that a model on nuScenes can rely mainly on ego state to obtain strong open-loop scores without fully using the surrounding scene. To avoid this shortcut and ensure that planning performance reflects the contribution of scene understanding and semantic guidance, we therefore do not provide ego-state inputs to MACA in our nuScenes experiments.
Table 2: Planning results on the nuScenes validation set. Lower is better.