acceptodds
Under review as a conference paper at ICLR 2027

Model-Agnostic Contrastive Adaptation for Slow-Fast End-to-End Autonomous Driving

Abstract

The dual-system architecture combining fast-reactive end-to-end (E2E) models with slow-reasoning Vision Language Models (VLMs) has emerged as a promising solution for autonomous driving in complex scenarios. However, the rapid iteration of E2E models demands a flexibly decoupled, model-agnostic framework. While injecting latent features from VLMs into E2E models preserves dense semantic guidance and inherently offers a higher theoretical upper bound for performance, naive implementations often fail, forcing researchers to adopt intricate, deeply coupled fusion structures. In this paper, we reveal that this plugin failure is not a structural deficiency but an optimization trap. Due to cross-model shortcut learning and gradient vanishing, pre-trained E2E models instinctively treat injected semantic priors as perturbation noise. To address this, we propose MACA, a novel model-agnostic cognitive plugin. MACA bypasses high-dimensional feature injection by extracting concise semantic intentions via a two-stage Sequential Reasoning-to-Decision Training (SRDT) strategy. To break optimization stagnation, we introduce Relative Advantage Preference Optimization (RAPO), which evaluates the VLM-augmented model against a frozen E2E baseline to generate dynamic comparative signals, compelling the network to deeply internalize VLM cognition. Extensive evaluations demonstrate MACA's generality across diverse state-of-the-art E2E architectures, including DiffusionDrive, SparseDrive, MomAD, and GoalFlow. For instance, integrating MACA into DiffusionDrive reduces the average L2 error from 0.57 m to 0.51 m and the collision rate from 0.08% to 0.05% on nuScenes, while improving the PDMS of DiffusionDrive and GoalFlow from 87.5 and 89.2 to 88.2 and 90.0, respectively, on the NAVSIM benchmark. Our approach ensures driving stability in regular scenarios while significantly boosting decision-making success in corner cases, comprehensively outperforming existing baselines. Code and models will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Trajectory Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eqAGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MACA, a slow-fast framework that integrates VLM semantic intentions into existing end-to-end autonomous-driving planners. Two-stage Sequential Reasoning-to-Decision Training (SRDT) produces compact intentions, while Relative Advantage Preference Optimization (RAPO) compares an augmented planner with a frozen E2E reference during training. Evaluations on nuScenes and NAVSIM report improvements across DiffusionDrive, SparseDrive, MomAD, and GoalFlow; for DiffusionDrive, average nuScenes L2 decreases from 0.57 to 0.51 m.

Critical strengths
  1. A meaningful problem formulation targets ineffective VLM-to-E2E semantic transfer. Figure 1 illustrates weak improvements and very small fusion gradients under naive injection. Such observations provide a specific motivation for optimization-oriented fusion.
  2. A lightweight integration strategy supports multiple existing planners. Feature reuse, compact SRDT intentions, and additive conditioning avoid extensive alterations to native planning modules (Sec. 3). This choice makes the approach practically attractive for adapting heterogeneous E2E systems.
  3. Reported gains cover multiple architectures and two driving benchmarks. DiffusionDrive improves nuScenes average L2 from 0.57 m to 0.51 m and collision rate from 0.08% to 0.05% (Table 1). Additional PDMS gains on NAVSIM for DiffusionDrive and GoalFlow strengthen the empirical evidence (Table 2).
Critical weaknesses
  1. Direct comparisons with closely related slow-fast and VLM–E2E systems are missing. Tables 1–2 emphasize conventional E2E baselines, whereas relevant cited approaches such as AdaDrive, SOLVE, VLMPlanner, and Senna have no matched quantitative comparisons. This omission makes it difficult to assess advantages over existing semantic-fusion frameworks.
  2. RAPO improvements are not clearly separated from generic trajectory-loss reweighting. Table 3 shows that RAPO alone, without semantic fusion, lowers DiffusionDrive average L2 from 0.57 m to 0.53 m, compared with 0.51 m for complete MACA. Simple weighted regression or hard-example baselines are needed to isolate the claimed preference-optimization benefit.
  3. Training and inference use different semantic inputs without sufficient robustness checks. Section 4 describes ground-truth semantics during RAPO training but VLM-predicted intentions at inference. Uncertain or incorrect intentions may alter driving decisions, yet this mismatch receives little direct analysis.
  4. Real-time applicability and reactive driving performance remain inadequately validated. Only CLIP text-encoding latency of roughly 4 ms is provided, rather than complete 3B-VLM inference and slow-fast update costs. Open-loop nuScenes and non-reactive NAVSIM scores cannot fully demonstrate interactive closed-loop safety.
  5. Collision-rate reporting is inconsistent and some steering scenarios regress. MACA–MomAD reports 0.08% average collision rate in Table 1 versus 0.11% in Appendix Table 9, and its left-turn collision rate rises from 0.20% to 0.32%. An explanation of metric aggregation and targeted safety analysis is needed to support broad stability claims.
  6. Statistical robustness is insufficiently demonstrated. Although the authors repeat training three times, all runs use the same default random seed, without reporting variance or confidence intervals (Appendix F). Given the relatively small NAVSIM PDMS improvements of 0.7–0.8 points (Table 2), experiments with different random seeds are necessary to establish the reliability and statistical significance of the reported gains.
Questions for authors
  1. Can the authors provide aligned comparisons with AdaDrive, VLMPlanner, or SOLVE, or explain why these are infeasible?
  2. How does RAPO compare with sample-weighted L2, baseline-normalized regression, and hard-example mining under matched budgets?
  3. What exactly are the ground-truth semantics used during training, and how does performance change with predicted or perturbed VLM intentions?
  4. What are the complete VLM-plus-planner latency and reasoning cadence? Please also reconcile the MACA–MomAD collision rates in Tables 1 and 9.
9 comments
AuthorsOpenReview ↗

Response to Reviewer eqAG (1/6)

We thank the reviewer for the careful reading and for recognizing the problem of ineffective VLM-to-E2E semantic transfer, the practical value of the lightweight integration, and the gains across planners on nuScenes and NAVSIM. We address each point below and will incorporate the clarifications into the revised manuscript.

W1 & Q1: Direct comparisons with closely related slow-fast and VLM–E2E systems are missing.

R1: We added a comparison of the guidance interfaces and cross-architecture validation of related dual-system methods (Table 1). MACA jointly encodes the driving decision and its scene-specific rationale. This guidance is integrated through a lightweight additive interface with RAPO, while continuous trajectory generation remains with the E2E planner. We validate this design on SparseDrive, DiffusionDrive, MomAD, and GoalFlow.

Table 1: Guidance interfaces and cross-architecture validation of dual-system methods.

MethodMeta-action guidanceTrajectory selection/refinementLLM hidden-state fusionExplicit decision-and-rationale guidanceCross-family E2E validation
Senna✓✗✗✗✗
SOLVE✗✓✗✗✗
AdaDrive✗✗✓✗✗
VLMPlanner✗✗✓✗✗
FASIONAD✓✗✗✗✗
NetRoller✗✗✓✗✗
MACA✗✗✗✓✓

The first four columns describe the guidance passed to the planner, rather than the VLM's reasoning capabilities. Explicit decision-and-rationale guidance jointly encodes the generated decision and its scene-specific reason; hidden-state fusion directly uses LLM hidden representations. Cross-family E2E validation refers to reported adaptation across distinct full E2E architecture families, rather than variants within one family or standalone motion planners. A cross in this column indicates that such validation was not reported, not that adaptation is impossible.

On the experimental comparison, we further add the reported nuScenes open-loop results of Senna, FASIONAD, and NetRoller. AdaDrive is evaluated on LangAuto, and VLMPlanner is evaluated on nuPlan. Their published scores therefore cannot be placed next to our nuScenes results. SOLVE is evaluated on nuScenes, but it uses ego state and historical trajectories to retrieve candidate trajectories, and it includes velocity and acceleration in trajectory refinement. Our nuScenes setting omits ego state. BEV-Planner shows that a model on nuScenes can rely mainly on ego state to obtain strong open-loop scores without fully using the surrounding scene. To avoid this shortcut and ensure that planning performance reflects the contribution of scene understanding and semantic guidance, we therefore do not provide ego-state inputs to MACA in our nuScenes experiments.

Table 2: Planning results on the nuScenes validation set. Lower is better.

MethodL2 1sL2 2sL2 3sL2 Avg.Col. 1sCol. 2sCol. 3sCol. Avg.
Senna0.370.540.860.590.090.120.330.18
FASIONAD0.190.621.250.690.020.090.440.18
NetRoller0.350.610.950.640.100.170.310.19
MACA-DiffusionDrive0.250.480.800.510.010.020.120.05
AuthorsOpenReview ↗

Response to Reviewer eqAG (2/6)

W2 & Q2: RAPO improvements are not clearly separated from generic trajectory-loss reweighting.

R2: Under the tested settings, RAPO outperforms the three alternative strategies. To distinguish RAPO from generic trajectory reweighting, we disable semantic fusion and replace the RAPO auxiliary term with sample-weighted L2, Baseline-normalized Regression, or hard-example mining, while retaining the original diffusion regression loss over all samples. All runs use the same training budget, auxiliary loss coefficient, activation epoch, and warm-up schedule.

Sample-weighted L2 uses the current training-step trajectory errors as weights, with gradients detached and weights normalized to a mean of one. Baseline-normalized Regression instead uses precomputed training-set errors from the original planner, with the same normalization. Hard-example mining applies the auxiliary loss only to the 50% of samples with the highest current trajectory errors in each batch.

The three alternatives achieve average L2 values of 0.63 m, 0.67 m, and 0.62 m, respectively, compared with 0.53 m for RAPO-only. These direct implementations do not yield improvements under the tested configuration; their weighting and sample-selection settings warrant further investigation. RAPO uses a dynamic relative advantage that captures whether the current prediction improves or deteriorates relative to a frozen reference model. Thank you for suggesting these comparisons.

Table 3: Comparison of RAPO with generic trajectory optimization strategies on nuScenes. Semantic fusion is off. Lower is better.

Optimization StrategyL2 1sL2 2sL2 3sL2 Avg.Col. 1sCol. 2sCol. 3sCol. Avg.
RAPO-only0.260.500.820.530.030.050.150.08
Sample-weighted L20.300.601.000.630.000.090.340.14
Baseline-normalized Regression0.310.641.050.670.000.160.430.20
Hard-example Mining0.300.590.980.620.000.070.240.10
AuthorsOpenReview ↗

Response to Reviewer eqAG (3/6)

W3 & Q3: Training and inference use different semantic inputs without sufficient robustness checks.

R3: Training uses the reference text, inference uses the VLM prediction for the current scene, and we added an intervention with incorrect semantics. The ground-truth semantics used in training are the reference texts from OmniDrive. The first SRDT stage uses multi-turn dialogue covering the scene description, perception questions, the key factors, and the final driving decision. The second stage rewrites the same samples as a single-turn question and merges the decision with its key factors into one compact sentence. RAPO training takes this reference text as the semantic input. At inference, the input is the VLM prediction for the current scene. With VLM-predicted semantics for the current scene, MACA achieves 0.51 m and a collision rate of 0.05%.

We keep the weights fixed and replace only the semantic input at inference. As the replacement ratio grows, planning performance declines. Specifically, we replace the current scene's VLM-generated textual answers with those from other scenes. At full replacement, average L2 rises from 0.51 m to 0.58 m and the collision rate from 0.05% to 0.10%. Scale-matched Gaussian noise degrades the result to 0.94 m and 0.29%. The intermediate ratios are reported in the table. Naive injection in Table 12 stays nearly unchanged when its features are swapped for random noise. The predicted semantics can be used for planning, and the quality of that semantic input affects the planning result.

Table 4: Semantic replacement at inference on MACA. The original input is the VLM prediction for the current scene. Lower is better.

Semantic InputReplacement (%)Avg. L2 (m)↓Col. Rate (%)↓
Original00.510.05
Cross-scene100.520.07
Cross-scene250.530.08
Cross-scene500.550.09
Cross-scene1000.580.10
Gaussian noise1000.940.29
Reviewer_oiLpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MACA is a plug-in framework for incorporating VLM-generated semantic guidance into existing end-to-end driving planners. It reuses the planner’s visual features and uses two-stage reasoning-to-decision training to produce compact driving intentions, which it encodes and injects through a lightweight additive interface. To improve semantic utilization, the method introduces RAPO, which compares trajectory errors against a frozen baseline and adds an advantage-conditioned semantic alignment loss. Experiments on nuScenes and NAVSIM report improvements across SparseDrive, MomAD, DiffusionDrive, and GoalFlow, supporting the framework’s applicability to multiple planning architectures.

Critical strengths
  1. Adaptability. MACA can be integrated into existing E2E planners through a simple semantic interface without redesigning their core architectures. The improvements across SparseDrive, MomAD, DiffusionDrive, and GoalFlow support its potential as a reusable plug-in, which is the main practical strength of this work.
Critical weaknesses
  1. The proposed remedy is not clearly connected to the diagnosed semantic optimization trap. Table 7 reports identical planning metrics for naive fusion with genuine VLM features and random noise, but an analogous intervention is not reported for MACA to establish that this failure has been resolved. Meanwhile, Table 3 shows substantial improvement from RAPO even without semantic fusion. The gradient comparison in Fig. 1 does not settle this issue because both the representation and training losses change. It therefore remains unclear whether MACA restores the use of semantic information or primarily improves trajectory regression. This missing link weakens the paper’s central explanation.

  2. The gains on relatively weak baselines leave the practical significance uncertain. On NAVSIM, MACA improves DiffusionDrive from 87.5 to 88.2 PDMS and GoalFlow from 89.2 to 90.0. These results remain substantially below stronger public planners such as DrivoR [R1] and ChainFlow-VLA [R2], which report 93.1 and 93.6 PDMS using navtrain alone, respectively. For a general plug-in, it is important to establish whether its benefits remain complementary to stronger representations and planning methods. The gains of 0.7-0.8 points on the evaluated baselines may diminish as baseline performance improves. Without such evidence, the current results provide limited support for MACA’s value for contemporary high-performing planners.

  • [R1] Ellington Kirby et al. Driving on Registers. CVPR 2026. arXiv:2601.05083. To my knowledge, this is a very strong open-source baseline available during the preparation of the ICLR 2027 submission.
  • [R2] Xiyang Wang et al. ChainFlow-VLA: Causal Flow Planning with Vision-Language Models. NeurIPS 2026. arXiv:2605.23270.
  • [R3] NavSim v1 Leaderboard. URL.
Questions for authors

Is the proposed method suitable for a broader range of methods? including:

  • Autoregressive planners or VLA/WAM-based systems. Evidence from one representative additional paradigm would strengthen the claimed generality; evaluating all of them is not necessary.
  • Higher baseline methods ~93 PDMS. To defend weakness 2, can MACA deliver further gains over a strong public baseline such as DrivoR, with matched training data, evaluation settings, and a comparable fine-tuning budget? A reproducible improvement in this setting would provide stronger evidence of the plug-in’s practical value and would support acceptance.

BTW, below is my suggestion rather than a question for future revision:

  • Consider reducing the emphasis on “slow-fast,” particularly in the title and abstract. The main contribution is semantic adaptation across planners, while the paper does not investigate inference scheduling or the accuracy-latency trade-off between the two systems. Framing the paper around model-agnostic semantic integration would better reflect its contribution and supporting evidence.
5 comments
AuthorsOpenReview ↗

Response to Reviewer oiLp (1/3)

We thank the reviewer for the careful reading, and for noting that MACA can be integrated into existing E2E planners through a simple semantic interface without redesigning their core architectures, and that the gains on SparseDrive, MomAD, DiffusionDrive, and GoalFlow support its use as a reusable plug-in. We answer each point below. The clarifications will be added to the revised manuscript.

W1: The proposed remedy is not clearly connected to the diagnosed semantic optimization trap.

R1: MACA improves trajectory regression and also makes the planner use semantics that match the current scene. We agree that the gradient comparison in Fig. 1 in the manuscript changes both the representation and the training losses, so it cannot by itself separate the two contributions. We therefore add a semantic replacement and a step-by-step comparison of the representation and the loss.

First, we replace the semantics. The weights stay fixed. At inference, replacing the semantics of the current scene with semantics from other scenes lowers planning performance. At full replacement, L2 rises from 0.51 m to 0.58 m. Gaussian noise raises it to 0.94 m. In Table 12, naive fusion with genuine VLM features and with random noise scores almost the same. The planning result of MACA depends on semantics that match the current scene, so the gain is not only trajectory regression.

Second, we separate the representation from the loss. Under the same standard regression loss, changing naive feature injection to semantic fusion lowers L2 from 0.56 m to 0.54 m. Adding RAPO then lowers it to 0.51 m, so both contribute. Adding the advantage-conditioned semantic alignment loss keeps the average L2 at 0.51 m at the reported precision and lowers the collision rate from 0.06% to 0.05%. After ranking frames by the prediction error of RAPO-only, full MACA still lowers L2 in every high-error bin. In the highest 5%, L2 falls from 1.85 m to 1.57 m. Semantic fusion still adds a gain on difficult samples after the regression improvement from RAPO.

Table 1: Semantic replacement at inference on MACA. The original input is the VLM prediction for the current scene. Lower is better.

Semantic InputReplacement (%)Avg. L2 (m)↓Col. Rate (%)↓
Original00.510.05
Cross-scene100.520.07
Cross-scene250.530.08
Cross-scene500.550.09
Cross-scene1000.580.10
Gaussian noise1000.940.29

Table 2: Injection choices under the same standard regression loss, and the result after adding RAPO. The first three rows do not use RAPO. Lower is better.

MethodFusionL_simL_RAPOAvg. L2 (m)↓Col. Rate (%)↓
Last-token Injection✓----0.580.12
Naive Feature Injection (F_mid)✓----0.560.06
Semantic Fusion✓----0.540.07
Semantic Fusion + RAPO✓--✓0.510.06
Full MACA (Ours)✓✓✓0.510.05

Table 3: High-error bins ranked by RAPO-only per-frame L2 (m). Both methods use the same 5% bin. Lower is better.

Percentile65-7070-7575-8080-8585-9090-9595-100
RAPO-only0.610.680.770.881.021.251.85
MACA0.550.600.700.730.931.141.57
Reduction0.060.080.070.150.090.110.28
AuthorsOpenReview ↗

Response to Reviewer oiLp (2/3)

Q1: Autoregressive planners and VLA/WAM-based systems.

R2: This paper studies adding VLM semantic guidance to an existing E2E planner. Current end-to-end driving methods fall into three groups. They are conventional E2E planners, VLM or VLA systems, and dual systems that pair a VLM with an E2E planner. MACA is in the third group. It adds VLM semantic guidance to an existing E2E planner through a lightweight interface. A VLA model already uses a vision-language model to produce actions, so another independent VLM branch sits outside the dual-system setting studied here. A WAM aims to plan from world dynamics without a vision-language model, and adding a VLM branch would depart from that goal. We therefore do not treat a WAM as a direct target for MACA. An autoregressive planner is closer to the setting we target. The discussion of VLA and WAM does not establish generality for that paradigm.

Following your suggestion, we added an adaptation experiment on the autoregressive planner VeteranAD. On stage 1 of NAVSAFE, a challenging evaluation set focused on safety-critical driving scenarios, MACA improves its PDMS from 67.6 to 72.5. Given the time and computational constraints of the rebuttal period, we use a matched training budget for the VeteranAD baseline and its MACA counterpart, keeping the training data and number of training steps identical and following the base planner's training hyperparameters in the official code. The 4.9-point improvement, measured using NAVSIM v1 PDMS, provides additional evidence of MACA's applicability to autoregressive planners. It is also consistent with our motivation for introducing VLM semantic guidance to improve planning in challenging driving scenarios. Thank you for suggesting this additional evaluation.

Table 4: Adaptation to the autoregressive planner VeteranAD on stage 1 of NAVSAFE, evaluated using NAVSIM v1 PDMS. Higher is better.

MethodNC↑DAC↑EP↑TTC↑Comf.↑PDMS↑
VeteranAD95.778.161.792.192.767.6
MACA-VeteranAD97.680.567.393.997.072.5
AuthorsOpenReview ↗

Response to Reviewer oiLp (3/3)

W2 & Q2: The gains on relatively weak baselines leave the practical significance uncertain.

W2 & Q2: Further gains on a strong planner at about 93 PDMS.

R3: Gains on standard benchmarks diminish as the baseline grows stronger. On stage 1 of NAVSAFE, scored with NAVSIM v1 PDMS, MACA raises the overall PDMS of DrivoR from 76.07 to 80.20 (+4.13). Following your recommendation, we evaluated MACA on the stronger open-source planner DrivoR. We trained MACA-DrivoR from scratch on navtrain following all official open-source training configurations and compared it against the official DrivoR checkpoint. Your intuition that gains on standard benchmarks diminish as the baseline grows stronger is well taken. On the full navtest split, which consists mostly of routine driving scenarios, DrivoR already reaches around 93.1 PDMS and leaves very little headroom for further gains under a matched training budget. To examine whether semantic guidance still helps a planner at about 93 PDMS when routine scenes are no longer the bottleneck, we evaluated both models on stage 1 of NAVSAFE, which specifically isolates safety-critical driving. On this set, MACA improves DrivoR's PDMS from 76.07 to 80.20, a gain of 4.13 points. This is consistent with our motivation for introducing VLM guidance to help E2E planners handle challenging corner cases. This higher PDMS on the safety-critical set provides evidence that MACA remains complementary to strong planners around 93 PDMS.

Table 5: Performance comparison between DrivoR and MACA-DrivoR on stage 1 of NAVSAFE using NAVSIM v1 PDMS. Higher is better.

MethodNC↑DAC↑EP↑TTC↑Comf.↑PDMS↑
DrivoR97.682.969.795.1100.076.1
MACA-DrivoR95.788.474.893.3100.080.2

Suggestion: Reducing the emphasis on slow-fast.

R4: Thank you for this helpful suggestion. We agree that model-agnostic semantic integration better reflects the main contribution of MACA. Our additional results on safety-critical scenarios further support this focus. On stage 1 of NAVSAFE, MACA improves DrivoR's PDMS from 76.07 to 80.20, consistent with our motivation to help E2E planners handle challenging corner cases through semantic guidance. Following your suggestion, we will reduce the emphasis on "slow-fast" in the title and abstract and place greater emphasis on semantic adaptation across planners.

Reviewer_soqQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to use VLM guidance to improve end-to-end autonomous driving. It proposes MACA, which injects compact semantic intentions into an existing planner and uses Relative Advantage Preference Optimization (RAPO) to improve training. The method is evaluated across several E2E architectures.

Critical strengths

The idea is interesting and addresses an important problem in autonomous driving. VLM-based methods do not always outperform E2E planners, and directly injecting VLM features can bring little improvement or even hurt performance. I have also observed this phenomenon. The proposed method is easy to follow, the loss design is straightforward, and the experiments across different planners support its applicability.

Critical weaknesses
  1. It is not fully clear whether the observed problem is shortcut learning or insufficient adaptation during training. The authors argue that E2E planners suppress the injected VLM information, but I wonder whether simply training the combined system longer, or adjusting the learning rates, could resolve this issue.
  2. The additional benefit of VLM guidance seems modest. In Table 3, RAPO alone reduces the average L2 error to 0.53 m, while adding semantic fusion reaches 0.51 m. This suggests that a substantial part of the improvement comes from the optimization objective itself. It remains unclear whether the additional VLM cost is justified by the gain.
  3. Sharing the image backbone reduces computation, but it may also limit the VLM’s visual understanding. A driving backbone such as ResNet is not necessarily the best encoder for language-based reasoning. The trade-off between efficiency and reasoning quality is not sufficiently explored.
Questions for authors
  1. With a longer training schedule or separately tuned learning rates for the planner and fusion module, does naive VLM injection still show the same suppression effect?
  2. Is the improvement from 0.53 m to 0.51 m consistent across different random seeds?
  3. Have you compared the shared backbone with a dedicated pretrained VLM image encoder? How does this choice affect planning performance and inference latency?
5 comments
AuthorsOpenReview ↗

Response to Reviewer soqQ (1/3)

We thank the reviewer for the careful reading and for recognizing the importance of the problem, the clarity of our method and loss design, and the applicability of MACA across different planners. We also appreciate the reviewer’s observation that directly injecting VLM features can bring little improvement or even hurt performance. We address each point below and will incorporate the clarifications into the revised manuscript.

W1 & Q1: It is not fully clear whether the observed problem is shortcut learning or insufficient adaptation during training.

R1: Within the tested range, longer training and separately tuned learning rates do not improve planning performance with naive VLM injection. We extended naive VLM injection from 15 epochs to 20, 25, and 30 epochs. We also lowered the learning rate of the E2E planner, raised the learning rate of the fusion module, and changed the two rates together. None of these settings improves on the original 0.56 m and 0.06%. Together with the noise-replacement results in Table 12, these findings support the interpretation that the model does not fully use the injected semantics.

Table 1: Naive VLM injection with a longer training schedule and separately tuned learning rates. Lower is better.

SettingE2E LRFusion LREpochsAvg. L2 (m)↓Col. Rate (%)↓
Naive injection3e-43e-4150.560.06
Longer training3e-43e-4200.560.07
Longer training3e-43e-4250.580.11
Longer training3e-43e-4300.580.07
Separate LR 11e-43e-4150.580.10
Separate LR 23e-53e-4150.570.07
Separate LR 33e-49e-4150.580.08
Separate LR 41e-49e-4150.570.09
AuthorsOpenReview ↗

Response to Reviewer soqQ (2/3)

W2: The additional benefit of VLM guidance seems modest. It remains unclear whether the additional VLM cost is justified by the gain.

R2: The full-set average includes many routine scenes that the E2E planner already handles well, leaving limited room for additional semantic guidance. Averaging over these scenes can dilute the larger benefits of VLM guidance on difficult samples. Our high-error analysis shows that semantic fusion provides larger gains beyond RAPO on these samples, while the semantic replacement experiment further confirms that MACA uses semantic information that matches the current scene. To check whether the gain comes only from the optimization objective, we add two checks. After ranking frames by the per-frame L2 of RAPO-only, MACA achieves lower L2 in every high-error bin. In the highest 5%, L2 falls from 1.85 m to 1.57 m. We then keep the model weights fixed and replace only the semantic input at inference. Full replacement with semantics from other scenes raises L2 from 0.51 m to 0.58 m, and Gaussian noise raises it to 0.94 m. In Table 12, naive fusion stays nearly unchanged when genuine VLM features are swapped for random noise. These results show that the gain of MACA does not come only from the optimization objective. VLM guidance still improves planning on difficult samples after RAPO. We therefore believe that the value of VLM guidance should also be assessed through its additional benefits on difficult samples, which are not fully reflected in the 0.02 m improvement averaged over the full evaluation set.

Table 2: High-error bins ranked by RAPO-only per-frame L2 (m). Both methods use the same 5% bin. Lower is better.

Percentile65-7070-7575-8080-8585-9090-9595-100
RAPO-only0.610.680.770.881.021.251.85
MACA0.550.600.700.730.931.141.57
Reduction0.060.080.070.150.090.110.28

Table 3: Semantic replacement at inference on MACA. The original input is the VLM prediction for the current scene. Lower is better.

Semantic InputReplacement (%)Avg. L2 (m)↓Col. Rate (%)↓
Original00.510.05
Cross-scene100.520.07
Cross-scene250.530.08
Cross-scene500.550.09
Cross-scene1000.580.10
Gaussian noise1000.940.29

Q2: Is the improvement from 0.53 m to 0.51 m consistent across different random seeds?

R3: The improvement from 0.53 m to 0.51 m is consistent across all three evaluated seeds at the reported precision. We compare RAPO-only and MACA on three distinct seeds, using the same seed in each paired comparison. All per-seed results are rounded to two decimal places, so small differences across seeds may not be visible. At the reported precision, average L2 decreases from 0.53 m to 0.51 m in each comparison, and the collision rate decreases from 0.08%–0.09% to 0.05%. The three paired runs show that the improvement is consistent.

Table 4: RAPO-only and MACA on three paired nuScenes seeds. Both methods use the same seed within each row. The mean and sample standard deviation are computed from the rounded values shown in the table. Lower is better.

SeedAvg. L2 (m) RAPO-onlyAvg. L2 (m) MACACol. Rate (%) RAPO-onlyCol. Rate (%) MACA
10.530.510.080.05
20.530.510.090.05
30.530.510.080.05
Mean ± SD0.530 ± 0.0000.510 ± 0.0000.083 ± 0.0060.050 ± 0.000
AuthorsOpenReview ↗

Response to Reviewer soqQ (3/3)

W3 & Q3: Sharing the image backbone reduces computation, but it may also limit the VLM's visual understanding.

R4: The dedicated SigLIP encoder does not improve overall planning performance and incurs higher computational cost, so we keep the shared backbone. Following your suggestion, we replaced the shared visual features used by the VLM with features from a dedicated SigLIP SO400M/14@384 encoder, while keeping the VLM training schedule unchanged. Average L2 stays at 0.51 m, and the collision rate changes from 0.05% to 0.06%. The visual token count rises from 1,056 to 4,374, VLM-branch latency at inference rises from 746.6 ms to 1006.9 ms. To check whether the full-set average masks differences on difficult samples, we rank frames by the prediction error of the shared ResNet-50 and compare the two encoders. SigLIP has slightly lower L2 in the four highest-error bins, so the dedicated encoder still helps on some difficult samples. These gains do not translate into better overall planning performance. Given this balance of planning performance and inference cost, we keep the shared backbone. We agree with the reviewer that exploring a shared representation space better suited to both visual-language reasoning and E2E planning is a worthwhile direction for future work.

Table 5: Shared ResNet-50 and dedicated SigLIP on nuScenes. Lower is better.

BackboneAvg. L2 (m)↓Col. Rate (%)↓
ResNet-500.510.05
ViT (SigLIP)0.510.06

Table 6: High-error frames by ResNet-50 L2. Each column compares the two encoders on the same 5% bin. Lower is better.

Percentile65-7070-7575-8080-8585-9090-9595-100
ResNet-500.590.650.740.851.001.221.80
ViT (SigLIP)0.580.670.750.840.981.181.77
Reduction0.01-0.02-0.010.010.020.040.03

Table 7: Visual tokens, latency, and GPU memory. The 25.2 ms of the shared ResNet-50 is already included in the E2E forward-pass latency and is not counted again in the VLM branch. Lower is better.

Visual EncodingVisual TokensEncoder (ms)LLM (ms)VLM Branch (ms)Peak Memory (MB)Process Memory (MB)
Shared ResNet-501,05625.2 (included in E2E)746.6746.68,63810,598
Independent SigLIP ViT4,37486.7920.21006.912,52915,054

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Reviewer Discussion

Dear Area Chair,

Thank you for overseeing the review of our submission. As the discussion draws to a close, we would like to briefly summarize the reviewers' feedback and our responses.

Reviewer soqQ confirmed that our response and additional experiments satisfactorily addressed their concerns and maintained their weak-accept rating.

Reviewer eqAG acknowledged that our RAPO comparisons, closed-loop results, and corrections addressed several concerns. We addressed their two remaining questions through matched fusion comparisons and a direct comparison of reference and predicted semantics at inference.

Reviewer oiLp asked whether MACA uses scene-specific guidance and generalizes to other planners. Our experiments showed that planning accuracy declined when guidance came from a different scene. As requested, we also evaluated DrivoR, the high-performing baseline they recommended, and VeteranAD, an autoregressive end-to-end planner. MACA improved both on stage 1 of NAVSAFE.

So far, we have received no further feedback from eqAG on our follow-up or from oiLp on our response. We would be grateful if you could take the additional experiments, our responses, and the feedback received so far into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.