acceptodds
Under review as a conference paper at ICLR 2027

Finding and Closing Coverage Holes in Vision-Language Motion Planners

Abstract

Vision-language motion planners have advanced rapidly, yet single-trajectory accuracy can conceal missing feasible alternatives in imitation-trained planners. We define a coverage hole as a state–direction pair with verified trajectory witnesses, but no feasible sample from a reference planner at a fixed budget. Our verifier checks drivable-area compliance and collision-freeness against recorded agents, not traffic-rule compliance. We diagnose candidate holes using retimed cross-scene trajectory prototypes. The same witnesses supply augmentation targets, while a separate human-supervised detector selects the evaluation set. Neither training nor detection requires HD-map or maneuver annotations. Witness-target augmentation is combined with rehearsal and prefix-scheduled decoding (PSD), which allocates inference samples across direction prefixes and free generation using a scene-independent schedule. On nuScenes, we evaluate a 10B vision-language planner on 254 reference-defined hole pairs at 200 training-split states (about half of the pairs received a training target) and on validation and held-out scenes. Augmentation raises closure@128 from 46.5% to 85.0%, while minADE@8 decreases from 2.39 to 1.72 m. Adding PSD reaches 90.9% closure [86.0, 95.1] at 1.91 m minADE and achieves 70% closure with about 29 samples. The off-the-shelf planner remains below 70% at 128. Validation closure under the official HD map reaches 96.0% versus 38.1%, with gains persisting under a lane-direction check. GT-only fine-tuning is more accurate (1.26 m) but recovers fewer reference-defined holes, and a matched-budget control with random witness targets closes most holes as well, so hole targeting mainly buys sampling efficiency at inference. These results show improved finite-budget candidate coverage under an open-loop verifier, with an accuracy trade-off relative to GT-only training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Vision-Language Models, Motion Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SbPJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies finite-budget directional coverage in vision-language motion planners. Verified cross-scene trajectory witnesses identify reference-relative omissions and provide augmentation targets, while prefix-scheduled decoding allocates samples across directions. Experiments on nuScenes demonstrate substantial open-loop coverage improvements, alongside accuracy and traffic-rule-compliance trade-offs.

Critical strengths

The witness-based formulation makes state-specific, finite-budget omissions measurable, providing a useful diagnostic beyond logged-trajectory accuracy. Coverage improvements are supported by official-map, lane-direction, and reference-independent evaluations. The non-targeted augmentation control and compliance audits help clarify the sources and limitations of these gains.

Critical weaknesses
  1. The incremental contribution over existing coverage methods is insufficiently established. There are no direct comparisons with CLOVER-style coverage supervision or dense/factorized trajectory vocabularies. Matched non-targeted augmentation already explains much of the improvement. Targeting provides meaningful low-budget validation gains, but its additional benefit is inconsistent across evaluation sets.
  2. The practical planning benefit remains unverified. Evaluation measures whether at least one acceptable candidate exists, without testing downstream selection or closed-loop execution. Meanwhile, the validation wrong-way share among verifier-feasible samples increases from 4.3% to 14.0%, leaving the net effect on driving quality uncertain.
Questions for authors
  1. Beyond reference-relative target selection and prefix scheduling, what methodological distinction from CLOVER’s pseudo-expert coverage supervision accounts for the claimed novelty?
  2. Can targeting gains over non-targeted augmentation be quantified with paired uncertainty estimates and training-seed repetitions, particularly on held-out scenes, alongside the additional diagnosis cost?
  3. Are matched-budget dense-vocabulary comparisons or fixed-selector/closed-loop results available to establish whether improved directional coverage yields better final planning decisions
6 comments

Dear Reviewer SbPJ,

Thank you for asking how our coverage diagnosis differs from CLOVER's supervision and for suggesting a matched-budget comparison with dense trajectory vocabularies. We will address your concerns and questions in the response below.

[W1 & Q1] Difference from CLOVER

We appreciate the opportunity to clarify how our directional coverage metric differs from CLOVER's coverage objective.

CLOVER uses set-level coverage as a training objective, whereas we measure directional coverage at a fixed sampling budget. The difference is as follows.

  1. CLOVER measures how close proposals are, not which directions are absent. CLOVER's pseudo-expert loss averages the L1 distance from each pseudo-expert to its nearest proposal (Sec. 3.2, Eq. 4). Its Oracle@64 PDMS reports the highest score among 64 proposals. A scene can therefore receive a high oracle score from a straight trajectory even if the planner generates no feasible left turn.

  2. Our metric measures which feasible directions are missing at a fixed budget. We test for precisely this kind of omission. A verified cross-scene witness establishes that a left turn satisfies σ in the current scene. If no reference sample satisfies σ in that direction within the diagnosis budget, we record a coverage hole. Any σ-feasible left turn can recover that hole, even if its path differs from the witness. Closure@N measures the fraction of diagnosed state–direction pairs for which the planner generates such a candidate within N samples.

[W1 & Q2] Paired uncertainty, seed repetition, and diagnosis cost

Thank you for asking how reliably targeting improves held-out coverage and at what computational cost.

Targeting gain and uncertainty. We compared targeted and non-targeted augmentation on the same state–direction pairs, holding the training recipe and target count fixed. The table below reports paired differences in closure under free sampling (targeted minus non-targeted, in percentage points) for the original training seed. We computed paired 95% confidence intervals by resampling scenes jointly for both models (11 scenes for Held-out-66).

Evaluation setΔclosure@8Δclosure@32Δclosure@128
Train-200+3.1 [−2.3, +8.8]+3.9 [−1.3, +9.5]+5.9 [+2.4, +9.5]
Val-153+10.2 [0.0, +19.5]+4.5 [−2.2, +12.1]+2.8 [−0.5, +6.8]
Held-out-66+1.2 [−13.1, +10.5]+1.2 [−8.1, +8.5]−1.2 [−6.0, +2.9]

At N = 8, the largest observed gain was on Val-153 (+10.2 pp). On Held-out-66, the gain was modest at N = 8 and 32 (+1.2 pp at both budgets) and did not persist at N = 128. All three held-out intervals include zero. These intervals quantify variation across scenes for one training seed.

Training-seed repetition. We retrained both the targeted and the non-targeted model with a second seed using the same training recipe. We evaluated all four runs with PSD on the same state–direction pairs in each split. The table below reports closure (%) and the estimated number of candidates needed to reach 70% closure.

PSDclosure@8closure@32closure@128Est. candidates for 70% closure
Train-200
Targeted (paper seed)44.171.390.929.3
Targeted (second seed)49.275.690.921.9
Non-targeted (paper seed)40.967.385.037.4
Non-targeted (second seed)39.065.087.439.3
Val-153
Targeted (paper seed)70.588.196.0< 8
Targeted (second seed)73.388.694.9< 8
Non-targeted (paper seed)60.283.593.212.9
Non-targeted (second seed)56.881.894.314.2
Held-out-66
Targeted (paper seed)43.583.595.315.8
Targeted (second seed)51.889.498.819.9
Non-targeted (paper seed)42.482.496.522.6
Non-targeted (second seed)41.287.197.617.1

On Train-200 and Val-153, both targeted runs achieved higher closure than both non-targeted runs at all three budgets. On Held-out-66, the targeted second seed led in closure at every budget, and the targeted paper seed reached 70% closure with the fewest candidates.

Across all three sets, targeted augmentation was effective and most effective when only a few candidates could be generated. At N = 8, both targeted runs exceeded both non-targeted runs on every set.

Diagnosis cost. Mining required 32 reference-planner rollouts per state across 13,220 states and took approximately 20 hours on four RTX 3090 GPUs. Using 3.7 GPU-seconds per inference candidate, we estimated how many planning queries would offset this one-off cost at a given closure target.

SetClosure targetEstimated candidates targeted / non-targetedEstimated break-even queries
Train-20070%29.3 / 37.49,600
Train-20090%≤ 128 / > 128Not estimable
Val-15370%≤ 8 / 12.9≤ 16,000
Val-15390%41.6 / 58.84,500
Held-out-6670%15.8 / 22.611,400
Held-out-6690%55.2 / 52.8No break-even

For the original training seed, targeting reduced the estimated budget for 70% closure on Held-out-66 from 22.6 to 15.8 candidates (30% fewer), yielding an estimated break-even point of 11,400 planning queries. At 90% closure, non-targeted augmentation required fewer candidates, so mining provided no inference savings.

[W1 & Q3] Dense-vocabulary comparison and fixed-scorer selection

We thank the reviewer for suggesting a matched-budget vocabulary comparison and asking whether broader coverage improves trajectory selection.

(a) Matched-budget dense vocabulary. We built two fixed vocabularies (K = 4,096 and K = 8,192) by applying k-means to the nuScenes training trajectories used for our 256-anchor bank. In the first table, we ranked prototypes within each direction by the number of training trajectories assigned to each cluster. The vocabulary baseline selected prototypes in descending frequency order using a left/right/straight/straight cycle and retimed each prototype to the current ego speed. PSD used left/right/free/free. Both methods received the same candidate budget. The vocabulary baseline used no learned scene-conditioned ranker.

In both tables, each budget column reports closure / feasible share (%). Closure is the fraction of diagnosed state–direction pairs recovered within the first N candidates. Feasible share is the fraction of those candidates that pass the same geometric verifier σ after generation by staying within the drivable area and avoiding collisions with recorded agents. Bold indicates the best value for each metric within each split.

Candidate source@8@16@32@64@128minADE@8 (m)
Train-200
K = 4,09640.2 / 30.655.9 / 32.364.6 / 30.376.8 / 30.287.4 / 28.13.50
K = 8,19233.9 / 31.446.5 / 30.161.4 / 28.076.8 / 27.486.6 / 27.94.12
Ours + PSD44.1 / 47.361.4 / 47.171.3 / 45.584.3 / 47.690.9 / 46.71.91
Val-153
K = 4,09652.8 / 46.372.7 / 46.783.5 / 43.096.0 / 41.598.3 / 38.73.16
K = 8,19235.2 / 47.146.0 / 43.569.9 / 39.889.2 / 37.997.2 / 38.23.24
Ours + PSD70.5 / 58.775.6 / 56.788.1 / 55.193.2 / 56.696.0 / 55.12.09
Held-out-66
K = 4,09656.5 / 36.970.6 / 39.380.0 / 33.090.6 / 34.092.9 / 30.13.00
K = 8,19263.5 / 45.376.5 / 40.980.0 / 34.392.9 / 30.298.8 / 30.43.13
Ours + PSD43.5 / 53.270.6 / 51.683.5 / 50.891.8 / 53.695.3 / 54.01.57

To test how the choice of candidates within each direction affects coverage, we repeated the comparison with the same vocabularies and direction cycle, drawing prototypes in random order within each direction.

Candidate source@8@16@32@64@128minADE@8 (m)
Train-200
K = 4,09627.6 / 22.242.5 / 20.551.2 / 21.469.3 / 23.178.7 / 20.33.22
K = 8,19223.2 / 15.746.9 / 20.663.0 / 21.280.3 / 21.089.8 / 21.24.00
Ours + PSD44.1 / 47.361.4 / 47.171.3 / 45.584.3 / 47.690.9 / 46.71.91
Val-153
K = 4,09621.6 / 25.838.6 / 25.054.5 / 26.273.9 / 28.289.8 / 25.02.70
K = 8,19221.6 / 21.150.0 / 26.667.6 / 27.095.5 / 26.298.3 / 26.23.71
Ours + PSD70.5 / 58.775.6 / 56.788.1 / 55.193.2 / 56.696.0 / 55.12.09
Held-out-66
K = 4,09627.1 / 26.531.8 / 20.942.4 / 21.081.2 / 22.589.4 / 19.72.99
K = 8,19231.8 / 17.061.2 / 23.869.4 / 23.782.4 / 22.195.3 / 22.73.31
Ours + PSD43.5 / 53.270.6 / 51.683.5 / 50.891.8 / 53.695.3 / 54.01.57

At N = 8, frequency ordering yielded higher vocabulary closure than random ordering for both vocabulary sizes on all three splits. Ours + PSD achieved the highest closure@8 on Train-200 and Val-153, while the frequency-ordered vocabulary led on Held-out-66. For example, at N = 128 on Held-out-66, the frequency-ordered K = 8,192 vocabulary recovered more holes than Ours + PSD (98.8% vs. 95.3%), but a larger share of its candidates failed the verifier (69.6% vs. 46.0%). At the same candidate budget, a dense vocabulary can match or exceed Ours + PSD in directional coverage. At every tested budget on all three splits, Ours + PSD achieved a higher verifier pass rate than either dense vocabulary under both frequency and random ordering.

Reviewer_8ekBOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies coverage holes in vision-language driving planners: feasible maneuvers that a planner fails to sample within a fixed budget. The authors construct 256 trajectory anchors using k-means, adjust them to the current ego speed, and check drivable-area compliance and non-collision. Feasible witnesses for missing directions are then used to fine-tune the planner with directional reasoning prompts. At inference, Prefix-Scheduled Decoding (PSD) allocates samples across left-turn, right-turn, and free-generation settings. The experiments focus on Alpamayo-1.5-10B and OmniDrive on nuScenes, with additional analysis of GenAD.

My main concerns are the limited technical novelty, the modest benefit of targeted mining over simpler augmentation at larger sampling budgets, and the decline in candidate quality. I would be more convinced by a clearer demonstration that targeting is worth its cost and that the additional candidates improve downstream selection. I am open to increasing the score if the concerns are tackled.

Critical strengths
  1. The evaluation includes useful controls, particularly non-targeted witness training in Table 3 and reference-independent evaluation states in Table 7. The authors also report wrong-way trajectories, split overlap, and other limitations of their pipeline. These analyses make the results easier to assess.

  2. Similar coverage problems appear in Alpamayo, OmniDrive, and GenAD. This suggests that the issue is not specific to one planner architecture.

Critical weaknesses
  1. The technical contribution and novelty are relatively limited. The method combines some familiar ideas: MultiPath-style trajectory anchors, feasibility filtering, supervision with alternative trajectories, and command-conditioned sampling. Its main distinction appears to be the diagnosis and recovery of reference-specific coverage holes. PSD is a fixed schedule over existing directional prompts. I would like a clearer account of what the diagnostic step adds beyond feasible-trajectory augmentation and directional conditioning.

  2. In Table 3, non-targeted witness training nearly matches targeted training at a budget of 128 samples. With PSD, closure is 93.2% versus 96.0% on validation and 96.5% versus 95.3% on held-out scenes. Under free generation, the non-targeted model also has higher held-out closure: 98.8% versus 96.5%. Targeting does help at smaller budgets, but the paper needs to show whether that benefit justifies the additional mining cost of 32 reference-planner rollouts per state.

  3. Under the default lane-direction criterion, Appendix A.3.3 flags 38.3% of training witnesses with a determinable lane pose as wrong-way, including 47.6% of left-turn witnesses. In Table 4, the validation wrong-way share among geometrically feasible samples is 4.3% for the off-the-shelf model with free generation and 14.0% for Ours + PSD. Table 7 reports a feasible-sample share of 53.1% for Ours + PSD, compared with 73.3% for GT-only fine-tuning. In Table 1, minADE@8 is also worse than GT-only fine-tuning: 1.91 m versus 1.26 m. These comparisons involve different evaluation settings, but together they raise a practical question: can a downstream selector turn the extra coverage into better selected trajectories? The paper does not test this.

  4. A direction label based on a 25-degree heading threshold can confuse road curvature or lane sweeps with a meaningful maneuver choice. The authors use a GBDT validity detector to select evaluation states, rejecting 74% of clear non-holes, but select the 5,768 training targets without applying that detector. It is unclear why this screening is not also used for training. In addition, 53.1% of the state-direction pairs in the main Train-200 evaluation received training targets for that exact state and direction. This limits what Table 1 establishes about generalization and places more weight on the independent evaluations.

Questions for authors
  1. Given the strong non-targeted baseline, when is reference-planner hole mining worth its additional cost? Can the authors quantify the trade-off between mining cost and inference sampling savings?

  2. Could ego and agent motion from the fused historical logs provide lane-direction estimates for filtering witnesses? Why was the GBDT validity detector not used to screen the training pool?

  3. With a fixed downstream trajectory scorer, such as PDM-Scorer, does the gain in mode coverage improve the selected trajectory despite the lower feasible-sample share and higher wrong-way rate?

3 comments

Dear Reviewer 8ekB,

Thank you for asking whether the inference savings from targeting justify the cost of hole mining. We will address your concerns and questions in the response below.


[W2 & Q1] When hole mining pays off

We appreciate the suggestion to quantify when hole mining pays off at inference. With the same number of training targets and the same PSD schedule, targeting reduced the estimated number of candidates needed for 70% closure on Held-out-66 from 22.6 to 15.8 (30% fewer). At 90% closure, non-targeted training required fewer candidates (52.8 vs. 55.2), so mining offered no inference savings at that target.

Mining required 32 reference-planner rollouts per state across 13,220 states and took approximately 20 hours on four RTX 3090 GPUs. We estimated the break-even query count from this one-off cost and the saving per query, using 3.7 GPU-seconds per inference candidate.

SetClosure targetEstimated candidates (targeted / non-targeted)Estimated break-even queries
Train-20070%29.3 / 37.49,600
Train-20090%≤ 128 / > 128Not estimable
Val-15370%≤ 8 / 12.9≤ 16,000
Val-15390%41.6 / 58.84,500
Held-out-6670%15.8 / 22.611,400
Held-out-6690%55.2 / 52.8No break-even

On Train-200, non-targeted augmentation did not reach 90% closure within 128 candidates, so its required budget could not be estimated. For a 70% closure target, the estimated inference savings offset the one-off mining cost after approximately 9,600 planning queries on Train-200 and 11,400 on Held-out-66.

[W4 & Q2] Label-free lane direction, training-pool screening, and evaluation validity

We thank the reviewer for suggesting a way to reduce wrong-way outputs without HD-map labels.

Lane direction from logged motion. We estimated lane direction from ego trajectories in the fused logs. Adding agent trajectories did not change the result. Against the official-map wrong-way labels, the flow filter achieved 98.5–100% precision and 18–33% recall. It removed only 2.2% of training witnesses. After retraining, the validation wrong-way share was 15.2%, compared with 14.0% without witness filtering.

At inference, the flow filter reduced the wrong-way share on all three sets. The table compares three verifiers on the same PSD candidates. Wrong-way share is measured among the candidates each verifier accepts.

N = 128, PSDσ (paper)σ ∧ flow (label-free)σ ∧ official map
Val-153closure (%)96.096.093.2
wrong-way (%)14.011.60.0
Held-out-66closure (%)95.395.387.1
wrong-way (%)13.19.10.0
Train-200closure (%)90.990.685.8
wrong-way (%)14.612.30.0

The cost in closure was at most 0.3 percentage points at N = 128 and 3.5–5.7 percentage points at N = 8.

GBDT screening. For evaluation, we prioritize precision because false-positive holes would distort the set of state–direction pairs on which closure is measured. For training, we prioritize recall while requiring each witness to pass the geometric verifier σ at the target state and match the target direction. Applying the GBDT at the state level could discard a genuine hole along with its verified training examples. In a post hoc check, the GBDT retained 62.6% of the 5,768 training targets and only 52.0% of left-turn targets.

Heading threshold. As reported in Appendix A.3.5 of the submitted manuscript, the model ranking remained unchanged when we redefined the hole pairs on the same 200 states using heading thresholds of ±20°, ±25°, and ±35°. We will explicitly acknowledge in the main text that a heading-based label can classify a bend as a turn even when the road offers no route choice.

Training overlap. We agree that the 53.1% overlap limits what Table 1 can establish about generalization. On Val-153 and Held-out-66, no evaluation pair received a training target for that exact state and direction. We will clarify in the main text that Table 1 measures recovery within the training split and give greater weight to Val-153 and Held-out-66 when discussing generalization.

[W1] What the diagnosis step adds

We appreciate the opportunity to clarify what hole diagnosis adds beyond witness augmentation and directional conditioning.

The diagnosis step adds three things.

  1. Measuring which directions a planner misses. Feasible-trajectory augmentation, including our non-targeted control, adds feasible alternatives to the training data without stating which directions a given planner fails to produce. Directional conditioning requests a direction without checking whether the planner can produce it. In contrast, we define a coverage hole relative to a reference planner and a sampling budget, and we introduce closure@N, the fraction of diagnosed holes recovered within N samples, to measure how many of these omissions a planner recovers at a given budget.

  2. Recovering directions that conditioning alone does not produce. Conditioning draws on directions the planner can already produce. The table below compares navigation text with CFG on the off-the-shelf model against PSD on our model, on the gold set (502 states, 731 pairs) at the same 128-sample allocation.

closure@128 (%)Off-the-shelfOurs
free (no navigation text)28.985.4
w/ navigation text + CFG (w = 3)53.483.6
w/ PSD46.290.0

Our model under free generation already exceeded the off-the-shelf model with navigation text and CFG, because diagnosis-guided augmentation added the missing directions to the planner rather than requesting them at inference. With PSD, our model reached the highest closure.

  1. Placing training targets where the reference planner fails. Non-targeted augmentation selects verified witnesses independently of the reference planner's omissions. Its targets therefore include directions that the planner already produces. In contrast, our targeted augmentation uses the diagnosis to select witnesses only for the omitted directions, and the same number of targets is therefore placed where the planner fails. The table below shows the difference between non-targeted and targeted augmentation under the same witnesses, verifier, target count, and PSD schedule.
AugmentationTrain-200 Closure@8 (%)Train-200 N*@70Val-153 Closure@8 (%)Val-153 N*@70Held-out-66 Closure@8 (%)Held-out-66 N*@70
Non-targeted40.937.460.212.942.422.6
Targeted (Ours)44.129.370.5≤ 843.515.8

Targeted augmentation achieved higher closure@8 and a lower N*@70 than non-targeted augmentation on all three sets. The difference is attributable to the placement of the witnesses, since both models received the same number of verified witnesses. Non-targeted augmentation allocated part of its targets to directions that the planner already produced, whereas the diagnosis assigned all targets to omitted directions. The planner therefore recovered the diagnosed holes within a smaller sampling budget.

[W3 & Q3] Does a fixed scorer still benefit from the extra coverage

Thank you for suggesting the PDM-Scorer to assess the quality of the best trajectory in each candidate set. minADE@8 measures how closely the best of eight candidates matches a single logged trajectory. On Train-200 it rises from 1.26 m (GT-finetuned) to 1.91 m (Ours + PSD), but it cannot assess candidates that follow other feasible paths. We report oracle best-of-eight PDMS on 6,076 NAVSIM navtest scenes. For each scene, we scored each model's first eight candidates with the same PDM-Scorer and retained the highest score for each model. The table shows the PDMS and sub-scores of the selected trajectories, averaged across scenes.

ModelPDMS_best@8NCDACDDCTLCEPTTCLKHC
Off-the-shelf (Alpamayo-1.5)83.498.994.296.8100.084.798.797.196.7
Off-the-shelf + PSD83.498.793.796.5100.086.498.497.196.5
GT-finetuned84.298.294.998.9100.084.197.895.796.2
Ours (free)84.698.595.799.3100.078.797.894.194.9
Ours + PSD85.298.795.899.4100.080.598.093.795.1

For the selected trajectories, Ours + PSD matched or exceeded GT-finetuning on NC, DAC, DDC and TLC. Despite the lower feasible share and higher wrong-way rate observed on nuScenes, Ours + PSD generated candidates with a higher oracle best-of-eight PDMS on NAVSIM than GT-finetuning (85.2 vs. 84.2). This gain measures the quality attainable within eight candidates.

I have carefully read the author's rebuttal. The clarification about novelty, training-test gap, extra ablations are convencing to me.

Thank you for the author's detailed response, which has addressed most of my concerns. I am therefore raising my score to weak accept.

Reviewer_aRgVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines a coverage hole as a state–direction pair for which a verified trajectory provides evidence that a maneuver is feasible, but a reference planner fails to produce that maneuver within a fixed sampling budget. It constructs witnesses from retimed trajectory prototypes, fine-tunes Alpamayo using witness trajectories paired with direction prompts, and introduces Prefix-Scheduled Decoding (PSD), which cycles through left, right, and unprompted samples. The paper reports substantial gains in its open-loop coverage metric, including 90.9% closure@128 on the main evaluation set.

Critical strengths
  • The paper identifies and formalizes a finite-sampling coverage metric for alternative maneuvers, rather than relying only on trajectory error.
  • It provides useful controls, including non-targeted witness augmentation, and reports trade-offs in accuracy and lane-direction compliance.
  • The observed issue is not specific to VLMs: the paper also reports high hole rates for non-language planners. This broadens the relevance of the phenomenon, though it also makes the contribution less specifically about VLMs.
Critical weaknesses
  1. The practical importance of the task is not established. The main setup explicitly removes external route commands and evaluates whether a planner can generate multiple directions at a state. In many autonomous-driving stacks, high-level navigation provides the intended route, and the local planner is expected to generate candidates consistent with it. Alternative-route generation can matter for contingencies, but the paper does not show that improving this particular unconditioned, open-loop coverage metric improves route following, contingency handling, downstream selection, or closed-loop driving performance. The paper therefore demonstrates gains on its chosen metric without establishing that the metric corresponds to a consequential deployment need.

  2. The methodological contribution appears limited. The core pipeline combines trajectory clustering, retiming, geometric feasibility filtering, supervised fine-tuning, and a fixed prompt schedule. These components produce a useful empirical result, but the paper offers limited methodological insight into constructing reliable alternative trajectories or ensuring that they are appropriate targets for autonomous driving.

  3. The ablation weakens the case for targeted hole diagnosis. Non-targeted witness augmentation achieves 82.7% closure@128, compared with 85.0% for targeted augmentation under free generation; with PSD, the corresponding results are 85.0% and 90.9%. The authors’ own analysis concludes that targeting mainly improves sampling efficiency, rather than being necessary for most of the final coverage gain. This narrows the contribution of the paper’s hole-diagnosis component.

  4. The feasibility predicate is not sufficient for safety-relevant claims. The base verifier checks drivable-area compliance and collision against recorded, non-reactive agents, but does not establish traffic-rule compliance or interactive safety. Under the lane-direction audit, the wrong-way share among validation samples that pass the base verifier rises from 4.3% for the off-the-shelf model to 14.0% for the augmented model with PSD. Coverage also remains higher under the stricter check, but the increased wrong-way output is a meaningful trade-off for the proposed application.

  5. The accuracy trade-off needs stronger justification. On the main set, the GT-finetuned reference has minADE@8 of 1.26 m, compared with 1.72 m for the augmented model and 1.91 m with PSD. Since the paper does not demonstrate a downstream benefit that compensates for this loss in logged-trajectory accuracy, it is difficult to assess the practical value of the trade-off.

Questions for authors
  1. What concrete deployment or benchmark scenario requires generating multiple uncommanded turn directions at the same state? Can the authors show that the added candidates improve a downstream selector or closed-loop driving outcome?
  2. How would the method behave when the route planner supplies a maneuver command? In that setting, is there a benefit over simply generating candidates conditioned on the commanded route?
  3. Can the authors reduce wrong-way outputs while retaining the coverage gains, without relying on HD-map labels?
3 comments

Dear Reviewer aRgV,

Thank you for recognizing the value of measuring coverage under a fixed sampling budget and for raising questions about its practical relevance to motion planning. We address your comments below.


[W2] Methodological contribution

We appreciate the opportunity to clarify our contribution.

Our contributions are as follows.

  1. Defining and measuring coverage holes. We define a coverage hole as a feasible state–direction pair that the reference planner misses within a fixed sampling budget. A verified witness establishes feasibility for that pair. Closure@NN measures the fraction of diagnosed holes recovered within NN samples.

  2. Verifying missing directions without HD-map labels. We propose a geometric verifier σ\sigma to identify feasible alternatives without HD-map labels. It checks whether a candidate trajectory stays within the drivable area and avoids collisions with recorded agents. A passing candidate in a direction missed by the reference planner serves as a witness for that direction.

  3. Distinguishing recovery gains from sampling-efficiency gains. We measure how much using verified witnesses as training targets and applying PSD improve closure in controlled comparisons. We also measure how targeting diagnosed holes changes the number of candidates needed to reach a given closure level. We quantify the accuracy and lane-direction trade-offs.

  4. Higher closure with few samples. With eight candidates, Ours + PSD recovered more diagnosed holes than both the off-the-shelf and GT-finetuned planners on all three evaluation sets. The table below reports closure at N=8N=8 and N=128N=128, along with the estimated number of samples needed to reach 70% closure (N∗@70N^{*}@70).

ModelClosure@8 (%)Closure@128 (%)N∗@70N^{*}@70
Train-200
Off-the-shelf13.446.5>128
GT-finetuned (reference planner)0.023.6>128
Ours + PSD44.190.929.3
Val-153
Off-the-shelf10.238.1>128
GT-finetuned (reference planner)2.814.2>128
Ours + PSD70.596.0<8
Held-out-66
Off-the-shelf16.557.6>128
GT-finetuned (reference planner)1.256.5>128
Ours + PSD43.595.315.8

[W1, W5 & Q1] Deployment scenario, selector benefit, and the accuracy trade-off

Thank you for asking when alternative maneuvers are useful and how they affect trajectory selection.

Scenario. In a proposal–selection planner, the local planner generates candidate trajectories before a downstream selector commits to a maneuver. For example, at an intersection where both going straight and turning left are feasible, the selector can compare them only if both are in the candidate set. Closure@NN measures the fraction of state–direction pairs missed by the reference planner for which a verified candidate is generated within NN samples.

Downstream selector. minADE@8 measures how closely the best of eight candidates matches a single logged trajectory. On Train-200 it rises from 1.26 m (GT-finetuned) to 1.91 m (Ours + PSD), but it cannot assess candidates that follow other feasible paths. We report oracle best-of-eight PDMS on 6,076 NAVSIM navtest scenes. For each scene, we scored each model's first eight candidates with the same PDM-Scorer and retained the highest score for each model. The table shows the PDMS and sub-scores of the selected trajectories, averaged across scenes.

ModelPDMSbest@8\mathrm{PDMS}_{\mathrm{best}}@8NCDACDDCTLCEPTTCLKHC
Off-the-shelf (Alpamayo-1.5)83.498.994.296.8100.084.798.797.196.7
Off-the-shelf + PSD83.498.793.796.5100.086.498.497.196.5
GT-finetuned84.298.294.998.9100.084.197.895.796.2
Ours (free)84.698.595.799.3100.078.797.894.194.9
Ours + PSD85.298.795.899.4100.080.598.093.795.1

Metrics. NC (no at-fault collision), DAC (drivable-area compliance), DDC (driving-direction compliance), TLC (traffic-light compliance), EP (ego progress), TTC (time to collision), LK (lane keeping), and HC (history comfort).

We evaluated all models zero-shot on NAVSIM navtest. We mapped the nuPlan cameras to the corresponding nuScenes camera slots and resampled each 10 Hz, 6.4 s trajectory to eight poses over 4 s before scoring (v2 devkit, single-stage, not comparable to v1 leaderboard PDMS).

Despite the higher minADE@8 on Train-200, Ours + PSD achieved the highest oracle PDMSbest@8\mathrm{PDMS}_{\mathrm{best}}@8 on NAVSIM. This shows that our model can generate high-scoring trajectories within a budget of just eight candidates.

[Q2] Route-conditioned generation versus PSD

Thank you for suggesting a comparison with route-conditioned generation. We evaluated the off-the-shelf model and our model on the gold set (502 states, 731 pairs) using the navigation input built into Alpamayo-1.5, both with and without classifier-free guidance (CFG, w=3w=3). For each state, we generated 128 samples in each navigation-text setting, using "Turn left ahead." for 32 samples, "Turn right ahead." for 32 samples, and no navigation text for the remaining 64, matching the sampling allocation used in PSD.

closure@128 (%)Off-the-shelfOurs
free (no navigation text)28.985.4
w/ navigation text44.0 [38.9, 49.3]82.8 [78.9, 86.7]
w/ navigation text + CFG (ww = 3)53.4 [48.0, 59.0]83.6 [79.7, 87.5]
w/ PSD46.2 [41.0, 51.7]90.0 [86.9, 93.1]

Navigation text raised off-the-shelf closure@128 from 28.9% to 44.0%. Navigation text + CFG (classifier-free guidance) achieved the highest off-the-shelf closure (53.4%), compared with 46.2% for PSD. Even under free generation, our model exceeded the off-the-shelf model's best result with navigation text and CFG by 32.0%p (85.4% vs. 53.4%). With PSD, our model reached 90.0% closure@128. Under the same sample allocation, PSD recovered more missing directions than navigation text did on our model, even with CFG.

[W4 & Q3] Reducing wrong-way outputs without HD-map labels

Thank you for asking whether we can reduce wrong-way outputs without HD-map labels while preserving coverage. We estimated a lane-direction field from ego trajectories in the fused logs and rejected candidates whose headings opposed the recorded direction. Against official HD-map labels, the filter achieved 98.5–100% precision and 18–33% recall. We compared the three verifiers on the same PSD candidates at N = 128.

N = 128, PSDσ (paper)σ ∧ flow (label-free)σ ∧ official map
Val-153closure (%)96.096.093.2
wrong-way (%)14.011.60.0
Held-out-66closure (%)95.395.387.1
wrong-way (%)13.19.10.0
Train-200closure (%)90.990.685.8
wrong-way (%)14.612.30.0

The official-map filter rejected all candidates flagged by the map criterion, reducing closure by approximately 3–8 percentage points at N = 128. The log-based filter reduced closure by at most 0.3 points at that budget and by 3.5–5.7 points at N = 8.

A filter based on logged motion can reduce the wrong-way share among accepted candidates without HD-map labels. At N = 128, this share fell from 14.0% to 11.6% on Val-153 and from 13.1% to 9.1% on Held-out-66, with closure unchanged on both sets. The remaining 11.6% on Val-153 was still above the 4.3% observed for off-the-shelf free generation.

[W3] What targeting adds beyond non-targeted augmentation

We thank the reviewer for pointing out the distinction between final coverage and sampling efficiency. We agree that non-targeted witness augmentation accounts for much of the final coverage gain. Targeting reduced the estimated number of candidates needed to reach 70% closure by 22% on Train-200 (37.4 to 29.3) and by 30% on Held-out-66 (22.6 to 15.8). To quantify the additional effect of targeting, we held the training-target count and PSD schedule fixed and compared closure at a given budget and the budget needed to reach 70% closure (N*@70).

N*@70Non-targetedTargeted
Train-20037.429.3
Val-15312.9≤ 8
Held-out-6622.615.8

The next table shows Δ = closure(targeted) − closure(non-targeted) in percentage points at each budget. Brackets give paired 95% confidence intervals obtained by resampling scenes.

Evaluation setΔclosure@8Δclosure@32Δclosure@128
Train-200+3.1 [−2.3, +8.8]+3.9 [−1.3, +9.5]+5.9 [+2.4, +9.5]
Val-153+10.2 [0.0, +19.5]+4.5 [−2.2, +12.1]+2.8 [−0.5, +6.8]
Held-out-66+1.2 [−13.1, +10.5]+1.2 [−8.1, +8.5]−1.2 [−6.0, +2.9]

At N = 8, closure was higher with targeting on all three sets, with the largest gain on validation (+10.2 pp). At N = 128, targeting still improved closure on Train-200 (+5.9 pp, paired 95% CI [2.4, 9.5]), while the held-out estimate slightly favoured non-targeted augmentation (−1.2 pp, [−6.0, 2.9]). The additional benefit observed across all three sets was reaching 70% closure with fewer candidates.

Dear Reviewer aRgV,

As the discussion period is closing soon, we wanted to kindly check whether our response has addressed your concerns.

If you have any further questions, we would be happy to answer them in the remaining time.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.