- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies finite-budget directional coverage in vision-language motion planners. Verified cross-scene trajectory witnesses identify reference-relative omissions and provide augmentation targets, while prefix-scheduled decoding allocates samples across directions. Experiments on nuScenes demonstrate substantial open-loop coverage improvements, alongside accuracy and traffic-rule-compliance trade-offs.
Critical strengths
The witness-based formulation makes state-specific, finite-budget omissions measurable, providing a useful diagnostic beyond logged-trajectory accuracy. Coverage improvements are supported by official-map, lane-direction, and reference-independent evaluations. The non-targeted augmentation control and compliance audits help clarify the sources and limitations of these gains.
Critical weaknesses
- The incremental contribution over existing coverage methods is insufficiently established. There are no direct comparisons with CLOVER-style coverage supervision or dense/factorized trajectory vocabularies. Matched non-targeted augmentation already explains much of the improvement. Targeting provides meaningful low-budget validation gains, but its additional benefit is inconsistent across evaluation sets.
- The practical planning benefit remains unverified. Evaluation measures whether at least one acceptable candidate exists, without testing downstream selection or closed-loop execution. Meanwhile, the validation wrong-way share among verifier-feasible samples increases from 4.3% to 14.0%, leaving the net effect on driving quality uncertain.
Questions for authors
- Beyond reference-relative target selection and prefix scheduling, what methodological distinction from CLOVER’s pseudo-expert coverage supervision accounts for the claimed novelty?
- Can targeting gains over non-targeted augmentation be quantified with paired uncertainty estimates and training-seed repetitions, particularly on held-out scenes, alongside the additional diagnosis cost?
- Are matched-budget dense-vocabulary comparisons or fixed-selector/closed-loop results available to establish whether improved directional coverage yields better final planning decisions
6 comments
[W1 & Q2] Paired uncertainty, seed repetition, and diagnosis cost
Thank you for asking how reliably targeting improves held-out coverage and at what computational cost.
Targeting gain and uncertainty. We compared targeted and non-targeted augmentation on the same state–direction pairs, holding the training recipe and target count fixed. The table below reports paired differences in closure under free sampling (targeted minus non-targeted, in percentage points) for the original training seed. We computed paired 95% confidence intervals by resampling scenes jointly for both models (11 scenes for Held-out-66).
| Evaluation set | Δclosure@8 | Δclosure@32 | Δclosure@128 |
|---|---|---|---|
| Train-200 | +3.1 [−2.3, +8.8] | +3.9 [−1.3, +9.5] | +5.9 [+2.4, +9.5] |
| Val-153 | +10.2 [0.0, +19.5] | +4.5 [−2.2, +12.1] | +2.8 [−0.5, +6.8] |
| Held-out-66 | +1.2 [−13.1, +10.5] | +1.2 [−8.1, +8.5] | −1.2 [−6.0, +2.9] |
At N = 8, the largest observed gain was on Val-153 (+10.2 pp). On Held-out-66, the gain was modest at N = 8 and 32 (+1.2 pp at both budgets) and did not persist at N = 128. All three held-out intervals include zero. These intervals quantify variation across scenes for one training seed.
Training-seed repetition. We retrained both the targeted and the non-targeted model with a second seed using the same training recipe. We evaluated all four runs with PSD on the same state–direction pairs in each split. The table below reports closure (%) and the estimated number of candidates needed to reach 70% closure.
| PSD | closure@8 | closure@32 | closure@128 | Est. candidates for 70% closure |
|---|---|---|---|---|
| Train-200 | ||||
| Targeted (paper seed) | 44.1 | 71.3 | 90.9 | 29.3 |
| Targeted (second seed) | 49.2 | 75.6 | 90.9 | 21.9 |
| Non-targeted (paper seed) | 40.9 | 67.3 | 85.0 | 37.4 |
| Non-targeted (second seed) | 39.0 | 65.0 | 87.4 | 39.3 |
| Val-153 | ||||
| Targeted (paper seed) | 70.5 | 88.1 | 96.0 | < 8 |
| Targeted (second seed) | 73.3 | 88.6 | 94.9 | < 8 |
| Non-targeted (paper seed) | 60.2 | 83.5 | 93.2 | 12.9 |
| Non-targeted (second seed) | 56.8 | 81.8 | 94.3 | 14.2 |
| Held-out-66 | ||||
| Targeted (paper seed) | 43.5 | 83.5 | 95.3 | 15.8 |
| Targeted (second seed) | 51.8 | 89.4 | 98.8 | 19.9 |
| Non-targeted (paper seed) | 42.4 | 82.4 | 96.5 | 22.6 |
| Non-targeted (second seed) | 41.2 | 87.1 | 97.6 | 17.1 |
On Train-200 and Val-153, both targeted runs achieved higher closure than both non-targeted runs at all three budgets. On Held-out-66, the targeted second seed led in closure at every budget, and the targeted paper seed reached 70% closure with the fewest candidates.
Across all three sets, targeted augmentation was effective and most effective when only a few candidates could be generated. At N = 8, both targeted runs exceeded both non-targeted runs on every set.
Diagnosis cost. Mining required 32 reference-planner rollouts per state across 13,220 states and took approximately 20 hours on four RTX 3090 GPUs. Using 3.7 GPU-seconds per inference candidate, we estimated how many planning queries would offset this one-off cost at a given closure target.
| Set | Closure target | Estimated candidates targeted / non-targeted | Estimated break-even queries |
|---|---|---|---|
| Train-200 | 70% | 29.3 / 37.4 | 9,600 |
| Train-200 | 90% | ≤ 128 / > 128 | Not estimable |
| Val-153 | 70% | ≤ 8 / 12.9 | ≤ 16,000 |
| Val-153 | 90% | 41.6 / 58.8 | 4,500 |
| Held-out-66 | 70% | 15.8 / 22.6 | 11,400 |
| Held-out-66 | 90% | 55.2 / 52.8 | No break-even |
For the original training seed, targeting reduced the estimated budget for 70% closure on Held-out-66 from 22.6 to 15.8 candidates (30% fewer), yielding an estimated break-even point of 11,400 planning queries. At 90% closure, non-targeted augmentation required fewer candidates, so mining provided no inference savings.
[W1 & Q3] Dense-vocabulary comparison and fixed-scorer selection
We thank the reviewer for suggesting a matched-budget vocabulary comparison and asking whether broader coverage improves trajectory selection.
(a) Matched-budget dense vocabulary. We built two fixed vocabularies (K = 4,096 and K = 8,192) by applying k-means to the nuScenes training trajectories used for our 256-anchor bank. In the first table, we ranked prototypes within each direction by the number of training trajectories assigned to each cluster. The vocabulary baseline selected prototypes in descending frequency order using a left/right/straight/straight cycle and retimed each prototype to the current ego speed. PSD used left/right/free/free. Both methods received the same candidate budget. The vocabulary baseline used no learned scene-conditioned ranker.
In both tables, each budget column reports closure / feasible share (%). Closure is the fraction of diagnosed state–direction pairs recovered within the first N candidates. Feasible share is the fraction of those candidates that pass the same geometric verifier σ after generation by staying within the drivable area and avoiding collisions with recorded agents. Bold indicates the best value for each metric within each split.
| Candidate source | @8 | @16 | @32 | @64 | @128 | minADE@8 (m) |
|---|---|---|---|---|---|---|
| Train-200 | ||||||
| K = 4,096 | 40.2 / 30.6 | 55.9 / 32.3 | 64.6 / 30.3 | 76.8 / 30.2 | 87.4 / 28.1 | 3.50 |
| K = 8,192 | 33.9 / 31.4 | 46.5 / 30.1 | 61.4 / 28.0 | 76.8 / 27.4 | 86.6 / 27.9 | 4.12 |
| Ours + PSD | 44.1 / 47.3 | 61.4 / 47.1 | 71.3 / 45.5 | 84.3 / 47.6 | 90.9 / 46.7 | 1.91 |
| Val-153 | ||||||
| K = 4,096 | 52.8 / 46.3 | 72.7 / 46.7 | 83.5 / 43.0 | 96.0 / 41.5 | 98.3 / 38.7 | 3.16 |
| K = 8,192 | 35.2 / 47.1 | 46.0 / 43.5 | 69.9 / 39.8 | 89.2 / 37.9 | 97.2 / 38.2 | 3.24 |
| Ours + PSD | 70.5 / 58.7 | 75.6 / 56.7 | 88.1 / 55.1 | 93.2 / 56.6 | 96.0 / 55.1 | 2.09 |
| Held-out-66 | ||||||
| K = 4,096 | 56.5 / 36.9 | 70.6 / 39.3 | 80.0 / 33.0 | 90.6 / 34.0 | 92.9 / 30.1 | 3.00 |
| K = 8,192 | 63.5 / 45.3 | 76.5 / 40.9 | 80.0 / 34.3 | 92.9 / 30.2 | 98.8 / 30.4 | 3.13 |
| Ours + PSD | 43.5 / 53.2 | 70.6 / 51.6 | 83.5 / 50.8 | 91.8 / 53.6 | 95.3 / 54.0 | 1.57 |
To test how the choice of candidates within each direction affects coverage, we repeated the comparison with the same vocabularies and direction cycle, drawing prototypes in random order within each direction.
| Candidate source | @8 | @16 | @32 | @64 | @128 | minADE@8 (m) |
|---|---|---|---|---|---|---|
| Train-200 | ||||||
| K = 4,096 | 27.6 / 22.2 | 42.5 / 20.5 | 51.2 / 21.4 | 69.3 / 23.1 | 78.7 / 20.3 | 3.22 |
| K = 8,192 | 23.2 / 15.7 | 46.9 / 20.6 | 63.0 / 21.2 | 80.3 / 21.0 | 89.8 / 21.2 | 4.00 |
| Ours + PSD | 44.1 / 47.3 | 61.4 / 47.1 | 71.3 / 45.5 | 84.3 / 47.6 | 90.9 / 46.7 | 1.91 |
| Val-153 | ||||||
| K = 4,096 | 21.6 / 25.8 | 38.6 / 25.0 | 54.5 / 26.2 | 73.9 / 28.2 | 89.8 / 25.0 | 2.70 |
| K = 8,192 | 21.6 / 21.1 | 50.0 / 26.6 | 67.6 / 27.0 | 95.5 / 26.2 | 98.3 / 26.2 | 3.71 |
| Ours + PSD | 70.5 / 58.7 | 75.6 / 56.7 | 88.1 / 55.1 | 93.2 / 56.6 | 96.0 / 55.1 | 2.09 |
| Held-out-66 | ||||||
| K = 4,096 | 27.1 / 26.5 | 31.8 / 20.9 | 42.4 / 21.0 | 81.2 / 22.5 | 89.4 / 19.7 | 2.99 |
| K = 8,192 | 31.8 / 17.0 | 61.2 / 23.8 | 69.4 / 23.7 | 82.4 / 22.1 | 95.3 / 22.7 | 3.31 |
| Ours + PSD | 43.5 / 53.2 | 70.6 / 51.6 | 83.5 / 50.8 | 91.8 / 53.6 | 95.3 / 54.0 | 1.57 |
At N = 8, frequency ordering yielded higher vocabulary closure than random ordering for both vocabulary sizes on all three splits. Ours + PSD achieved the highest closure@8 on Train-200 and Val-153, while the frequency-ordered vocabulary led on Held-out-66. For example, at N = 128 on Held-out-66, the frequency-ordered K = 8,192 vocabulary recovered more holes than Ours + PSD (98.8% vs. 95.3%), but a larger share of its candidates failed the verifier (69.6% vs. 46.0%). At the same candidate budget, a dense vocabulary can match or exceed Ours + PSD in directional coverage. At every tested budget on all three splits, Ours + PSD achieved a higher verifier pass rate than either dense vocabulary under both frequency and random ordering.
Dear Reviewer SbPJ,
Thank you for asking how our coverage diagnosis differs from CLOVER's supervision and for suggesting a matched-budget comparison with dense trajectory vocabularies. We will address your concerns and questions in the response below.
[W1 & Q1] Difference from CLOVER
We appreciate the opportunity to clarify how our directional coverage metric differs from CLOVER's coverage objective.
CLOVER uses set-level coverage as a training objective, whereas we measure directional coverage at a fixed sampling budget. The difference is as follows.
CLOVER measures how close proposals are, not which directions are absent. CLOVER's pseudo-expert loss averages the L1 distance from each pseudo-expert to its nearest proposal (Sec. 3.2, Eq. 4). Its Oracle@64 PDMS reports the highest score among 64 proposals. A scene can therefore receive a high oracle score from a straight trajectory even if the planner generates no feasible left turn.
Our metric measures which feasible directions are missing at a fixed budget. We test for precisely this kind of omission. A verified cross-scene witness establishes that a left turn satisfies σ in the current scene. If no reference sample satisfies σ in that direction within the diagnosis budget, we record a coverage hole. Any σ-feasible left turn can recover that hole, even if its path differs from the witness. Closure@N measures the fraction of diagnosed state–direction pairs for which the planner generates such a candidate within N samples.