- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WMKD, a teacher–student framework that learns future prediction targets for end-to-end autonomous driving. A privileged teacher encodes ground-truth future semantic BEV sequences under joint planning and semantic-reconstruction supervision, producing discrete codes that a current-observation student learns to predict. The framework incorporates residual feature fusion and planning-response distillation, together with a Trajectory Repropagation Planner (TRP) that reconstructs planning queries from progressively refined trajectories. Experiments on NAVSIM and Bench2Drive demonstrate the effectiveness of the complete framework.
Critical strengths
1.The target-construction approach is well motivated. Jointly training the privileged representation for future semantic reconstruction and current planning provides a concrete way to incorporate planning supervision into the world-model target. The student operates without privileged future inputs at inference.
2.The empirical evaluation is reasonably comprehensive. The paper includes both NAVSIM and interactive closed-loop Bench2Drive results, alongside inference-efficiency measurements. The appendix provides useful controls for teacher supervision, privileged timesteps, continuous versus discrete prediction targets, codebook size, and planner architecture.
Critical weaknesses
1.The proposed joint planning and segmentation supervision improves PDMS by only 0.3 over segmentation-only supervision (Table A3). This marginal gain does not convincingly substantiate the paper's central claim that planning-oriented targets are substantially more effective than conventional future semantic representations.
2.TRP alone improves PDMS from 84.7 to 90.3, while the full WMKD achieves 91.3 (Table A5). Most of the improvement thus comes from trajectory refinement rather than world modeling, weakening the significance of the claimed core contribution.
3.The predicted trajectories are represented only by (x, y) coordinates, without explicit heading information. Could the authors clarify how vehicle orientations are recovered for NAVSIM evaluation?
Questions for authors
1.How does the teacher reconstruct temporally resolved future semantic maps from a single aggregated BEV representation? Please clarify the decoder architecture and temporal supervision.
2.Why is the predicted representation reconstructed using softmax-weighted codebook embeddings rather than discrete code selection? How does this choice affect planning performance?
3.In Eq. (5), does the Global Multi-Head Cross-Attention (G-MHCA) operate on the current BEV features at a spatial resolution of 8×8 or 64×64? Please clarify the feature resolution used for cross-attention. Is WMKD built upon WoTE?
11 comments
Response to Reviewer Y77c - Part 1
We sincerely thank the reviewer Y77c for the constructive feedback and for the follow-up suggestions. In this response, we (1) provide matched comparisons that show the effect of the planning-oriented discrete target (Tables R1 to R3), (2) separate the contribution of the world model from that of the planner, (3) clarify heading prediction, the reconstruction decoder, soft codebook reconstruction (Table R4), the G-MHCA resolution, and the relationship to WoTE, and (4) answer the follow-up questions on the offline PDMS scores with refined anchors and on query-based trajectory generation. We hope these additional results and clarifications address the reviewer's concerns.
Response to Reviewer Y77c - Part 2
[W1] Contributions of Planning-Oriented World Model Target
We thank the reviewer for raising this concern. We examine the proposed target (i) against conventional future semantic representations under the full WMKD architecture and (ii) against a segmentation-only codebook in a matched setting.
1. Discrete codes vs. future BEV labels and continuous latents
Setting. We keep TRP, RFM, and PRD fixed and change only the world-model target; the same jointly supervised teacher provides PRD in every row. Row (a) keeps the identical architecture but removes world-model prediction and future BEV reconstruction supervision.
Results. Future BEV Label (b) gives no gain over (a) (both 90.7 PDMS), and Continuous Latent (c) is slightly worse (90.5). Only our planning-oriented codebook (d) improves, reaching 91.3 PDMS (+0.6). Since the planning architecture is identical, this gain comes from our discrete world-model target.
Table R1: Ablation of world-model prediction targets under the full WMKD architecture on NAVSIM.
2. Planning-supervised codes vs. a segmentation-only codebook
Setting. In Table A3, the segmentation-only teacher had no PRD, since it has no planning outputs to distill, so the 0.3 PDMS gap also includes the effect of PRD. We therefore compare the two teachers with PRD disabled on both sides (Table R2 (a) vs. (b)).
NAVSIM: the gain is concentrated in turning scenes. The average gain is small (91.0 to 91.1 PDMS). When split by driving command, both reach 91.6 PDMS in straight driving (8,070 scenes, 66%), while in turning scenes (4,076) planning supervision improves PDMS by +0.5 (89.7 to 90.2) and DAC by +0.6 (96.8 to 97.4). Straight driving mostly requires lane keeping and car following, so reconstructing future semantics may suffice. In contrast, turning is more challenging for end-to-end planners [S1, S2, S3]: it is rarer in driving data [S1, S2] and requires quick, accurate direction changes [S3], whereas straight driving can often be handled by extending the current motion [S1]. In a turn, the ego path often crosses other agents' paths, and which part of the future scene matters depends on the intended path. A reconstruction-only target treats all regions equally, whereas a planning-shaped target keeps the information relevant to the intended path.
Planning supervision also enables PRD. With PRD, PDMS improves from 91.1 to 91.3 (Table R2 (b) vs. (c)), extending the gain to straight driving (91.6 to 91.8 PDMS, 98.0 to 98.3 DAC) while keeping it in turns.
Table R2: Ablation of privileged teacher tasks on NAVSIM, by driving command.
| Scenario (#scenes) | ID | Plan | Seg. | PRD | PDMS ↑ | NC ↑ | DAC ↑ | EP ↑ | TTC ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Turn (4,076) | (a) | - | ✓ | - | 89.7 | 98.7 | 96.8 | 82.2 | 98.0 |
| (b) | ✓ | ✓ | - | 90.2 | 98.7 | 97.4 | 82.6 | 97.9 | |
| (c) | ✓ | ✓ | ✓ | 90.2 | 98.5 | 97.5 | 82.6 | 98.1 | |
| Straight (8,070) | (a) | - | ✓ | - | 91.6 | 98.4 | 97.9 | 85.7 | 97.8 |
| (b) | ✓ | ✓ | - | 91.6 | 98.2 | 98.0 | 85.6 | 97.7 | |
| (c) | ✓ | ✓ | ✓ | 91.8 | 98.2 | 98.3 | 85.8 | 97.8 | |
| All (12,146) | (a) | - | ✓ | - | 91.0 | 98.5 | 97.5 | 84.6 | 97.9 |
| (b) | ✓ | ✓ | - | 91.1 | 98.4 | 97.8 | 84.5 | 97.8 | |
| (c) | ✓ | ✓ | ✓ | 91.3 | 98.3 | 98.1 | 84.8 | 97.9 |
HUGSIM: the gain also holds in unseen closed-loop environments. On zero-shot HUGSIM, planning supervision improves average RC from 39.3 to 40.3 and HD-Score from 25.4 to 26.2 (Table R3 (a) vs. (b)), and PRD adds a further gain to 41.4 RC and 27.6 HD-Score ((b) vs. (c)). A segmentation-only codebook reconstructs the whole future scene, so its codes depend on the layouts seen in training. Planning supervision keeps the future information that affects the ego decision, which is shared across environments and transfers better. This matters because deployed systems meet unseen environments where retraining is often impractical.
Table R3: Ablation of privileged teacher tasks on the HUGSIM benchmark (zero-shot, closed-loop).
| ID | Plan | Seg. | PRD | RC | HD-Score | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| E | M | H | X | Avg. | E | M | H | X | Avg. | ||||
| (a) | - | ✓ | - | 72.3 | 44.5 | 25.6 | 18.4 | 39.3 | 63.6 | 27.9 | 11.5 | 4.8 | 25.4 |
| (b) | ✓ | ✓ | - | 77.2 | 42.3 | 29.6 | 18.4 | 40.3 | 67.3 | 26.4 | 14.7 | 4.8 | 26.2 |
| (c) | ✓ | ✓ | ✓ | 78.6 | 44.7 | 30.9 | 17.1 | 41.4 | 68.6 | 29.1 | 16.2 | 4.1 | 27.6 |
Summary. With PRD disabled, planning supervision improves PDMS by +0.5 in turning scenes on NAVSIM and RC/HD-Score by +1.0/+0.8 on HUGSIM. Including PRD, which planning supervision enables, the total gain over the segmentation-only teacher is +0.3 PDMS on NAVSIM and +2.1 RC and +2.2 HD-Score on HUGSIM. We will include these results in the revised manuscript.
[S1] UAD: End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation, TPAMI 2026
[S2] S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation, CVPR 2025
[S3] MomAD: Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving, CVPR 2025
Suggestion for Rebuttal
1.The authors do not need to conduct additional experiments on Bench2Drive, as I have removed this request.
2.I recently spent some time reproducing the TRP module and noticed a potential issue. The PDMS scores are precomputed offline using WoTE's trajectory anchors, but these anchors are updated during training. Could this lead to a mismatch between the PDMS scores and the updated trajectories? Additionally, why did the authors not explore query-based trajectory generation, as adopted in iPad?
3.If these concerns are adequately addressed, I would be happy to raise my score!