acceptodds
Under review as a conference paper at ICLR 2027

WMKD: Planning-Oriented World Model via Future-Guided Knowledge Distillation for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving requires anticipating how the surrounding scene will evolve to make safe and effective planning decisions. World models provide this context by learning to predict future scene evolution, yet a fundamental question remains: "What should a world model predict for planning?" Existing approaches largely formulate world modeling as reconstructing future observations or scene representations, often without explicitly modeling how these targets should be constructed to support the current planning decision. To this end, we present World Model Knowledge Distillation (WMKD), a teacher–student framework in which a future-guided teacher constructs a planning-oriented prediction target and a student learns to predict it from sensor observations. The teacher encodes ground-truth future semantic evolution under joint supervision from future semantic reconstruction and ego planning, and quantizes the resulting scene representation into discrete codes. The student's Discrete World Model (DWM) predicts these codes to anticipate planning-relevant future knowledge without privileged inputs at inference. To incorporate the predicted representation into planning, the Residual Fusion Module (RFM) fuses it with the observed BEV and their residual, while Planning Response Distillation (PRD) transfers the teacher's planning responses. The Trajectory Repropagation Planner (TRP) further re-encodes refined trajectory geometry to guide successive refinement. Extensive experiments across NAVSIM and Bench2Drive validate WMKD, achieving 92.0 PDMS and 90.6 EPDMS on NAVSIM, as well as a Driving Score (DS) of 87.56 and a Success Rate (SR) of 72.27% on Bench2Drive.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End autonomous driving, world model, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Y77cOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WMKD, a teacher–student framework that learns future prediction targets for end-to-end autonomous driving. A privileged teacher encodes ground-truth future semantic BEV sequences under joint planning and semantic-reconstruction supervision, producing discrete codes that a current-observation student learns to predict. The framework incorporates residual feature fusion and planning-response distillation, together with a Trajectory Repropagation Planner (TRP) that reconstructs planning queries from progressively refined trajectories. Experiments on NAVSIM and Bench2Drive demonstrate the effectiveness of the complete framework.

Critical strengths

1.The target-construction approach is well motivated. Jointly training the privileged representation for future semantic reconstruction and current planning provides a concrete way to incorporate planning supervision into the world-model target. The student operates without privileged future inputs at inference.

2.The empirical evaluation is reasonably comprehensive. The paper includes both NAVSIM and interactive closed-loop Bench2Drive results, alongside inference-efficiency measurements. The appendix provides useful controls for teacher supervision, privileged timesteps, continuous versus discrete prediction targets, codebook size, and planner architecture.

Critical weaknesses

1.The proposed joint planning and segmentation supervision improves PDMS by only 0.3 over segmentation-only supervision (Table A3). This marginal gain does not convincingly substantiate the paper's central claim that planning-oriented targets are substantially more effective than conventional future semantic representations.

2.TRP alone improves PDMS from 84.7 to 90.3, while the full WMKD achieves 91.3 (Table A5). Most of the improvement thus comes from trajectory refinement rather than world modeling, weakening the significance of the claimed core contribution.

3.The predicted trajectories are represented only by (x, y) coordinates, without explicit heading information. Could the authors clarify how vehicle orientations are recovered for NAVSIM evaluation?

Questions for authors

1.How does the teacher reconstruct temporally resolved future semantic maps from a single aggregated BEV representation? Please clarify the decoder architecture and temporal supervision.

2.Why is the predicted representation reconstructed using softmax-weighted codebook embeddings rather than discrete code selection? How does this choice affect planning performance?

3.In Eq. (5), does the Global Multi-Head Cross-Attention (G-MHCA) operate on the current BEV features at a spatial resolution of 8×8 or 64×64? Please clarify the feature resolution used for cross-attention. Is WMKD built upon WoTE?

11 comments
Reviewer_Y77cOpenReview ↗

Suggestion for Rebuttal

1.The authors do not need to conduct additional experiments on Bench2Drive, as I have removed this request.

2.I recently spent some time reproducing the TRP module and noticed a potential issue. The PDMS scores are precomputed offline using WoTE's trajectory anchors, but these anchors are updated during training. Could this lead to a mismatch between the PDMS scores and the updated trajectories? Additionally, why did the authors not explore query-based trajectory generation, as adopted in iPad?

3.If these concerns are adequately addressed, I would be happy to raise my score!

AuthorsOpenReview ↗

Response to Reviewer Y77c - Part 1

We sincerely thank the reviewer Y77c for the constructive feedback and for the follow-up suggestions. In this response, we (1) provide matched comparisons that show the effect of the planning-oriented discrete target (Tables R1 to R3), (2) separate the contribution of the world model from that of the planner, (3) clarify heading prediction, the reconstruction decoder, soft codebook reconstruction (Table R4), the G-MHCA resolution, and the relationship to WoTE, and (4) answer the follow-up questions on the offline PDMS scores with refined anchors and on query-based trajectory generation. We hope these additional results and clarifications address the reviewer's concerns.

AuthorsOpenReview ↗

Response to Reviewer Y77c - Part 2

[W1] Contributions of Planning-Oriented World Model Target

We thank the reviewer for raising this concern. We examine the proposed target (i) against conventional future semantic representations under the full WMKD architecture and (ii) against a segmentation-only codebook in a matched setting.

1. Discrete codes vs. future BEV labels and continuous latents

Setting. We keep TRP, RFM, and PRD fixed and change only the world-model target; the same jointly supervised teacher provides PRD in every row. Row (a) keeps the identical architecture but removes world-model prediction and future BEV reconstruction supervision.

Results. Future BEV Label (b) gives no gain over (a) (both 90.7 PDMS), and Continuous Latent (c) is slightly worse (90.5). Only our planning-oriented codebook (d) improves, reaching 91.3 PDMS (+0.6). Since the planning architecture is identical, this gain comes from our discrete world-model target.

Table R1: Ablation of world-model prediction targets under the full WMKD architecture on NAVSIM.

2. Planning-supervised codes vs. a segmentation-only codebook

Setting. In Table A3, the segmentation-only teacher had no PRD, since it has no planning outputs to distill, so the 0.3 PDMS gap also includes the effect of PRD. We therefore compare the two teachers with PRD disabled on both sides (Table R2 (a) vs. (b)).

NAVSIM: the gain is concentrated in turning scenes. The average gain is small (91.0 to 91.1 PDMS). When split by driving command, both reach 91.6 PDMS in straight driving (8,070 scenes, 66%), while in turning scenes (4,076) planning supervision improves PDMS by +0.5 (89.7 to 90.2) and DAC by +0.6 (96.8 to 97.4). Straight driving mostly requires lane keeping and car following, so reconstructing future semantics may suffice. In contrast, turning is more challenging for end-to-end planners [S1, S2, S3]: it is rarer in driving data [S1, S2] and requires quick, accurate direction changes [S3], whereas straight driving can often be handled by extending the current motion [S1]. In a turn, the ego path often crosses other agents' paths, and which part of the future scene matters depends on the intended path. A reconstruction-only target treats all regions equally, whereas a planning-shaped target keeps the information relevant to the intended path.

Planning supervision also enables PRD. With PRD, PDMS improves from 91.1 to 91.3 (Table R2 (b) vs. (c)), extending the gain to straight driving (91.6 to 91.8 PDMS, 98.0 to 98.3 DAC) while keeping it in turns.

Table R2: Ablation of privileged teacher tasks on NAVSIM, by driving command.

Scenario (#scenes)IDPlanSeg.PRDPDMS ↑NC ↑DAC ↑EP ↑TTC ↑
Turn (4,076)(a)-✓-89.798.796.882.298.0
(b)✓✓-90.298.797.482.697.9
(c)✓✓✓90.298.597.582.698.1
Straight (8,070)(a)-✓-91.698.497.985.797.8
(b)✓✓-91.698.298.085.697.7
(c)✓✓✓91.898.298.385.897.8
All (12,146)(a)-✓-91.098.597.584.697.9
(b)✓✓-91.198.497.884.597.8
(c)✓✓✓91.398.398.184.897.9

HUGSIM: the gain also holds in unseen closed-loop environments. On zero-shot HUGSIM, planning supervision improves average RC from 39.3 to 40.3 and HD-Score from 25.4 to 26.2 (Table R3 (a) vs. (b)), and PRD adds a further gain to 41.4 RC and 27.6 HD-Score ((b) vs. (c)). A segmentation-only codebook reconstructs the whole future scene, so its codes depend on the layouts seen in training. Planning supervision keeps the future information that affects the ego decision, which is shared across environments and transfers better. This matters because deployed systems meet unseen environments where retraining is often impractical.

Table R3: Ablation of privileged teacher tasks on the HUGSIM benchmark (zero-shot, closed-loop).

IDPlanSeg.PRDRCHD-Score
EMHXAvg.EMHXAvg.
(a)-✓-72.344.525.618.439.363.627.911.54.825.4
(b)✓✓-77.242.329.618.440.367.326.414.74.826.2
(c)✓✓✓78.644.730.917.141.468.629.116.24.127.6

Summary. With PRD disabled, planning supervision improves PDMS by +0.5 in turning scenes on NAVSIM and RC/HD-Score by +1.0/+0.8 on HUGSIM. Including PRD, which planning supervision enables, the total gain over the segmentation-only teacher is +0.3 PDMS on NAVSIM and +2.1 RC and +2.2 HD-Score on HUGSIM. We will include these results in the revised manuscript.

[S1] UAD: End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation, TPAMI 2026

[S2] S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation, CVPR 2025

[S3] MomAD: Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving, CVPR 2025

Reviewer_2NMEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WMKD investigates how to construct prediction targets for autonomous driving world models that support current planning decisions. A teacher learns discrete BEV targets from future semantic information under joint planning and reconstruction supervision. The student predicts these codes from current observations and combines residual fusion, response distillation, and trajectory re-encoding for planning. WMKD achieves 92.0 PDMS on NAVSIM and 87.56 DS on Bench2Drive.

Critical strengths

Joint future semantic reconstruction and planning supervision explicitly connects target construction to the driving task, making the problem practically relevant.

With ResNet-34, WMKD achieves 92.0 PDMS on NAVSIM v1, outperforming the reported DiffusionDrive and WoTE results with the same backbone and providing quantitative support for the overall framework.

The appendix provides extensive ablations beyond incrementally adding modules, including TRP-only, different teacher objectives, a current-frame-only teacher, codebook sizes, and query propagation strategies. These comparisons help examine the sources of improvement.

Critical weaknesses

TRP alone improves the baseline from 84.7 to 90.3 PDMS, while the complete WMKD framework reaches 91.3. A substantial share of the overall gain therefore comes from the planner, and the contribution of future-target design needs clearer attribution. Using a segmentation-only teacher already yields 91.0 PDMS, while adding planning supervision raises this to 91.3. This improvement does not fully establish the importance of a planning-oriented target. Moreover, the ablation does not clarify whether PRD remains active when the teacher has no planning supervision or how its planning responses are obtained, making it unclear whether the comparison conditions are matched.

PDMS sub-scores are computed offline for the initial trajectory anchors, but Equation (6) progressively changes their geometry, and inference selects the final refined trajectories. Collision, progress, and drivable-area labels for the initial anchors may no longer apply to the refined trajectories. Equation (17) also aligns teacher and student scores by candidate index, although their corresponding refined trajectories may differ. The paper should clarify whether trajectories are rescored, whether refinement magnitude is constrained, and how the scoring targets remain semantically consistent.

Section 3.1 describes the teacher’s semantic task as future semantic reconstruction, whereas the text following Equation (18) describes (L_{\mathrm{map}}) as an auxiliary perception objective shared by the teacher and student. It is unclear whether the teacher reconstructs all future frames, how frames are weighted, and whether future BEV maps are aligned to the current ego coordinate frame. The decoder used to produce the future sequences in Figure 4 is also insufficiently described.

The teacher uses nearest-neighbor hard quantization, while student planning uses probability-weighted codebook embeddings. These mixtures may lie between codewords, but the effects of this difference on prediction and planning are not evaluated separately.

Component ablations focus on NAVSIM, while the closed-loop experiments mainly report the complete model. The individual contributions of future targets, PRD, and TRP to interactive driving therefore remain unclear.

The contributions should be better summarized. The statement "Extensive experiments across NAVSIM benchmarks and the interactive closed-loop Bench2Drive benchmark validate the proposed framework, achieving 92.0 PDMS and 90.6 EPDMS on NAVSIM, as well as 87.56 DS and 72.27% SR on Bench2Drive." cannot be considered as a contribution. Conducting experiments is a means to verify the effectiveness of your technical innovations, not an end in itself.

Questions for authors

Could you clarify the attribution of gains to TRP, PRD, and WMKD? Please provide matched ablations and closed-loop evaluations that isolate the contributions of planning-oriented supervision and future-target design from improvements due to the planner.

Are refined trajectories rescored, and how is teacher–student scoring alignment ensured in Equation (17)? Please also specify which future frames are reconstructed, how their coordinates are aligned, and which decoder is used, and explain how these choices affect future semantic reconstruction.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 2NME - Part 1

We sincerely thank the reviewer 2NME for the careful and detailed feedback. In this response, we (1) separate the contribution of the world model from that of the planner (Table R1), (2) provide a matched comparison between segmentation-only and planning-oriented teacher supervision with PRD disabled, on NAVSIM and on zero-shot closed-loop HUGSIM (Tables R2 and R3), (3) clarify how the offline PDMS scores are used with trajectory refinement and how teacher and student scores are aligned, (4) clarify the reconstruction objective, the coordinate frame, and the decoder, (5) compare hard index selection with soft codebook reconstruction (Table R4), and (6) add component-wise closed-loop ablations on Bench2Drive and HUGSIM (Tables R5 and R6). We hope these additional results and clarifications address the reviewer's concerns.

AuthorsOpenReview ↗

Response to Reviewer 2NME - Part 2

[W1-1] Independent and Complementary Contributions of DWM and Planning Modules

We thank the reviewer for raising this concern. We first clarify that the DWM-only configuration (M1\mathcal{M}_1) in Table 4 uses single-trajectory planning, which we will state explicitly in the revision. Introducing DWM alone improves PDMS from 84.7 to 89.5 (+4.8). This single-trajectory model already reaches a higher PDMS than several recent multi-trajectory planners, including DiffusionDrive (88.1), SeerDrive (88.9), WoTE (88.3), ResWorld (89.0), ResAD (88.8), MeanFuser (89.0), and WorldRFT (87.0). DWM therefore improves planning independently of multi-trajectory generation and selection.

Meanwhile, multi-trajectory planning offers advantages in considering diverse driving hypotheses. This motivated us to design our planning modules, including TRP, so that the future knowledge learned by DWM can be used effectively across multiple trajectory candidates through trajectory refinement. As shown in Table A5, the multi-trajectory planner alone achieves 90.3 PDMS, and the complete WMKD framework reaches 91.3 PDMS (+1.0). We believe this gain is meaningful because little room for improvement remains at this level. The human reference on NAVSIM, which can be seen as a practical upper bound, is 94.8 PDMS, only 4.5 PDMS above the planner alone, and gains become harder to obtain as performance gets closer to this bound.

Finally, Table R1 compares different world-model prediction targets under the full WMKD architecture, where TRP, RFM, and PRD are fixed and only the target changes. We also add a no-prediction baseline (a), which keeps the identical architecture but removes world-model prediction and future BEV reconstruction supervision. Under this full architecture, Future BEV Label prediction gives the same performance as the model without world-model prediction supervision (both 90.7 PDMS), and Continuous Latent prediction is even lower (90.5 PDMS). In contrast, our target reaches 91.3 PDMS (+0.6). This suggests that predicting the future does not always lead to better planning, even with the same planning modules. These results show that our planning-oriented discrete world-model target is what allows the full architecture to benefit from future knowledge, and that the planning modules work best when combined with it.

Table R1: Ablation of world-model prediction targets under the full WMKD architecture on NAVSIM.

IDWorld-Model TargetPDMS ↑NC ↑DAC ↑EP ↑TTC ↑
(a)None90.798.397.784.197.7
(b)Future BEV Label90.798.397.684.397.7
(c)Continuous Latent90.598.297.583.997.6
(d)Codebook (Ours)91.398.398.184.897.9
AuthorsOpenReview ↗

Response to Reviewer 2NME - Part 3

[W1-2, Q1-1] Contributions of Planning-Oriented Teacher Supervision

We thank the reviewer for raising this concern. We first clarify the setting of Table A3 and then provide a matched comparison.

Setting. PRD is applied only when the teacher has planning supervision, since a segmentation-only teacher has no planning outputs to distill. In Table A3, the segmentation-only teacher was thus trained without PRD while the jointly supervised teacher used PRD, so the two rows were not matched, as the reviewer points out. We now compare the two teachers with PRD disabled on both sides (Table R2 (a) vs. (b)).

NAVSIM: the gain is concentrated in turning scenes. The average gain is small (91.0 to 91.1 PDMS). When split by driving command, both reach 91.6 PDMS in straight driving (8,070 scenes, 66%), while in turning scenes (4,076) planning supervision improves PDMS by +0.5 (89.7 to 90.2) and DAC by +0.6 (96.8 to 97.4). Straight driving mostly requires lane keeping and car following, so reconstructing future semantics may suffice. In contrast, turning is more challenging for end-to-end planners [S1, S2, S3]: it is rarer in driving data [S1, S2] and requires quick, accurate direction changes [S3], whereas straight driving can often be handled by extending the current motion [S1]. In a turn, the ego path often crosses other agents' paths, and which part of the future scene matters depends on the intended path. A reconstruction-only target treats all regions equally, whereas a planning-shaped target keeps the information relevant to the intended path.

Planning supervision also enables PRD. With PRD, PDMS improves from 91.1 to 91.3 (Table R2 (b) vs. (c)), extending the gain to straight driving (91.6 to 91.8 PDMS, 98.0 to 98.3 DAC) while keeping it in turns.

Table R2: Ablation of privileged teacher tasks on NAVSIM, by driving command.

Scenario (#scenes)IDPlanSeg.PRDPDMS ↑NC ↑DAC ↑EP ↑TTC ↑
Turn (4,076)(a)-✓-89.798.796.882.298.0
(b)✓✓-90.298.797.482.697.9
(c)✓✓✓90.298.597.582.698.1
Straight (8,070)(a)-✓-91.698.497.985.797.8
(b)✓✓-91.698.298.085.697.7
(c)✓✓✓91.898.298.385.897.8
All (12,146)(a)-✓-91.098.597.584.697.9
(b)✓✓-91.198.497.884.597.8
(c)✓✓✓91.398.398.184.897.9

HUGSIM: the gain also holds in unseen closed-loop environments. On zero-shot HUGSIM, planning supervision improves average RC from 39.3 to 40.3 and HD-Score from 25.4 to 26.2 (Table R3 (a) vs. (b)), and PRD adds a further gain to 41.4 RC and 27.6 HD-Score ((b) vs. (c)). A segmentation-only codebook reconstructs the whole future scene, so its codes depend on the layouts seen in training. Planning supervision keeps the future information that affects the ego decision, which is shared across environments and transfers better. This matters because deployed systems meet unseen environments where retraining is often impractical.

Table R3: Ablation of privileged teacher tasks on the HUGSIM benchmark (zero-shot, closed-loop).

IDPlanSeg.PRDRCHD-Score
EMHXAvg.EMHXAvg.
(a)-✓-72.344.525.618.439.363.627.911.54.825.4
(b)✓✓-77.242.329.618.440.367.326.414.74.826.2
(c)✓✓✓78.644.730.917.141.468.629.116.24.127.6

Summary. With PRD disabled, planning supervision improves PDMS by +0.5 in turning scenes on NAVSIM and RC/HD-Score by +1.0/+0.8 on HUGSIM. Including PRD, which planning supervision enables, the total gain over the segmentation-only teacher is +0.3 PDMS on NAVSIM and +2.1 RC and +2.2 HD-Score on HUGSIM. We will include these results in the revised manuscript.

[S1] UAD: End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation, TPAMI 2026

[S2] S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation, CVPR 2025

[S3] MomAD: Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving, CVPR 2025

Reviewer_2CuvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what an auxiliary world model inside an end-to-end driving planner should predict, and answers with a teacher–student scheme, WMKD. The paper designs a teacher that utilises privileged input from future frames. The teacher uses a codebook to quantise the BEV for downstream tasks. The student is additionally trained by using the 2 distillation from the teacher.

Critical strengths

What a planning world model should predict is a real design question. It shows strong numbers on two benchmarks. The ablation study is broad and clearly explains each module's effect.

Critical weaknesses

The empirical evidence for the central "planning-oriented world-model target" claim is weaker than the overall system performance suggests. While WMKD substantially outperforms the baseline, the gain specifically attributable to the proposed planning-oriented discrete code target appears modest (89.5 PDMS vs 89.2 for Future BEV Segmentation in Table 5). The current ablations demonstrate that future knowledge is valuable, but do not clearly establish that planning-oriented discrete codes are the critical factor.

Moreover, the strongest architectural contribution appears to be the TRP planner and associated planning modules, whose improvements are substantial. It remains unclear how much of the final performance originates from the proposed planning-oriented target construction versus improvements in planning architecture.

The paper would be strengthened by comparing the proposed target against closer alternatives, including (1) continuous planning-oriented latent targets without quantisation and (2) reconstruction-only VQ targets without planning supervision, evaluated within the full WMKD system. Such experiments would directly test the central hypothesis that planning-oriented discrete targets are superior, rather than merely demonstrating that future knowledge is useful.

Questions for authors

Which semantic classes and which coordinate frame do the privileged maps ST:T+NS_{T:T+N} use? Do they contain the ego vehicle's future footprint, or are future maps rendered in the future ego frame? If so, what do the teacher and the student score with the ego removed and all maps in the current ego frame?

With TRP, RFM and PRD fixed, what does the student reach when DWM predicts (a) future BEV segmentation, (b) codes from a segmentation-only codebook, or (c) a continuous planning-shaped latent, instead of the planning-supervised codes?

Does LmapL_map in the teacher objective reconstruct the future frames T+1…T+NT+1…T+N, or only the current frame? Is PRD kept on for the segmentation-only teacher in Table A3(b), and if so, which planning outputs does that teacher provide?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2Cuv - Part 1

We sincerely thank the reviewer 2Cuv for the constructive and detailed feedback. In this response, we (1) compare our planning-oriented discrete codes against future BEV labels, continuous latents, and a segmentation-only codebook under the full WMKD setting (Tables R1 and R2), (2) show where planning-oriented teacher supervision is beneficial, through a per-command breakdown on NAVSIM and zero-shot closed-loop evaluation on HUGSIM (Tables R2 and R3), (3) separate the contribution of the world model from that of the planner, and (4) clarify the privileged map representation, the reconstruction objective, and the PRD setting. We hope these additional results and clarifications address the reviewer's concerns.

AuthorsOpenReview ↗

Response to Reviewer 2Cuv - Part 2

[W1, W3, Q2] Effect of the Planning-Oriented Discrete World-Model Target (1/2)

We thank the reviewer for raising this concern. The reviewer's question has two parts: (i) whether discrete codes are better than other target representations, and (ii) whether planning supervision adds value over a reconstruction-only codebook. Following the reviewer's suggestion, we test both within the full WMKD system.

1. Discrete codes vs. future BEV labels and continuous latents

Setting. We keep TRP, RFM, and PRD fixed and change only the world-model prediction target. The same jointly supervised teacher provides the planning responses for PRD in every configuration. We also add a no-prediction baseline, Table R1 (a), which keeps the identical architecture but removes world-model prediction and future BEV reconstruction supervision, so the world-model network only acts as a feature refiner.

Results. As shown in Table R1, Future BEV Label prediction (b) gives no improvement over the no-prediction baseline (both 90.7 PDMS), and Continuous Latent prediction (c) is slightly worse (90.5 PDMS). Only our planning-oriented discrete codebook (d) improves over the baseline, reaching 91.3 PDMS (+0.6), which is also +0.6 over Future BEV Labels and +0.8 over Continuous Latents. Since the planning architecture is identical in all rows, this gain can be attributed to the prediction target. These results show that our discrete representation is an effective world-model target for planning.

Table R1: Ablation of world-model prediction targets under the full WMKD architecture on NAVSIM.

IDWorld-Model TargetPDMS ↑NC ↑DAC ↑EP ↑TTC ↑
(a)None90.798.397.784.197.7
(b)Future BEV Label90.798.397.684.397.7
(c)Continuous Latent90.598.297.583.997.6
(d)Codebook (Ours)91.398.398.184.897.9

2. Planning-supervised codes vs. a segmentation-only codebook

Setting. We compare a segmentation-only teacher with a jointly supervised teacher, with PRD disabled on both sides (Table R2 (a) vs. (b)). PRD must be disabled for this comparison because a segmentation-only teacher has no planning outputs to distill.

NAVSIM: the gain is concentrated in turning scenes. On average, planning-oriented supervision is slightly better (91.1 vs. 91.0 PDMS), but this average hides a difference. Table R2 splits navtest by driving command into turning scenes (left and right turns, 4,076 scenes) and straight-driving scenes (8,070 scenes). In straight driving, which makes up 66% of navtest, both targets reach 91.6 PDMS. In turning scenes, planning-oriented supervision improves PDMS by +0.5 (89.7 to 90.2), with higher drivable-area compliance (96.8 to 97.4 DAC, +0.6).

This matches the purpose of planning-oriented supervision. Straight driving mostly requires keeping the lane and following the vehicle ahead, so reconstructing future semantics may already be enough. In contrast, turning scenes are more challenging for end-to-end planners, as reported in UAD [S1], S4-Driver [S2], and MomAD [S3]. Turning is much less frequent than straight driving in driving data [S1, S2], and it requires the ego vehicle to adjust its direction quickly and accurately [S3], whereas straight driving can often be handled by simply extending the current motion [S1]. This is where planning supervision becomes important. In a turn, the ego path changes and often crosses the paths of other agents, so the planner has to consider interactions with them, and which part of the future scene matters depends on where the ego vehicle intends to go. A reconstruction-only target treats all regions of the future scene equally, whereas a target shaped by the planning objective keeps the future information that is relevant to the intended path.

Planning supervision also makes PRD possible, which is not available with a segmentation-only teacher. With PRD, PDMS further improves from 91.1 to 91.3 and DAC from 97.8 to 98.1 (Table R2 (b) vs. (c)). As shown in Table R2, PRD extends the benefit to straight driving (91.6 to 91.8 PDMS, 98.0 to 98.3 DAC) while keeping the gain in turning scenes. As a result, planning-oriented supervision with PRD yields consistent gains over the segmentation-only teacher in both turning scenes (+0.5 PDMS) and straight driving (+0.2 PDMS).

Table R2: Ablation of privileged teacher tasks on NAVSIM, by driving command.

Scenario (#scenes)IDPlanSeg.PRDPDMS ↑NC ↑DAC ↑EP ↑TTC ↑
Turn (4,076)(a)-✓-89.798.796.882.298.0
(b)✓✓-90.298.797.482.697.9
(c)✓✓✓90.298.597.582.698.1
Straight (8,070)(a)-✓-91.698.497.985.797.8
(b)✓✓-91.698.298.085.697.7
(c)✓✓✓91.898.298.385.897.8
All (12,146)(a)-✓-91.098.597.584.697.9
(b)✓✓-91.198.497.884.597.8
(c)✓✓✓91.398.398.184.897.9
AuthorsOpenReview ↗

Response to Reviewer 2Cuv - Part 3

[W1, W3, Q2] Effect of the Planning-Oriented Discrete World-Model Target (2/2)

HUGSIM: the gain also holds in unseen closed-loop environments. We evaluate the NAVSIM-trained models on HUGSIM in a zero-shot closed-loop setting. Here, planning-oriented supervision increases average RC from 39.3 to 40.3 and average HD-Score from 25.4 to 26.2 over the segmentation-only target (Table R3 (a) vs. (b)). PRD adds a further gain here as well, improving average RC from 40.3 to 41.4 and average HD-Score from 26.2 to 27.6 (Table R3 (b) vs. (c)).

We believe planning-oriented supervision is especially useful in unseen environments. A segmentation-only codebook is trained to reconstruct the whole future scene, so its codes depend on the scene layouts seen during training, which differ in a new environment. In contrast, planning supervision trains the codebook to keep the future information that affects the ego decision, which is shared across driving environments and therefore transfers better.

This zero-shot result is important because a deployed driving system is likely to meet environments that are not covered by its training data, and retraining for each of them is often impractical. A world-model target is therefore more valuable when its benefit remains in unseen environments. The gain on HUGSIM shows that planning-oriented supervision gives the student future knowledge that generalizes to unseen domains.

Table R3: Ablation of privileged teacher tasks on the HUGSIM benchmark (zero-shot, closed-loop).

IDPlanSeg.PRDRCHD-Score
EMHXAvg.EMHXAvg.
(a)-✓-72.344.525.618.439.363.627.911.54.825.4
(b)✓✓-77.242.329.618.440.367.326.414.74.826.2
(c)✓✓✓78.644.730.917.141.468.629.116.24.127.6

Summary. With PRD disabled on both sides, planning-oriented teacher supervision improves PDMS by +0.5 in turning scenes on NAVSIM and improves average RC by +1.0 and HD-Score by +0.8 on zero-shot HUGSIM. Planning supervision also makes PRD possible, which brings the total gain over the segmentation-only teacher to +0.3 PDMS on NAVSIM and +2.1 RC and +2.2 HD-Score on HUGSIM. We will include these results and clarify the corresponding settings in the revised manuscript.

[S1] UAD: End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation, TPAMI 2026

[S2] S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation, CVPR 2025

[S3] MomAD: Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving, CVPR 2025

Comments on the paper

2 comments
Area_Chair_bpg2OpenReview ↗

Please Review the Author Rebuttal and Reassess Your Rating

Dear Reviewers,

Thank you for your time and effort in reviewing the submissions.

The authors have now submitted their rebuttals. I kindly ask you to carefully review the authors' responses and assess whether they have adequately addressed your questions and concerns.

If the rebuttal resolves your concerns, please consider updating your review and adjusting your rating accordingly. If any concerns remain, please feel free to raise them in the discussion.

Your timely feedback and active participation are greatly appreciated, as they are essential to reaching a fair, well-informed decision.

Thank you again for your valuable contributions!

Best, Your AC

AuthorsOpenReview ↗

General Response

We would first like to express our gratitude to the Program Chairs, Senior Area Chairs, and Area Chairs for their efforts, as well as to the reviewers for their careful reading and insightful comments on our paper.

We appreciate the reviewers for highlighting the strengths of our work:

  • A well-motivated question and target construction. The reviewers noted that what a planning world model should predict is a real design question, and that jointly training the privileged representation for future semantic reconstruction and current planning is a concrete way to connect the world-model target to the driving task.

  • Strong results on benchmarks. With a ResNet-34 backbone, WMKD achieves 92.0 PDMS on NAVSIM, and it is also evaluated in interactive closed-loop driving on Bench2Drive, together with inference-efficiency measurements.

  • Broad ablation studies. The reviewers found that the ablations clearly explain the effect of each module, including controls for teacher supervision, privileged timesteps, continuous versus discrete targets, codebook size, and planner architecture.

Based on the reviewers' constructive feedback, we conducted additional experiments and provided clarifications during the rebuttal. We summarize them below and incorporate them into the revised manuscript.

  • Benefit of planning-oriented supervision over segmentation-only supervision. Planning-oriented supervision is more effective in interactive scenarios such as turning, where it improves PDMS from 89.7 to 90.2 on NAVSIM, and in the zero-shot setting, where it improves average RC and HD-Score from 39.3 and 25.4 to 40.3 and 26.2 on HUGSIM.

  • Contribution of the world model and its discrete target. The world model alone improves PDMS from 84.7 to 89.5 with single-trajectory planning. With the full planner, simply predicting the future does not lead to better planning: Future BEV Label and Continuous Latent prediction reach 90.7 and 90.5 PDMS, the same as or lower than the model without world-model prediction (90.7). When combined with our discrete representation, the full planner reaches 91.3 PDMS.

  • Effect of soft codebook reconstruction. We use softmax-weighted reconstruction to keep the uncertainty of the student's future prediction and to give a more stable gradient flow from the planning objectives to the world model. It improves PDMS from 90.3 to 91.3 over hard index selection.

  • Contribution of each component in closed-loop driving. We also evaluated each component in closed-loop driving on Bench2Drive and on zero-shot HUGSIM, and the results show a trend consistent with that on NAVSIM.

  • Consistency of trajectory scoring. We clarified that our formulation extends the existing anchor-based score distillation to a teacher–student framework, where teacher and student scores are aligned by anchor index.

  • Details of the method. We clarified the privileged map representation, the reconstruction objective, the decoder architecture, heading prediction, and the PRD setting for each teacher.

We hope our responses have addressed the concerns raised, and we would be glad to discuss any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.