acceptodds
Under review as a conference paper at ICLR 2027

PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving

Abstract

World models in end-to-end autonomous driving predict future scene evolution to provide foresight for trajectory planning. Existing methods mainly study how to predict the future and how to use it, but less often ask which future representation is actually most useful for planning. To this end, we propose PlanWAM, a Planning-Shaped World Action Model. The key idea is to let the planning task shape the future-state representation, so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning. Specifically, we first use a Temporal Register Pyramid to compress multi-frame historical information in a recency-aware manner, learning a compact history representation oriented toward future reasoning and planning. We then introduce a privileged future posterior branch that observes ground-truth future frames, and shape its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation. Hindsight-to-Foresight Distillation trains a prior branch that depends only on history to predict this future latent representation. The predicted future latent representation serves as planning context and guides trajectory generation and selection. PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations. Extensive experiments further demonstrate that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, World-Action Models, Latent World Models, Trajectory Planning, Predictive Representation Learning, Privileged Future Supervision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JCcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PlanWAM, a novel World Action Model for end-to-end autonomous driving. Instead of directly applying future prediction supervision to the world model, the authors use planning objectives to learn future representations that are more useful for trajectory planning. Specifically, they first introduce a Temporal Register Pyramid to compress historical observations, then propose a planning-shaped future representation learned from privileged future observations through planning supervision. A Hindsight-to-Foresight Distillation strategy is further introduced to transfer this representation to a history-only world model, whose predicted future features guide trajectory generation and selection. The proposed method demonstrates strong performance on NAVSIM and HUGSIM, validating its effectiveness in both open-loop and closed-loop evaluations.

Critical strengths
  1. Strong empirical performance. The authors evaluate the proposed method on multiple benchmarks, achieving state-of-the-art results on NAVSIM and HUGSIM, demonstrating strong planning capabilities.

  2. Clear presentation and detailed implementation. The paper is well-written and provides comprehensive methodological and implementation details, making the proposed framework easy to understand and reproduce. Extensive experimental comparisons and ablation studies are also provided.

  3. Effective methodological designs. The authors introduce a Temporal Register Pyramid for efficient historical information compression and a Hindsight-to-Foresight Distillation strategy for learning planning-shaped future representations. The effectiveness of these components is demonstrated through ablation studies.

Critical weaknesses
  1. Limited generalization of the planning-shaped future representation. The main motivation of this paper is to learn future representations that are more useful for planning through planning objectives. However, compared with generic future-feature supervision, the proposed planning-shaped representation only improves PDMS by 0.3 points. Moreover, the DrivoR baseline already achieves a relatively high PDMS of 93.1 with a trajectory scorer. It would be more convincing if the authors could demonstrate the effectiveness of the proposed representation on other planning architectures, such as VADv2, to further validate its generality.

  2. Limited improvement in trajectory generation. As shown in Table 6, introducing the predicted future latent into trajectory generation alone does not improve PDMS, while most of the improvement comes from trajectory selection. The failure analysis in Figure 5 also shows that the reduction in failures mainly comes from fewer selection errors. This raises the question of why the planning-shaped future representation provides limited benefits for trajectory generation, which seems to restrict the applicability of the proposed method.

  3. Lack of ablation studies on HUGSIM. Although the paper reports strong zero-shot closed-loop performance on HUGSIM, the ablation studies are mainly conducted on NAVSIM. Given the relatively saturated performance and limitations of NAVSIM, it would be valuable to provide additional ablations on HUGSIM, particularly comparing generic future representations with planning-shaped representations and examining their contributions to trajectory generation and selection. Such results would further strengthen the evidence for the effectiveness of the proposed method in closed-loop driving scenarios.

Questions for authors
  1. As shown in Table 6, conditioning trajectory generation on the predicted future latent does not improve the final PDMS. Could the authors clarify whether the future latent improves the quality or diversity of generated trajectory candidates, for example, in terms of oracle PDMS or best-of-K trajectory error?
9 comments
AuthorsOpenReview ↗

(1/3) Re: the Reviewer JCcM

We thank you for the careful reading of our paper and constructive comments in detail.

1. Effectiveness and Cross-Architecture Generalization of Planning-Shaped Future Representations (W1)

We thank the reviewer for this important suggestion. To further validate the effectiveness and transferability of planning-shaped representations, we conduct additional experiments on multi-seed stability, alternative future supervision objectives, and cross-architecture transfer.

(1) Performance gains and independent contributions of planning-shaped supervision.

First, to assess the stability of the 0.3 PDMS gain, we repeat the experiments with three random seeds. Generic Future Latent and Planning-Shaped Latent achieve 93.54 ± 0.05 and 93.82 ± 0.03 PDMS, respectively, yielding a consistent average improvement of 0.28 PDMS.

To further distinguish planning-shaped supervision from generic future supervision, we introduce a Future BEV Occupancy Prediction control following WoTE [1], which supervises future latents through predicted BEV occupancy. Under identical model configurations, we compare four supervision strategies:

Table R1. Future Representation Supervision Ablations.

Future RepresentationSupervisionPDMS (Mean ± Std)
Null Future–93.41 ± 0.03
Generic Future LatentL_scene93.54 ± 0.05
Future BEV-Shaped LatentL_bev93.63 ± 0.04
Planning-Shaped Latent (Ours)L_plan + L_score93.82 ± 0.03

Generic future-feature supervision and BEV occupancy supervision improve PDMS by 0.13 and 0.22, respectively, over Null Future. Planning-shaped supervision still achieves the best performance, suggesting that directly shaping future representations with planning objectives provides additional benefits beyond generic future supervision.

We also clarify that DrivoR [2] already achieves 93.1 PDMS, making a 0.3-point gain meaningful but potentially sensitive to architectural factors. The controlled comparisons above, which vary only the future supervision objective under identical model configurations, more directly isolate the contribution of planning-shaped representations.

(2) Cross-architecture transferability.

We agree that evaluation across planning architectures is important. Since PlanWAM adopts a trajectory generation-and-selection paradigm, we further integrate the proposed future representation learning mechanism into iPad [3], which follows the same paradigm but uses a different planning architecture. We compare the original iPad, generic future-feature supervision, and planning-shaped supervision:

Table R2. Transferability of Planning-Shaped Future Representations to iPad.

MethodFuture Latent SupervisionNAVSIM-v1 PDMS ↑HUGSIM HD-Score ↑
iPad–91.7 ± 0.0521.2 ± 0.9
PlanWAM-Generic Future (iPad)L_scene92.0 ± 0.0722.7 ± 1.3
PlanWAM (iPad)L_plan + L_score92.6 ± 0.0425.1 ± 1.0

Under the iPad architecture, planning-shaped supervision improves performance over generic future-feature supervision by 0.6 PDMS and 2.4 HD-Score, and over the original iPad by 0.9 PDMS and 3.9 HD-Score. These results indicate that the benefits of planning-shaped representations are not specific to PlanWAM's original planning architecture, but can transfer to other architectures following the same trajectory generation-and-selection paradigm.

Overall, these additional experiments provide further evidence for the independent contribution and transferability of planning-shaped future representations, through multi-seed stability, controlled supervision comparisons, and cross-architecture evaluation. We have incorporated these results and analyses into the revised manuscript.

AuthorsOpenReview ↗

(2/3) Re: the Reviewer JCcM

2. Does the Future Latent Improve the Quality or Diversity of Generated Trajectory Candidates? (W2 and Q1)

We thank the reviewer for raising this important question. To further examine the effect of future representations on trajectory generation, we conduct additional analyses of candidate quality and geometric diversity, including overall candidate quality, Oracle Top-K performance, and geometric dispersion.

(1) Candidate trajectory quality.

We evaluate all 64 candidate trajectories generated by DrivoR and PlanWAM, as reported in Table R3.

Table R3. Candidate Trajectory Quality Comparison.

MetricDrivoRPlanWAMGain
Mean PDMS over 64 Candidates79.2182.39+3.18
Oracle Top-1 PDMS98.9599.27+0.32
Oracle Top-5 Mean PDMS98.0998.67+0.58
Oracle Top-10 Mean PDMS97.1797.95+0.78

The results show that planning-shaped future representations improve the overall quality of generated candidates. Mean PDMS across all 64 candidates increases by 3.18 points, while Oracle Top-1, Top-5, and Top-10 mean PDMS improve by 0.32, 0.58, and 0.78 points, respectively. This indicates that the improvement extends beyond the best candidate to a broader set of proposals.

Regarding why generation-only conditioning in Table 6 does not improve final PDMS, DrivoR already achieves 98.95 Oracle Top-1 PDMS, leaving limited room to improve the best candidate. Moreover, the generation-only setting conditions the Planner on the predicted future latent without explicitly enhancing the Scorer. Thus, improving candidate quality alone does not guarantee better final trajectory selection or a higher PDMS.

(2) Candidate trajectory diversity.

We further measure the standard deviations of candidate endpoints, lateral positions, and headings, together with the nearest-neighbor endpoint distance, as shown in Table R4.

Table R4. Geometric Dispersion of Candidate Trajectories.

MetricDrivoRPlanWAM
Endpoint XY Std. (m)2.2412.142
Lateral Y Std. (m)1.0881.042
Heading Std. (°)3.152.52
Nearest-Neighbor Endpoint Distance (m)0.3540.348

The results show no increase in geometric diversity, consistent with our design objective. Our goal is to improve candidate quality through planning-shaped supervision rather than explicitly expand geometric coverage. The main benefit therefore lies in higher-quality candidates, not increased diversity. Further improving trajectory diversity remains a direction for future work.

We have incorporated these candidate quality and diversity analyses into the revised manuscript.

AuthorsOpenReview ↗

(3/3) Re: the Reviewer JCcM

3. Additional Ablation Studies on HUGSIM (W3)

We thank the reviewer for this valuable suggestion. Following the reviewer's recommendation, we conduct additional zero-shot closed-loop ablations on HUGSIM, comparing generic and planning-shaped future representations and examining their respective contributions to trajectory generation and selection. The results are reported below as means ± standard deviations over three random seeds.

Table R5. HUGSIM Ablation Study of Planning-Shaped Future Representations.

ConfigurationGenerationSelectionHD-Score ↑RC ↑
DrivoR Baseline——33.5 ± 1.145.3 ± 1.9
Generic Future Latent✓✓35.0 ± 1.247.6 ± 2.1
Planning-Shaped Latent (Generation Only)✓×34.6 ± 1.247.1 ± 2.0
Planning-Shaped Latent (Selection Only)×✓37.1 ± 1.151.3 ± 1.9
Planning-Shaped Latent (Full)✓✓38.7 ± 1.353.7 ± 2.0

First, planning-shaped future representations outperform generic future representations in closed-loop evaluation. With both generation and selection enabled, replacing generic future supervision with planning-shaped supervision improves HD-Score from 35.0 to 38.7 (+3.7) and RC from 47.6 to 53.7 (+6.1), demonstrating that the benefits of planning-shaped representations extend to closed-loop driving.

Second, both trajectory generation and selection benefit from planning-shaped future representations. Compared with the DrivoR baseline, conditioning generation alone improves HD-Score and RC by 1.1 and 1.8 points, respectively, while conditioning selection alone yields gains of 3.6 and 6.0 points. Combining both achieves the best performance, improving HD-Score by 4.1 and 1.6 points and RC by 6.6 and 2.4 points over the generation-only and selection-only settings, respectively. These results indicate that planning-shaped representations benefit both generation and selection, with their joint use yielding the strongest closed-loop performance.

Overall, these additional HUGSIM ablations further validate the advantages of planning-shaped supervision over generic future supervision and clarify its contributions to trajectory generation and selection. We have incorporated these results and analyses into the revised manuscript.

References

[1] Y. Li et al., "End-to-End Driving with Online Trajectory Evaluation via BEV World Model," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 27137–27146, 2025.

[2] E. Kirby et al., "Driving on Registers," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 32058–32069, 2026.

[3] K. Guo et al., "iPad: Iterative Proposal-Centric End-to-End Autonomous Driving," IEEE Robotics and Automation Letters (RA-L), vol. 11, no. 10, pp. 11142–11149, 2026.

Reviewer_euhyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Most recent world-action models study how to predict the future and how to use the prediction. PlanWAM studies a different question: which future representation should be predicted. The method uses a privileged teacher in latent space. A Temporal Register Pyramid compresses four frames from four cameras into 128 tokens. It uses [4, 4, 8, 16] queries per camera across four temporal levels, so recent frames get more queries. During training, a posterior branch sees ground-truth future frames and builds a 256-token future latent. This latent is not used for reconstruction. It is shaped only by the trajectory generation loss and the trajectory scoring loss. A prior branch sees history only. It is regressed onto the posterior latent with MSE and stop-gradient. At test time the posterior is removed. The prior prediction is concatenated with the history tokens and given to the planner and the scorer. Training has two stages. Stage two freezes the posterior and initialises some prior parameters with posterior weights. The latent loss weight is 0.05. Reported results: 93.8 PDMS on NAVSIM v1 navtest, 90.9 EPDMS on NAVSIM v2, and 38.7 HD-Score with 53.7 RC on HUGSIM in a zero-shot setting with no simulator fine-tuning.

Critical strengths

The closed-loop result is the main reason I lean towards acceptance. NAVSIM v1 and v2 are close to saturated. The difference between 93.8 and 93.1 for DrivoR is small and I do not weight it heavily. The HUGSIM result is different. The gap is 38.7 against 28.9 for Latent-WAM. That gap is large. Open-loop gains of this size often disappear in closed-loop evaluation, so seeing a closed-loop number matters here. The second strength is the problem framing. Most papers in this area group methods into pixel reconstruction, latent prediction, and future conditioning. PlanWAM asks which representation is useful for planning instead. Two implementation details matter here. The prior is initialised from the posterior. The prior queries and the posterior queries follow the same order over future time, camera view, and register slot. I think these two details are the reason the distillation reaches 0.93 cosine similarity.

Critical weaknesses

The effect sizes are the first problem. Every ablation that supports the main claim changes PDMS by 0.1 to 0.3. The paper reports no error bars, no standard deviation, and no multiple seeds. In Table 5, the planning-shaped latent beats generic future-feature supervision by 0.3 PDMS. The main claim of the paper depends on this single 0.3 value. The second problem is that the ablation tables do not agree with each other. Table 4 uses a base model at 93.1. Table 5 uses a null-future model at 93.4. Table 8 uses a prior-only model at 93.4. Table 4 reports 93.5 for adding H2F, and Table 8 reports 93.7 for the same component. I had to check the three tables several times to find out which rows use Foresight-Conditioned Planning. The ablation set also misses the control I want most: a shaping objective that is neither planning nor reconstruction. Future agent motion, occupancy, or BEV semantics would work. Without it, I cannot tell whether planning-shaping is the cause of the 0.3 gain.

The HUGSIM baseline set is the second main weakness. It contains UniAD, VAD, LTF, GTRS-Dense, and Latent-WAM. Four of these are from 2023 or earlier. DiffusionDrive, DrivoR, and DriveFuture all appear in the NAVSIM tables, and none of them appear in the HUGSIM table. The HUGSIM result is the main generalisation claim of the paper, so this comparison set is too weak.

Questions for authors

Can you report two or three seeds for the main NAVSIM comparisons? The component gains are 0.1 to 0.3 PDMS. Without a variance estimate I cannot separate signal from noise. This is the item most likely to change my rating.

Can you add a shaping objective to Table 5 that is neither planning nor reconstruction? Future BEV occupancy or motion forecasting of surrounding agents would be my first choices.

Can you add DrivoR and DiffusionDrive to the HUGSIM comparison?

About latency: 342 ms at 128 tokens, 346 ms at 128 tokens with uniform allocation, 335 ms at 120 tokens. Reducing the token count by 6 percent reduces latency by 2 percent. The compression in TRP does not seem to affect runtime. Where does the time actually go, and does this weaken the efficiency motivation for the pyramid? Does 342 ms include the DINOv2 backbone over all sixteen camera frames, and what is the input resolution?

What exactly is copied between the posterior and the prior in stage 2 under "structurally compatible parameters"? The two branches use different queries, so I do not know what is shared.

The oracle posterior reaches 95.0 and the prior reaches 93.8. Is the remaining 1.2 worth discussing as the part of the future that cannot be inferred from history? Is 0.93 cosine similarity a meaningful value for latents of this shape, or does it saturate easily?

L_reg, the diversity regularisation term, is mentioned once and is not ablated. What happens without it?

7 comments
AuthorsOpenReview ↗

(1/5) Re: the Reviewer euhy

We thank you for the careful reading of our paper and constructive comments in detail.

1. Multi-Seed Evaluation of Main NAVSIM Ablations (W1 and Q1)

We thank the reviewer for highlighting the need to validate the statistical stability of small performance gains. To address this concern, we independently retrained the main NAVSIM-v1 ablation configurations using three random seeds, while keeping the training settings and evaluation protocol identical. The per-seed PDMS results, means, and standard deviations are reported below.

Table R1. Multi-Seed Evaluation of Main Component Ablations on NAVSIM-v1.

TRPH2FFCPSeed 1Seed 2Seed 3Mean ± Std
×××93.1393.0593.1893.12 ± 0.07
✓××93.3893.3393.4093.37 ± 0.04
✓✓×93.5393.4693.5793.52 ± 0.06
✓✓✓93.8293.7993.8593.82 ± 0.03

Table R2. Multi-Seed Evaluation of Future Representations on NAVSIM-v1.

Future RepresentationSeed 1Seed 2Seed 3Mean ± Std
Null Future93.4193.3893.4493.41 ± 0.03
Generic Future Latent93.5493.4993.5893.54 ± 0.05
Planning-Shaped Latent93.8293.7993.8593.82 ± 0.03

The results show consistent gains across all three seeds. In the incremental component ablation, TRP, H2F, and FCP contribute average improvements of +0.25, +0.15, and +0.30 PDMS, respectively, yielding an overall gain of +0.70 PDMS over the baseline.

Moreover, in the reviewer-highlighted comparison between Planning-Shaped and Generic Future Latents, the improvements are +0.28, +0.30, and +0.27 PDMS across the three seeds, averaging +0.28 ± 0.02 PDMS. The consistent gains further support the effectiveness of shaping future representations with planning objectives.

We have added these multi-seed results and standard deviations to the revised manuscript to further substantiate the stability of our main ablation findings.

2. Clarification of Ablation Configurations in Tables 4, 5, and 8 (W1)

We thank the reviewer for pointing out this ambiguity. We acknowledge that the original descriptions of the ablation settings were insufficiently clear. We clarify that all reported results are correct; the differences arise from distinct experimental configurations, rather than inconsistent results or experimental errors. The key distinction is whether Foresight-Conditioned Planning (FCP) is enabled. We summarize the configurations below (H2F distillation denotes the standard setting; parentheses indicate ablation variants).

Table R3. Ablation Configuration Summary.

TableConfigurationTRPH2FFCPPDMS
Table 4Baseline×None×93.1
Table 4+ TRP✓None×93.4
Table 4+ TRP + H2F✓H2F distillation×93.5
Table 4Full✓H2F distillation✓93.8
Table 5Null Future✓None✓93.4
Table 5Generic Future Latent✓H2F distillation (generic future)✓93.5
Table 5Planning-Shaped Latent✓H2F distillation✓93.8
Table 8Prior-only✓None✓93.4
Table 8+ H2F Distillation✓H2F distillation (w/o posterior init.)✓93.7
Table 8Full✓H2F distillation✓93.8
  • Table 4 incrementally introduces the proposed modules. The baseline contains none of them. Adding TRP and then H2F increases PDMS to 93.4 and 93.5, respectively. Neither configuration enables FCP, so the predicted future latent is not explicitly used by the Planner or Scorer. The full model further enables FCP, reaching 93.8.

  • Table 5 enables FCP in all configurations and compares different future representation supervision targets: Null Future, Generic Future Latent, and Planning-Shaped Latent.

  • Table 8 also enables FCP throughout and examines H2F distillation and posterior initialization. The Prior-only configuration uses no H2F distillation (93.4). Adding H2F distillation increases PDMS to 93.7, and further introducing posterior initialization reaches 93.8.

Therefore, the 93.5 PDMS in Table 4 and 93.7 in Table 8 are not contradictory: the former disables FCP, whereas the latter enables it, allowing predicted future latents to guide trajectory generation and selection. Similarly, Null Future in Table 5 and Prior-only in Table 8 share the same configuration (93.4). The +TRP setting in Table 4 also achieves 93.4 but differs because FCP is disabled.

We have clarified these configurations in the revised manuscript, particularly the use of FCP and the different future representation supervision targets, to avoid further ambiguity.

AuthorsOpenReview ↗

(2/5) Re: the Reviewer euhy

3. Additional Non-Planning, Non-Reconstruction Future Representation Supervision (W1 and Q2)

We thank the reviewer for this valuable suggestion. To further distinguish planning-shaped supervision from generic future supervision, we introduce an additional shaping objective that relies on neither planning nor feature reconstruction: Future BEV Occupancy Prediction. Following WoTE [1], we use future BEV occupancy prediction to supervise the future latent, providing structured future-scene supervision. Keeping all other model configurations unchanged, we compare four supervision strategies across three random seeds.

Table R4. Future Representation Supervision Ablations.

Future RepresentationSupervisionPDMS (Mean ± Std)
Null Future–93.41 ± 0.03
Generic Future LatentL_scene93.54 ± 0.05
Future BEV-Shaped LatentL_bev93.63 ± 0.04
Planning-Shaped Latent (Ours)L_plan + L_score93.82 ± 0.03

Future BEV occupancy supervision achieves 93.63 ± 0.04 PDMS, improving upon generic future-feature reconstruction (93.54 ± 0.05) by 0.09 points. This indicates that structured future-scene supervision provides additional planning benefits. Nevertheless, planning-shaped supervision achieves the best performance, outperforming Future BEV-Shaped and Generic Future Latents by 0.19 and 0.28 PDMS, respectively. These results suggest that structured future-scene supervision alone is insufficient to achieve the best performance, whereas jointly shaping future representations through trajectory generation and candidate scoring objectives better aligns future information with downstream planning.

We have incorporated this additional controlled experiment and the corresponding analysis into the revised manuscript. We thank the reviewer again for the constructive suggestion.

4. Additional Strong Baseline Comparisons on HUGSIM (W2 and Q3)

We thank the reviewer for the valuable suggestion. To further validate PlanWAM's zero-shot closed-loop generalization, we additionally evaluate DrivoR [2], DiffusionDrive [3], and iPad [4] under the same HUGSIM scenarios, simulator configurations, and evaluation protocol as PlanWAM. We do not include DriveFuture [5] because it has not been evaluated on HUGSIM and its implementation is not publicly available. All methods are evaluated zero-shot, without HUGSIM-specific fine-tuning. The results are shown below.

Table R5. Additional Zero-Shot Closed-Loop Comparisons on HUGSIM.

(a) Route Completion

MethodEasyMediumHardExtremeAvg.
iPad69.434.825.021.835.9
DiffusionDrive75.130.730.430.338.7
DrivoR79.348.129.329.345.3
PlanWAM (Ours)87.648.351.937.253.7

(b) HD-Score

MethodEasyMediumHardExtremeAvg.
iPad56.118.011.18.421.2
DiffusionDrive65.713.013.814.923.3
DrivoR72.635.416.216.133.5
PlanWAM (Ours)75.332.935.622.038.7

The results show that PlanWAM outperforms all newly evaluated baselines in overall closed-loop performance. Compared with DrivoR, PlanWAM improves average RC and HD-Score by 8.4 and 5.2 points, respectively; compared with DiffusionDrive, the gains are 15.0 and 15.4 points. Notably, on Hard scenarios, PlanWAM achieves 51.9 RC and 35.6 HD-Score, substantially outperforming all added baselines, while maintaining its advantage on Extreme scenarios.

These results further demonstrate that PlanWAM's performance gains extend beyond NAVSIM open-loop evaluation and remain substantial in zero-shot closed-loop comparisons against recent strong baselines under a unified protocol. We have included the additional baselines and complete evaluation results in the revised manuscript.

AuthorsOpenReview ↗

(3/5) Re: the Reviewer euhy

5. Additional Latency Breakdown and Efficiency Analysis of the Temporal Register Pyramid (TRP) (Q4)

We thank the reviewer for the insightful analysis of TRP's computational efficiency. We provide a module-wise latency breakdown and an additional comparison with DrivoR under identical inference settings.

First, the reported 342 ms includes DINOv2-S/14 processing all 16 historical images. Specifically, the input consists of four camera views (front, left, right, and rear) at four historical timestamps, processed together in a single batched forward pass. Each image has a resolution of 1148 × 672 with a patch size of 14, yielding 82 × 48 = 3,936 patch tokens. Including 16 scene-query tokens, one CLS token, and four register tokens, each image contains 3,957 ViT input tokens.

Second, the latency breakdown confirms that most computational overhead comes from the visual backbone. As shown below, DINOv2 accounts for 301 ms (88.0%) of PlanWAM's total inference time, while TRP and the Prior World Model together require only 18 ms.

Table R6. Module-Wise Inference Latency Breakdown.

ModulePlanWAM (ms)DrivoR (ms)
DINOv2-S/1430176
Temporal Register Pyramid (TRP)8—
Prior World Model10—
Planner1110
Scorer1211
Total34297

This also explains the modest latency differences across token budgets in Table 7. At 128 tokens, pyramid and uniform allocations require 342 ms and 343 ms, respectively. Reducing the budget to 120 tokens (−6.25%) lowers latency to 335 ms (−2.0%). This is because token reduction primarily reduces the cost of downstream modules, while DINOv2 still processes all 16 images and dominates end-to-end latency.

We clarify that TRP's efficiency benefit lies primarily in compressing multi-frame, multi-view visual features into compact representations under a limited token budget, while preserving historical evidence useful for future planning, rather than substantially reducing overall inference latency. We have added the input specifications and module-wise latency breakdown to the revised manuscript to clarify this distinction.

6. Clarification of Posterior-to-Prior Parameter Initialization in Stage 2 (Q5)

We thank the reviewer for raising this important question. Stage 2 initializes the Prior with corresponding network weights from the Stage 1 checkpoint, including the history visual encoder, Temporal Register Pyramid, World Model Transformer, Trajectory Planner, and Trajectory Scorer.

The corresponding Posterior and Prior modules share the same architecture and parameter dimensions, differing primarily in the source of World Model queries. The Posterior uses visual registers extracted from ground-truth future scenes as queries, whereas the Prior uses learnable future queries organized in the same time–camera–register-slot order. Both have identical query counts and feature dimensions and attend to the same history representation as keys and values. Thus, the World Model Transformer weights can be directly transferred. Importantly, we copy the Transformer weights, not the Posterior's future visual queries; the Prior retains its own learnable future queries.

Moreover, the Planner and Scorer in both branches receive a Planning Memory formed by concatenating the history representation with the future latent. Since their architectures and input dimensions are identical, their corresponding weights can also be directly initialized from Stage 1. During Stage 2, the Posterior remains frozen as the teacher, while the initialized Prior is independently fine-tuned, without continued parameter sharing between the two branches.

We have clarified this initialization procedure in the revised manuscript.

Reviewer_n7N9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors argue that for e2e planning, it is better to let world action model to predict the feature representation that is actually most useful for planning. To validate the idea, author design a new pipeline : 1. Temporal Register Pyramid helps compresses history registers, 2. Privileged posterior is trained with trajectory generation scoring loss plus regularization, instead of a reconstruct loss to get a "planning-shaped" future latent. 3. Hindsight-to-Foresight Distillation trains a prior branch only on history, then the prediction can be used for planning context and guide the trajectory generation and selection.

Critical strengths
  1. The formulation is very clear with an upper bound and sufficient ablation. The oracle first quantifies how far the design can reach. Then the paper ablates representation type, where the latent is used, the distillation design and history allocation. The contribution of the work is easy to check.

  2. Strong result with a light backbone. Based on ViT-S, it reach 93.8 PDMS and 90.9 EPDMS and transfers zero-shot to HUGSIM, with a notable gain on the hard subset.

Critical weaknesses
  1. the central claim is not separated from a simpler explanation, the model is just doing distillation of privilege ego plan. During training, the posterior gets to look at the future frame from the ego camera, where the viewpoint changes can implicitly offer information about expert's future trajectory. The network thus is highly use the visual shift to decode and memorize those expert's future trajectory. if this is the case, then the method can be a standard privilege distillation not about discovering the "most useful future representation".

  2. Figure 5 shows about 177 and 190 failures cased by DAC. However based on 12146 test scenes, and the claim (98.9 and 98.8) from Table 4, it would allow at most 134 and 151 cases with DAC = 0.

Questions for authors
  1. which HUGSIM evaluator version and scenario are used, could you also add DrivoR under same settings?

  2. EC in Table 1 is only 70.5 which is lowest among all baselines. Could you please interpret that?

  3. Paper claim the foresight is "deployable", however the Tabel 7 shows that the latency is about 342 ms per frame, can you also compared with DrivoR under same setting.

5 comments
AuthorsOpenReview ↗

(1/3) Re: the Reviewer n7N9

We thank you for the careful reading of our paper and constructive comments in detail.

1. Distinction Between Planning-Shaped Future Representations and Privileged Ego-Trajectory Distillation (W1)

We thank the reviewer for raising this important concern. We clarify that PlanWAM does not simply distill the expert trajectory implicitly encoded by viewpoint changes in future ego-camera observations. The key distinction is that the model learns not only how the expert drives, but also how to evaluate the planning quality of different candidate trajectories.

Specifically, in addition to best-of-K regression against the expert trajectory, the Posterior receives scoring supervision over all 64 candidate trajectories, using simulator-provided labels for safety, progress, and comfort. These labels reflect the actual planning quality of each candidate, rather than its similarity to the expert trajectory. The expert future trajectory alone cannot determine the safety or planning quality of other candidates, as such evaluations also depend on surrounding traffic participants and their interactions with each candidate. Therefore, planning supervision encourages the future latent to retain decision-relevant information beyond that contained in a single expert trajectory. The learning objective is not to imitate a single expert trajectory, but to preserve scene information relevant to driving decisions through direct planning-task supervision. This representation is subsequently distilled into a history-only Prior branch via Hindsight-to-Foresight Distillation (H2F).

Our ablations further support this distinction. In Table 5, under the same privileged future observations, generic future-feature reconstruction and planning-shaped supervision achieve 93.5 and 93.8 PDMS, respectively. Table 6 further shows that using the predicted future latent solely for candidate trajectory selection achieves 93.7 PDMS. These results support the contribution of planning objectives to shaping future representations and improving downstream decisions, rather than attributing the gains solely to the introduction of privileged future information.

2. Correction of Failure Counts in Figure 5(a) (W2)

We sincerely thank the reviewer for carefully identifying the numerical inconsistency in Figure 5(a), and apologize for this error. We rechecked all 12,146 per-scene evaluation records in NAVSIM-v1 navtest and confirmed that the error occurred during data preparation for plotting Figure 5(a), while the original evaluation records and results were correct.

Specifically, the DAC-related counts were corrected from 177/190 to 137/147, and the TTC-related counts from 143/68 to 246/117. The complete corrected results are presented below.

Table R1. Corrected Failure Counts in Figure 5(a)

Failure TypeID0ID3 (Ours)Relative Change
NC-related13852−62.3%
TTC-related246117−52.4%
DDC-related3010−66.7%
DAC-related137147+7.3%

The corrected DAC counts are consistent with the DAC scores reported in Table 4 (98.9/98.8). We further verified that all evaluation metrics in Table 4 (NC, DAC, TTC, C, EP, and PDMS), the number of failure scenes with PDMS = 0 in Figure 5(b) (314 vs. 237), and the corresponding failure attribution analysis remain unchanged. The relative changes across the four failure categories in Figure 5(a) are also preserved.

Therefore, this error was limited to the display of certain absolute failure counts in Figure 5(a) and does not affect any other experimental results or the overall conclusions of the paper. We have corrected Figure 5(a) in the revised manuscript and carefully rechecked the results.

AuthorsOpenReview ↗

(2/3) Re: the Reviewer n7N9

3. HUGSIM Evaluation Protocol and Additional Strong Baselines (Q1)

We thank the reviewer for the valuable suggestion. We clarify the HUGSIM evaluation protocol and provide additional comparisons with recent strong baselines.

Evaluation Protocol. We use the official HUGSIM simulator [1] (hyzhou404/HUGSIM, commit 62c690d) and all 436 public scenarios from nuScenes (88), Waymo (108), KITTI-360 (113), and PandaSet (127), covering Easy (80), Medium (157), Hard (96), and Extreme (103) difficulties. Following Appendix C.4, we report Route Completion (RC) and HD-Score, with overall scores averaged across all scenarios. All models are evaluated zero-shot, without HUGSIM-specific fine-tuning.

Additional Baselines. Following the reviewer's suggestion, we additionally evaluate DrivoR [2], DiffusionDrive [3], and iPad [4] under identical HUGSIM scenarios, simulator configurations, and evaluation protocols.

Table R2. Additional Zero-Shot Closed-Loop Comparisons on HUGSIM.

(a) Route Completion

MethodEasyMediumHardExtremeAvg.
iPad69.434.825.021.835.9
DiffusionDrive75.130.730.430.338.7
DrivoR79.348.129.329.345.3
PlanWAM (Ours)87.648.351.937.253.7

(b) HD-Score

MethodEasyMediumHardExtremeAvg.
iPad56.118.011.18.421.2
DiffusionDrive65.713.013.814.923.3
DrivoR72.635.416.216.133.5
PlanWAM (Ours)75.332.935.622.038.7

Compared with DrivoR, PlanWAM improves average RC and HD-Score by 8.4 and 5.2 points, respectively. The gains are particularly pronounced on Hard scenarios, where RC increases from 29.3 to 51.9 (+22.6) and HD-Score from 16.2 to 35.6 (+19.4), with consistent advantages on Extreme scenarios. These results demonstrate PlanWAM's superior overall closed-loop performance under a unified evaluation protocol, particularly in challenging scenarios, further supporting its zero-shot generalization capability.

We have added the evaluator version, scenario details, and new baseline comparisons to the revised manuscript.

4. Interpretation of Extended Comfort (EC) on NAVSIM-v2 (Q2)

We thank the reviewer for highlighting the EC metric. We note that 70.5 is not the lowest reported EC, as DriveVLA-W0 achieves 58.9. Nevertheless, we reexamined the per-scene evaluation results to investigate the relatively low EC.

First, EC primarily measures temporal consistency between consecutive plans, rather than the comfort of individual trajectories. Specifically, it evaluates changes in the overlapping segments of successive planned trajectories. Thus, the EC of 70.5 mainly reflects insufficient inter-plan consistency. In contrast, PlanWAM achieves 98.1 History Comfort (HC), comparable to other baselines, indicating good comfort within individual trajectories.

Further per-scene analysis reveals how this inconsistency manifests. Compared with EC-passing scenes, EC-failing scenes show larger median longitudinal discrepancies between consecutive plans (0.67 vs. 0.41 m), while lateral (6.3 vs. 5.2 cm) and heading discrepancies (0.44° vs. 0.45°) remain similar. Average route progress is nearly identical (0.909 vs. 0.910), and approximately 88% of EC-failing scenes still receive full scores on all other submetrics. These findings indicate that the lower EC mainly reflects longitudinal adjustments between planning cycles, rather than substantial lateral or heading deviations.

The lower EC is primarily associated with our independent candidate selection mechanism. At each planning step, PlanWAM selects the highest-scoring trajectory among 64 candidates. The scorer considers safety, progress, and single-trajectory comfort, but does not explicitly enforce consistency with the previous plan. Thus, even when consecutive plans are individually smooth, changes in their longitudinal speed profiles may trigger EC penalties. Incorporating cross-cycle consistency constraints could further improve this metric.

AuthorsOpenReview ↗

(3/3) Re: the Reviewer n7N9

5. Deployability of Foresight and Inference Latency Analysis (Q3)

We thank the reviewer for raising this efficiency concern. We clarify that “deployable foresight” refers to the availability of the predicted future representation at inference time, rather than claiming that the current implementation meets high-frequency real-time requirements for on-vehicle deployment. Specifically, the Privileged Posterior is used only during training and completely removed at inference. The Prior predicts planning-shaped future latents solely from historical observations, without accessing ground-truth future information.

To identify the sources of computational overhead, we conducted module-wise latency measurements for PlanWAM and DrivoR under identical settings (single NVIDIA A800, batch size 1, FP32, averaged over 100 inference runs).

Table R3. Module-Wise Inference Latency Comparison.

ModulePlanWAM (ms)DrivoR (ms)
DINOv2-S/1430176
Temporal Register Pyramid8—
World Model (Prior)10—
Planner1110
Scorer1211
Total34297

The results show that the additional latency mainly comes from multi-frame visual feature extraction, rather than foresight prediction itself. PlanWAM processes four historical timestamps from four cameras (16 images), whereas DrivoR processes only four images from the current timestamp. Consequently, DINOv2 latency increases by 225 ms, accounting for 91.8% of the total 245 ms increase. In contrast, the Temporal Register Pyramid and Prior World Model together add only 18 ms, while the Planner and Scorer contribute merely 2 ms of additional latency. Thus, the 342 ms latency primarily reflects the cost of richer historical visual inputs, rather than the computational overhead of learning and predicting planning-shaped future representations.

We have clarified the meaning of “deployable foresight” and added the module-wise latency breakdown to the revised manuscript.

References

[1] H. Zhou et al., "HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving," IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 48, no. 4, pp. 4673–4691, 2026.

[2] E. Kirby et al., "Driving on Registers," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 32058–32069, 2026.

[3] B. Liao et al., "DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12037–12047, 2025.

[4] K. Guo et al., "iPad: Iterative Proposal-Centric End-to-End Autonomous Driving," IEEE Robotics and Automation Letters (RA-L), vol. 11, no. 10, pp. 11142–11149, 2026.

Comments on the paper

3 comments
Area_Chair_2CdGOpenReview ↗

Please Review the Author Rebuttal and Reassess Your Rating

Dear Reviewers,

Thank you for your time and effort in reviewing the submissions.

The authors have now submitted their rebuttals. I kindly ask you to carefully review the authors' responses and assess whether they have adequately addressed your questions and concerns.

If the rebuttal resolves your concerns, please consider updating your review and adjusting your rating accordingly. If any concerns remain, please feel free to raise them in the discussion.

Your timely feedback and active participation are greatly appreciated, as they are essential to reaching a fair, well-informed decision.

Thank you again for your valuable contributions!

Best,

Your AC

AuthorsOpenReview ↗

(2/2) Global Rebuttal

Summary of Status by Reviewer

Reviewer n7N9

  • Addressed Main Concerns: Clarified the distinction between planning-shaped future representations and privileged ego-trajectory distillation; corrected the Figure 5(a) failure counts; and added HUGSIM evaluation details and comparisons with DrivoR, DiffusionDrive, and iPad.
  • Additional Analyses: Provided a detailed interpretation of EC and a module-wise inference latency breakdown.
  • Follow-up on W1: Further introduced controlled Future GT Ego Motion Only and future-representation supervision experiments to examine the contribution of privileged ego-motion and planning-shaped supervision.

Reviewer euhy

  • Addressed Main Concerns: Added three-seed NAVSIM ablations, clarified the configurations of Tables 4, 5, and 8, and introduced non-planning future BEV occupancy supervision.
  • Additional Experiments: Added strong HUGSIM baselines, detailed latency analysis, and an ablation of latent regularization.
  • Implementation Clarifications: Explained Posterior-to-Prior parameter initialization, the Posterior–Prior performance gap, and the interpretation of latent cosine similarity.

Reviewer JCcM

  • Addressed Main Concerns: Added cross-architecture transfer experiments using iPad, candidate trajectory quality and diversity analyses, and HUGSIM closed-loop ablations separating generation and selection.
  • Follow-up on C1–C3: Further provided HUGSIM component-wise ablations including the DrivoR-based ID0 baseline, controlled evidence for the generalizability of planning-shaped representations, and zero-shot Bench2Drive evaluation.
  • Additional Transparency: Released the detailed per-route Bench2Drive evaluation results for ID0 and Full PlanWAM, including route completion and infraction breakdowns.

We sincerely thank all reviewers again for their thoughtful feedback. Their comments have helped us improve the clarity of our methodology, strengthen the experimental evidence, and better articulate the contributions and limitations of PlanWAM.

AuthorsOpenReview ↗

(1/2) Global Rebuttal

Global Rebuttal

We sincerely thank the reviewers and the Area Chair for their careful assessment and constructive feedback. Their comments have helped us strengthen the empirical evidence, clarify the methodological contribution, and improve the presentation of PlanWAM. To facilitate the assessment, we first summarize the Key Common Concerns and our corresponding responses, followed by a brief status overview for each individual reviewer.

Key Common Concerns

1. Planning-Shaped Future Representations Beyond Privileged Ego-Trajectory Distillation (Raised by n7N9, euhy, and JCcM)

We clarify that PlanWAM uses privileged future observations only to train the Posterior, while H2F distills a planning-shaped future representation, rather than directly reproducing a single expert trajectory. Beyond best-of-K trajectory regression, the Posterior receives simulator-based scoring supervision over all 64 candidate trajectories across six dimensions (NC, DAC, TTC, EP, DDC, and comfort), which requires information about future traffic interactions and the quality of alternative driving decisions.

To further examine the privileged ego-motion explanation, we introduce a Future GT Ego Motion Only control, achieving 93.61 ± 0.04 PDMS, compared with 93.82 ± 0.03 using full future observations under the same planning and scoring supervision. Moreover, under identical future observations, planning-shaped supervision outperforms both generic future-feature reconstruction and future BEV occupancy supervision (93.82 ± 0.03 vs. 93.54 ± 0.05 and 93.63 ± 0.04, respectively). These controlled experiments demonstrate that privileged ego-motion alone cannot explain the full improvement, and that planning objectives provide additional value in shaping future representations. For details, please refer to our responses to Reviewers n7N9, euhy, and JCcM.

2. Statistical Reliability and Ablation Clarity (Raised by euhy and JCcM)

We independently retrained the main NAVSIM-v1 ablation configurations using three random seeds. The full model consistently achieves 93.82 ± 0.03 PDMS, improving over the baseline (93.12 ± 0.07), with successive gains from TRP (+0.25), H2F (+0.15), and FCP (+0.30). The key comparison between generic and planning-shaped future representations also shows a consistent improvement of 0.28 ± 0.02 PDMS across three seeds (93.54 ± 0.05 vs. 93.82 ± 0.03).

We further clarify the different FCP configurations underlying Tables 4, 5, and 8, and introduce an additional BEV occupancy supervision control and an ablation of the latent regularization term. These results provide further evidence for the stability of the improvements and the complementary roles of the proposed components.

3. Planning Utility, Generation–Selection Contributions, and Generalization (Raised by n7N9, euhy, and JCcM)

To better understand how future representations benefit planning, we analyze the quality of all 64 candidate trajectories. PlanWAM improves their mean PDMS from 79.21 to 82.39, with consistent gains in Oracle Top-1, Top-5, and Top-10 PDMS. We also clarify that these gains do not come from increased geometric diversity, but primarily from improved candidate quality.

Beyond NAVSIM, we conduct additional zero-shot closed-loop evaluations and ablations on HUGSIM. Under the same 436-scenario evaluation protocol, PlanWAM achieves 53.7 Route Completion and 38.7 HD-Score, outperforming DrivoR by 8.4 and 5.2 points, respectively. Additional three-seed ablations demonstrate that planning-shaped future representations benefit both trajectory generation and selection, with their joint use achieving the best performance. We further validate transferability by integrating the proposed mechanism into iPad and evaluating NAVSIM-trained checkpoints on Bench2Drive without target-domain fine-tuning, where Full PlanWAM achieves 66.5 ± 1.2 DS and 38.2 ± 2.1 SR, compared with 54.2 ± 1.2 and 28.1 ± 2.0 for the DrivoR-based baseline.

4. Evaluation Consistency, Runtime, and Metric Interpretation (Raised by n7N9 and euhy)

We rechecked all 12,146 NAVSIM-v1 navtest evaluation records and corrected the plotting error in Figure 5(a). The DAC-related counts were corrected from 177/190 to 137/147, while the original evaluation records, Table 4 metrics, and Figure 5(b) results remain unchanged. We also provide a detailed analysis of the EC metric and a module-wise latency comparison with DrivoR. Specifically, the 342 ms inference time is dominated by DINOv2 processing 16 historical images (301 ms), whereas TRP and the Prior World Model together introduce only 18 ms. We further clarify that deployable foresight refers to the history-only Prior being available at inference time, rather than claiming that the current implementation already meets high-frequency real-time deployment requirements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.