acceptodds
Under review as a conference paper at ICLR 2027

FPD-Drive: Future Policy Distillation for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving models are typically trained on real-world driving logs, which record only the trajectory actually executed in each scene, providing limited supervision for alternative valid behaviors. Multi-target supervision can mitigate this limitation, but broader trajectory coverage makes safe candidate selection more challenging. We propose FPD-Drive, an end-to-end planner that distills privileged future knowledge into a visual driving policy to enrich trajectory supervision and improve safety-aware candidate selection. Specifically, Future-Privileged Multi-Target Distillation transfers diverse, high-quality trajectory targets from a teacher with access to recorded future observations, extending student supervision beyond the single recorded human trajectory. Safety-Aware Score Correction further corrects ranking errors between nearby safe and unsafe candidates and raises underestimated safety scores of safe candidates. At deployment, the student requires neither future observations nor teacher inference. Our base model achieves 94.9 PDMS on NAVSIM v1 navtest and 55.9 EPDMS on NAVSIM v2 navhard. The scaled FPD-Drive-Pro improves these scores to 95.5 PDMS and 58.1 EPDMS, respectively, achieving state-of-the-art performance among the compared methods on both benchmarks. Zero-shot closed-loop evaluation on HUGSIM further yields an HD-Score of 45.1 and RC of 55.3.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Planning, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_C1g8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FPD-Drive, an end-to-end autonomous driving planner that distills privileged future information into a visual policy. The method uses a teacher with access to recorded future camera observations to generate multiple high-quality trajectory targets, which are filtered by planning metrics and assigned to different student proposals. This Future-Privileged Multi-Target Distillation extends supervision beyond the single human trajectory available in each driving log. The paper further introduces Safety-Aware Score Correction to explicitly penalize highly ranked unsafe candidates relative to nearby safe alternatives and raise underestimated safety scores for safe candidates. FPD-Drive achieves strong results on NAVSIM v1 and v2, while its zero-shot HUGSIM evaluation demonstrates generalization to closed-loop settings.

Critical strengths
  1. The use of recorded future camera observations to improve a privileged teacher is well motivated. Prior work such as Hydra-MDP has explored multi-target distillation from human and rule-based teachers. FPD-Drive extends this direction by conditioning its teacher on future visual observations to construct scene-adaptive alternatives, while retaining a student that only uses current observations at deployment.

  2. The empirical performance is strong. FPD-Drive and FPD-Drive-Pro obtain 94.9/95.5 on NAVSIM v1 and 55.9/58.1 on NAVSIM v2, respectively. The zero-shot HUGSIM results are also encouraging, with an average HD-Score of 45.1 and route completion of 55.3. The paper includes useful component and design ablations, and the deployed student has no dependence on future observations or teacher inference.

Critical weaknesses
  1. The related-work discussion of multi-target distillation could be strengthened. Hydra-MDP [1] is cited in the context of metric-based supervision, but it is also relevant as prior work that distills multiple targets from human and rule-based teachers. A more explicit discussion would help distinguish FPD-Drive's contributions, such as the future-conditioned teacher model, from prior literature.

  2. There is a potential issue in using recorded future observations to generate alternative trajectories. Those observations reflect how the scene evolved under the recorded human action, whereas some distilled alternatives may induce different reactions from other agents. The PDMS-based filtering mitigates this issue but does not fully model interactive counterfactual futures. It would be helpful to analyze when future-conditioned targets become inconsistent with the alternative ego behavior.

[1] Li Z, Li K, Wang S, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation[J]. arXiv preprint arXiv:2406.06978, 2024.

Questions for authors

I don't have other questions. Please refer to the Critical Weaknesses section.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer C1g8

We thank the reviewer for the positive assessment and constructive feedback.

W1 - Relation to Hydra-MDP and Prior Multi-Target Distillation: we will expand the distillation paragraph in Related Work to explicitly acknowledge Hydra-MDP's contributions to multi-teacher, multi-target supervision from human and rule-based teachers. We will clarify how FPD-Drive builds on this direction through a future-privileged teacher that generates scene-adaptive trajectory targets, while the deployed student uses only current observations. We will also consistently distinguish multi-metric supervision, multi-trajectory supervision, and multi-teacher distillation to make the relationship to prior work clearer.

W2 - Consistency of Future-Privileged Targets with Alternative Ego Behaviors: we acknowledge this limitation of NAVSIM v1 navtest, where surrounding agents replay their recorded trajectories without reacting to alternative ego actions. To assess performance beyond this limitation, we also evaluate on NAVSIM v2 navhard, whose Stage 2 uses synthetic follow-up scenes and whose evaluation incorporates reactive traffic agents that respond to ego behavior. As reported in Table 2, FPD-Drive and FPD-Drive-Pro achieve the highest score among the compared methods. Furthermore, our zero-shot HUGSIM closed-loop evaluation achieves the best average HD-Score and route completion among the compared methods (Table 3). These results provide empirical evidence that the distilled policy remains effective under reactive traffic and closed-loop evaluation. We have added a corresponding discussion to the Limitations section, acknowledging these evaluation limitations and the need for further real-world validation.

Following the reviewers' suggestions, we have revised the Limitations section in the appendix and added a new More Discussion section, with changes highlighted in blue.

Thanks for the reply. My concerns have been addressed and I will keep my score.

Reviewer_KyEiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a key limitation of end-to-end autonomous driving: models trained on human driving logs mainly imitate the single trajectory that was executed and receive limited supervision about other valid and safe alternatives. It proposes FPD-Drive, a teacher–student policy distillation framework. The teacher has access to privileged future observations that are unavailable at test time. Future-Privileged Multi-Target Distillation transfers multiple diverse trajectory candidates from the teacher to the student, while Safety-Aware Score Correction addresses biased safety ranking among candidate trajectories. Only the student model is used during inference. The method is evaluated on NAVSIM v1/v2 and in zero-shot closed-loop experiments on HUGSIM. The paper also provides ablations of the distillation targets, teacher architecture, and score-correction components.

Critical strengths
  1. The Future-Privileged Multi-Target Distillation method described in Section 3.3 transfers multiple geometrically diverse trajectory candidates instead of only imitating the recorded human trajectory. This addresses the limited coverage of human demonstrations and provides supervision for alternative behaviours that may also be valid and safe.
  2. The Safety-Aware Score Correction module in Section 3.4 targets ranking errors that arise during multi-target distillation. In particular, it penalizes highly ranked unsafe candidates and increases the scores of safe candidates that are underestimated by the scorer. This is a relevant complement to multi-target trajectory supervision, since diverse candidate generation alone does not guarantee reliable candidate selection.
  3. The evaluation covers two NAVSIM benchmarks and includes zero-shot closed-loop testing on HUGSIM. The paper also provides ablations of the teacher architecture, future-token injection, target budget, target-source selection, distillation strategy, and score-correction components. These experiments help analyse the contribution of several design choices.
Critical weaknesses
  1. The contributions of future privilege and multi-target supervision are not fully disentangled. Table 5(a) studies the effect of the target budget K, but these experiments retain the future-privileged teacher. Table 5(b) compares self-distillation without future information against distillation from a future-privileged teacher, which provides useful evidence for the value of future observations. However, the comparison does not fully control the number, quality, and selection procedure of teacher targets. A more tightly controlled experiment with the same target budget, target-selection rule, and student architecture would be useful for separating the gain from future information from the gain provided by multi-target supervision.
  2. The analysis of the two SASC correction terms remains limited. Table 11 reports the combined effect of the two regularization terms, but the paper does not provide sufficient case-level analysis showing which types of ranking failures are primarily corrected by each term. It is therefore difficult to understand whether the two terms address genuinely distinct failure modes or mainly provide complementary regularization. Examples of unsafe over-ranking and safe under-estimation, together with ablations of each term on those cases, would make the mechanism more interpretable.
  3. Some qualitative figures should be improved for readability. Although the visualizations are informative, Figures 2 and 5 overlay multiple predicted trajectories, recorded trajectories, vehicle footprints, and safety-related annotations within compact subplots. The dense use of line styles and colour patches makes it difficult to distinguish the different elements without repeatedly consulting the captions. Some subplot labels and metric annotations also appear cramped. Enlarged examples or simplified visualizations would make the qualitative comparisons easier to interpret.
Questions for authors

1 Table 5(b) already compares self-distillation without future information with distillation from a future-privileged teacher. Could you provide a more tightly controlled comparison using the same target budget, target-selection rule, and student architecture in both settings? Please also report whether the number and quality of retained targets are comparable, since these factors may otherwise confound the measured benefit of future privilege. 2 SASC contains two separate correction objectives for over-ranked unsafe candidates and under-estimated safe candidates. Could you provide case-level analysis showing what type of planning failure each term addresses? In what driving scenarios would removing either term lead to a distinct model behaviour or ranking error? 3 Zero-shot HUGSIM evaluation is an important result of the paper. How similar are the NAVSIM training scenes and HUGSIM test scenes in terms of road layouts, traffic patterns, and scenario composition? Could you provide results on HUGSIM scenarios that are more clearly out of distribution relative to NAVSIM, in order to evaluate the generalization boundary of the proposed distillation pipeline? 4 In the multi-target distillation setup, bipartite matching assigns teacher trajectories to student proposals. What happens when none of the student proposals is close to a high-quality teacher trajectory generated using privileged future observations? How does the matching procedure avoid propagating misleading supervision in such teacher–student mismatch cases? Please provide statistics on matching distances and analyse the effect of poorly matched targets.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer KyEi

We thank the reviewer for the positive assessment and constructive feedback.

Q1 - Controlled Comparison of Future Privilege and Multi-Target Supervision: As detailed in updated Appendix G.2 (Table 13), we compare self-distillation without future information and future-privileged distillation at identical target budgets, using the same student architecture, initialization, training schedule, candidate count (M=64M=64), and PDM-Closed safety and diversity selection rules. We also report the average retained count Jˉ\bar{J} and the mean PDMS of retained targets. Since target selection starts with the highest-scoring trajectory and then adds safety-checked candidates based on geometric diversity, increasing KK may decrease the mean retained-target PDMS while covering a broader range of valid behaviors. Despite this, the improvement in student PDMS suggests that diverse valid targets provide useful complementary supervision beyond the highest-scoring trajectory alone. These results support the benefit of future-privileged multi-target supervision under matched target budgets and selection rules.

Target budget KKDistillationRetained targets Jˉ\bar{J}Mean retained-target PDMS ↑Student PDMS ↑
1w/o Future (Self-distillation)198.194.2
1w/ Future (FPD)199.694.6
2w/o Future (Self-distillation)1.797.394.2
2w/ Future (FPD)1.999.394.7
4w/o Future (Self-distillation)396.994.3
4w/ Future (FPD)3.198.794.9
6w/o Future (Self-distillation)3.196.494.3
6w/ Future (FPD)3.398.794.9

Q2 - Case-Level Analysis of SASC Correction Objectives: Updated Appendix G.3 provides case-level visualizations with and without the two SASC correction objectives. The visualized comparisons illustrate these complementary roles: combining both objectives helps avoid unsafe trajectory selection while reducing excessive conservatism.

Q3 - Zero-Shot Generalization on HUGSIM: HUGSIM is a photorealistic closed-loop simulator that reconstructs scenes from KITTI-360, Waymo, nuScenes, and PandaSet, providing heterogeneous scene sources beyond the nuPlan-derived NAVSIM logs. These source datasets differ in camera configuration and image resolution: KITTI-360 combines two front-facing perspective cameras with two side fisheye cameras, with rectified perspective images at 1408×3761408 \times 376; Waymo uses five cameras, with its three forward-facing views at 1920×12801920 \times 1280; nuScenes uses six surround-view cameras at 1600×9001600 \times 900; and PandaSet uses six cameras at 1920×10801920 \times 1080, including wide-angle and long-focus views. These specifications describe the source imagery used for reconstruction, while HUGSIM generates observations through configurable virtual cameras. In addition to varied street layouts, lighting conditions, and imaging characteristics, HUGSIM supports both normal and aggressive traffic behaviors through editing agents, providing complementary evaluation beyond recorded driving logs. Our model, trained on NAVSIM and SimScale, is evaluated zero-shot on 345 HUGSIM scenarios across Easy, Medium, Hard, and Extreme difficulty levels, as reported in Table 3. Our supplementary video demo contains visualizations of these scenarios and the model's closed-loop behavior.

Q4 - Bipartite Matching and Teacher-Student Mismatch: we clarify that bipartite matching assigns supervision to learnable proposals and does not require them to already be close to the teacher targets. When all proposals are distant, Eq. (5) still assigns each target to a distinct proposal by minimizing the total ADE. Crucially, in Eq. (6), Lprop\mathcal{L}_{\mathrm{prop}} directly trains the proposal decoder to approach its assigned target, while Ldiff\mathcal{L}_{\mathrm{diff}} and Ltraj\mathcal{L}_{\mathrm{traj}} supervise residual denoising and trajectory reconstruction toward that same target. For multiple targets, one-to-one assignment ensures that each matched proposal receives one trajectory target per training instance, and these trajectory losses apply only to the JJ matched pairs. Target quality is assessed through evaluator-based filtering before matching, independently of the student's current prediction accuracy. Thus, a large matching distance indicates a gap in the student's current coverage, which alone does not establish misleading supervision.

Following the reviewers' suggestions, we have revised the Limitations section in the appendix and added a new More Discussion section, with changes highlighted in blue.

Thanks for the detailed response. I will keep the original rating.

Reviewer_H8vsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FPD-Drive, an end-to-end driving planner that transfers privileged future information into a policy using only current observations at deployment. A future-conditioned teacher generates trajectory candidates, from which evaluator-based filtering and diversity selection construct multiple supervision targets. These targets supervise a student through bipartite matching and residual diffusion. Safety-Aware Score Correction additionally addresses highly ranked unsafe candidates and underestimated safe alternatives. The method achieves competitive results on NAVSIM v1 and v2, with supplementary zero-shot closed-loop evaluation on HUGSIM.

Critical strengths

Expanding supervision beyond a single demonstration is interesting, but current method with broader candidate coverage also increases the difficulty of reliable selection. FPD-Drive is promising in addresses this gap. Beyond SOTA performance, the HUGSIM results also provide encouraging evidence that the benefits extend beyond open-loop setting.

Critical weaknesses

Future-informed teacher refinement and distillation already appear in EvoDriveVLA, while evaluator-filtered multi-target supervision and coverage-aware target selection are central to CLOVER. The contribution therefore appears to lie primarily in integrating these ideas with residual diffusion and targeted scoring losses.

Questions for authors

How do you assess the appropriateness of teacher targets when their success depends on future events unavailable to the current-frame student? Can you provide results specifically for occlusion and interactive scenarios?

Does the denoiser receive the proposal coordinates or proposal-specific features? Are scoring losses computed on full DDIM samples or on target-conditioned reconstructions, and are candidate trajectories detached before scoring?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer H8vs

We thank the reviewer for the positive assessment and constructive feedback.

W1 - Distinctions from EvoDriveVLA and CLOVER: our distinctions concern both the distillation objective and the source of supervision. EvoDriveVLA aligns token distributions and hidden states for a selected teacher candidate, whereas we explicitly supervise multiple complete trajectories selected for planning quality and diversity. Table 9 supports this choice: adapting token-level KL distillation to our framework yields 94.5 PDMS, compared with 94.9 for our multi-target trajectory distillation. CLOVER constructs its initial pseudo-experts by instantiating hand-designed motion families along each scene’s route and filtering them with the evaluator. It subsequently refines its generator through scorer-guided self-distillation. Our teacher uses recorded future observations to generate candidate trajectories. We then select multi supervision targets from these trajectories. In Table 10, teacher-generated targets outperform predefined trajectories, using the same student, 64 candidates, target budget, and filtering criteria. These controlled ablations support the benefits of explicit multi-trajectory distillation and learned, future-privileged target generation. Beyond target generation, we introduce Safety-Aware Score Correction (SASC) to address the increased difficulty of learning reliable safety rankings under broader candidate coverage. Its two targeted safety regularizers penalize over-ranked unsafe candidates and raise underestimated safety scores for safe alternatives, helping reduce unsafe trajectory selection without overlooking valid alternatives. Table 11 supports this contribution: combining both regularizers reduces unsafe cases. We will explicitly clarify these distinctions in the final version of the manuscript.

Q1 - Teacher Target Appropriateness under Occlusion and Interaction: we assess teacher targets through PDM-based safety filtering and examine whether the distilled student performs well using only current observations, especially for those occlusion or interactive scenarios. As visualization results shown in updated Appendix G.1, the teacher selects a trajectory with higher progress than the recorded human trajectory. Crucially, the distilled student also achieves better PDMS than the recorded human trajectory. This illustrates that the student can achieve effective planning under partial occlusion or interactive scenarios, beyond merely demonstrating a higher-scoring privileged teacher.

Q2 - Denoiser Inputs, Scoring, and Gradient Flow: as specified in Sec. 3.1, the denoiser receives noisy residuals, timesteps, ego status, and shared scene tokens, with proposals entering indirectly through the training residuals without explicit proposal coordinates or proposal-specific features. Scoring losses are computed on target-conditioned, single-pass reconstructions from Eq. (3) during training, while full 8-step DDIM sampling is used at inference (Appendix A). Regarding gradient flow, Eq. (3) explicitly applies stop-gradient to proposals through sg(bm)\mathrm{sg}(b_m), and Sec. 3.4, following Eq. (9), specifies that candidate selection and evaluator labels are detached. To further clarify the scorer interface, candidate coordinates are also detached before scoring, preventing scoring gradients from propagating through trajectories into the denoiser or proposal decoder.

Following the reviewers' suggestions, we have revised the Limitations section in the appendix and added a new More Discussion section, with changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.