acceptodds
Under review as a conference paper at ICLR 2027

PreFTO: Reinforcement Fine-Tuning of Diffusion Planners with Local Execution Feedback

Abstract

Imitation-trained diffusion planners generate long reference trajectories, but de- ployed vehicles execute only a short segment before replanning. Reinforcement fine-tuning therefore needs feedback that connects generated plans to their long- term driving consequences. Scoring only a short execution can miss delayed hazards, whereas following an entire plan can penalize continuations that deploy- ment would replace. Full rollouts with repeated replanning are costly, and local scores must also distinguish candidates under realistic execution conditions. We study how finite local execution feedback can improve a replanning planner while retaining its pretrained trajectory generator. Under a safety-preserving continuation model with a common expected quality decrement, our analysis identifies group- centered local scores as relative advantages with an additional safety-deficit penalty. A separate sufficient condition connects score gains to deployment improvement through candidate-dependent score–value discrepancy and execution discrepancy. We introduce Prefix-Rewarded Full-Trajectory Optimization (PreFTO). It compares plans from shared policy-visited states through surrogate controller execution, uses continuous safety margins with minimum aggregation, and applies group-relative feedback to the complete original plans with DiffusionNFT. This separates the feed- back window from both the generated trajectory and the deployment replanning interval. On reactive nuPlan, PreFTO improves the pretrained Diffusion Planner on all three tested splits, including a 4.58-point gain on Test14-hard. Ablations demonstrate that proper choices of feedback length, learning targets, and loss coverage can effectively improve the closed-loop behavior of an RL post-trained planner.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement learning, Diffusion models, Autonomous driving, Action chunking, Policy fine-tuning, Receding-horizon planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_rCGtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces PreFTO, a reinforcement fine-tuning method for an imitation-trained Diffusion Planner. Its central design separates four horizons: the 8s full plan H_p, 4s evaluation window, 1.5s replanning interval H_a, and 8 reinforcement-loss coverage H_g. At policy-visited states, PreFTO samples 16 plans, executes each through the TwoStage controller for 4s with reactive IDM traffic, and scores the resulting motion without additional planner calls.

The theoretical contribution has two parts. Proposition 1 interprets centered local scores as relative advantages plus a safety-deficit preference, conditional on a nonempty safety-preserving continuation class and a candidate-independent expected quality decrement. Proposition 2 gives a sufficient condition relating local-score improvement to deployment improvement through score–value and execution-discrepancy terms.

Validation uses reactive and non-reactive nuPlan on Val14, Test14-random, and Test14-hard. Relative to the locally evaluated IL initialization, PreFTO improves the aggregate score on all six comparisons. Ablations support the selected 4 s feedback window, original-plan targets, shared-state centering, controller-based scoring, and continuous minimum-aggregated safety, although several effects vary by split.

Critical strengths
  1. The method addresses a real mismatch between long-horizon generation and short-horizon replanning with a coherent, testable design. PreFTO distinguishes prediction, evaluation, execution, and loss-coverage horizons instead of treating them as interchangeable. Its shared-state, controller-executed comparisons also expose tracking and interaction effects while avoiding repeated planner inference during scoring.

  2. The empirical study is diagnostic about which parts of the local-feedback pipeline matter. Beyond aggregate performance, the paper varies horizon, target, state source, tracking model, score centering, safety continuity, aggregation, reward composition, group size, expert mixing, etc.

Critical weaknesses
  1. The principal empirical claim is substantially weaker than its presentation suggests, and the evaluation does not establish reliable generalization. The highlighted 4.58-point gain on reactive Test14-hard is computed against the authors’ locally evaluated Diffusion Planner initialization (68.21 -> 72.79), not against the strongest entries in Table 1. PreFTO remains below G2DP (73.91) and PDM-Closed (75.19) on that column. At the same time, these external results were imported from different published protocols rather than reproduced under the authors’ evaluation setup, so Table 1 cannot support a controlled state-of-the-art comparison. More importantly, most principal results use one optimization seed and one evaluation seed, while Test14-hard informed training sampling, method settings, and checkpoint selection. The supplement consequently acknowledges that Hard is not an independent model-selection test. Because the largest reported gain occurs on this selection-informed split, and the gains on Val14 and Random are comparatively small, the current evidence is insufficient to establish robust or held-out improvement.

  2. Critical components of the experimental protocol are insufficiently specified, preventing reproduction and obscuring the source of the reported gains. Controller execution is central to PreFTO: the TwoStage controller transforms generated references into the motion used for reward calculation, and the perfect-tracking ablation shows that this transformation materially affects performance. Nevertheless, the paper does not provide a self-contained description of the TwoStage controller, its tracking law and parameters, state initialization, or whether its operation differs between candidate scoring and deployment. Likewise, Test14-hard is repeatedly reported as “Hard,” but the paper does not explain how its 272 scenarios were selected or what operational criteria make them hard.

  3. The algorithmic novelty relative to closely related diffusion-planner fine-tuning methods is insufficiently isolated. PreFTO combines policy-visited state collection, shared-state candidate sampling, controller-executed scoring, group-relative feedback, and diffusion-policy reinforcement fine-tuning. However, the paper itself states that PlannerRFT already combines closed-loop state collection, controller-executed candidate groups, adaptive exploration, diffusion-policy optimization, and comparisons of multiple reward windows. PreFTO additionally adopts the existing DiffusionNFT optimization objective rather than introducing a new underlying RL algorithm. The apparent new elements are therefore the specific continuous minimum-aggregated safety reward, the use of local execution scores to weight complete original-plan targets, and the accompanying horizon/discrepancy analysis. While potentially useful, these contributions are largely a particular scoring and target-design configuration built upon existing collection and optimization techniques.

Questions for authors
  1. Can you report independently repeated end-to-end runs (including policy-state collection, candidate generation, optimization, and evaluation) with uncertainty estimates on a split that was not used for training-scenario sampling, method design, checkpoint selection, or other model-selection decisions? A reliable positive result under such a protocol would materially change my assessment.

  2. How exactly were the 272 Test14-hard scenarios selected, and what criteria distinguish this split from Test14-random? Please also enumerate how Hard influenced training sampling, hyperparameter or reward design, ablation decisions, and checkpoint selection. This information is necessary to determine whether the 4.58-point improvement should be interpreted as held-out generalization or selection-set performance.

  3. Please provide a reproducible description of the TwoStage controller, including its tracking mechanism, parameters, controller-state initialization, waypoint lookahead, and any differences between candidate scoring and deployment. How sensitive are PreFTO’s conclusions to this particular controller? Since the reward is computed from controller-executed motion and perfect tracking changes the results, the answer could materially affect whether the gain is attributable to PreFTO itself or to controller-specific interactions.

  4. Please explicitly identify which comparisons in Table 1 are performed under identical scenario manifests, simulator settings, replanning intervals, controllers, and evaluation code. Is the intended claim only that PreFTO improves the local Diffusion Planner initialization, or do the authors claim superiority over external methods? Under the reported numbers, PreFTO does not outperform G2DP or PDM-Closed on reactive Test14-hard.

  5. It would be useful to study whether PreFTO can cooperate with structured test-time sampling methods such as [1] for example, by using tree-guided trajectories during candidate generation or applying tree-guided inference to the fine-tuned model.

  6. What precisely is the algorithmic distinction between PreFTO and PlannerRFT beyond the reward definition, learning-target choice, and application of DiffusionNFT?

[1]: Hyeonseong Jeon et al. Tree-Guided Diffusion Planner. 2025.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer rCGt

We thank you for your detailed review. We now address the concerns here.

  1. Repetitions on a split free of selection (W1/Q1). See Common Comment 1. Common Comments 5 and 6 show significant gains under the same protocol.
  2. Test14-hard (Q2). Test14-hard follows PlanTF. For each of the 14 scenario types, 100 test-split scenarios are simulated with PDM-Closed, and the 20 lowest-scoring ones are kept, giving 272 scenarios after availability filtering. Test14-random instead samples scenarios at random within each type. In the submission, Hard informed checkpoint selection. As stated in Appendix E.1, it also informed the main model and method settings, and Mini/Hard type frequencies informed training sampling. Training scenarios never overlap the evaluation manifests.
  3. Controller (W2/Q3). We use nuPlan's official TwoStageController, an LQR tracker followed by a kinematic bicycle model.
  • Tracking: the tracker looks 1 s ahead (10 points at 0.1 s) and fits velocity and curvature profiles by regularized least squares, with a jerk penalty of 1e-4 and a curvature-rate penalty of 1e-2.
  • Longitudinal LQR: Q = 10, R = 1.
  • Lateral LQR: Q = diag(1, 10, 0) on lateral error, heading error and steering, and R = 1 on steering rate.
  • Stopping: below 0.2 m/s, a proportional stopping controller with gain 0.5 takes over.
  • Vehicle model: Pacifica parameters (wheelbase 3.089 m), a steering limit of ±π/3, and an integration step of 0.1 s.
  • Initialization: scoring starts from the captured ego state (pose, velocities, accelerations and steering), with IDM agents initialized from the detections at that time. Scoring and deployment use the same controller and parameters. The only difference is that scoring tracks one fixed plan for 4 s without replanning, whereas deployment replans every 1.5 s. Using perfect tracking for scoring instead lowers Val14 from 84.27 to 83.33 and Test14-hard from 72.79 to 70.72 (Table 3 of the submission).
  1. Table 1 (Q4). IL and PreFTO are evaluated in our harness. PDM-Closed was re-simulated in the same harness and matches its published row. All other rows are published scores on the same manifests (Common Comment 2).
  2. Tree-Guided Diffusion Planner (Q5). We now cite it in the related work (Section 2) and list combining it with PreFTO as future work (Section 7). It is complementary to PreFTO: tree-guided sampling could supply candidates during collection, or run at inference on the fine-tuned model.
  3. PlannerRFT (W3/Q6). See Common Comment 7. We also corrected our related-work description of PlannerRFT: according to its Sec. 4.4 and App. C, its candidate groups are scored open-loop with PDMS rather than by controller execution.

Thank you for answering raised questions and running the experiments. I raised my rating, but I am still leaning toward rejection due to the limited algorithmic novelty and marginal improvement.

Reviewer_sFc6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies reinforcement fine-tuning of diffusion planners for autonomous driving, focusing on the mismatch between long trajectory generation and short execution before replanning. They argues that short execution feedback can miss delayed hazards, while executing an entire plan without replanning can penalize continuations that deployment would correct. To address this, the paper proposes PreFTO, which evaluates candidate plans from the same policy-visited state through local controller execution and uses group-relative feedback to optimize the original full trajectories with DiffusionNFT. Its reward combines continuous safety margins with driving quality, using minimum aggregation to retain sensitivity to brief hazards. Experiments show improvements over the various pretrained Diffusion Planner across all three splits.

Critical strengths

The paper is well motivated and addresses a concrete problem in reinforcement fine-tuning for diffusion-based autonomous driving. It clearly distinguishes the generation horizon, evaluation window, replanning interval, and loss coverage, making the problem formulation precise and the relevant trade-off explicit. The proposed approach is reasonable, using local controller execution to evaluate candidate plans while retaining the original full trajectories as learning targets. This design accounts for how generated plans translate into executed motion and provides a coherent way to incorporate execution feedback while retaining the pretrained planner’s full trajectory representation.

Critical weaknesses

There are some explanations for why an intermediate scoring horizon can outperform both shorter and longer horizons: short evaluation windows may miss delayed hazards, while long windows may penalize continuations that deployment replanning could correct. However, Table 2 compares only the final performance of models trained with different scoring horizons and does not directly verify that this trade-off explains the advantage of 4-second scoring. In particular, it remains unclear whether 4-second scores rank candidate plans more accurately according to their deployment returns. This interpretation would be strengthened by comparing local scores at different horizons with returns obtained by executing each candidate for the deployment interval and then continuing with replanning. Reporting some metric that can measure this would provide more direct evidence for the proposed explanation.

Questions for authors

Could the authors include an expert-only continued fine-tuning baseline, starting from the same imitation-trained Diffusion Planner checkpoint and using only expert demonstrations throughout fine-tuning? Keeping the batch size, number of optimizer updates, loss for expert examples, and pretrained-model anchor settings consistent with PreFTO would help isolate the contribution of reward-guided policy updates. Although Table 13 includes a no-expert ablation, it remains unclear whether continued expert imitation alone could achieve comparable gains over the initial IL checkpoint.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer sFc6

We thank you for your detailed review. We now address the concerns here.

  1. Horizons vs. deployment return (W). As suggested, we executed each candidate for 1.5 s and then replanned every 1.5 s up to 12 s. For local scores at 1.5, 2, 4 and 8 s: The within-group Spearman correlation with this deployment return is 0.58, 0.62, 0.43 and 0.32. The within-group standard deviation of the local score, which measures the learning signal, is 0.0075, 0.0107, 0.0335 and 0.0504. The share of locally safe candidates that later violate a safety margin is 27%, 25%, 19% and 14%. Longer windows thus give more learning signal and catch more delayed hazards, but they agree less with the deployment return. Common Comment 4 tests this trade-off directly.
  2. Expert-only fine-tuning (Q). We ran it as specified; see Common Comment 6.

Thank you for the additional experiments and my concerns are addressed. I'll keep my score.

Reviewer_stnyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PreFTO which is a reinforcement fine-tuning strategy for diffusion planner for autonomous driving planners. The method evaluates candidate trajectories using short execution feedback and applies group-relative rewards to fine-tune complete trajectories through DiffusionNFT. It also propose continuous safety rewards and separates prediction, execution, evaluation, and learning horizons. Experiments demonstrate consistent improvements over the imitation-trained diffusion planner, particularly in challenging driving scenarios.

Critical strengths

(1) Practical and well-motivated framework

  • The paper addresses an important mismatch between long-horizon trajectory generation and short-horizon execution in autonomous driving. The proposed components are reasonably motivated and integrated into a coherent framework.

(2) Solid empirical evaluation

  • The paper provides extensive ablation studies examining reward horizons, learning targets, candidate comparisons, and safety aggregation. The results demonstrate meaningful improvements over the pretrained baseline, especially on challenging scenarios.
Critical weaknesses

(1) Limited methodological novelty

  • Although the framework combines several reasonable design choices, most of its key components build directly on existing approaches, including group-relative rewards, controller-based trajectory evaluation, and DiffusionNFT. Related work such as PlannerRFT already explores several similar ideas. Consequently, the contribution appears largely incremental, and it is unclear what fundamentally new methodological insight PreFTO provides beyond combining and refining existing techniques.

(2) Limited evidence of generalizability

  • All experiments are conducted using a single pretrained Diffusion Planner architecture. Since the proposed method is presented as a general approach to reinforcement fine-tuning of diffusion planners, its effectiveness across different planner architectures remains unclear. Evaluating at least one additional diffusion-based planner would substantially strengthen the contribution.

(3) Insufficient justification for full-trajectory optimization using partial feedback.

  • A central assumption is that rewards obtained from 4 seconds of execution can guide optimization of the entire 8-second trajectory. However, a low reward during the first 4 seconds does not necessarily imply that the entire trajectory is undesirable, and the reverse is also possible. Although the empirical results support the effectiveness of this design in some settings, they do not establish that partial execution rewards adequately reflect the quality of complete trajectories. The theoretical analysis also does not directly validate this assumption under realistic driving conditions.
Questions for authors

(1) Methodological novelty: What is the most important methodological difference between PreFTO and existing approaches such as PlannerRFT, beyond the combination of established components?

(2) Generalizability: Is there evidence that PreFTO also improves other diffusion-based planners? How sensitive is the method to the underlying planner architecture?

(3) Validity of partial feedback: Have the authors examined the correlation between 4-second rewards and the actual quality of complete 8-second trajectories? Such an analysis would help justify using partial rewards to reinforce or suppress entire trajectories.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer stny

We thank you for your detailed review. We now address the concerns here.

  1. Novelty (W1/Q1). See Common Comment 7.
  2. Generalizability (W2/Q2). See Common Comment 5.
  3. Partial feedback (W3/Q3). Within groups, the 4 s score has a Spearman correlation of 0.60 with the 8 s fixed-plan score and 0.43 [0.30, 0.55] with the deployment return. A low 4 s score thus predicts a worse complete trajectory, although not perfectly. The 8 s fixed-plan score is not the right target either, because deployment replaces the plan after 1.5 s. Replanning changes the 8 s score much more than the 4 s score (Common Comment 4, Prop. 2).

Thank you for your thoughtful responses to my concerns and questions. Although some of my concerns have been addressed, I still believe that the algorithmic novelty is incremental. Therefore, I will maintain my original score.

Reviewer_2eMrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PreFTO, a reinforcement fine-tuning framework for diffusion-based driving planners that uses local execution feedback to update complete predicted trajectories. It scores controller-executed candidate plans using safety and driving-quality metrics and applies group-relative feedback through DiffusionNFT. The paper reports Diffusion Planner’s performance before and after fine-tuning.

Critical strengths
  1. The authors explicitly acknowledge limitations of the theoretical assumptions and evaluation protocol.
  2. The supplementary material documents training settings and the sources of reported baseline scores.
Critical weaknesses
  1. Model selection on the test set. Test14-hard (Hard) is a test set on which the abstract highlights a 4.58-point gain. However, Section 7 states that “Hard informed checkpoint selection.” Appendix E.1 further confirms that Hard informed both model and method-setting selection. The highlighted gain is therefore subject to selection bias and cannot serve as independent evidence of generalization. Evaluation on a test set excluded from model and method selection is needed.
  2. Unclear advantage over existing methods. The paper does not compare PreFTO with alternative fine-tuning methods such as “PlannerRFT” and “M-Diffusion Planner” under matched conditions. The authors explicitly state in Appendix E.1 that the external baselines in Table 1 retain their published evaluation protocols and were not rerun under their protocol. PreFTO also leads none of Table 1’s six evaluation columns. Its advantage over existing methods therefore remains unclear.
  3. Unvalidated link between theory and practice. The continuation assumptions and discrepancy bounds are central to the paper’s theoretical interpretation and policy-improvement guarantee, yet the authors acknowledge in Section 7 that neither is experimentally verified. In Appendix A.2, they also state that the analysis “does not establish that local scores accurately rank the reference policy’s values.” Its applicability to the implemented method therefore remains unvalidated.
  4. Dense and difficult-to-follow writing. The Introduction (p. 2) packs “safety-preserving continuation,” “common expected quality decrement,” and “safety-deficit penalty” into a single sentence without intuitive explanation. Although formalized later, these concepts obscure the central claim when introduced together. A plain-language explanation or concrete example is needed.
  5. Limited evidence of generalization across planner families. Section 7 acknowledges that the evidence “covers one planner family.” Experiments on multiple data splits do not establish whether PreFTO’s benefits extend beyond the tested Diffusion Planner backbone. Evaluation on other planner architectures is needed to assess whether the gains generalize beyond Diffusion Planner.
Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 2eMr

We thank you for your detailed review. We now address the concerns here.

  1. Test-set selection. See Common Comment 1.
  2. Existing methods. See Common Comment 2.
  3. Theory–practice. See Common Comment 4.
  4. Writing. We have polished the writing. In plain language, the dense sentence in the Introduction makes two simplifying assumptions. First, a candidate that is safe over the scoring window stays safe afterwards. Second, from a given state, the remaining driving quality drops by the same expected amount for every candidate. Under these assumptions, comparing local scores within a group compares long-term advantages, with an extra preference for safer candidates.
  5. Planner families. See Common Comment 5.

Comments on the paper

2 comments
AuthorsOpenReview ↗

common comments - part1

We thank all reviewers. We address the shared concerns here.

  1. Model selection (2eMr, rCGt; Appendix F.1). In the submission, Test14-hard informed checkpoint selection. We now select checkpoints only on val-hard, which consists of 79 hard scenarios from the nuPlan val split. Each selected checkpoint is evaluated once on Test14-hard and on a held-out split of 482 scenarios. The held-out split was frozen before any evaluation, excludes every scenario within 30 s of a Val14 or Test14 scenario, and was never used for any decision. We ran three independent end-to-end seeds, each covering state capture, candidate collection, optimization and evaluation. The selected epochs are 10, 4 and 6. On Test14-hard, the three seeds score 69.6, 70.4 and 70.7, compared with 68.63 for IL, a gain of +1.58 [−0.92, 4.15]. On held-out, they score 82.7, 81.8 and 81.8, compared with 81.69 for IL, a gain of +0.40 [−0.80, 1.57]. All seeds score above IL.

  2. Existing methods (2eMr, rCGt; Section 6.1). PlannerRFT, M-Diffusion Planner and DFP release no code or checkpoints, and PlannerRFT's nuMax simulator is also unreleased, so we can only cite their published scores. All rows of Table 1 use the standard Val14, Test14-random and Test14-hard manifests. Re-simulating the open-source PDM-Closed in our harness reproduces its published scores within 0.01, so our evaluation protocol matches the published one.

PDM-Closed is a rule-based planner that scores its own proposals with the metric. G2DP adds inference-time guidance with explicit occupancy and progress costs, and DFP and KnowDiffuser also change inference. These methods are orthogonal to post-training, which keeps the architecture and inference unchanged. Among RL post-training methods, PreFTO is above all M-Diffusion Planner variants. It is comparable to PlannerRFT under reactive traffic: 84.27 vs. 84.46 on Val14, 85.38 vs. 85.80 on Test14-random, and 72.79 vs. 72.21 on Test14-hard. We claim only improvement over the same IL initialization, which is the official Diffusion Planner checkpoint, and we do not claim superiority over G2DP or PDM-Closed.

  1. Protocol details (rCGt). The controller is nuPlan's official TwoStageController (https://github.com/motional/nuplan-devkit). Full details are in our reply to rCGt.

  2. Theory–practice (2eMr, stny, sFc6; Appendix F.3). We used 288 IL states from 96 Val14 scenarios, with 16 candidates per state. Proposition 1 says that the local score reflects the global return if the continuation keeps locally safe candidates safe. We recorded one common set of rollouts: every candidate is executed for 1.5 s, and the deployment policy then replans every 1.5 s. For each H, the evaluated safety of a candidate is its local score over the first H seconds of the fixed plan, as in training. Its actual safety is that of the rollout over the next 8 s. Because all horizons share these rollouts, the same 87.6% of candidates are actually safe (no collision or off-road) for every H; only the evaluated split changes.

For H = 1.5, 2, 4 and 8 s, the results are as follows:

  • Evaluated safe and actually safe: 87.6%, 87.6%, 83.5% and 74.3%.
  • Evaluated safe but actually unsafe: 7.1%, 7.0%, 4.7% and 2.1%.
  • Evaluated unsafe but actually safe: 0.0%, 0.0%, 4.1% and 13.3%.
  • Evaluated unsafe and actually unsafe: 5.3%, 5.3%, 7.7% and 10.2%. Short windows miss hazards that appear later, whereas long windows penalize plans that replanning keeps safe. The probability that a locally safe candidate stays safe rises from 0.93 to 0.97 with the horizon. When any safety-margin violation counts as unsafe, agreement between evaluated and actual safety is highest at 4 s (0.84, 0.86, 0.89 and 0.84). With a privileged PDM-Closed planner as the continuation instead, 92–96% of locally safe candidates stay safe at every horizon, so a safety-preserving continuation exists for most candidates.

Proposition 2 attributes the remaining error to evaluating beyond the 1.5 s replanning interval. We compared the local score before and after replacing the fixed plan with deployment replanning over the same window. For H = 1.5, 2, 4 and 8 s, the within-group Spearman correlation is 1.000, 0.947, 0.631 and 0.365, and the mean absolute difference is 0.000, 0.007, 0.062 and 0.164. Longer windows catch delayed hazards (Prop. 1) but enlarge this execution error, and this trade-off favors 4 s.

AuthorsOpenReview ↗

common comments - part2

  1. Other planners (2eMr, stny; Section 6.2). We applied PreFTO unchanged to the released Flow Planner, which uses flow matching and chunked action tokens. Starting from its official checkpoint, Flow-PreFTO (epoch 10, seed 0) improves over Flow Planner on every split:
  • Val14: +2.29 [1.07, 3.55]
  • Held-out: +2.59 [0.77, 4.57]
  • Test14-hard: +1.62 [−0.71, 4.18]
  • Test14-random: +0.95
  • Held-out-hard: +0.15 The Val14 and held-out gains are significant. Flow matching is not strictly diffusion, but Flow Planner is the closest released generative planner whose backbone and tokenization differ from Diffusion Planner's.
  1. Expert-only fine-tuning (sFc6; Appendix F.2). This control starts from the same IL checkpoint and keeps the batch, number of updates, expert loss and SFT anchor, but removes the scored candidates (epoch 10). Compared with IL, expert-only fine-tuning changes the score on Val14, Test14-random, Test14-hard, held-out and held-out-hard by −2.55, −3.43, −3.10, −2.65 and −1.63. The drops on Val14, Test14-random and held-out are significant. PreFTO exceeds this control on the same five splits by +2.65, +3.06, +4.03, +3.61 and +3.07, all significant. The gain therefore comes from reward-guided updates, not from more expert data.

  2. Novelty (stny, rCGt). First, we give a systematic analysis of horizons in diffusion-policy RL. We explain why a prefix reward can train the full trajectory (Prop. 1) and where its error comes from (Prop. 2), and both are now measured (item 4). We also tune the horizon choices. In contrast, previous work such as PlannerRFT compared 2-, 4- and 6-second reward windows only empirically, without explaining the outcome. Second, we build the group data specifically for NFT. Candidates are generated from closed-loop, policy-visited states. Each candidate is scored by its controller execution in reactive traffic, while its raw plan is the learning target. Groups whose candidates all receive the same reward carry no learning signal and are discarded, so the planner learns only from meaningful reward differences. This sampling procedure is tailored to the NFT paradigm. In contrast, PlannerRFT scores its GRPO candidates open-loop with PDMS (its Sec. 4.4 and App. C).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.