acceptodds
Under review as a conference paper at ICLR 2027

Beyond Imitation: Planner-Agnostic Reinforcement Learning Post-Training for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving has advanced rapidly on large-scale closed-loop benchmarks such as nuPlan, with planners commonly based on regression, diffusion, or flow matching. Although imitation learning (IL) provides strong behavioral priors, it primarily optimizes similarity to logged expert trajectories rather than task-level driving metrics such as safety, progress, comfort, and interaction quality. Reinforcement learning (RL) can address this limitation, but existing RL post-training methods are often coupled to planner-specific generation mechanisms, resulting in fragmented solutions across planner families. We propose Planner-Agnostic Reinforcement Learning (PARL), a unified post-training framework that isolates planner-specific differences in a lightweight log-probability adapter while sharing the remaining RL pipeline, including reward computation, KL regularization, and policy optimization. PARL supports regression-based, diffusion-based, and flow-matching-based planners without modifying their architectures, inference interfaces, or action spaces, and requires no privileged information. Under the full nuPlan closed-loop protocol, covering non-reactive and reactive settings on Val14, Test14-hard, and Test14, PARL consistently improves all six evaluated IL planners across all six settings, with the largest gains in interaction-heavy long-tail scenarios. These results demonstrate the feasibility and effectiveness of unified, planner-agnostic RL post-training for end-to-end autonomous driving. Code and pretrained checkpoints will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Motion Planning, Reinforcement Learning, Imitation Learning, Planner-Agnostic Framework

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HxUWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PARL, a planner-agnostic post-training framework for nuPlan end-to-end planners. It breaks down post-training of RL into a shared GRPO pipeline and a log-probability adapter that is replaced by planner family.

Critical strengths

The experiments were very thorough, including six planners, three parameterizations, and all six closed-loop settings were positive. Furthermore, the pure RL rows in Tables 4 and 5, without any regularization terms, still showed a gain of +1.4 to +1.5. This indicates that "open-loop metric reward + GRPO" can indeed improve the closed-loop performance of both regression and diffusion IL planners.

Critical weaknesses

The paper's descriptions at the methodological, experimental, and evidentiary levels are contradictory. The objective function of the regression adapter in Fig. 1, Eq. (4) in the main text, and the sampling K=4 per scene described in Appendix A.3 are inconsistent. The expected value of all modes in the figure is a sampled score-function estimate, while the appendix describes a group sampling process. Sec. 3.3 states "strictly on-policy, single-update," but Sec. 4.3 and Table 8 show z^old, stored rollout, and replay-implied noise, indicating no replay in a single update. DIAMOND is classified as a diffusion planner and applies the DDIM adapter, but Appendix A.2 describes it as an IL planner that "synthesizes training data using diffusion," without specifying whether its policy head is a denoising chain. ReflexDiffusion is an inference-time module, not a planner. The underlying training mechanism is not explained, and the 3.65-point difference between ReflexDiffusion-PARL and Diffusion Planner-PARL is also unexplained. Furthermore, the framework of "sharing the GRPO pipeline and only replacing log-prob computation" was proposed by DDPO, Flow-GRPO, and DanceGRPO in 2025. The latter two explicitly unified diffusion and flow matching, yet this paper does not cite any of them.

Questions for authors
  1. What objective does the regression adapter actually optimize? Does it compute the expected reward over all MM proposals, or sample K=4K=4 proposals for REINFORCE? Is there only one update after each rollout, or is the same batch reused for multiple updates?

  2. Does the DIAMOND policy use a denoising chain? If not, why is it classified as a DM-family model and assigned the DDIM adapter? Which planner serves as the base for ReflexDiffusion post-training? If it is Diffusion Planner, why does ReflexDiffusion-PARL score 3.65 points lower than Diffusion Planner-PARL?

  3. In Fig. 3 and Fig. 4, do the three rows in each column show the same scene? The road geometry suggests otherwise. Please provide renderings from all three planners on the same scene and explain how the examples were selected. How exactly does the reward correct the rear-end collision in Fig. 4(b)? Is the red circle misplaced?

  4. Please supplement the comparison with scorer-based test optimization methods, including PLUTO with post-processing, Diffusion Planner with refinement, and using your own PyTorch scorer for best-of-K. Is PARL intended to replace these methods, or can it be combined with them? What are the results of adding post-processing after PARL?

  5. In Table 7, row 2, adding only the KL anchor lowers the score by 1.01 points; Table 4, row 2, also shows a decrease of 0.22 points. If this term is intended to anchor the model to IL, why does it move performance further away from the IL baseline?

  6. Table 8, row 1, scores 84.67 without the RL gradient, while Table 7, row 3, scores 84.68 with the RL gradient but without KL. These results are almost identical. Does the RL gradient contribute virtually nothing to FlowDrive?

  7. Is the reported 0.45-point improvement entirely due to the KL term with β=0.001\beta=0.001?

  8. Which sampler and how many sampling steps were used for the closed-loop evaluation of the post-trained Diffusion Planner and FlowDrive? Did evaluation use the original checkpoint settings or the post-training setting of L=4L=4?

  9. Is the clean-trajectory ℓ2\ell_2 IL-retention loss used for diffusion consistent with Diffusion Planner's original training objective? If not, please explain the difference.

  10. How many epochs were actually used for RL post-training? Does “follow the original IL configuration” include the number of training epochs, meaning the same number used for Diffusion Planner's original IL training?

  11. Appendix A.1 states “15 s planning horizon,” but Sec. 3.1 specifies T=80T=80 frames at 10,Hz10\\,\mathrm{Hz}, which corresponds to 8,s8\\,\mathrm{s}. Please clarify the actual setting.

  12. The captions of Table 1 and Table 2 say that improvements are shown in parentheses, but the tables use superscripts. Please make these consistent.

1 comment

We thank the reviewer for the careful reading and constructive suggestions. All points refer to the submitted version;

C1 (regression-adapter objective; on-policy consistency; Q1)

  • Answer: The adapter optimizes Eq. (4): each scene samples K=4K=4 trajectories from the current policy, the advantage of Eq. (2) is computed from their rewards, and the expectation is over these samples, not all modes. Each batch drives one gradient step (Sec. 3.3). The stored trajectory (Sec. 4.3, Table 8) is the current group's own sample; its implied noise is recomputed (Eq. 10) to retain the gradient through the velocity field, not reused across updates; single-update and on-policy are consistent.
  • Evidence: Eqs. 2-4; Secs. 3.3, 4.3.
  • Boundary: the expectation symbol in Fig. 1 renders Eq. (4) compactly, not enumeration over modes; it will be corrected.

C2 (DIAMOND and ReflexDiffusion; Q2)

  • Answer: DIAMOND plans by iterative denoising, so it receives the DDIM adapter; the synthesized rollouts of App. A.2 are additional data, not the policy. ReflexDiffusion is a planner variant: its checkpoint couples the base planner with a denoising-time reflection adjustment; the base will be named. The 3.653.65 points on Val14 NR (91.4091.40 vs 87.7587.75, Table 1) follow from different IL initializations (89.8789.87 vs 87.2387.23) and sampling; the comparable quantities are the within-planner gains +1.53+1.53 and +0.52+0.52.
  • Evidence: Table 1; App. A.2.
  • Boundary: the base planner is not named in the paper; this will be added.

C3 (novelty relative to DDPO, Flow-GRPO, DanceGRPO)

  • Answer: DDPO is cited in Sec. 2.2; Flow-GRPO and DanceGRPO are not cited and will be added. The contribution is not the adapter-only principle but its extension to multi-hypothesis regression planners, categorical over modes with IL-supervised coordinates (Eq. 4, Sec. 4.3), under an unchanged inference interface, with closed-loop evaluation over three families (3636 cells, Table 1).
  • Evidence: Secs. 2.2, 3.3, 4.1, 4.3; Table 1.
  • Boundary: the shared-pipeline idea overlaps with them; we will state the overlap.

C4 (Figs. 3 and 4; Q3)

  • Answer: Within a row the IL and PARL panels share one scene and initial conditions; rows are different scenes chosen for each failure mode, and captions will say so. In Fig. 4(b) the IL policy drives below traffic speed and is struck from behind; the reward weighs progress (Eq. 1, weight 55) and penalizes collisions, so post-training raises speed and removes the conflict. The red-circle annotation will be re-checked and corrected.
  • Evidence: Figs. 3 and 4; Sec. 5.2; App. B.1; Eq. 1.
  • Boundary: renderings of all three planners on one scene are unavailable; the paired panels are the controlled evidence.

C5 (scorer-based test-time optimization; Q4)

  • Answer: Scorer-based test-time optimization was not run, alone or after PARL. PARL is training-time and leaves the inference interface unchanged (Sec. 4.1), so the two are complementary; no replacement is claimed.
  • Boundary: no number can be reported for post-processing after PARL.

C6 (KL-only rows; Q5-Q7)

  • Answer: The gains are not attributable to KL. In Table 7 KL alone lowers the average by 1.011.01 (68.9268.92 to 67.9167.91) while IL retention alone raises it by 5.765.76 (68.9268.92 to 84.6884.68), and the full configuration reaches 85.1285.12; for PLUTO KL alone costs 0.220.22 (78.9978.99 to 78.7778.77) and both terms together add 1.011.01. The anchor constrains the policy distribution, not the closed-loop score, and in isolation suppresses the metric-driven update; it helps once IL retention holds the policy near the reference. For FlowDrive, 84.6784.67 (Table 8 row 1) and 84.6884.68 (Table 7 row 3) are two single-factor removals, each costing 0.450.45, not an inert RL gradient.
  • Evidence: Tables 4, 7 and 8.

C7 (evaluation sampler; retention objective; epochs; Q8-Q10)

  • Answer: Evaluation uses each planner's original inference procedure and interface (Sec. 4.1, App. A.2); the L=4L=4 steps of App. A.3 discretize the adapter likelihood, not the evaluation sampler. The clean-trajectory retention loss anchors the deployed output; it does not re-implement each original objective; the frozen copy is the reference. Epochs follow the original IL configuration (App. A.3).
  • Evidence: Secs. 4.1, 4.2; App. A.2, A.3.
  • Boundary: the sampler, its step count, and the epoch numbers are not listed; all three will be added.

C8 (format corrections; Q11-Q12)

  • Answer: The horizon is 8 s8\text{ s}, from T=80T=80 frames at 10 Hz10\text{ Hz} (Sec. 3.1); 15 s15\text{ s} in App. A.1 is a typo and will be corrected. The captions of Tables 1 and 2 will match the format used.
  • Evidence: Sec. 3.1; App. A.1; Tables 1 and 2.

We would welcome your view on the bar for acceptance: which clarifications, comparisons, and additional experiments above would be decisive to substantiate our claims. We will take your suggestions seriously in our revision, look forward to your reply, and wish you all the best in your work.

Reviewer_oYr1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PARL combines planner-specific log-probability adapters with a shared reward-based post-training objective, KL regularization, and imitation retention. Training scores sampled trajectories without simulator rollouts. Evaluation on six planners across three families shows consistent improvements under six nuPlan closed-loop settings.

Critical strengths
  1. Evaluation spans six planners across regression, diffusion, and flow matching, providing substantive evidence of applicability across different trajectory parameterizations.

  2. Improvements persist under both reactive and non-reactive traffic. Collision and TTC gains on Test14-hard support practical safety benefits beyond better imitation accuracy.

Critical weaknesses
  1. Figure 1 appears to be AI-generated, with many distorted and illegible characters. This falls short of the care expected in scientific communication.

  2. Comparisons include only IL initializations, without existing RL post-training methods or matched additional IL training. This limits evidence of PARL’s advantage over established approaches and isolates its contribution insufficiently.

  3. The method essentially combines three existing RL approaches and lacks methodological novelty. I do not believe it meets ICLR’s acceptance standards or makes a meaningful contribution to the community. An open-source implementation could provide a useful RL codebase, but its value is difficult to assess without access to the code.

Questions for authors
  1. How does PARL compare with existing RL post-training methods and continued IL training under matched computational budgets?

  2. What methodological novelty does PARL introduce beyond integrating existing RL approaches, and can you provide anonymized code for assessment?

1 comment

We thank the reviewer for the careful reading of our paper and for the constructive suggestions. We clarify the three points below with reference to the submitted version.

C1 (Figure 1 legibility)

  • Answer: We agree that the labels in Fig. 1 are not legible at print size; the figure was embedded as a raster and its text is distorted. The figure contains no results, and its content is fully defined by Eqs. 1-12 and Sec. 4.1; we will replace it with a vector rendering that uses the same notation.
  • Revision: Fig. 1 replaced; no change of content.

C2 (comparisons with existing RL post-training; matched additional IL training)

  • Answer: All comparisons are within-planner and within one evaluation harness, each post-trained planner against its own IL initialization (Table 1), which isolates post-training from differences in evaluation. The metric-driven update alone, with both regularizers removed, already accounts for +1.4+1.4 on PLUTO (77.577.5 to 78.978.9, Table 4 row 1) and +1.5+1.5 on Diffusion Planner (81.681.6 to 83.183.1, Table 5 row 1). Existing RL post-training methods are single-family by construction, DPPO within diffusion policies, DiffusionDriveV2 within its truncated model, and CarPlanner for autoregressive planners, so a common harness for them does not exist without re-implementation.
  • Evidence: Table 1; Tables 4 and 5, row 1.
  • Boundary: we did not run a matched-budget continued-IL control, nor re-implement the single-family RL methods, so we cannot report a head-to-head number; this is the comparison the paper lacks.

C3 (methodological novelty; code)

  • Answer: The novelty is not the optimizer: GRPO is adopted as the base and stated as such (Sec. 3.3). It is the factorization and what the factorization requires. Planner-specific differences are confined to a log-probability adapter, with architecture, inference interface and action space unchanged (Sec. 4.1). For flow matching, the likelihood is built by SDE lifting with replay-implied noise recomputation (Eqs. 8-10); removing it costs 0.40.4 points (85.185.1 against 84.684.6, Table 8). For diffusion, the likelihood uses a σ\sigma floor and a temporal discount over denoising steps (Eqs. 6-7, 84.084.0 against 83.583.5, Table 6). For regression, the gradient enters the mode policy while coordinates remain IL-supervised (Eq. 4, Sec. 4.3). The empirical form of the claim is that one recipe improves six planners across three families in all 36 planner-setting cells (Table 1), which no single-family method has shown. Code and post-trained checkpoints will be released upon acceptance, as stated in the Abstract, in accordance with institutional restrictions that currently prevent release before acceptance; if the reviewer wishes, we can provide the resulting evaluation tables in anonymized form during the discussion.
  • Evidence: Secs. 3.3, 4.1, 4.3; Eqs. 4, 6-10; Tables 1, 6, 8.
  • Boundary: the individual ingredients are established; the claim rests on the factorization and on its empirical validity across families, not on a new optimization algorithm.

We would welcome the reviewer's view of the bar: which of these would be decisive for you, namely a matched-budget continued-IL control, a re-implementation of a single-family RL method, or a specific ablation isolating the adapters from the shared recipe, and which experiments you would consider necessary to substantiate the claims. We will take your suggestions seriously in preparing the revision. We look forward to your reply and wish you all the best in your work.

Reviewer_f5dUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PARL applies GRPO-style trajectory-level reward post-training to six nuPlan planners across regression, diffusion and flow-matching families. A family-specific likelihood adapter supports a shared reward, reference KL and imitation-retention loss. The paper reports improvements across three full nuPlan splits in reactive and non-reactive modes and provides implementation details for each planner family.

Critical strengths
  1. The empirical coverage is valuable: six planners, three families, all three full splits and both closed-loop modes. Positive changes across the reported settings make the recipe potentially useful to practitioners beyond one architecture.

  2. The exposition and implementation details are clear, including how gradients are maintained for flow models. A common experimental framework and post-trained checkpoints would be useful practical outputs even where individual optimization ingredients are established.

Critical weaknesses
  1. The contribution beyond established post-training methods is not yet isolated. The categorical, diffusion and flow adapters use familiar techniques and family-specific stabilizations, so the unification is mainly empirical. Comparisons with planner-specific RL, reward-weighted or best-of-K fine-tuning, and inference-time selection using the same available scorer would establish what benefit the shared recipe provides. For regression planners, RL updates mode selection rather than trajectory coordinates, making a selection baseline particularly important.

  2. The model-selection protocol and statistical reliability need clarification. Ablations report results on the same named splits used for the headline evaluation, but it is unclear which results determined the final coefficients. Several gains are small, and no run-to-run variance is provided. Re-evaluation of the IL baselines and validation of the PyTorch scorer against the official metric are also needed. I regard the paper as borderline and provisionally lean toward weak rejection pending these clarifications.

Questions for authors
  1. Which data and criterion selected each family's regularization and stabilization settings? Were test14-hard and test14 untouched until selection was complete? Please clarify baseline provenance and provide seed variation for the smaller gains.

  2. How does post-training compare with planner-specific RL and a matched-budget reward-weighted or best-of-K control? For inference-time selection, which scorer inputs are available at deployment, and can a fair control use only those inputs?

  3. How closely does the PyTorch reward agree with the official scorer? What future information about other agents is used during reward computation, and how does this relate to reactive evaluation? Please also distinguish improved hypothesis selection from improved trajectory generation in regression planners.

1 comment

We thank the reviewer for the detailed comments. Below we respond with reference to the submitted version; all table and section numbers refer to it.

W1/Q2 (isolation of the contribution; planner-specific RL, reward-weighted and best-of-K controls)

  • Answer: The staged ablations isolate the shared recipe. Removing both regularizers leaves only the metric-driven update, which alone raises the six-setting average of PLUTO from 77.577.5 to 78.978.9 and of Diffusion Planner from 81.681.6 to 83.183.1 (Table 4 and Table 5, row 1); the full recipe adds +1.0+1.0 and +0.8+0.8 over those rows (row 4). The stabilizers are family-specific rather than uniform: FlowDrive collapses without IL retention (68.968.9 and 67.967.9, Table 7 rows 1-2, against 84.384.3 for its IL initialization) and reaches 85.185.1 with it. For regression, the update acts on the categorical mode policy (Eq. 4) while trajectory coordinates stay IL-supervised (Sec. 4.3), so the gain is realized through mode reweighting, which changes the deployed candidate as in the PLUTO pair of Fig. 3(a), where the IL plan collides and the post-trained plan decelerates earlier.
  • Evidence: Tables 4, 5, 7; Fig. 3(a); Eq. 4.
  • Boundary: we did not re-implement planner-specific RL (DPPO, DiffusionDriveV2, CarPlanner) or a matched-budget reward-weighted or best-of-K control. Each is architecture-coupled and needs its own training and reward harness, so we make no numerical claim against them.

W2/Q1 (model-selection protocol; baseline provenance; variance)

  • Answer: Each family is configured from a four-point grid over β\beta and λIL\lambda_{\mathrm{IL}}, with the adopted settings being those of the last row in Tables 4, 5 and 7. The conclusion does not hinge on the grid point: all PLUTO configurations lie above its IL initialization (78.77–80.078.77\text{--}80.0 against 77.577.5) and all Diffusion Planner configurations above 81.681.6 (83.17–84.083.17\text{--}84.0); for FlowDrive, the two configurations with IL retention give 84.684.6 and 85.185.1. The adapter settings are equally flat: four diffusion-adapter configurations span 83.32–84.083.32\text{--}84.0 (Table 6) and four flow-matching-adapter configurations span 84.37–85.184.37\text{--}85.1 (Table 8). All 36 planner-setting cells in Table 1 are positive, the smallest being +0.1+0.1 (PlanTF, Test14-hard R-CLS) and +0.2+0.2 (FlowDrive, Val14 R-CLS), each reported with its variation in the same table. All IL baselines are released checkpoints evaluated in the same protocol and harness as their post-trained counterparts (App. A.2, A.3), so the comparison is within one evaluation rather than across published numbers.
  • Evidence: Table 3; Tables 4-8; Table 1.
  • Boundary: the reimplemented reward shapes training only; every reported score comes from the official closed-loop protocol (Sec. 3.2, Sec. 5), so disagreement between the two scorers cannot inflate the evaluation. A direct agreement study is not part of the paper.

W3/Q3 (reward inputs; reactive evaluation; selection versus generation)

  • Answer: The reward uses the inputs of the corresponding official scoring logic, including other agents' recorded trajectories; the planner's own input is unchanged (Secs. 3.1-3.2). In reactive evaluation the simulator moves the surrounding agents, so recorded trajectories approximate what the scorer would see in closed loop; the gains remain positive in every R-CLS cell (Table 1). For regression planners the improvement is hypothesis selection by construction: the gradient enters the mode distribution (Eq. 4) while coordinates remain IL-supervised (Sec. 4.3); for diffusion and flow-matching planners the update acts on the generative process itself (Eqs. 7 and 12).
  • Evidence: Secs. 3.1-3.2, 4.3; Table 1; Eqs. 4, 7, 12.
  • Boundary: a fair inference-time selection control would need the inputs available on the vehicle, i.e. predicted rather than recorded neighbor trajectories; we did not run one.

We would welcome the reviewer's view of the bar: which of these would be decisive for you, namely a matched-budget comparison against planner-specific RL, an agreement study between the reward and the official scorer, seed-level statistics for the small gains, or an inference-time selection control, and which experiments you would consider necessary to substantiate the claims. We will take your suggestions seriously in preparing the revision.

We look forward to your reply and wish you all the best in your work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.