acceptodds
Under review as a conference paper at ICLR 2027

Don’t Rewrite, But Correct: Closed-Loop Manifold Correction via RL Post-Training for Generative Driving Policies

Abstract

Generative driving policies have become a dominant source of pretrained driving priors in autonomous driving. In particular, flow-based planners trained by imitation learning capture expert behavior well on routine scenarios, whereas a few long-tail states can trigger compounding errors during closed-loop interaction, leading to a substantial gap between offline generative quality and closed-loop performance. To this end, we propose Neighborhood Action Correction, a closed-loop neighborhood post-training framework for generative driving planners. Different from the typical RL fine-tuning approach that updates the whole pretrained planner and may reshape its behavior on routine states, NAC freezes the generative planner and only trains a lightweight corrector to adaptively correct its prior action, particularly on long-tail states. The trainable correction is bounded by a neighborhood and confined to the only variable that enters vehicle dynamics under receding-horizon execution. NAC enables efficient improvement of the planner’s closed-loop performance while preserving the strong prior manifold of the generative policy. Experiments on nuPlan show that NAC achieves state-of-the-art closed-loop performance with flow-matching planner and outperforms the evaluated RL fine-tuning methods.

Primary area:
reinforcement learning
Keywords:
Autonomous Driving, Reinforcement Learning Post-training, Flow-matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tA2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Neighborhood Action Correction (NAC), an RL-based post-training framework for generative autonomous driving planners. Instead of fine-tuning the pretrained trajectory generator, NAC freezes the planner and learns a lightweight residual policy that corrects the controller-projected action within a bounded neighborhood. The motivation is that, under receding-horizon execution, only the executed action directly influences vehicle dynamics, making action-space correction potentially more efficient and stable than trajectory-level optimization. The authors provide a theoretical analysis based on controller-null directions and policy-gradient variance, and evaluate NAC on several nuPlan benchmarks using flow-matching and diffusion-based planners. The reported results show notable closed-loop improvements, particularly on challenging scenarios. While the approach is practically interesting, I have several concerns regarding its methodological novelty, theoretical justification, and the extent to which the experiments substantiate its central claims.

Critical strengths

-The paper addresses an important problem in autonomous driving: improving closed-loop behavior without substantially modifying a pretrained generative planner.

-The action-space correction design is simple and computationally lightweight at inference time, making it relatively easy to integrate into existing trajectory-based planning pipelines.

-The experiments cover multiple nuPlan evaluation settings and include comparisons with trajectory-level correction, direct RL fine-tuning, and different generative planner backbones.

Critical weaknesses

-The theoretical analysis relies on restrictive assumptions, and some conclusions appear stronger than the derivations justify. Lemma 1 argues that action-space correction has lower policy-gradient variance because it avoids controller-null trajectory directions. However, the analysis assumes an isotropic Gaussian trajectory correction and matched executed-action distributions, whereas the actual implementation uses a bounded Beta policy and a nonlinear controller. The appendix proof also relies on a local linearization, making the stated covariance relationship difficult to interpret as a general result for the implemented PPO procedure. More importantly, the extension of this variance argument to direct planner fine-tuning is not fully convincing, since parameter-space gradients depend on the planner Jacobian and may not preserve the claimed strict variance dominance. Similarly, the proof of Lemma 2 assumes that an (O(\alpha)) action perturbation implies an (O(\alpha)) change in state occupancy under total variation, which does not generally follow without additional assumptions on the dynamics and transition distributions. These issues weaken the theoretical foundation used to motivate the method.

-The central claim of preserving the pretrained driving manifold is not convincingly verified. Freezing the trajectory generator preserves its parameters and output distribution for a fixed observation, but this does not necessarily mean that the corrected closed-loop policy remains on the original driving manifold. Even small action corrections can accumulate over multiple planning cycles and lead to substantially different state distributions. The bounded action neighborhood therefore provides a local constraint, not a direct guarantee of trajectory-level manifold preservation. The analysis in Appendix H is a useful attempt to examine this issue, but the reported evidence is limited. In particular, the main task-improvement group has a sign-test p-value of 0.0835, while the action-space analysis indicates that corrected action sequences can move farther from the prior-policy action manifold.

-The baseline comparisons do not fully isolate the benefits of correcting actions rather than trajectories. Table 3 compares NAC with several RL fine-tuning approaches and a trajectory-correction baseline, but the training formulations differ in important ways. For example, the PPO fine-tuning baseline uses an FPO surrogate, while SAC and TD3 adopt different off-policy optimization pipelines. The trajectory-correction baseline also operates in a substantially higher-dimensional output space, without matching the distribution of executed actions induced by NAC. The paper itself acknowledges that the matched-action-marginal condition in Lemma 1 is not enforced experimentally. Consequently, the performance differences cannot be attributed solely to the location or dimensionality of the correction.

-The generalization and statistical reliability of the reported improvements remain uncertain. The large improvements on Flow-Planner are encouraging, but the cross-family results in Table 2 are less convincing. With the original hyperparameters, NAC reduces performance on two non-reactive Diffusion-Planner settings. After planner-specific tuning, the improvements become positive but remain relatively small on several evaluation splits. This suggests greater sensitivity to the underlying planner than the paper's general framing implies.

Questions for authors

-What is the essential methodological difference between NAC and an existing residual RL method applied directly to the controller-projected action?

-Can the authors clarify whether Lemma 1 is intended as an exact covariance result or a local approximation? How do its conclusions extend to the actual Beta-policy parameterization and PPO optimization used in NAC?

-How do the authors verify that corrected closed-loop trajectories remain close to the pretrained driving manifold? Could they report correction magnitudes, action-distribution shifts, and performance changes separately for routine and long-tail scenarios?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tA2D (1/2)

We thank the reviewer for the careful review and the comments and suggestions on the theoretical analysis, the scope of manifold preservation, the methodological contribution, baseline comparisons, and the reliability of the results.

Theoretical assumptions and scope (W1/Q2)

The revision makes both lemmas' assumptions and scope explicit. At a fixed state, Lemma 1 gives an exact covariance decomposition for Gaussian mean scores under a linear controller, isotropic Gaussian trajectory perturbations, and matched executed-action distributions, identifying an additional positive semidefinite term from the controller kernel. A common fixed parameter Jacobian preserves the semidefinite comparison but can eliminate that term, while a nonlinear controller's Jacobian provides only a local linear interpretation within a smooth branch.

The bounded Beta policy has a different score, and PPO clipping weights do not automatically satisfy the kernel independence required by the weighted Gaussian decomposition, so we do not claim general covariance dominance for the implemented PPO parameter gradients. The analysis motivates correction placement, while the closed-loop comparisons in our W3 response evaluate the implemented scheme.

Lemma 2 retains the corrected policy's state occupancy and applies the Performance Difference Lemma and an action Taylor expansion. Bounded executed corrections and a uniform Hessian bound yield ∣Rα∣≤Mα2/(1−γ)|R_\alpha|\leq M\alpha^2/(1-\gamma) without assuming an O(α)O(\alpha) change in occupancy under total variation. Section 3.3 and Appendices A.2–A.3 provide the full assumptions, weighted case, and proofs. The radius analysis guides local interpretation, while closed-loop experiments evaluate its practical effect.

Scope of manifold preservation (W2)

Following the data manifold hypothesis discussed by Li and He, we view a generative planner's conditional trajectory prior learned from demonstrations as an approximation to their behavior manifold. Freezing preserves this prior at a fixed observation, and NAC learns bounded corrections around its LQR-projected actions. The local band constrains exploration at each planning cycle, with a fresh prior anchor at each observation, but visited states and executed trajectories can still change through interaction.

Appendix H reports that corrected action prefixes are farther from the prior-action manifold in both outcome groups. For 0.5 s closed-loop trajectory snippets aggregated by scenario, task_better projection error falls from 0.0363 to 0.0253 (30.4%), with lower error in 68.4% of scenarios, whereas the corresponding task_worse values are 5.6% and 46.7%. With headline p=0.0835p=0.0835, these results provide only directional evidence about the relationship between short-snippet geometry and task improvement, rather than a general preservation guarantee.

Correction placement and attribution of gains (W3)

Table 3's PPO trajectory-correction control shares NAC's Flow-Planner prior, reward, optimizer settings, and corrector depth and width, learning trajectory corrections before LQR while NAC learns action corrections after LQR. NAC scores higher in all five settings, including 79.30 versus 77.36 on test14-hard-R and 41.39 versus 34.33 on InterPlan10. The RL methods underwent the same tuning procedure, with optimization details in Appendix G. This comparison evaluates complete policies learned in closed loop at different interfaces under shared main training conditions, without enforcing matched executed-action distributions. Trajectory perturbations still pass through LQR mappings that vary with state and controller branch, so matching residual radii or mean executed magnitudes would not by itself ensure matched full action distributions. These comparisons assess practical post-training performance, while the exact variance-mechanism comparison remains within the theoretical model described in W1/Q2.

The supplementary Policy Decorator transfer results show that its post-LQR variant scores higher than its pre-LQR variant in all five settings, while NAC scores higher in all five settings at the same post-LQR interface. The full transfer table is in the FQwm-W4/Q4 response. These comparisons provide complementary evidence through trajectory versus action correction under shared main training conditions, transfer of the same residual method before and after the controller, and different learning schemes at the same interface. Together, they support execution-side adaptation and the effectiveness of NAC's overall design in the tested configurations, without attributing all score differences to correction placement alone.

AuthorsOpenReview ↗

Response to Reviewer tA2D (2/2)

Cross-prior transfer and stability (W4)

The same execution-side correction recipe already transfers, while tuning for each prior explores further performance potential. In Table 2, the unchanged recipe improves three reactive Diffusion-Planner settings but reduces the two non-reactive scores. A light planner-specific configuration yields scores above the original Diffusion-Planner in all five settings, with smaller gains than on Flow-Planner. Hyperparameters are selected using mean training episode return, without test scores. Table 5 also reports planTF and pre-RL Plan-R1 results, including the latter's gains in the first four settings and its InterPlan10 decrease from 42.85 to 42.38.

Results for three independent NAC training seeds on Flow-Planner are reported in the seed table in reviewer 7FTe-Q3, with evaluation covering nominal, difficult, and highly interactive conditions within nuPlan.

Application contribution within residual RL (Q1)

We agree that NAC belongs to the residual RL family, and an existing residual method applied after the same tracking controller would share its basic action-correction structure. Our main contribution is to address the application challenges of autonomous driving planning by developing an effective closed-loop post-training scheme for pretrained generative driving planners. This transfer is nontrivial: the generative prior outputs trajectories, while vehicle execution also involves a tracking controller, so correction placement, exploration bounds, and the learning scheme all need to be designed for this composed system.

To this end, NAC freezes the generative driving prior, reuses planner hidden features and ego state, and learns bounded two-dimensional corrections after LQR with a Beta policy and PPO. In the accompanying theoretical analysis, Lemma 1 identifies an additional variance term from the controller's null space under an idealized model, motivating correction placement, while Lemma 2 characterizes the local correction-radius trade-off. The full conditions and scope are given in our W1/Q2 response.

NAC scores higher in all five settings in the trajectory-correction comparison described in W3. In the supplementary Policy Decorator transfer, we retain its SAC, Gaussian corrections, tanh bounds, and progressive exploration. With the same Flow-Planner prior and post-LQR interface, NAC also scores higher than Policy Decorator in all five settings, including 79.30 versus 77.29 on test14-hard-R and 41.39 versus 27.42 on InterPlan10. The full transfer table is in the FQwm-W4/Q4 response. Together, these comparisons support the effectiveness of NAC's overall design for autonomous driving planning. This conclusion is limited to the tested configurations and neither isolates the benefit of each design choice nor establishes general superiority over all residual RL methods.

Trajectory proximity and scenario diagnostics (Q3)

Evidence for closed-loop trajectory proximity comes from the short trajectory snippet diagnostic in Appendix H, with the results and the limits of the preserved object described in W2. This diagnostic provides a supporting interpretation of the relationship between short-term motion geometry and task improvement, with conclusions limited to a directional association rather than a general guarantee of closed-loop manifold preservation. The main performance conclusions are supported by complete-scenario task and safety evaluations and do not depend on this geometric diagnostic reaching statistical significance.

In the main table, gains over the same Flow-Planner prior are 4.05, 8.88, and 10.85 points on val14-R, test14-hard-R, and InterPlan10, covering nominal, difficult, and highly interactive conditions. A separate diagnostic uses a fixed checkpoint from training seed 0 on 209 Test14-hard-R scenarios, reporting correction magnitudes, frequencies, signed shifts, constraint events, task/safety regression rates under routine/difficult proxy strata, and overall longitudinal jerk. The tables and aggregation definitions are in the FQwm-Q3 response, and these diagnostic results are reported separately from the main-table runs.

In the rebuttal, the authors acknowledge that the PPO parameter gradients do not strictly satisfy the independence conditions of Lemma 1, and that the corrected action prefixes can actually move farther from the prior-policy action manifold (Appendix H). This essentially concedes that the core theoretical motivations and strict manifold-preservation claims are not rigorously sustained in practice.

Given that NAC ultimately functions as a practical engineering design for post-controller residual adjustment, the authors should consider toning down their strong theoretical framing. Furthermore, since action-space residuals inherently lead to state and trajectory distribution shifts during closed-loop rollouts, could the authors clarify: do NAC's empirical gains stem from its architectural constraint, or are they largely attributable to more comprehensive hyperparameter tuning at the post-LQR interface compared to conventional residual RL baselines?

This essentially concedes that the core theoretical motivations and strict manifold-preservation claims are not rigorously sustained in practice.

As clarified in the revised manuscript and our first-round response, Section 3.3 provides supporting design analysis. It explains correction placement and local exploration within explicitly stated comparison models, motivating architectural choices such as post-LQR correction, low-dimensional parameterization, and neighborhood constraints. The framework’s definition and implementation do not require the assumptions of these comparison models to hold. The limitations on extending Lemma 1 to the implemented PPO procedure delimit the scope of the covariance result. They do not invalidate the structural constraints directly enforced by the framework: retaining the frozen conditional trajectory prior and generating bounded correction proposals around its controller-projected actions. The effectiveness of the implemented method is evaluated through closed-loop experiments.

do NAC's empirical gains stem from its architectural constraint, or are they largely attributable to more comprehensive hyperparameter tuning at the post-LQR interface compared to conventional residual RL baselines?

Exploration range, learning rate, and related hyperparameters are indeed important for both NAC and conventional residual RL methods. However, based on our experimental comparisons, we believe that more extensive hyperparameter tuning is not the primary source of NAC’s performance gains. At the same post-LQR interface, Policy Decorator was also tuned using the same hyperparameter-selection procedure as NAC. This comparison therefore does not pit a tuned NAC against an untuned residual RL baseline, and NAC still achieves higher scores in all five settings. NAC’s core design improves closed-loop performance by learning low-dimensional, bounded action corrections around the controller-projected actions of a frozen generative driving prior. Planner-specific adjustments focus on a small number of standard optimization parameters, including the learning rate and PPO clipping threshold, to accommodate differences in how each prior interacts with the controller. The gains already emerge without these specialized adjustments: applying the Flow-Planner recipe unchanged to Diffusion-Planner improves performance in most evaluated settings.

More directly, the benefit of correction placement is observed within two learning schemes. In the PPO comparison, which shares the prior, reward, optimizer settings, and corrector depth and width, post-LQR action correction outperforms pre-LQR trajectory correction in all five settings. Policy Decorator also improves in all five settings when moved to the post-LQR interface while retaining its own learning and correction design. These complementary results support our conclusion that execution-side action correction is a core architectural contributor to the gains, with benefits observable even without additional prior-specific tuning.

Finally, we would like to emphasize three aspects of NAC’s contribution: repeatable closed-loop gains, applicability across planner priors, and empirically grounded insight into post-training design. All three independent training seeds outperform the same prior in all five settings, and the correction scheme has also been extended to multiple prior models. Together with trajectory-versus-action correction controls sharing the main training conditions and comparisons at the same post-LQR interface, these results support NAC as an effective and extensible closed-loop post-training approach. They also offer a reusable design insight: correction placement matters in a pretrained planning and control system, and low-dimensional, bounded action correction at the execution interface provides an effective adaptation path.

Reviewer_FQwmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Neighborhood Action Correction (NAC), which freezes a pretrained generative driving planner and learns a bounded residual on the acceleration and steering-rate commands produced by its tracking controller. A PPO-trained corrector uses the frozen planner representation and ego state, with executed action at=at0+αδϕ(ht,xt)a_t=a_t^0+\alpha\delta_\phi(h_t,x_t). The motivation is that adapting the two-dimensional executed action avoids optimizing trajectory directions that the controller does not use. The paper supports this with a policy-gradient variance argument and a local performance expansion.

On a Flow-Planner prior, NAC improves all five reported nuPlan-family settings, including test14-hard reactive score from 70.42 to 79.30 and InterPlan10 from 30.54 to 41.39. The corrector adds approximately 2.33 ms per planning cycle and is trained for 100 million environment steps. The paper also evaluates diffusion and other planner priors, correction radius, direct fine-tuning, trajectory correction, and a diagnostic of adherence to a prior-derived trajectory manifold.

Critical strengths
  1. The correction location is well motivated by the actual execution stack. Adapting the controller output creates a small action space and allows the pretrained planner to supply representations and nominal behavior throughout post-training.

  2. The gains over the frozen Flow-Planner are substantial on difficult closed-loop scenarios. Evaluation includes reactive and non-reactive traffic, as well as inserted interaction cases, and InterPlan10 is evaluated without benchmark-specific training or retuning (Section 4.1).

  3. The radius sweep demonstrates a meaningful locality trade-off: very small corrections limit gains, while broad corrections can degrade the prior's performance (Table 2). The trajectory-correction control also directly tests the proposed intervention location.

  4. The manuscript supplies per-metric results, implementation details, and inference overhead. It also reports less successful transfer cases, including the need to tune the diffusion variant, rather than presenting the method as uniformly effective with unchanged settings.

  5. The comparison with residual-action methods is explicitly discussed in Section 2.3. This gives a reasonable basis for positioning the contribution around the controller interface and driving evaluation rather than the residual-learning idea alone.

Critical weaknesses
  1. Lemma 1 is stronger than the derivation supports. Section 3.3 states an exact covariance decomposition for a nonlinear controller, but Appendix A.2 obtains independence of controller-null components only after linearizing the controller and working to leading order. A direction in the Jacobian's kernel can still change the action at second order. The inverse (JCJC⊤)−1(J_CJ_C^\top)^{-1} also requires full row rank, which needs justification, particularly on the stopping branch where steering output is fixed. Extending the result to learned parameter gradients and clipped PPO needs additional argument: a planner Jacobian can eliminate the excess component, and likelihood/clipping weights can depend on the nominally null variables. The Gaussian matched-marginal analysis is informative, but does not establish the general exact dominance stated for the implemented bounded Beta policy.

  2. Lemma 2 uses an unjustified distribution-shift step. Appendix A.3 asserts that a uniformly α\alpha-small action change implies O(α)O(\alpha) total-variation distance between trajectory distributions. This does not follow from the stated smoothness of QQ in action. Under deterministic dynamics and a fixed initial state, arbitrarily close but different action sequences can produce distinct deterministic paths whose distributions have total-variation distance one. The proof therefore needs stronger transition/occupancy assumptions or a different continuity argument before replacing expectations under the corrected policy with expectations under the prior and claiming an O(α2)O(\alpha^2) remainder.

  3. The preservation claim needs a more precise scope. Freezing the generator preserves its conditional output distribution at a fixed observation, and the residual bounds each action locally. Repeated corrections can nevertheless change the visited-state distribution and executed behavior substantially. Appendix H acknowledges that its trajectory-manifold result is a directional trend, with headline sign-test p=0.0835p=0.0835, and finds action prefixes farther from the prior-action manifold in both outcome groups. These diagnostics do not establish general preservation of the executed closed-loop trajectory manifold.

  4. The baseline comparisons do not fully isolate the claimed mechanism. The matched-pipeline trajectory-correction experiment is useful, but its executed-action marginals are not matched to NAC's, as Section 5.3 acknowledges. The fine-tuning baselines also use different optimization procedures, including an FPO surrogate for PPO and distinct off-policy update budgets (Appendix G). Their failures support the tested configurations, but do not establish that generator fine-tuning is generally inferior. The Policy Decorator transfer is described in Appendix G, yet its numerical result is not included in the presented tables. Applying that residual method at the same post-controller interface would help isolate the benefit of correction location from the choice of learner.

  5. Cross-family gains are uneven and statistical support is incomplete. Untuned NAC reduces Diffusion-Planner's non-reactive scores, and the tuned improvements are much smaller than those on Flow-Planner (Table 2). The pre-RL Plan-R1 checkpoint also drops from 42.85 to 42.38 on InterPlan10 after correction (Table 5). Main results lack seed dispersion or confidence intervals, which particularly affects small comparisons such as 41.39 versus Plan-R1's 40.95. The three benchmark subsets all remain within the nuPlan ecosystem.

  6. Efficiency and safety details need fuller accounting. The corrector adds 5.729M parameters to a 1.244M-parameter Flow-Planner, so its small latency overhead should be distinguished from model size (Table 4). Training uses 100M steps without a corresponding learning curve or end-to-end GPU-hour comparison. The action-normalization scales, treatment of commands outside physical limits after residual addition, and exact dense training-reward formula also need specification. A local normalized-action bound alone does not establish physical safety or routine-behavior preservation.

Questions for authors
  1. Restate Lemma 1 as an exact result for an appropriate linear, full-rank controller model, or as a local asymptotic result with an explicit remainder. State which assumptions survive nonlinear controller branches, bounded policies, learned parameterizations, and PPO clipping. Validate the mechanism with measured gradient variance under matched executed-action distributions.

  2. Repair the occupancy-distribution step in Lemma 2. Supply assumptions that justify the claimed continuity, or retain the corrected-policy occupancy in the expansion. Address the stochastic generative prior separately from the deterministic prior assumed by the lemma.

  3. Report correction magnitudes and frequency on routine versus difficult states, long-horizon drift, comfort/jerk, and the fraction of scenarios whose safety or task score regresses. Extend the existing manifold diagnostic with a clearly held-out reference bank and uncertainty, keeping its conclusions proportional to its statistical support.

  4. Add matched-budget comparisons with controller retuning and an existing residual method applied after the controller. Ablate the corrector's access to pretrained planner features to quantify their benefit. Publish the missing transfer result and control the effective executed-action correction scale when comparing action and trajectory residuals.

  5. Report learning curves, total training GPU-hours, multiple training seeds, and paired scenario-level uncertainty. Explain planner-specific tuning and checkpoint selection, and present the smaller or negative transfer results alongside the headline Flow-Planner gains.

  6. Specify acceleration/steering-rate normalization, physical limits, clipping and associated likelihood treatment, and the full per-step reward. Since the actor receives no explicit prior-action input while the generator is stochastic, ablate conditioning it on the sampled prior action or trajectory to test whether corrections should depend on the sampled mode.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FQwm (1/5)

We thank the reviewer for the careful review and the comments and suggestions on the theoretical analysis, closed-loop behavior, experimental reliability, computational cost, and implementation details.

Lemma 1 (W1/Q1)

We have restated Lemma 1 as an exact Gaussian mean-score covariance comparison at a fixed state under a linear controller, isotropic Gaussian trajectory perturbations, and matched executed-action distributions. The kernel decomposition yields an additional positive semidefinite covariance term, while a pseudoinverse and effective rank handle rank deficiency, including the stopping branch with r≤1r\leq1. For a nonlinear controller, its Jacobian provides only a local linear model within a smooth branch. A common fixed parameter mapping preserves the semidefinite comparison but can eliminate the excess term, while weighted extensions additionally require kernel independence and finite second moments. The full assumptions and proof are given in Section 3.3 and Appendix A.2.

The bounded Beta policy has a different score, and PPO clipping weights do not automatically satisfy these independence conditions, so we do not claim general covariance dominance for the implemented PPO parameter gradients. We have not added a gradient-variance measurement under matched executed-action distributions, and the covariance comparison is established by the exact proof within the stated model. The implemented scheme is evaluated in closed loop using trajectory correction with a shared prior, reward, and optimizer settings, together with Policy Decorator at the same post-LQR interface. The results are reported in Section 5.3 and our W4/Q4 response and support the performance of the tested configurations.

Lemma 2 (W2/Q2)

Revised Lemma 2 uses the Performance Difference Lemma and an executed-action Taylor expansion under the corrected-policy occupancy dπαd_{\pi_\alpha}, without the TV-based occupancy substitution. With the same execution mapping as the zero-correction baseline, two-dimensional executed-action changes bounded by α\alpha per channel, and a uniform Hessian operator norm bound MM independent of α\alpha, the expansion gives a local linear term with remainder ∣Rα∣≤Mα2/(1−γ)|R_\alpha|\le M\alpha^2/(1-\gamma).

For a stochastic prior, π0(⋅∣s)\pi_0(\cdot\mid s) denotes the executed prior-action distribution at state ss. The identity Eaˉ0∼π0(⋅∣s)[Aπ0(s,aˉ0)]=0\mathbb E_{\bar a^0\sim\pi_0(\cdot\mid s)}[A^{\pi_0}(s,\bar a^0)]=0 cancels the zeroth-order term after taking expectation under the executed prior-action distribution, while subsequent expectations remain under the corrected-policy occupancy. The full conditions and proof are given in Appendix A.3. This analysis describes the local radius trade-off, with radius selection guided by training returns and the resulting gains evaluated through closed-loop experiments.

Scope of manifold preservation (W3)

Following the data manifold hypothesis discussed by Li and He, we view the conditional trajectory prior learned from demonstrations as an approximation to their behavior manifold. Freezing preserves this conditional distribution at a fixed observation, while fixed LQR maps prior trajectories to nominal actions, and NAC constrains each correction proposal to their closed α\alpha neighborhood. The prior supplies a fresh anchor at each observation, but repeated corrections can still change state visitation and executed trajectories. This defines the objects and limits of the preservation claim, with short trajectory snippets examined in Appendix H.

For 0.5 s snippets in task_better, projection error falls from 0.0363 to 0.0253 (30.4%), with reductions in 68.4% of scenarios, whereas the task_worse group shows 5.6% and 46.7%, respectively. The headline p=0.0835p=0.0835 supports only a directional interpretation. Moreover, corrected action prefixes are farther from the prior-action manifold in both groups, so this diagnostic does not establish general closed-loop manifold preservation.

AuthorsOpenReview ↗

Response to Reviewer FQwm (2/5)

Baseline and interface comparisons (W4/Q4)

Table 3 evaluates the implemented post-training configurations, in which PPO trajectory correction and NAC share the Flow-Planner prior, reward, optimizer settings, and corrector depth and width. The RL methods underwent the same tuning procedure, with optimization settings in Appendix G. The PPO trajectory-correction comparison with NAC shares the main training conditions and evaluates complete policies learned in closed loop at their respective interfaces, without enforcing matched executed-action distributions. The distribution-matching condition in Lemma 1 serves the comparison of variance mechanisms due to parameterization, while these experiments evaluate the learning schemes' practical closed-loop performance. The supplementary Policy Decorator transfer results below show that its post-LQR variant scores higher than its trajectory variant in all five settings. NAC also scores higher at the same post-LQR interface, supporting the tested correction and learning design.

MethodCorrection interfaceval14 NRval14 Rtest14-hard NRtest14-hard RInterPlan10 R
Policy DecoratorTrajectory, before LQR83.9271.3773.5563.1223.30
Policy DecoratorAction, after LQR91.2184.8081.9777.2927.42
Trajectory-correction (PPO)Trajectory, before LQR89.9086.3979.7177.3634.33
NACAction, after LQR93.2887.3683.5679.3041.39

We retain the official default LQR and its parameters to fix the base tracking conditions in these controlled comparisons, with the configuration given in reviewer 7FTe's evaluation-interface (W1/Q1) response. We have not added controller retuning because that comparison would assess optimization of the base tracker, while this paper tests gains from learned correction of a frozen prior under common tracking conditions. This conclusion does not assume that the default controller is optimal. Environment interactions, optimization updates, and GPU time measure different budgets, and fixing one does not automatically align the other costs. We therefore retain each learner's optimization procedure and use a common tuning procedure, the PPO control with shared training conditions, and same-interface comparisons to evaluate closed-loop performance under the reported configurations, with NAC's own training cost reported separately in Q5. This conclusion concerns gains at the corresponding training investment and does not depend on an advantage at matched compute.

We have not added a planner-feature ablation. Because removing the hidden features also removes scene information such as vehicles and lanes, isolating the benefit of pretraining requires encoders with equivalent observations.

We have also not added a performance experiment calibrated to matched effective action-correction scales. LQR varies with state and controller branch, and downstream execution processing affects both correction types, so matching a mean magnitude would not by itself ensure matched full executed-action distributions. Execution details are in Appendix B and Q6, while measured magnitudes and constraint events are in Q3 and the conditional variance comparison is in W1/Q1. These experiments provide complementary evidence through trajectory versus action correction under shared main training conditions, Policy Decorator transfer before and after the controller, and different learning schemes at the same interface. They support execution-side adaptation and the closed-loop gains of NAC's overall design, evaluating the complete scheme without attributing all score differences to correction placement alone.

AuthorsOpenReview ↗

Response to Reviewer FQwm (3/5)

Cross-prior transfer and statistical interpretation (W5)

Table 2 retains the uneven cross-family results: the unchanged recipe improves the three reactive Diffusion-Planner settings but reduces the two non-reactive scores. Light planner-specific tuning then improves all five over that prior, with smaller gains than on Flow-Planner. Table 5 includes planTF and pre-RL Plan-R1, with the latter improving in the first four settings but decreasing from 42.85 to 42.38 on InterPlan10. All these evaluations remain within the nuPlan ecosystem.

Q5 reports the training-seed results and explains that no paired scenario-level confidence intervals have been added. The main evidence is the consistent same-prior improvement, including 8.88 points on test14-hard-R and 10.85 on InterPlan10 for the main run. The 41.39 versus 40.95 comparison with full Plan-R1 is a 0.44-point difference in point estimates, not a claim of statistically significant superiority.

Computational efficiency and safety scope (W6)

The corrector reuses planner features and adds one forward pass, with most new parameters in feature projections, and post-training updates only this corrector. Table 4 reports 5.729M added parameters and 2.331 ms added latency (9.48%), totaling 6.973M parameters and 26.911 ms. Latency per million parameters is 0.407 ms for the corrector versus 19.759 ms for Flow-Planner, with the former being 2.06% of the latter. Thus, the small measured inference overhead should be distinguished from model size.

The training curve and the scope of the reported 96-GPU-hour cost for one post-training run are given in Q5, and physical action scales, clipping, and the full per-step reward are given in Q6 and Appendix B. The local proposal bounds constrain the magnitude of each exploration step, while actual vehicle behavior also depends on execution processing and closed-loop interaction, so physical safety or preservation of routine behavior cannot follow from these bounds alone. We assess the resulting behavior through official closed-loop task and safety metrics, together with the correction, constraint-event, and regression statistics in Q3.

Reviewer_7FTeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NAC freezes a generative nuPlan planner and trains a small PPO policy to add bounded residuals to the acceleration and steering-rate outputs of its LQR tracking controller. The paper compares action correction with trajectory correction and planner fine-tuning, reports large gains on difficult scenarios, and provides a null-space variance argument for correcting after the controller.

Critical strengths
  1. The reported improvements on hard and long-tail scenarios are substantial, and the small corrector has little stated runtime overhead. This makes the approach practically interesting if the evaluation is comparable.

  2. The same-prior comparison between fine-tuning, trajectory correction and action correction is useful for examining where adaptation should occur. The correction-radius sweep also exposes an important design trade-off.

Critical weaknesses
  1. The evaluation interface needs clarification before the headline comparison can be interpreted. NAC changes the action produced by the tracking controller, whereas standard trajectory-output planners are evaluated through a fixed controller. This introduces a capability difference that may contribute to the gain. The paper should state the applicable benchmark rules, identify methods using the same action interface, and distinguish planner quality from an improved tracking stack. I would reconsider my recommendation if a matched-interface comparison supports the claimed advantage.

  2. The evidence does not yet isolate a robust contribution beyond established residual RL. Lemma 1 relies on assumptions not shown to describe the compared policies. The correction-radius sweep is highly non-monotone, no seed variability is reported, and the split used for hyperparameter selection needs clarification. Weak RL baselines and limited transfer of the unchanged recipe to Diffusion-Planner further complicate attribution. I currently lean toward weak rejection.

Questions for authors
  1. Is modification of the controller output permitted in the reported official evaluation protocol? Which baselines have equivalent action-level access? Can you provide a matched-interface control or a comparison under an unchanged tracking stack?

  2. Which split selected the correction radius and the Diffusion-Planner hyperparameters? Were the test14-hard and InterPlan10 results used only after selection? Please identify training scenarios and log-level separation from evaluation.

  3. How stable are the gains across seeds, and were the RL baselines comparably tuned? Please report comfort, progress and TTC components to show whether the action residual trades one objective against another.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7FTe (1/2)

We thank the reviewer for the careful review and the comments and suggestions on the methodological contribution, evaluation setup, and experimental evidence.

Evaluation interface (W1/Q1)

The 2023 nuPlan Planning Challenge opened on January 30 and closed on May 26, 2023. Its closed-loop tracks specified trajectory output followed by LQR tracking. Our experiments use the publicly available nuPlan benchmark. Section IV-B of the official benchmark paper explicitly supports alternative trackers, including iLQR. Subsequent work has used this flexibility: Gen-Drive (ICRA 2025, Section IV-A) uses iLQR for nuPlan evaluation. The default LQR also has documented tracking limitations: CaRL (CoRL 2025, Appendix E.2) reports that replacing LQR with iLQR raises the same human log-replay trajectory's val14-NR score from 93.53 to 96.24. Direct control output also has a public nuPlan precedent: CaRL predicts acceleration and a steering target, converted to steering rate. Its released implementation executes these controls directly through vehicle dynamics.

NAC retains the default LQR and all its parameters, adding a bounded residual to its acceleration and steering-rate commands before downstream execution processing and vehicle dynamics. Our controlled runs use the same LQR wherever tracking applies, with identical nuPlan metrics and score aggregation. Table 1 compares complete closed-loop systems. Appendix B and the official configuration specify the tracker:

Fixed LQR configuration

ParameterValue
q_longitudinal[10.0]
r_longitudinal[1.0]
q_lateral[1.0, 10.0, 0.0]
r_lateral[1.0]
discretization_time0.1 s
tracking_horizon10
jerk_penalty1e-4
curvature_rate_penalty1e-2
stopping_proportional_gain0.5
stopping_velocity0.2 m/s

Baseline interface comparison (W1/Q1)

Appendix E, Table 5 already reports planTF + NAC and pre-RL Plan-R1 + NAC. We reproduce their prior and corrected scores below. To align the interface conditions, we also evaluated Policy Decorator with corrections applied after LQR. The prior/+NAC pairs measure the effect of adding action correction to each planner. planTF improves in all five settings, while pre-RL Plan-R1 improves in the first four, with InterPlan10 changing from 42.85 to 42.38. With the same Flow-Planner prior and post-LQR interface, NAC exceeds Policy Decorator by 2.07, 2.56, 1.59, 2.01, and 13.97 points, respectively.

MethodOutput / correction interfaceval14 NRval14 Rtest14-hard NRtest14-hard RInterPlan10 R
planTF (prior)Trajectory output84.2776.9569.7061.6133.70
planTF + NACAction correction after LQR89.1481.2277.0367.9636.42
Plan-R1 (pre-RL prior)Trajectory output85.5080.6972.1565.3842.85
Plan-R1 (pre-RL) + NACAction correction after LQR90.6387.4179.4975.1942.38
Flow-Planner + Policy DecoratorAction correction after LQR91.2184.8081.9777.2927.42
Flow-Planner + NACAction correction after LQR93.2887.3683.5679.3041.39
AuthorsOpenReview ↗

Response to Reviewer 7FTe (2/2)

Rationale and applicability (W2)

We agree that NAC belongs to the residual RL family. Our main contribution is to combine theoretical analysis of correction placement with engineering design to develop a closed-loop post-training scheme for pretrained generative driving planners. Applying residual learning to this composed planning and control system is nontrivial: both the trajectory-correction control and the Policy Decorator transfer at the same post-LQR interface score below NAC. The specific challenges, design, and comparisons are discussed in our response to reviewer tA2D's Q1.

Lemma 1 explains the sources of variance due to parameterization under an idealized model and motivates correction placement. The theoretical assumptions and proof are given in Section 3.3 and Appendix A.2, while the closed-loop effectiveness of the implemented Beta/PPO scheme is evaluated through the empirical comparison in Section 5.3. An excessively small radius limits correction capacity, making effective adjustments to the prior behavior difficult. An excessively large radius may make RL exploration overly aggressive and convergence more difficult, so performance does not improve monotonically as the radius increases. For Diffusion-Planner, the original configuration improves the three reactive settings. With a light planner-specific configuration, the scores in all five settings in Table 2 exceed those of the original Diffusion-Planner, supporting cross-prior applicability of neighborhood correction with light tuning.

Hyperparameter selection and data splits (Q2)

Hyperparameter selection: The correction radius and Diffusion-Planner hyperparameters were selected using the mean episode return during training.

Use of test sets: The test14-hard and InterPlan10 scores were used only for final evaluation after configuration selection and did not enter hyperparameter selection.

Training scenarios: NAC closed-loop post-training uniformly samples cached scenarios from the nuPlan training split.

Log-level separation: The training sampler filters scenarios using a list of 13,180 training log identifiers, with no overlap with the configured lists of 1,381 validation and 1,349 test log identifiers.

Reliability of the results (Q3)

Stability across seeds: We have completed the NAC experiments on Flow-Planner with three independent training seeds. The table below reports individual scores, together with means and sample standard deviations computed from the displayed values. Seed 0 retains the original main-experiment scores, and seeds 1/2 are the supplementary training runs. All three seeds outperform the same prior in all five settings.

SettingPriorOriginal NAC(seed 0)Seed 1Seed 2Mean ± sample SD
val14 NR90.4393.2893.8693.5393.56 ± 0.29
val14 R83.3187.3687.9087.1187.46 ± 0.40
test14-hard NR76.4783.5683.6483.6483.61 ± 0.05
test14-hard R70.4279.3079.3878.9479.21 ± 0.23
InterPlan10 R30.5441.3941.6741.3441.47 ± 0.18

Baseline tuning: The RL baselines underwent the same hyperparameter tuning procedure as NAC. PPO trajectory correction and NAC share the Flow-Planner prior, reward, and optimizer settings, with the optimization configurations described in Appendix G.

Component metrics: NAC's Comfort, Progress, and TTC component scores for each evaluation setting are already reported in Appendix F (Detailed closed-loop metrics).

Thanks for the reply. I still have a few questions.

  1. You now clarify that Lemma 1 is only an exact result for a linear controller with Gaussian perturbations, and does not cover the implemented Beta/PPO policy. Should this limitation be stated more explicitly in the main theorem and conclusion?

  2. Since the matched-interface comparisons still use different planners, and the executed-action distributions are not matched, how should we interpret the gain from placing NAC after LQR? Is the evidence enough to attribute it to the correction location itself?

  3. You report 24 out of 209 Test14-hard-R scenarios with either task or safety regression, while scenario-level uncertainty is unavailable. What safety claim do you think the paper can still make from these results?

Thanks again for the clarification.

  1. Should this limitation be stated more explicitly in the main theorem and conclusion?

Thank you for this suggestion. We have updated the relevant statements in the latest revision. The Introduction already limits the analysis to a linear Gaussian comparison model. The lead-in, title, and formal assumptions of Lemma 1 in Section 3.3 also specify the linear controller, Gaussian perturbations, and matched executed-action distributions. To make this scope more explicit in the main text and conclusion, we have added two further statements.

  • In Section 3.3, we have added: “This exact covariance result is specific to the stated linear Gaussian comparison model and does not establish covariance dominance for the implemented Beta/PPO parameter gradients. Closed-loop comparisons evaluate the effectiveness of the implemented scheme.”

  • In Conclusion, we have added: “The covariance analysis provides a design rationale within the linear Gaussian comparison model and does not establish a general variance guarantee for Beta/PPO. The implemented scheme is supported by closed-loop experiments.”

These additions clarify the relationship between the analysis and the implemented method. The covariance result holds within the stated comparison model and motivates correction placement. The framework directly retains the frozen conditional trajectory prior at a fixed observation and bounds each local correction proposal, while closed-loop experiments support the effectiveness of the implemented scheme.

  1. the matched-interface comparisons still use different planners

Our direct matched-interface comparison evaluates NAC and Policy Decorator using the same frozen Flow-Planner prior and the same post-LQR acceleration and steering-rate interface. NAC scores higher in all five settings respectively. This direct comparison therefore holds both the planner prior and correction interface fixed.

the executed-action distributions are not matched

Trajectory correction passes through an LQR mapping that depends on the state and controller branch. Official LQR tracking uses reference velocity and curvature fits with soft jerk and curvature-rate penalties, which shape the command response to trajectory perturbations. Post-LQR correction instead adjusts the nominal command directly, so this adjustment follows a different mapping. With the benchmark setup and the architectures under comparison held fixed, guaranteeing matched full executed-action distributions through hyperparameter or experimental-configuration adjustments alone is therefore difficult.

how should we interpret the gain from placing NAC after LQR? Is the evidence enough to attribute it to the correction location itself?

The benefit of correction placement is observed within two learning schemes. Under the same Flow-Planner prior, the PPO comparison shares the reward, optimizer settings, and corrector depth and width, while the Policy Decorator comparison retains its own learning and correction design. Both comparisons show that post-LQR action correction outperforms pre-LQR trajectory correction in all five settings. These complementary results support the conclusion stated in our latest response to reviewer tA2D that execution-side action correction is a core architectural contributor to the gains. This design focuses learning on a low-dimensional action neighborhood around the controller-projected actions of a frozen prior, providing an effective adaptation path for closed-loop post-training.

  1. What safety claim do you think the paper can still make from these results?

Our safety conclusion concerns the observed closed-loop behavior of the evaluated model on these 209 Test14-hard-R scenarios, based on official safety components and scenario-level metric changes. This supplementary diagnostic uses a fixed checkpoint from training seed 0 and is reported separately from the main-table runs. In 185 scenarios (88.52%), the overall score and all three examined safety components are maintained or improved relative to both the default-LQR execution reference and the zero-correction reference at the same action interface.

The scenario-level results also show that, among the 22 scenarios with lower overall scores, 11 still have improvements in other components, including nine with higher route progress and two with higher speed-limit compliance scores. Among the 10 scenarios with a decrease in at least one safety component, five also achieve higher route progress. These component-level changes reflect trade-offs among driving objectives.

The local bound specifies the magnitude of each correction proposal, while official safety components and full-scenario statistics assess the resulting closed-loop behavior. Balancing route progress, compliance, and safety is a common challenge in closed-loop reinforcement learning. Future work will explore how to better balance these objectives and improve overall closed-loop post-training performance across scenarios.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.