- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Neighborhood Action Correction (NAC), an RL-based post-training framework for generative autonomous driving planners. Instead of fine-tuning the pretrained trajectory generator, NAC freezes the planner and learns a lightweight residual policy that corrects the controller-projected action within a bounded neighborhood. The motivation is that, under receding-horizon execution, only the executed action directly influences vehicle dynamics, making action-space correction potentially more efficient and stable than trajectory-level optimization. The authors provide a theoretical analysis based on controller-null directions and policy-gradient variance, and evaluate NAC on several nuPlan benchmarks using flow-matching and diffusion-based planners. The reported results show notable closed-loop improvements, particularly on challenging scenarios. While the approach is practically interesting, I have several concerns regarding its methodological novelty, theoretical justification, and the extent to which the experiments substantiate its central claims.
Critical strengths
-The paper addresses an important problem in autonomous driving: improving closed-loop behavior without substantially modifying a pretrained generative planner.
-The action-space correction design is simple and computationally lightweight at inference time, making it relatively easy to integrate into existing trajectory-based planning pipelines.
-The experiments cover multiple nuPlan evaluation settings and include comparisons with trajectory-level correction, direct RL fine-tuning, and different generative planner backbones.
Critical weaknesses
-The theoretical analysis relies on restrictive assumptions, and some conclusions appear stronger than the derivations justify. Lemma 1 argues that action-space correction has lower policy-gradient variance because it avoids controller-null trajectory directions. However, the analysis assumes an isotropic Gaussian trajectory correction and matched executed-action distributions, whereas the actual implementation uses a bounded Beta policy and a nonlinear controller. The appendix proof also relies on a local linearization, making the stated covariance relationship difficult to interpret as a general result for the implemented PPO procedure. More importantly, the extension of this variance argument to direct planner fine-tuning is not fully convincing, since parameter-space gradients depend on the planner Jacobian and may not preserve the claimed strict variance dominance. Similarly, the proof of Lemma 2 assumes that an (O(\alpha)) action perturbation implies an (O(\alpha)) change in state occupancy under total variation, which does not generally follow without additional assumptions on the dynamics and transition distributions. These issues weaken the theoretical foundation used to motivate the method.
-The central claim of preserving the pretrained driving manifold is not convincingly verified. Freezing the trajectory generator preserves its parameters and output distribution for a fixed observation, but this does not necessarily mean that the corrected closed-loop policy remains on the original driving manifold. Even small action corrections can accumulate over multiple planning cycles and lead to substantially different state distributions. The bounded action neighborhood therefore provides a local constraint, not a direct guarantee of trajectory-level manifold preservation. The analysis in Appendix H is a useful attempt to examine this issue, but the reported evidence is limited. In particular, the main task-improvement group has a sign-test p-value of 0.0835, while the action-space analysis indicates that corrected action sequences can move farther from the prior-policy action manifold.
-The baseline comparisons do not fully isolate the benefits of correcting actions rather than trajectories. Table 3 compares NAC with several RL fine-tuning approaches and a trajectory-correction baseline, but the training formulations differ in important ways. For example, the PPO fine-tuning baseline uses an FPO surrogate, while SAC and TD3 adopt different off-policy optimization pipelines. The trajectory-correction baseline also operates in a substantially higher-dimensional output space, without matching the distribution of executed actions induced by NAC. The paper itself acknowledges that the matched-action-marginal condition in Lemma 1 is not enforced experimentally. Consequently, the performance differences cannot be attributed solely to the location or dimensionality of the correction.
-The generalization and statistical reliability of the reported improvements remain uncertain. The large improvements on Flow-Planner are encouraging, but the cross-family results in Table 2 are less convincing. With the original hyperparameters, NAC reduces performance on two non-reactive Diffusion-Planner settings. After planner-specific tuning, the improvements become positive but remain relatively small on several evaluation splits. This suggests greater sensitivity to the underlying planner than the paper's general framing implies.
Questions for authors
-What is the essential methodological difference between NAC and an existing residual RL method applied directly to the controller-projected action?
-Can the authors clarify whether Lemma 1 is intended as an exact covariance result or a local approximation? How do its conclusions extend to the actual Beta-policy parameterization and PPO optimization used in NAC?
-How do the authors verify that corrected closed-loop trajectories remain close to the pretrained driving manifold? Could they report correction magnitudes, action-distribution shifts, and performance changes separately for routine and long-tail scenarios?
4 comments
Response to Reviewer tA2D (2/2)
Cross-prior transfer and stability (W4)
The same execution-side correction recipe already transfers, while tuning for each prior explores further performance potential. In Table 2, the unchanged recipe improves three reactive Diffusion-Planner settings but reduces the two non-reactive scores. A light planner-specific configuration yields scores above the original Diffusion-Planner in all five settings, with smaller gains than on Flow-Planner. Hyperparameters are selected using mean training episode return, without test scores. Table 5 also reports planTF and pre-RL Plan-R1 results, including the latter's gains in the first four settings and its InterPlan10 decrease from 42.85 to 42.38.
Results for three independent NAC training seeds on Flow-Planner are reported in the seed table in reviewer 7FTe-Q3, with evaluation covering nominal, difficult, and highly interactive conditions within nuPlan.
Application contribution within residual RL (Q1)
We agree that NAC belongs to the residual RL family, and an existing residual method applied after the same tracking controller would share its basic action-correction structure. Our main contribution is to address the application challenges of autonomous driving planning by developing an effective closed-loop post-training scheme for pretrained generative driving planners. This transfer is nontrivial: the generative prior outputs trajectories, while vehicle execution also involves a tracking controller, so correction placement, exploration bounds, and the learning scheme all need to be designed for this composed system.
To this end, NAC freezes the generative driving prior, reuses planner hidden features and ego state, and learns bounded two-dimensional corrections after LQR with a Beta policy and PPO. In the accompanying theoretical analysis, Lemma 1 identifies an additional variance term from the controller's null space under an idealized model, motivating correction placement, while Lemma 2 characterizes the local correction-radius trade-off. The full conditions and scope are given in our W1/Q2 response.
NAC scores higher in all five settings in the trajectory-correction comparison described in W3. In the supplementary Policy Decorator transfer, we retain its SAC, Gaussian corrections, tanh bounds, and progressive exploration. With the same Flow-Planner prior and post-LQR interface, NAC also scores higher than Policy Decorator in all five settings, including 79.30 versus 77.29 on test14-hard-R and 41.39 versus 27.42 on InterPlan10. The full transfer table is in the FQwm-W4/Q4 response. Together, these comparisons support the effectiveness of NAC's overall design for autonomous driving planning. This conclusion is limited to the tested configurations and neither isolates the benefit of each design choice nor establishes general superiority over all residual RL methods.
Trajectory proximity and scenario diagnostics (Q3)
Evidence for closed-loop trajectory proximity comes from the short trajectory snippet diagnostic in Appendix H, with the results and the limits of the preserved object described in W2. This diagnostic provides a supporting interpretation of the relationship between short-term motion geometry and task improvement, with conclusions limited to a directional association rather than a general guarantee of closed-loop manifold preservation. The main performance conclusions are supported by complete-scenario task and safety evaluations and do not depend on this geometric diagnostic reaching statistical significance.
In the main table, gains over the same Flow-Planner prior are 4.05, 8.88, and 10.85 points on val14-R, test14-hard-R, and InterPlan10, covering nominal, difficult, and highly interactive conditions. A separate diagnostic uses a fixed checkpoint from training seed 0 on 209 Test14-hard-R scenarios, reporting correction magnitudes, frequencies, signed shifts, constraint events, task/safety regression rates under routine/difficult proxy strata, and overall longitudinal jerk. The tables and aggregation definitions are in the FQwm-Q3 response, and these diagnostic results are reported separately from the main-table runs.
In the rebuttal, the authors acknowledge that the PPO parameter gradients do not strictly satisfy the independence conditions of Lemma 1, and that the corrected action prefixes can actually move farther from the prior-policy action manifold (Appendix H). This essentially concedes that the core theoretical motivations and strict manifold-preservation claims are not rigorously sustained in practice.
Given that NAC ultimately functions as a practical engineering design for post-controller residual adjustment, the authors should consider toning down their strong theoretical framing. Furthermore, since action-space residuals inherently lead to state and trajectory distribution shifts during closed-loop rollouts, could the authors clarify: do NAC's empirical gains stem from its architectural constraint, or are they largely attributable to more comprehensive hyperparameter tuning at the post-LQR interface compared to conventional residual RL baselines?
This essentially concedes that the core theoretical motivations and strict manifold-preservation claims are not rigorously sustained in practice.
As clarified in the revised manuscript and our first-round response, Section 3.3 provides supporting design analysis. It explains correction placement and local exploration within explicitly stated comparison models, motivating architectural choices such as post-LQR correction, low-dimensional parameterization, and neighborhood constraints. The framework’s definition and implementation do not require the assumptions of these comparison models to hold. The limitations on extending Lemma 1 to the implemented PPO procedure delimit the scope of the covariance result. They do not invalidate the structural constraints directly enforced by the framework: retaining the frozen conditional trajectory prior and generating bounded correction proposals around its controller-projected actions. The effectiveness of the implemented method is evaluated through closed-loop experiments.
do NAC's empirical gains stem from its architectural constraint, or are they largely attributable to more comprehensive hyperparameter tuning at the post-LQR interface compared to conventional residual RL baselines?
Exploration range, learning rate, and related hyperparameters are indeed important for both NAC and conventional residual RL methods. However, based on our experimental comparisons, we believe that more extensive hyperparameter tuning is not the primary source of NAC’s performance gains. At the same post-LQR interface, Policy Decorator was also tuned using the same hyperparameter-selection procedure as NAC. This comparison therefore does not pit a tuned NAC against an untuned residual RL baseline, and NAC still achieves higher scores in all five settings. NAC’s core design improves closed-loop performance by learning low-dimensional, bounded action corrections around the controller-projected actions of a frozen generative driving prior. Planner-specific adjustments focus on a small number of standard optimization parameters, including the learning rate and PPO clipping threshold, to accommodate differences in how each prior interacts with the controller. The gains already emerge without these specialized adjustments: applying the Flow-Planner recipe unchanged to Diffusion-Planner improves performance in most evaluated settings.
More directly, the benefit of correction placement is observed within two learning schemes. In the PPO comparison, which shares the prior, reward, optimizer settings, and corrector depth and width, post-LQR action correction outperforms pre-LQR trajectory correction in all five settings. Policy Decorator also improves in all five settings when moved to the post-LQR interface while retaining its own learning and correction design. These complementary results support our conclusion that execution-side action correction is a core architectural contributor to the gains, with benefits observable even without additional prior-specific tuning.
Finally, we would like to emphasize three aspects of NAC’s contribution: repeatable closed-loop gains, applicability across planner priors, and empirically grounded insight into post-training design. All three independent training seeds outperform the same prior in all five settings, and the correction scheme has also been extended to multiple prior models. Together with trajectory-versus-action correction controls sharing the main training conditions and comparisons at the same post-LQR interface, these results support NAC as an effective and extensible closed-loop post-training approach. They also offer a reusable design insight: correction placement matters in a pretrained planning and control system, and low-dimensional, bounded action correction at the execution interface provides an effective adaptation path.
Response to Reviewer tA2D (1/2)
We thank the reviewer for the careful review and the comments and suggestions on the theoretical analysis, the scope of manifold preservation, the methodological contribution, baseline comparisons, and the reliability of the results.
Theoretical assumptions and scope (W1/Q2)
Scope of manifold preservation (W2)
Correction placement and attribution of gains (W3)