acceptodds
Under review as a conference paper at ICLR 2027

FieldDrive: Field-Guided One-Step Trajectory Generation and Reward Optimization for Autonomous Driving

Abstract

Generative models have been introduced as trajectory decoders for autonomous driving to model the conditional multimodal distribution of reasonable driving behaviors in complex traffic scenes. However, existing generative decoders typically rely on multiple denoising or flow-integration steps, resulting in high inference overhead. Meanwhile, existing reinforcement learning methods are mainly designed around multi-step generation chains and are structurally mismatched with one-step generators. Combining one-step multimodal trajectory decoding with the optimization capability of reinforcement learning remains a central challenge in generative autonomous driving planning. To address this challenge, we propose FieldDrive, a Drift-based one-step trajectory decoder with a corresponding reinforcement learning method. We cluster a trajectory vocabulary and select high-scoring trajectories to construct a feasible trajectory distribution, which guides the generated trajectory distribution by the Drift Field Vector. We train a critic network to provide a Reward Field Vector that directs the model toward high-reward regions. Theoretically, we show that the Drift Field Vector and Reward Field Vector share a common origin: both arise as optimization directions of distributional energy function. Through a two-stage training paradigm, the model explores different distributions of reasonable trajectories and progressively converges to high-reward regions through reinforcement learning, while enabling efficient decoding with a single forward pass at inference time. Experiments on NAVSIM v1 and v2 demonstrate that FieldDrive enables efficient and high-quality trajectory decoding.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Generative Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_WNXyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FieldDrive applies the Drifting paradigm to one-step trajectory generation for end-to-end autonomous driving. During imitation-learning pretraining, the method constructs a scene-conditioned reference distribution from the expert trajectory and high-PDMS trajectories selected from a clustered vocabulary. During reinforcement-learning fine-tuning, it combines the Drift Field with trajectory gradients from an online critic to improve planning performance while retaining a reference-behavior constraint. At inference, the direct variant produces a single trajectory using zero noise and one generator forward pass. A separately fine-tuned variant supports candidate generation and scorer-based selection. The paper reports competitive planning results and efficient decoding on NAVSIM.

Critical strengths
  1. An interesting application of Drifting to autonomous driving. Adapting this recent generative modeling paradigm to end-to-end planning is a worthwhile direction. The combination of distribution matching and reward-guided refinement forms a coherent framework for training a one-step planner.

  2. Strong planning performance with efficient one-step decoding. The direct variant achieves 92.1 PDMS without candidate scoring at inference, requiring only one generator forward pass at a reported decoding speed of 546 FPS. The scorer-assisted variant further reaches 93.3 PDMS. The positive-sample and field-component ablations provide useful evidence for the training design.

Critical weaknesses
  1. The overall framework is heavily tailored to PDMS. PDMS influences both stages of training: it is used to select positive trajectories during imitation-learning pretraining, while the critic supervision and reward potential are constructed around its component metrics during RL fine-tuning. Although these choices produce strong benchmark results, they leave uncertainty about whether the improvements reflect broader driving quality. Moreover, the v1 and v2 results use different fine-tuning configurations, so the paper does not establish how the same inferred trajectories perform under both PDMS and EPDMS.

  2. The effectiveness of the framework under broader and more complex reward combinations remains insufficiently explored. According to Appendix B.3, the v2 adaptation adds DDC guidance and reduces the EP weight, but does not report explicitly incorporating TLC, LK into the reward optimization. This is relevant to Table 2: despite achieving the highest aggregate EPDMS, FieldDrive has substantially lower EC, lower LK and TLC than most listed methods. These results motivate examining whether the reward-field framework can effectively optimize a more comprehensive set of driving objectives and achieve a better balance across them.

Questions for authors
  1. Could the authors report both PDMS and EPDMS for the same fixed inference trajectories from each variant, particularly the EPDMS of the model achieving 92.1 PDMS? Please clarify any differences between the evaluation protocols.

  2. Could the authors provide the complete v2 reward formula and explore a broader reward combination that explicitly includes additional objectives such as TLC and LK? Reporting the full metric breakdown before and after fine-tuning, together with an analysis of the low EC, would help establish whether the framework can address these weaknesses while retaining its planning performance.

  3. How sensitive is performance to the nonzero α and β weights? Table 5 establishes the benefit of combining the two fields, but a controlled sensitivity analysis would help assess the robustness of their balance.

8 comments
AuthorsOpenReview ↗

Response (Part 1)

We appreciate your careful examination of the reward design and evaluation and present cross-evaluation results, broader reward experiments, and an analysis of the field-weight sensitivity.

W1/Q1: Cross-Evaluation Results and Generalization.

Cross-evaluation under PDMS and EPDMS.

We agree that reward hacking is a potential concern during RL post-training [1]. Designing rewards that lead to better performance across a broad range of driving tasks remains an open problem. According to your suggestion, we evaluate both the main PDMS-oriented model and the EPDMS-oriented variant under both scoring protocols on the same navtest split. For each variant, the same inference trajectories are evaluated under PDMS and EPDMS. The main model achieves 92.14 PDMS and 86.63 EPDMS, while the EPDMS-oriented variant achieves 91.51 PDMS and 89.71 EPDMS. In our manuscript, we use the same PDM evaluator and metric definitions as in navtest to compute rewards on training scenes like ReCogDrive [2], SimVTA [3], and DriveFine [4].

Table W1 reports the metric breakdown under PDMS evaluation.

Table W1. Cross-evaluation of both model variants under PDMS.

VariantNCDACEPTTCCDDCPDMS
EPDMS-oriented98.5599.0586.1695.3899.9598.7091.51
Main97.8598.6390.3693.42100.0095.8792.14

Table W2 reports the metric breakdown under EPDMS evaluation.

Table W2. Cross-evaluation of both model variants under EPDMS.

VariantNCDACEPTTCDDCHCECLKTLCEPDMS
EPDMS-oriented98.5599.0588.7098.0499.6798.1873.7294.3799.6389.71
Main97.8598.6393.7397.0997.6898.0357.6292.5299.4286.63

Zero-shot generalization to closed-loop HUGSIM evaluation.

We additionally evaluate transfer beyond NAVSIM using the closed-loop HUGSIM benchmark. Because some HUGSIM evaluation datasets do not support rear-camera input, we retrain a version using the current front, left, and right views together with the front view from the previous frame (0.5 seconds earlier). All other configurations remain unchanged with EPDMS-oriented variant. This version is trained on the original NAVSIM training data and evaluated on HUGSIM without HUGSIM training or fine-tuning. Table W3 reports the closed-loop results.

Table W3. Zero-shot closed-loop evaluation on HUGSIM.

MethodRC ERC MRC HRC XRC Avg.HD-Score EHD-Score MHD-Score HHD-Score XHD-Score Avg.
LTF [5]67.8035.1026.2040.5040.4258.9018.009.8025.9025.57
UniAD [6]78.4060.5033.6017.8047.7764.9045.8020.606.6034.50
VAD [7]51.3031.1025.3026.5032.4436.309.508.0011.5014.56
GTRS-Dense [8]64.2050.0020.7022.3039.6155.5039.0011.7014.3030.18
Latent-WAM [9]84.2042.5030.6035.5045.8872.5024.0012.2018.1028.91
MeanFuser [10]76.7041.1037.9028.4043.9067.1024.3021.5011.0028.40
FieldDrive88.5445.4833.8131.8047.5874.2329.1017.7714.8131.51
AuthorsOpenReview ↗

Response (Part 2)

W2/Q2: Exploring EPDMS as a Training Reward.

For the NAVSIM v2 variant, the complete reward is

Rv2=NC⋅DAC⋅DDC⋅(1⋅EP+5⋅TTC+2⋅C).R_{\mathrm{v2}} = \mathrm{NC}\cdot\mathrm{DAC}\cdot\mathrm{DDC}\cdot\left(1\cdot\mathrm{EP}+5\cdot\mathrm{TTC}+2\cdot\mathrm{C}\right).

In implementation, we compute the Reward Field Vector by differentiating the logarithmic reward constructed from the critic's metric predictions.

Adding TLC and LK improves EPDMS.

We explored adding TLC and LK to the reward, extending the variant with adjusted EP and DDC weights. With these objectives included and a larger α to keep trajectories closer to the expert reference, EPDMS improves from 89.71 to 90.40. TLC increases from 99.63 to 99.79, and LK increases from 94.37 to 97.79. The stronger expert constraint also encourages a more consistent trajectory style, which we associate with the improvement in EC from 73.72 to 84.30. Specifically, we set α = 2.5 in this experiment: the additional reward terms contribute gradients that are summed in the Reward Field Vector, so we increase α to balance their combined influence with the expert constraint. However, NC, DAC, and HC decrease. Table W4 reports the full breakdown, showing that this reward configuration improves overall performance while introducing trade-offs among individual objectives.

Table W4. EPDMS evaluation with additional reward objectives.

Reward configurationNCDACEPTTCDDCHCECLKTLCEPDMS
Main PDMS-based reward97.8598.6393.7397.0997.6898.0357.6292.5299.4286.63
PDMS-based reward with adjusted EP and DDC weights98.5599.0588.7098.0499.6798.1873.7294.3799.6389.71
+ TLC and LK98.0498.1390.0597.4399.5797.8684.3097.7999.7990.40
+ TLC, LK, EC, and HC, with historical conditioning for the critic97.3295.7589.3095.8999.5897.2968.2196.1999.8385.24

Adding EC and HC degrades performance under misaligned conditioning.

We further incorporated EC and HC and repeated both training stages. We tried different α values, but none yielded satisfactory performance. The configuration reported in Table W4 achieves 85.24 EPDMS, compared with 90.40 for the TLC-and-LK configuration. EC requires a historical model prediction, while HC requires ego states from the preceding three frames. To obtain EC labels, we used a frozen copy of the current model to generate historical trajectories, paired them with current-frame trajectories generated from the same initial noise samples, and computed EC using the official NAVSIM v2.2 procedure. For the critic, we encoded the historical prediction for EC and the preceding three ego states for HC, then concatenated each encoding with the current trajectory token before its corresponding scoring head.

AuthorsOpenReview ↗

Response (Part 3)

Adding EC and HC degrades performance under misaligned conditioning.

We further incorporated EC and HC and repeated both training stages. We tried different α values, but none yielded satisfactory performance. The configuration reported in Table W4 achieves 85.24 EPDMS, compared with 90.40 for the TLC-and-LK configuration. EC requires a historical model prediction, while HC requires ego states from the preceding three frames. To obtain EC labels, we used a frozen copy of the current model to generate historical trajectories, paired them with current-frame trajectories generated from the same initial noise samples, and computed EC using the official NAVSIM v2.2 procedure. For the critic, we encoded the historical prediction for EC and the preceding three ego states for HC, then concatenated each encoding with the current trajectory token before its corresponding scoring head.

Table W5 highlights the resulting actor–critic conditioning mismatch: the critic receives the historical information needed to predict EC and HC, while the actor retains its current-scene inputs. In particular, conditioning the actor on its own historical prediction introduces a recursive dependency. Generating the prediction for the previous frame would require an even earlier prediction, which in turn would require another preceding prediction. Our current training setup does not provide a rollout and initialization mechanism to resolve this chain, so we did not add these historical conditions to the actor.

We consider this conditioning mismatch a likely cause of the poorer result. The critic supplies reward gradients that depend on a particular historical prediction or ego-state history, but the actor cannot associate these directions with the historical conditions it does not observe. Consequently, the additional gradients, particularly those from EC, may act partly as noise during actor optimization and interfere with learning.

Table W5. Metric-wise conditioning availability for the Actor Net and Critic Net.

Current-frame conditions comprise scene features, current ego state, and the navigation command. Historical-frame conditions comprise the historical model prediction for EC and ego states from the preceding three frames for HC. ✓: the required conditioning is available and the metric is optimized by the actor or predicted by the critic. ✗: the metric is required, but its historical conditioning is unavailable. Blank: the metric is not used by that network's objective. In the main model, DDC supervises the critic but has zero reward weight for the actor.

M1: Main PDMS-based model; M2: Adjusted EP + DDC; M3: + TLC + LK; M4: + TLC + LK + EC + HC. Actor and Critic denote Actor Net and Critic Net, respectively.

MetricM1 ActorM1 CriticM2 ActorM2 CriticM3 ActorM3 CriticM4 ActorM4 CriticConditioning information
NC✓✓✓✓✓✓✓✓Current-frame information
DAC✓✓✓✓✓✓✓✓〃
DDC✓✓✓✓✓✓✓〃
TTC✓✓✓✓✓✓✓✓〃
EP✓✓✓✓✓✓✓✓〃
C✓✓✓✓✓✓✓✓〃
TLC✓✓✓✓〃
LK✓✓✓✓〃
EC✗✓Historical-frame information
HC✗✓〃

〃 denotes the same conditioning category as the preceding row.

Our reward setting aligns with mainstream NAVSIM RL methods.

Our PDMS-based reward follows the setting used by most existing RL post-training methods on NAVSIM like ReCogDrive [2], SimVTA [3], and DriveFine [4]. We consider the historical dependence of some EPDMS metrics a plausible reason for this preference: PDMS objectives can be evaluated against the current scene and candidate trajectory, whereas optimizing EC and HC additionally requires historical conditions that may be unavailable to a planner conditioned on the current scene. For our current architecture, PDMS therefore provides a training reward better aligned with the actor's available information, while EPDMS provides a broader evaluation of driving quality.

Reviewer_zbTYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FieldDrive, which applies Drifting-based one-step generation to autonomous driving trajectory planning. The authors first select high-scoring trajectories from a vocabulary for each scene and combine them with the expert trajectory as positive samples. A Drift field then trains the generator to model diverse feasible trajectories. In the second stage, the authors combine the Drift field with trajectory gradients from a learned critic to further improve planning performance. At inference, the model can directly produce a final trajectory in a single forward pass.

The paper reports competitive results on NAVSIM and presents ablations showing the benefits of combining distribution matching with reward optimization.

Critical strengths
  1. The problem is practically relevant. Improving planning performance through driving rewards while retaining the efficiency of one-step generation is a worthwhile goal. The two-stage training procedure is easy to follow and does not require additional reward-gradient updates at inference.

  2. The ablations provide evidence that the approach is effective. In Table 5, the full method achieves 92.14 PDMS, compared with 85.58 for Drift-only and 84.05 for Reward-only. These substantial differences are convincing and support combining the two update directions in the evaluated setting.

  3. The planning results are competitive. FieldDrive achieves 92.1 PDMS with direct decoding and 93.3 with an additional scorer on NAVSIM v1, as well as 89.7 EPDMS under the v2 metrics. The experiments on positive-sample counts and candidate counts also help explain its behavior under different settings.

Critical weaknesses
  1. The novelty appears mainly incremental and combinatorial, applying existing ideas to the recently introduced Drifting framework. I view this primarily as a characterization of the contribution rather than a reason to dismiss the work.

  2. The comparison with MeanFuser needs further clarification regarding architecture and timing settings. Table 1 lists DINOv2 ViT-S as the backbone for FieldDrive and ResNet-34 for MeanFuser. Although both use one-step generation, the paper reports a 1.26x speedup over MeanFuser. It is unclear how much of this difference comes from the generation method and how much comes from the backbone, decoder, trajectory reconstruction, or other implementation choices. Please clarify whether both methods were measured with the same hardware, batch size, numerical precision, and timing scope, and provide a breakdown of the main components. A comparison with matched architectures would further help establish the source of the speed advantage.

  3. Further analysis of whether critic gradients improve the actual reward would be helpful. The method guides the generator using trajectory gradients from a learned critic, but an increase in predicted reward does not necessarily imply an improvement in the actual driving score. The paper refers to a validation in Appendix B.4; it would be helpful to summarize those findings in the response, particularly the changes in actual PDMS after following the gradient on scenes not used to fit the critic, and the effect of different update magnitudes. This would help assess the reliability of the update direction and whether predicted reward can increase while the actual score decreases.

  4. The smaller performance gains would benefit from evidence of stability. I find the larger differences, such as the approximately 6.6-point PDMS improvement over Drift-only, convincing evidence that the method is effective. However, under the NAVSIM v2 metrics, FieldDrive achieves 89.7 EPDMS compared with 89.5 for MeanFuser, a difference of only 0.2 points. For such small gains, reporting the mean and standard deviation across independent training seeds would help establish whether the advantage is consistent. This concern applies mainly to these small differences and does not negate the larger improvements already demonstrated in the paper.

Questions for authors
  1. Was MeanFuser's speed remeasured under exactly the same conditions as FieldDrive? Do both measurements include visual encoding, trajectory decoding, and post-processing?
  2. After adjusting trajectories along the critic gradient, what fraction show an improvement in actual PDMS? Were there cases where predicted reward and actual score changed in opposite directions?
  3. Could the authors provide the mean and standard deviation across training seeds for the 0.2-point EPDMS improvement over MeanFuser under the NAVSIM v2 metrics?
5 comments
AuthorsOpenReview ↗

Response (Part 1)

We appreciate your recognition of the practical value of efficient one-step planning and address your concerns about novelty, timing comparisons, critic-gradient reliability, and training stability.

W1: Novelty and Main Contributions

The gap between one-step generation and existing RL formulations.

We agree that FieldDrive builds on the existing Drift framework. Our core contribution is to bridge the gap between one-step generative modeling and reinforcement learning. Diffusion and flow-matching models learn denoising or transport dynamics across different noise levels, ultimately mapping Gaussian noise to the target data distribution. Established RL methods such as DDPO [1] optimize generation indirectly through intermediate stochastic transitions, increasing the probability of generation paths that yield higher rewards. When generation is reduced to a single step that directly maps noise to a final trajectory, this formulation based on a multi-step stochastic chain does not carry over directly.

From distribution matching to reward optimization

The recently introduced Drift framework formulates generation directly as distribution matching. Its appendix connects the unnormalized Drift [2] update to a direction that reduces the maximum mean discrepancy (MMD) between the generated and target distributions, setting aside the normalization as an engineering trick used in practice. This perspective inspired three connected insights: (i) generative modeling extends point-to-point prediction to a mapping between distributions; (ii) imitation learning for a generative model aligns its generated distribution with a target distribution; and (iii) reinforcement learning for a generative model therefore should be formulated as optimizing a reward functional of its generated distribution.

Our methodological and practical contribution.

Building on these insights, we derive the optimization direction of the distributional reward objective, obtaining the trajectory gradient of a differentiable critic as the Reward Field Vector (Lemma 2). This unifies the optimization directions of imitation learning and reinforcement learning under the common concept of the Field Vector and leads to a direct RL post-training method for one-step generation. We regard this unified formulation and the resulting training method as our main methodological contribution. Autonomous driving provides a practical setting where multiple feasible trajectories coexist with strict inference-time requirements. The results and ablations demonstrate the advantages of this approach in planning quality while retaining efficient single-pass trajectory generation.

AuthorsOpenReview ↗

Response (Part 2)

W2/Q1: Fairness and attribution of the speed advantage over MeanFuser.

We are happy to provide the details of our planning-speed measurements and supplement them with an end-to-end inference-speed comparison.

Timing scope and architectural details.

We follow the evaluation scope and settings used for the planning-speed metric in the MeanFuser [3] paper. Specifically, planning latency measures the time from the scene tokens produced by the visual encoder to the final output trajectory, including trajectory decoding and, where applicable, trajectory reconstruction. It excludes visual encoding. We focus on this metric because our contribution concerns generative trajectory decoding and its RL post-training.

Planning latency depends on the number of generation steps, as well as the token count, feature dimension, attention structure, and number of Transformer layers. Both FieldDrive and MeanFuser use one-step generation; Table Z1 compares their decoder architectures.

Table Z1. Decoder architecture comparison.

ComponentDenoise stepFeature dimensionConditioning token numberTransformer layersAttention structureReconstruction module
MeanFuser decoder11281,0541Self-attention and cross-attentionneed
FieldDrive decoder1256644Cross-attention onlyno need

For hardware and inference settings, we follow MeanFuser by using a single NVIDIA H20 GPU with batch size 1. All methods in our timing experiments use FP32 precision. We evaluated both methods over 10,000 scenes. On our hardware, MeanFuser's planning latency is 2.66 ms, while FieldDrive's is 1.83 ms. In the submitted paper, we directly used the planning-speed results reported in MeanFuser's paper. FieldDrive outputs the final trajectory directly and requires no reconstruction module. Eliminating reconstruction is the direct source of the planning-speed advantage in this comparison: MeanFuser spends 1.16 ms on decoding and a further 1.50 ms on reconstruction, whereas FieldDrive directly produces the final trajectory in 1.83 ms.

With DINOv2 ViT-S as the encoder, however, FieldDrive is slower than MeanFuser in end-to-end inference because of the visual-encoding cost. To assess the effect of changing the backbone, we retrained FieldDrive using ResNet-34. This variant uses a single visual backbone and omits the redundant LiDAR branch retained in MeanFuser's TransFuser-style encoder. In the camera-only setting, that branch is randomly initialized and receives no LiDAR observations, yet still incurs computation. Removing it reduces encoding latency from 10.78 ms to 6.26 ms.

FieldDrive achieves planning gains over MeanFuser with both backbones, showing that the gains extend beyond the choice of DINOv2. In particular, the ResNet-34 variant achieves 91.2 PDMS with only 31.7M parameters and an end-to-end inference speed of approximately 135.5 FPS (7.38 ms per scene). It outperforms MeanFuser in PDMS, end-to-end inference speed, and planning speed. This result provides strong evidence of our method's practical advantages.

Table Z2 reports the component latency breakdown. All latency values are in milliseconds. Planning latency is the sum of decoding and reconstruction latency, and end-to-end latency is computed as encoding plus planning latency.

Table Z2. Inference latency breakdown and planning performance. Latencies are in milliseconds.

MethodVisual encodingEncodingDecodingReconstructionPlanningEnd-to-endPDMS
MeanFuserTransFuser-style dual ResNet-3410.781.161.502.6613.4489.0
FieldDriveDINOv2 ViT-S128.471.83Not required1.83130.3092.1
FieldDrive–ResNetStandard ResNet-346.261.12Not required1.127.3891.2
AuthorsOpenReview ↗

Response (Part 3)

W3/Q2: Reliability of critic gradients in improving actual PDMS.

Appendix Table 8 in the submitted paper provides an initial evaluation of whether following these gradients improves actual PDMS. We expand this analysis in Tables Z3 and Z4 with the full subscore breakdown on held-out test scenes and results at different update magnitudes.

Evaluation with a frozen actor and critic.

For each scene in the full navtest set, we use the frozen first-stage Actor Net to generate 64 initial trajectories. We then use the frozen first-stage Critic Net to compute the reward gradient with respect to each trajectory and update the trajectory along this direction. After each update, we recompute the gradient at the updated trajectory to obtain the next update direction. Both networks remain frozen throughout; only the trajectories are updated. The test scenes are not used to fit either network.

At each step, we evaluate the updated trajectories using the actual PDM evaluator and average the scores over the 64 trajectories and all test scenes. Thus, Tables Z3 and Z4 measure changes in actual driving scores. This experiment isolates the critic's trajectory-update direction from changes in the actor or critic parameters.

Actual PDMS and subscore improvements.

Our reference step size is 0.000625, computed as the product of the reward-field weight (0.05), trajectory-loss weight (500), and mean second-stage learning rate (2.5 × 10⁻⁵). With this step size, actual PDMS increases from 75.12 to 80.21 over five updates, a gain of 5.09 points. Every reported subscore in Table Z3 also improves throughout the update sequence. All scores in Table Z3 are reported on a 0–100 scale.

Table Z3. Actual PDMS and subscores over five critic-gradient updates at step size 0.000625.

Update stepPDMSNCDACEPTTCCDDC
075.1290.8989.5676.1480.5099.2089.78
177.3791.4491.4778.3081.2599.2389.88
278.4391.7992.2679.3181.7199.2789.96
379.1792.0692.7780.0082.0899.3090.00
479.7592.2893.1680.5382.3999.3290.05
580.2192.4693.4780.9682.6499.3590.09

Effect of update magnitude.

We repeat the five-step experiment with four step sizes, using the same initial trajectories and frozen critic. Table Z4 reports actual mean PDMS on the same 0–100 scale.

Table Z4. Actual PDMS over five critic-gradient updates with different step sizes.

Step sizeStep 0Step 1Step 2Step 3Step 4Step 5
0.62575.1242.8731.8828.9027.5126.84
0.0062575.1280.3581.4882.4683.0983.61
0.00062575.1277.3778.4379.1779.7580.21
0.000062575.1275.3675.5975.8075.9976.18

Actual mean PDMS improves at every update for the three tested step sizes from 0.0000625 to 0.00625. Smaller steps produce more gradual improvements, while a step size of 0.00625 reaches 83.61 PDMS after five updates. The much larger step size of 0.625 causes a sharp deterioration, showing that the local gradient requires an appropriate update magnitude. Together, these results show that the frozen critic provides useful reward-improvement directions on held-out scenes across a range of step sizes, supporting its use to guide actor optimization during RL post-training.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response (Part 1)

Thank you for your careful review and constructive comments on our approach to one-step generation and reward optimization; we welcome the opportunity to clarify our contribution and discuss reward generalization, exploration, and the training configurations.

W: Novelty and Main Contributions beyond Component Combination.

From distribution matching to reward optimization.

Our core contribution is to bridge the gap between one-step generation and reinforcement learning, where existing formulations such as DDPO [1] rely on intermediate stochastic transitions that do not carry over directly to one-step generators. The recently introduced Drift [2] framework formulates generation directly as distribution matching. Its appendix connects the unnormalized Drift update to a direction that reduces the maximum mean discrepancy (MMD) between the generated and target distributions, setting aside the normalization used in practice. This perspective inspired three connected insights: (i) generative modeling extends point-to-point prediction to a mapping between distributions; (ii) imitation learning for a generative model aligns its generated distribution with a target distribution; and (iii) reinforcement learning for a generative model can therefore be formulated as optimizing a reward functional of its generated distribution.

Our methodological and practical contribution.

Building on these insights, we derive the optimization direction of the distributional reward objective, obtaining the trajectory gradient of a differentiable critic as the Reward Field Vector (Lemma 2). This unifies the optimization directions of imitation learning and reinforcement learning under the common concept of the Field Vector and leads to a direct RL post-training method for one-step generation. We regard this unified formulation and the resulting training method as our main methodological contribution. Each component in our method serves a specific role: Drift provides the distribution-matching generative foundation and the capacity to model multiple trajectory modes, which supports exploration during RL; the Critic Net provides the reward-improvement direction; the selected positive trajectories engage this multimodal modeling capacity and establish a feasible exploration space; and RL enables the generator to optimize the specified reward beyond matching the given positive samples. Table T1 summarizes the ablation settings from the paper and our supplementary experiments. The reported results show that RL improves the GT-only model from 84.65 to 90.64 PDMS and the vocabulary-plus-GT model from 79.34 to 92.14 PDMS. Moreover, the richer positive set leads to a further 1.50-point gain after RL, supporting the role of feasible multimodal exploration in reward optimization. A deterministic model that predicts a single trajectory from one query provides limited coverage of feasible behaviors, making direct reward-gradient optimization more dependent on the initial trajectory and limiting exploration of alternative high-reward modes.

Table T1. Ablation configurations and planning performance.

SettingTrajectory modelIL positive setRL post-trainingPDMS
(a)Deterministic query-based mappingGT onlyNo85.88
(b)Deterministic query-based mappingGT onlyYes87.99
(c)Drift generative modelGT onlyNo84.65
(d)Drift generative modelGT onlyYes90.64
(e)Drift generative modelVocabulary + GTNo79.34
(f)Drift generative modelVocabulary + GTYes92.14
AuthorsOpenReview ↗

Response (Part 2)

Q1: Reward Design and Potential Overfitting.

Reward design and metric over-optimization.

We use the same PDM evaluator and metric definitions as in navtest to compute rewards on training scenes. This follows the PDMS-based reward design used in prior RL post-training methods such as ReCogDrive [3], SimVTA [4], and DriveFine [5]. The component metrics measure different dimensions of driving performance: NC measures collision avoidance, DAC measures drivable-area compliance, and EP measures ego progress, for example. We agree that reward hacking is a potential concern during RL post-training [6]. As you noted, we also observe signs of over-optimization on some metrics. Designing rewards that lead to better performance across a broad range of driving tasks remains an open problem; PDMS provides a reasonable starting point, while the balance among its objectives still matters.

Adjusting reward weights and evaluation on navtest.

In our experiments, reducing the EP weight and setting the DDC weight to 1 yields a model with strong performance under both PDMS and EPDMS on navtest, as reported in Tables T2 and T3.

Table T2. PDMS evaluation of the main and adjusted-reward variants.

VariantNCDACEPTTCCDDCPDMS
Adjusted EP + DDC98.5599.0586.1695.3899.9598.7091.51
Main Exp97.8598.6390.3693.42100.0095.8792.14

Table T3. EPDMS evaluation of the main and adjusted-reward variants.

VariantNCDACEPTTCDDCHCECLKTLCEPDMS
Adjusted EP + DDC98.5599.0588.7098.0499.6798.1873.7294.3799.6389.71
Main Exp97.8598.6393.7397.0997.6898.0357.6292.5299.4286.63

Zero-shot transfer to closed-loop HUGSIM evaluation.

To examine performance beyond the NAVSIM evaluation, we additionally evaluate FieldDrive in the closed-loop HUGSIM benchmark. Some of its evaluation datasets do not support a rear-camera input. We therefore retrain a version of FieldDrive using the current front, left, and right views together with the front view from the previous frame (0.5 seconds earlier) , keeping all other settings the same as those of the EPDMS-oriented variant. This model is trained on the original NAVSIM training data and transferred to HUGSIM without training or fine-tuning on HUGSIM. Table T4 reports its closed-loop results.

Table T4. Zero-shot closed-loop evaluation on HUGSIM.

MethodRC ERC MRC HRC XRC Avg.HD-Score EHD-Score MHD-Score HHD-Score XHD-Score Avg.
LTF [7]67.8035.1026.2040.5040.4258.9018.009.8025.9025.57
UniAD [8]78.4060.5033.6017.8047.7764.9045.8020.606.6034.50
VAD [9]51.3031.1025.3026.5032.4436.309.508.0011.5014.56
GTRS-Dense [10]64.2050.0020.7022.3039.6155.5039.0011.7014.3030.18
Latent-WAM [11]84.2042.5030.6035.5045.8872.5024.0012.2018.1028.91
MeanFuser [12]76.7041.1037.9028.4043.9067.1024.3021.5011.0028.40
FieldDrive88.5445.4833.8131.8047.5874.2329.1017.7714.8131.51

On HUGSIM, FieldDrive achieves 47.58 RC Avg. and 31.51 HD-Score Avg., exceeding MeanFuser's reported results by 3.68 RC points and 3.11 HD-Score points, respectively. These closed-loop results provide evidence that the learned policy can transfer beyond the NAVSIM benchmark used for training. Together with the navtest results, they support the generalization potential of reward-field optimization, while the remaining metric imbalance, including the low EC, motivates further work on more comprehensive reward design.

AuthorsOpenReview ↗

Response (Part 3)

Q2: How Exploration Benefits Reinforcement Learning.

With zero noise as a fixed input, the decoding process is indeed deterministic. The role of multimodality and exploration is primarily to provide a foundation for RL post-training: the model first learns a diverse distribution of feasible trajectories, allowing reward optimization to explore a broader feasible space before concentrating the distribution in high-reward regions.

Multimodal exploration after IL improves the outcome of RL.

Table T5 reproduces the relevant results from Table 3 of the submitted paper, showing that greater trajectory diversity after imitation learning leads to better performance after RL. Increasing the positive set from GT only to 32 vocabulary trajectories plus GT raises pairwise ADE from 0.65 to 2.25 and improves the final RL PDMS from 90.64 to 92.14. The more diverse initialization leads to a better outcome after reward optimization. This supports our design: the purpose of first-stage multimodality is to expose the model to more feasible behaviors that the second stage can explore and improve.

The diversity metrics in Table T5 are measured after IL, following the definitions in the paper.

Table T5. Effect of IL positive-sample count on trajectory diversity and RL performance.

IL positive setPairwise ADEPairwise FDEMax ADEMax FDEIL PDMSRL PDMS
GT only0.651.482.165.1284.6590.64
4 voc traj + GT0.921.862.755.8683.9791.63
16 voc traj + GT1.522.774.568.4181.2191.70
32 voc traj + GT2.253.876.8111.6979.3492.14

Concentrating the distribution enables direct inference.

After RL, the generated trajectories concentrate in high-reward regions, and different noise inputs produce closely clustered trajectories, as illustrated in Figure 3. The exploration performed during training therefore benefits the final deterministic policy in inference. We directly use zero noise to generate a high-quality trajectory in a single forward pass, avoiding subsequent trajectory reconstruction or candidate selection. Appendix Table 9 further explains this choice: different initial noise inputs yield trajectories of similar quality, making zero-noise inference a reasonable choice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.