acceptodds
Under review as a conference paper at ICLR 2027

Driving-OPD: Learning Complementary Driving Experts with Reward Variance-Aware RL and On-Policy Distillation

Abstract

Reinforcement learning (RL) is increasingly adopted for post-training of Vision- Language-Action (VLA) models in end-to-end autonomous driving. However, existing approaches mainly optimize a single policy, leaving the effect of differ- ent RL treatments on driving capabilities underexplored. Under structured driv- ing rewards, we find that GRPO’s variance-dependent scaling changes the rela- tive optimization emphasis across rollout groups. Motivated by this, we intro- duce reward-variance-aware RL, which explores different treatments of variance- dependent scaling to induce complementary driving specializations. Specifically, Dr.GRPO removes standard-deviation normalization and yields a safety-oriented expert, while our Variance-Gated Scaling GRPO (VGS-GRPO) retains controlled variance-dependent scaling and produces a progress-oriented expert from the same base policy and RL data. Building on this finding, we propose Driving- OPD, a three-stage post training framework consisting of supervised fine-tuning (SFT), reward-variance-aware RL for complementary expert learning, and On- Policy Distillation (OPD) for expert integration. We use Win-Margin-Based Sce- nario Selection and Dual-Teacher Routing to improve multi-expert distillation by providing more effective expert-specific supervision. Without explicit Chain-of- Thought reasoning or additional planning modules, Driving-OPD directly gener- ates trajectories autoregressively and achieves 92.8 PDMS on NAVSIM v1 and 91.4 EPDMS on NAVSIM v2, highlighting the strong post-training potential of autoregressive Driving VLAs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Autonomous Driving, Reinforcement Learning, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bwKXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose Driving-OPD, a post-training framework for autonomous driving. Built on observation on variance-dependent advantages of GRPO, authors proposed VGS-GRPO to train progress expert and use Dr. GRPO to train safety expert. The progress expert is further trained with OPD with supervision from two experts. The final VLA achieves SOTA performance on NAVSIM v1 and v2.

Critical strengths
  1. The paper is built on clear motivations from the observations of advantage scaling and reward group variance in GRPO. This motivates the proposed VGS-GRPO to learn a progress-oriented expert. The subsequent integration of two experts to single student policy is also natural and well motivated.
  2. The paper provides clear empirical results and ablations for OPD stage. The experiments clearly show the complementary of two experts and provide empirical support for the major design of OPD stage choice.
Critical weaknesses
  1. Though authors conduct analysis on reward variance and advantage magnitude, it still somehow remain unclear why VGS-GRPO achieved specially high EP and Dr. GRPO achieved high NC. A clear and deeper analysis of the mechanism of the relationship of advantage and resulting behavior could better motivate the design.
  2. The VGS-GRPO mainly adopt chosen hyper parameters like τ, k, s_max. Some are selected from the training distribution statistics, and some are chosen empirically. More ablations on these parameters could better support the generalization of method across different data distribution.
Questions for authors
  1. Have author consider utilizing OPSD in the training procedure? It could be interesting to see whether the student policy could serve as teacher to teach itself to replace the multi-teacher OPD?
  2. The two experts have different specializations on EP and NC. What if the two complementary capabilities of progress expert and safety expert conflict during OPD? Will it leads to degradation on other skills or performance?
  3. The title may overemphasize OPD since OPD is only a part of the overall pipeline and RL stage is also an important part of the method.
4 comments
AuthorsOpenReview ↗

Response to W1 and W2

We sincerely thank the reviewer for the positive assessment of our work and the thoughtful, constructive suggestions. We greatly appreciate the reviewer’s recognition and believe that these valuable comments have helped us further improve the quality and clarity of our paper.

W1:

We appreciate the reviewer’s suggestion to further clarify the relationship between reward variance, advantage magnitude, and the resulting driving behaviors, particularly the distinct performance characteristics of VGS-GRPO and Dr.GRPO in terms of EP and NC.

Low-variance groups are dominated by fine-grained EP differences.

According to the official NAVSIM PDMS scoring formula, EP and TTC have the same weight (5/12), but differ in granularity: EP is continuous, whereas TTC is binary ({0, 1}). NC and DAC introduce additional discontinuities through multiplicative zeroing, while Comfort contributes little variation since nearly all trajectories satisfy this criterion.

To examine their effects on rollout variance, we analyze VGS-GRPO on-policy rollout groups (K=8).

Rollout group patternProportionMean σMedian σ
All TTC = 162.90%0.0110.008
All TTC = 01.60%0.0130
Mixed TTC (0/1)18.90%0.1860.193
All-zero reward (invalid-gradient groups)16.60%00
Total100.00%——

Groups with mixed TTC states exhibit substantially higher reward variance, whereas groups with uniform TTC states (all 0 or all 1) exhibit similarly low variance. This suggests that although TTC and EP have identical scoring weights, advantage scaling in low-variance groups primarily acts on the fine-grained differences provided by continuous EP. Moreover, NC and TTC both reflect collision-related safety, suggesting a degree of synergy between these two metrics.

These findings also explain the distinct optimization tendencies of the two methods: VGS-GRPO preferentially amplifies low-variance groups dominated by EP differences, whereas Dr.GRPO retains the relative contribution of larger advantage signals from high-variance groups, which are mainly associated with safety-related metrics. Consequently, VGS-GRPO exhibits stronger EP performance, whereas Dr.GRPO achieves higher scores on safety-related metrics, including NC, DAC, and TTC.

W2:

Ablation Study on VGS-GRPO Hyperparameters

We thank the reviewer for this valuable suggestion. In VGS-GRPO, the gating coefficient is defined as:

βq=11+exp⁡(k(σq−τ)).\beta_q=\frac{1}{1+\exp(k(\sigma_q-\tau))}.

Here, τ\tau determines the transition threshold of the variance gate, while kk controls the sharpness of the transition. Increasing τ\tau extends the region where the gate retains stronger variance-dependent scaling to groups with higher reward variance, allowing more groups to receive additional scaling for fine-grained reward differences. In contrast, decreasing τ\tau restricts this effect to lower-variance groups. Increasing kk makes the transition sharper, strengthening the preferential scaling of low-variance groups, whereas decreasing kk produces a smoother transition and weakens this distinction. Since low-variance groups in our training distribution mainly reflect fine-grained differences among otherwise successful trajectories, retaining their scaling sensitivity is important for preserving progress-related optimization signals.

We set τ=0.06\tau=0.06 based on the training distribution, where approximately 75.8% of rollout groups have a within-group standard deviation below this threshold. We choose k=40k=40 empirically and set smax⁡=8s_{\max}=8 to prevent excessive advantage amplification.

Following the reviewer's suggestion, we conduct additional ablation experiments on τ\tau and kk, while keeping smax⁡=8s_{\max}=8. We also include the main experimental configuration for reference.

ConfigurationPDMSNCDACTTCEPC
Main: τ=0.06,k=40\tau=0.06, k=4090.5697.6097.2991.5190.3399.96
τ=0.05,k=40\tau=0.05, k=4091.1898.2898.0793.8688.2699.93
τ=0.06,k=30\tau=0.06, k=3091.2398.3998.0593.9588.2799.93
τ=0.07,k=40\tau=0.07, k=4090.1997.3497.1291.1090.2299.93

The results illustrate how the gate parameters regulate the relative emphasis on fine-grained reward differences. Increasing τ\tau expands the range of groups receiving variance-dependent scaling, whereas increasing kk sharpens the distinction between low- and high-variance groups. Conversely, decreasing τ\tau narrows this range, while decreasing kk produces a smoother transition. The observed changes in EP and safety-related metrics reflect the resulting shift in optimization preference. These ablations provide empirical support for the intended roles of τ\tau and kk in controlling variance-dependent scaling.

AuthorsOpenReview ↗

Response to Q1, Q2 and Q3

Q1:

We appreciate the reviewer’s suggestion. We agree that exploring OPSD in autonomous driving trajectory generation is an interesting direction. OPSD relies on privileged information to construct stronger and weaker variants of the same policy, which serve as the teacher and student, respectively, for self-distillation.

However, our current experimental setting does not involve such privileged information, making it difficult to construct the corresponding teacher–student pair required by OPSD.

In contrast, this work focuses on introducing OPD as a training paradigm into autonomous driving trajectory planning, providing an alternative approach to addressing the challenge of simultaneously improving driving progress (EP) and safety (NC), which are difficult to optimize together through conventional RL. We hope this study provides useful insights for further exploring OPD in autonomous driving. Meanwhile, our investigations into scenario selection, teacher routing, and training-data analysis may also offer valuable references for the future application of OPSD in this domain.

Q2:

Since the two experts exhibit substantially different driving behaviors, their supervision signals inherently involve certain conflicts. To mitigate this issue, we route each training sample to the corresponding expert based on its PDMS advantage, rather than directly mixing the probability distributions of the two teachers. Thus, each sample receives supervision from a single expert, providing a clear learning target during distillation.

Furthermore, the trade-off between the two experts can be adjusted by changing their respective proportions in the distillation data. We evaluate different data ratios while keeping the total distillation pool fixed at 6K scenarios.

Progress Expert : Safety ExpertPDMSNCDACTTCEPComfortΔ vs. 3:3
1:592.4699.1398.6396.7187.3899.94-0.32
2:492.6199.1498.2896.8188.1699.95-0.17
3:392.7899.0098.2796.5388.8299.95—
4:292.7398.9598.1395.7689.7599.95-0.05
5:192.4098.7497.8095.1290.0899.94-0.38

Increasing the progress-expert data proportion improves EP, while increasing the safety-expert proportion generally favors safety-related metrics.

These results show that OPD seeks a relative balance between safety and progress when learning from the two teachers, and this balance can be adjusted through the data ratio. Regarding the potential degradation of other capabilities, although individual metrics vary across different ratios, we observe no substantial degradation in the evaluated metrics. Notably, all tested ratios achieve higher overall PDMS than both teacher experts.

These findings suggest that expert-specific data allocation provides a practical way to balance competing driving objectives during distillation, offering useful insights for future multi-expert distillation research in autonomous driving.

Q3:

We appreciate the reviewer’s observation. We would like to clarify that reward variance-aware RL and OPD play equally important and complementary roles in our framework. The effectiveness of OPD relies on first obtaining specialized experts through different RL treatments; without this expert specialization, it would be difficult for the subsequent OPD stage to achieve substantial gains from knowledge integration. Accordingly, our paper title places reward variance-aware RL and OPD on equal footing, reflecting their complementary contributions to the overall framework.

We use Driving-OPD as the abbreviation for our proposed framework to highlight our exploration of OPD as a complementary training paradigm beyond mainstream RL-based optimization in autonomous driving. Our intention is to explore complementary training strategies for autonomous driving and provide useful insights for future research, rather than to imply that OPD constitutes the entire contribution of the framework.

Thanks for the reply. I find the new results to be interesting and I will keep my current positive score.

Thank you for your follow-up and for maintaining your positive assessment. We appreciate your constructive feedback, which has helped us strengthen the analysis and experiments. We will incorporate the additional results and clarifications into the revised manuscript.

Reviewer_K23rOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper presents Driving-OPD, a three-stage framework comprising supervised fine-tuning, expert specialization, and on-policy distillation to improve trajectory generation in autoregressive driving VLAs.
  2. The authors analyze the effects of reward standard-deviation normalization in GRPO. They use Dr.GRPO and VGS-GRPO to obtain safety-oriented and progress-oriented experts, respectively, and integrate their capabilities through win-margin-based scenario selection and dual-teacher routing.
  3. Driving-OPD achieves 92.8 PDMS on NAVSIM v1 and 91.4 EPDMS on NAVSIM v2. Ablations examine reward-variance treatments, distillation objectives, scenario selection, and teacher configurations.
Critical strengths
  1. Well-motivated problem with informative empirical analysis. Figure 2 and Appendix A connect reward variance, advantage magnitude, and safety-critical failures. The recovery obtained by bounding the scaling factor provides useful evidence for the proposed explanation of GRPO degradation and motivates the subsequent expert-specialization strategy.
  2. Effective integration of complementary capabilities. The final model outperforms both individual experts in PDMS, while Appendix C demonstrates additional performance headroom from expert complementarity. Ablations on teacher routing, scenario selection, and progress-expert replacement provide meaningful support for the overall framework beyond its headline benchmark results.
Critical weaknesses
  1. The robustness of several key improvements remains unclear. Win-margin-based selection improves PDMS by only 0.3 points, while using VGS-GRPO instead of GRPO+Cap as the progress expert improves the distilled model by 0.4 points. Without repeated runs and parameter-sensitivity analyses, it is difficult to determine whether these differences consistently support the corresponding design choices.
  2. The necessity of the specific specialization strategy is not fully established. The existing ablations support dual-teacher integration, but do not include two teachers trained with the same RL method using different random seeds. Consequently, the results do not fully distinguish the benefits of the proposed specialization mechanism from those attributable to general policy diversity and additional training.
Questions for authors
  1. Could the authors report results across multiple random seeds for the key ablations, particularly the comparisons between 92.8 and 92.4/92.5 PDMS, and explain how the gating parameters were selected?
  2. Would distilling two teachers trained with the same RL method but different random seeds yield similar gains under a matched training budget? This comparison would directly affect my assessment of the contribution of the proposed specialization strategy.
4 comments
AuthorsOpenReview ↗

Response to W1 and Q1

We thank the reviewer for the positive assessment of our work and the constructive feedback, which helps us further clarify and strengthen the paper.

W1:

To address the reproducibility concern, we repeat three key distillation settings—the main configuration, random scenario selection, and replacing VGS-GRPO with GRPO+Cap as the progress teacher—with two additional random seeds, 567 and 1234. We also clarify the rationale for the gating parameters.

(1) Repeated experiments on key distillation ablations.

In addition to the original seed 42, we repeat the three key distillation settings with seeds 567 and 1234, keeping the model architecture and distillation hyperparameters unchanged:

Distillation settingSeed 42Seed 567Seed 1234
VGS-GRPO + Dr.GRPO (Win-margin selection)92.7892.6892.85
VGS-GRPO + Dr.GRPO (Random selection)92.5192.4492.56
GRPO+Cap + Dr.GRPO (Win-margin selection)92.3592.3292.32

The performance ordering remains consistent across all three distillation seeds. Win-margin selection improves PDMS over random selection by 0.27, 0.24, and 0.29 points, respectively. Using VGS-GRPO instead of GRPO+Cap as the progress teacher improves PDMS by 0.43, 0.36, and 0.53 points, respectively. Both design choices therefore yield consistent gains across the tested runs, providing repeated evidence for the corresponding distillation comparisons.

(2) Gating parameter selection and scaling behavior.

We set τ=0.06\tau=0.06 based on training-distribution statistics: approximately 75.8% of rollout groups have a within-group standard deviation below this threshold. We set smax⁡=8s_{\max}=8 to match the scaling upper bound of GRPO+Cap, while k=40k=40 is an empirical setting controlling the steepness of the gating transition. We did not select these parameters based on final PDMS. We further examine the scaling behavior produced by this configuration.

We compare GRPO+Cap and VGS-GRPO on the same 27,943 non-zero-variance rollout groups collected over 279 on-policy training steps:

Scaling factorGRPO+Cap (%)VGS-GRPO (%)
≈11.620.3
[2, 3)8.41.7
[3, 4)2.83.1
[4, 5)1.74.9
[5, 6)1.67.3
[6, 7)1.910.4
[7, 8)1.213.6
=8 (clipped)80.838.8
Mean / Median7.18 / 8.005.79 / 7.20

VGS-GRPO reduces the proportion of groups reaching the scaling upper bound from 80.8% to 38.8%. Among the 11,750 groups clipped by GRPO+Cap but not by VGS-GRPO, the scaling factors have a mean of 5.94 and a median of 6.32. Thus, many groups assigned the same upper-bound weight by GRPO+Cap receive differentiated scaling after gating.

Q1:

As detailed in our response to W1, the three distillation settings maintain consistent performance ordering across all three random seeds. The repeated experiments demonstrate the reproducibility of the observed gains from win-margin selection and VGS-GRPO-based expert integration. We also clarify the training-distribution-based choice of τ\tau, the scaling bound smax⁡s_{\max}, and the empirical selection of kk. Together, these additional experiments and analyses address the concerns regarding experimental reproducibility and hyperparameter selection.

AuthorsOpenReview ↗

Response to W2 and Q2

W2:

Thank you for suggesting this important control. We add two complementary experiments: distillation using two teachers trained with the same RL method but different random seeds, and continued single-policy RL training. We match the distillation data size, training epochs, and optimization hyperparameters where applicable to examine the contributions of policy diversity and knowledge integration.

(1) Same-method teachers trained with different random seeds.

We use two VGS-GRPO teachers trained with RL seeds 42 and 1234. For each teacher, we select the top 3K scenarios by its PDMS win margin, forming a 6K distillation pool. We use the same JSD configuration as the main experiment (β=0.5\beta=0.5, top-100, 2 epochs, and a learning rate of 2×10−62\times10^{-6}). The teachers’ metrics are:

TeacherRL seedPDMSNCDACTTCEP
VGS-GRPO123490.7697.5597.8391.3090.36
VGS-GRPO4290.5697.6097.2991.5190.33

The two teachers have similar aggregate safety and progress metrics, without the pronounced specialization observed between VGS-GRPO and Dr.GRPO, although they may still be complementary on individual scenarios. The corresponding student results are:

Distillation settingPDMS
Student initialization90.56
VGS-GRPO + VGS-GRPO (Different RL seeds)91.38
VGS-GRPO + Dr.GRPO (Main OPD)92.78

Same-method dual-teacher OPD improves student PDMS from 90.56 to 91.38, a gain of 0.82 points, showing that policy diversity from different RL seeds provides useful supervision. However, the heterogeneous combination achieves 92.78 PDMS, a further improvement of 1.40 points. Thus, under the tested teacher combinations and distillation configuration, the same-method teachers do not achieve comparable performance.

(2) Comparison with additional single-policy RL training.

To further examine the role of additional training, we continue RL training of both original experts for 2 epochs on exactly the same 6K scenarios used by the main OPD setting. Full component metrics are provided in our response to Reviewer BfGB’s W2.

Training settingPDMS
Continued RL (VGS-GRPO)92.43
Continued RL (Dr.GRPO)92.08
Dual-teacher OPD (Ours)92.78

Continued RL substantially improves the original experts, but dual-teacher OPD still achieves 0.35 and 0.70 points higher PDMS, respectively.

Q2:

As reported in W2, we distill two VGS-GRPO teachers trained with different RL seeds, matching the main experiment’s distillation data size (6K), training epochs (2), and optimization hyperparameters. The same-method combination achieves 91.38 PDMS, improving over the initialization of 90.56 but remaining 1.40 points below the heterogeneous combination’s 92.78.

Thank you for the detailed responses and additional experiments. Results across three distillation seeds consistently support the benefits of win-margin selection and VGS-GRPO, while the gating rationale and sensitivity analysis clarify the design choices. The same-method teacher control also shows that heterogeneous specialization provides gains beyond generic policy diversity. I will consider raise my score.

Thank you for your thoughtful feedback and positive assessment of our work. We have incorporated the key additional experiments and analyses into the final version of the manuscript. We sincerely appreciate your recognition and constructive suggestions, which have been instrumental in strengthening our experimental validation and improving the overall quality of the paper. Thank you again for your time and valuable contributions.

Reviewer_BfGBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Driving-OPD, a three-stage post-training framework that learns complementary safety- and progress-oriented driving experts using Dr.GRPO and VGS-GRPO, then integrates them through on-policy distillation. The method achieves 92.8 PDMS on NAVSIM v1 and 91.4 EPDMS on NAVSIM v2.

Critical strengths
  1. The paper provides an interesting analysis of how reward-variance normalization affects GRPO optimization and driving specialization.
  2. The method achieves strong benchmark performance, supported by ablations on RL strategies, teacher selection, and distillation.
Critical weaknesses
  1. The incremental contribution of VGS-GRPO is limited. Simple scaling-factor clipping already achieves 91.8 PDMS, outperforming VGS-GRPO (90.6). After distillation, replacing VGS-GRPO with the clipped baseline only reduces PDMS from 92.8 to 92.4. The necessity of the proposed gating formulation is therefore not fully established.
  2. The benefit of multi-expert distillation is not sufficiently isolated from additional training. The OPD stage introduces further optimization on 6K selected scenarios, but lacks compute- and data-matched comparisons with continued single-policy RL or fine-tuning.
  3. The distilled policy does not fully preserve the strengths of either expert. Compared with the safety expert, Driving-OPD slightly reduces NC, DAC, and TTC. Meanwhile, its EP drops from 90.3 to 88.8 compared with the progress expert. Although aggregate PDMS improves, the claim of integrating complementary capabilities requires a clearer safety-progress trade-off analysis.
  4. The scenario selection and teacher routing strategies rely on heuristic choices. Teachers are assigned using overall PDMS win margins rather than explicit safety or progress criteria. Moreover, the fixed selection of 3K scenarios per expert does not reflect their highly imbalanced winning distributions. The sensitivity to these choices remains unclear.
Questions for authors
  1. What advantages does VGS-GRPO offer over simpler bounded advantage-scaling strategies?
  2. How does dual-teacher OPD compare with continued single-policy training under the same computational budget?
  3. Can the authors provide a safety-progress trade-off analysis comparing Driving-OPD with both experts and single-policy alternatives?
  4. How sensitive is the final performance to the teacher-specific scenario selection ratio and routing criteria?
4 comments
AuthorsOpenReview ↗

Response to W1 and Q1

We sincerely thank the reviewer for the thoughtful and constructive review. We appreciate the positive assessment of our analysis of reward-variance effects and the empirical performance of Driving-OPD. We provide point-by-point responses below.

W1:

Thank you for raising this important concern. We agree that GRPO+Cap is an effective and competitive baseline. As shown in Tab. 3, under the same SFT initialization, RL data, and optimization steps, VGS-GRPO achieves lower PDMS than GRPO+Cap (90.6 vs. 91.8), but higher EP (90.3 vs. 89.1), indicating a stronger emphasis on driving progress. Sec. 5.4 further shows that VGS-GRPO yields a student PDMS of 92.8 as a progress teacher, compared with 92.4 for GRPO+Cap.

The two methods differ in how they scale advantages across groups with different reward standard deviations. GRPO+Cap retains inverse-standard-deviation scaling and clips factors above 8. VGS-GRPO uses a gate to retain bounded amplification in low-standard-deviation groups while gradually approaching the unit scaling of Dr.GRPO in high-standard-deviation groups. Here, τ=0.06 and k=40 control the transition location and steepness, respectively, with the same upper bound of 8. On the same 27,943 non-zero-variance rollout groups, VGS-GRPO reduces the proportion reaching this upper bound from 80.8% to 38.8%, assigning differentiated weights to groups that GRPO+Cap clips to the same value.

We provide two additional analyses: a paired progress comparison on a common test subset and teacher-replacement experiments with two additional distillation seeds.

(1) Progress comparison on a common subset.

In NAVSIM v1, EP is set to zero whenever NC or DAC is zero. Thus, mean EP on the full test set reflects both progress and safety-related gating. We further compare the policies on the 11,460 common NAVTEST scenarios where both have non-zero NC and DAC.

MetricVGS-GRPOGRPO+CapΔ
PDMS (NAVTEST)90.691.8−1.2
Mean EP (NAVTEST)90.389.1+1.2
Mean EP (common subset†)94.9992.37+2.62
EP win rate (common subset†)48.0%6.7%+41.3 pp

† Common NAVTEST subset where both policies have non-zero NC and DAC (n=11,460). Δ = VGS-GRPO − GRPO+Cap; pp denotes percentage points.

VGS-GRPO achieves 1.2 points higher mean EP on the full NAVTEST set and 2.62 points higher on the common subset, with an EP win rate of 48.0% versus 6.7%. These results support its progress advantage, while the full-test-set PDMS difference reflects the trade-off between overall performance and progress.

(2) Teacher-configuration comparison across distillation seeds.

We keep Dr.GRPO and the OPD configuration fixed, supplementing the original seed 42 with distillation runs using seeds 567 and 1234. Both configurations use win-margin-based scenario selection and dual-teacher routing.

SeedVGS-GRPO + Dr.GRPOGRPO+Cap + Dr.GRPOGain
4292.7892.35+0.43
56792.6892.32+0.36
123492.8592.32+0.53
Average gain——+0.44

VGS-GRPO as the progress teacher improves student PDMS by 0.43, 0.36, and 0.53 points across the three seeds, respectively, with an average gain of 0.44. This reproduces the performance ordering in the original paper and supports the reproducibility of the teacher-configuration advantage.

Q1:

As discussed in W1, VGS-GRPO achieves higher mean EP on both the full NAVTEST set and the common subset. As a progress teacher, it also consistently improves student PDMS across three distillation seeds, with an average gain of 0.44 points. These results support its progress performance and downstream value as a teacher, despite its lower standalone PDMS.

AuthorsOpenReview ↗

Response to W2 and Q2

W2:

Thank you for this valuable suggestion. We conduct an additional ablation by continuing RL training of the original progress expert (VGS-GRPO) and safety expert (Dr.GRPO) for 2 epochs on exactly the same 6K scenarios selected for OPD. We report the best-performing checkpoint for each setting, evaluated on the official NAVSIM v1 NAVTEST (n=12,147).

ModelTrainingPDMSNCDACTTCComfortEP
VGS-GRPOOriginal expert90.5697.6097.2991.5199.9690.33
VGS-GRPOContinued RL92.4398.7097.4495.4199.7990.37
Dr.GRPOOriginal expert91.7599.1298.4996.6399.9485.98
Dr.GRPOContinued RL92.0899.4198.7197.8999.9485.06
Driving-OPDDual-teacher OPD (Ours)92.7899.0098.2796.5399.9588.82

After 2 epochs of continued RL on the same 6K scenarios, VGS-GRPO and Dr.GRPO improve to 92.43 and 92.08 PDMS, respectively. Driving-OPD still achieves the highest PDMS of 92.78, exceeding these baselines by 0.35 and 0.70 points. Thus, its advantage persists when compared with additional RL training.

Moreover, OPD samples one rollout per scenario, whereas continued RL samples eight. Under the same scenario count and training epochs, OPD therefore achieves higher overall PDMS with fewer rollout samples.

Q2:

As reported in W2, with the same 6K scenarios and 2 training epochs, continued RL achieves 92.43 and 92.08 PDMS for VGS-GRPO and Dr.GRPO, respectively, while Driving-OPD achieves 92.78, exceeding them by 0.35 and 0.70 points. OPD samples one rollout per scenario, compared with eight for continued RL. This comparison therefore matches the scenario count and training epochs and shows higher overall PDMS with fewer rollout samples.

AuthorsOpenReview ↗

Response to W3 and Q3

W3:

Thank you for pointing this out. We agree that Driving-OPD does not simultaneously preserve the maximum value of every metric from both experts. By “integrating complementary capabilities,” we mean improving safety-related performance relative to the progress expert and improving progress relative to the safety expert, while achieving higher overall PDMS. The specific comparisons are provided below; full metrics are reported in the table in W2.

Compared with the original experts, Driving-OPD improves NC, DAC, and TTC over VGS-GRPO by 1.40, 0.98, and 5.02 points, respectively, at the cost of a 1.51-point decrease in EP. Relative to Dr.GRPO, it improves EP by 2.84 points, while NC, DAC, and TTC decrease by 0.12, 0.22, and 0.10 points, respectively. Thus, the student exhibits safety–progress characteristics between those of the two experts, while achieving 92.78 PDMS, exceeding their scores of 90.56 and 91.75.

Compared with continued single-policy RL, Driving-OPD improves NC, DAC, and TTC over continued VGS-GRPO by 0.30, 0.83, and 1.12 points, respectively, while EP decreases by 1.55 points and PDMS increases by 0.35 points. Relative to continued Dr.GRPO, it improves EP by 3.76 points, with decreases in safety-related metrics and a 0.70-point increase in PDMS. These results show that OPD achieves a safety–progress trade-off with higher overall PDMS than the tested single-policy alternatives.

We will therefore state our conclusion more precisely: Driving-OPD integrates complementary capabilities from both experts, while retaining trade-offs among individual metrics. The data-ratio experiments in W4 further illustrate how this trade-off changes with the allocation of expert-specific scenarios.

Q3:

As discussed in W3, Driving-OPD improves NC, DAC, and TTC relative to the progress expert, while EP decreases by 1.51 points. Relative to the safety expert, it improves EP by 2.84 points, with slight decreases in safety-related metrics. Compared with continued single-policy RL for VGS-GRPO and Dr.GRPO, Driving-OPD also exhibits trade-offs among individual metrics, but achieves 0.35 and 0.70 points higher PDMS, respectively. Thus, complementary integration here means achieving a safety–progress trade-off with higher overall PDMS, rather than simultaneously preserving every expert’s best individual metrics.

AuthorsOpenReview ↗

Response to W4 and Q4

W4:

Thank you for raising this concern. We assign teachers based on their PDMS win margins to select the better-performing teacher for each scenario under a unified overall evaluation objective. An expert’s overall specialization in safety or progress does not imply that it performs better in every scenario. Meanwhile, the balanced 3K:3K allocation aims to ensure supervision coverage from both teachers rather than reproduce their natural winning frequencies. We clarify the ratio sensitivity and routing rationale below.

(1) Sensitivity to the scenario selection ratio.

Keeping the total distillation pool fixed at 6K scenarios and all other OPD settings unchanged, we vary the scenario-count ratio between VGS-GRPO and Dr.GRPO from 1:5 to 5:1:

VGS-GRPO : Dr.GRPO (scenario count)PDMSNCDACTTCEPComfortΔ PDMS
1000 : 500092.4699.1398.6396.7187.3899.94−0.32
2000 : 400092.6199.1498.2896.8188.1699.95−0.17
3000 : 300092.7899.0098.2796.5388.8299.95—
4000 : 200092.7398.9598.1395.7689.7599.95−0.05
5000 : 100092.4098.7497.8095.1290.0899.94−0.38

As the proportion of progress-expert scenarios increases, EP rises from 87.38 to 90.08, while safety-related metrics generally decrease, showing that data allocation can adjust the student’s safety–progress preference. PDMS ranges from 92.40 to 92.78 across all five settings and from 92.61 to 92.78 for the intermediate ratios of 2:4, 3:3, and 4:2. The 3:3 setting achieves the highest observed PDMS, while neighboring ratios maintain similar overall performance.

Tab. 6 reports that VGS-GRPO and Dr.GRPO win on 55.1% and 5.1% of the training scenarios, respectively. Allocating supervision directly according to these winning frequencies would assign a relatively small proportion to the safety expert. We therefore use balanced allocation to ensure sufficient coverage of these less frequent but complementary scenarios. The results support this choice and indicate that the method does not depend on a single data ratio within the tested range.

(2) Rationale and component-level analysis of PDMS-based routing.

Our main 6K pool consists of 3K scenarios where VGS-GRPO strictly wins and 3K scenarios where Dr.GRPO strictly wins, ranked and selected by PDMS win margin. To examine whether overall PDMS identifies teachers with different strengths, we analyze component-level performance in both selected pools. The table below reports safety-related failures of the other expert in each pool:

Selected poolFailure indicator of the other expertCountProportion
Dr.GRPO-winningVGS-GRPO: NC=073524.50%
VGS-GRPO: DAC=044314.77%
VGS-GRPO: NC=0 or DAC=01,17439.13%
VGS-GRPO: TTC=02,58486.13%
VGS-GRPO-winningDr.GRPO: NC=0341.13%
Dr.GRPO: DAC=0541.80%
Dr.GRPO: NC=0 or DAC=0882.93%
Dr.GRPO: TTC=02909.67%

In the 3K Dr.GRPO-winning scenarios, VGS-GRPO has zero NC or DAC in 39.13% of cases and zero TTC in 86.13%, indicating that this pool is enriched with scenarios requiring safety-related improvements. On the 3K VGS-GRPO-winning scenarios, the progress comparison is:

StatisticResult
Mean EP (VGS-GRPO)0.957
Mean EP (Dr.GRPO)0.661
Mean EP difference+0.30
Median EP difference+0.28
Scenarios where VGS-GRPO achieves higher EP94%
Scenarios where VGS-GRPO’s EP is higher by more than 0.291%

In the VGS-GRPO-winning pool, its mean EP is 0.957, compared with 0.661 for Dr.GRPO, and it achieves higher EP in 94% of scenarios, demonstrating a clear progress advantage. Moreover, among the 1,176 NAVTRAIN scenarios where VGS-GRPO has zero NC or DAC and Dr.GRPO strictly wins, 1,174 are included in the safety-expert pool, yielding 99.83% coverage.

These results show that PDMS win margins capture safety- and progress-related complementarity in the selected scenarios without additional component-specific selection thresholds. Tab. 4 of the original paper also shows that win-margin selection improves student PDMS from 92.5 to 92.8 compared with random selection, providing downstream evidence for this selection strategy.

Q4:

As reported in W4, varying the scenario ratio from 1:5 to 5:1 yields PDMS scores of 92.40–92.78, with the three intermediate ratios achieving 92.61–92.78. Meanwhile, EP and safety-related metrics exhibit different trends as the ratio changes. Component-level statistics show that the two pools are enriched with scenarios requiring safety improvements and scenarios exhibiting progress advantages, respectively, supporting the rationale for the current design.

Comments on the paper

1 comment
Area_Chair_Ed1KOpenReview ↗

Reminder: Author Response and Reviewer Discussion

Dear Reviewers,

We are halfway through the discussion period. Please review the authors’ responses, address any remaining concerns, and engage in the discussion where appropriate. If your assessment remains unchanged, please briefly acknowledge that you have considered the authors’ responses.

Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.