- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose Driving-OPD, a post-training framework for autonomous driving. Built on observation on variance-dependent advantages of GRPO, authors proposed VGS-GRPO to train progress expert and use Dr. GRPO to train safety expert. The progress expert is further trained with OPD with supervision from two experts. The final VLA achieves SOTA performance on NAVSIM v1 and v2.
Critical strengths
- The paper is built on clear motivations from the observations of advantage scaling and reward group variance in GRPO. This motivates the proposed VGS-GRPO to learn a progress-oriented expert. The subsequent integration of two experts to single student policy is also natural and well motivated.
- The paper provides clear empirical results and ablations for OPD stage. The experiments clearly show the complementary of two experts and provide empirical support for the major design of OPD stage choice.
Critical weaknesses
- Though authors conduct analysis on reward variance and advantage magnitude, it still somehow remain unclear why VGS-GRPO achieved specially high EP and Dr. GRPO achieved high NC. A clear and deeper analysis of the mechanism of the relationship of advantage and resulting behavior could better motivate the design.
- The VGS-GRPO mainly adopt chosen hyper parameters like τ, k, s_max. Some are selected from the training distribution statistics, and some are chosen empirically. More ablations on these parameters could better support the generalization of method across different data distribution.
Questions for authors
- Have author consider utilizing OPSD in the training procedure? It could be interesting to see whether the student policy could serve as teacher to teach itself to replace the multi-teacher OPD?
- The two experts have different specializations on EP and NC. What if the two complementary capabilities of progress expert and safety expert conflict during OPD? Will it leads to degradation on other skills or performance?
- The title may overemphasize OPD since OPD is only a part of the overall pipeline and RL stage is also an important part of the method.
4 comments
Response to Q1, Q2 and Q3
Q1:
We appreciate the reviewer’s suggestion. We agree that exploring OPSD in autonomous driving trajectory generation is an interesting direction. OPSD relies on privileged information to construct stronger and weaker variants of the same policy, which serve as the teacher and student, respectively, for self-distillation.
However, our current experimental setting does not involve such privileged information, making it difficult to construct the corresponding teacher–student pair required by OPSD.
In contrast, this work focuses on introducing OPD as a training paradigm into autonomous driving trajectory planning, providing an alternative approach to addressing the challenge of simultaneously improving driving progress (EP) and safety (NC), which are difficult to optimize together through conventional RL. We hope this study provides useful insights for further exploring OPD in autonomous driving. Meanwhile, our investigations into scenario selection, teacher routing, and training-data analysis may also offer valuable references for the future application of OPSD in this domain.
Q2:
Since the two experts exhibit substantially different driving behaviors, their supervision signals inherently involve certain conflicts. To mitigate this issue, we route each training sample to the corresponding expert based on its PDMS advantage, rather than directly mixing the probability distributions of the two teachers. Thus, each sample receives supervision from a single expert, providing a clear learning target during distillation.
Furthermore, the trade-off between the two experts can be adjusted by changing their respective proportions in the distillation data. We evaluate different data ratios while keeping the total distillation pool fixed at 6K scenarios.
| Progress Expert : Safety Expert | PDMS | NC | DAC | TTC | EP | Comfort | Δ vs. 3:3 |
|---|---|---|---|---|---|---|---|
| 1:5 | 92.46 | 99.13 | 98.63 | 96.71 | 87.38 | 99.94 | -0.32 |
| 2:4 | 92.61 | 99.14 | 98.28 | 96.81 | 88.16 | 99.95 | -0.17 |
| 3:3 | 92.78 | 99.00 | 98.27 | 96.53 | 88.82 | 99.95 | — |
| 4:2 | 92.73 | 98.95 | 98.13 | 95.76 | 89.75 | 99.95 | -0.05 |
| 5:1 | 92.40 | 98.74 | 97.80 | 95.12 | 90.08 | 99.94 | -0.38 |
Increasing the progress-expert data proportion improves EP, while increasing the safety-expert proportion generally favors safety-related metrics.
These results show that OPD seeks a relative balance between safety and progress when learning from the two teachers, and this balance can be adjusted through the data ratio. Regarding the potential degradation of other capabilities, although individual metrics vary across different ratios, we observe no substantial degradation in the evaluated metrics. Notably, all tested ratios achieve higher overall PDMS than both teacher experts.
These findings suggest that expert-specific data allocation provides a practical way to balance competing driving objectives during distillation, offering useful insights for future multi-expert distillation research in autonomous driving.
Q3:
We appreciate the reviewer’s observation. We would like to clarify that reward variance-aware RL and OPD play equally important and complementary roles in our framework. The effectiveness of OPD relies on first obtaining specialized experts through different RL treatments; without this expert specialization, it would be difficult for the subsequent OPD stage to achieve substantial gains from knowledge integration. Accordingly, our paper title places reward variance-aware RL and OPD on equal footing, reflecting their complementary contributions to the overall framework.
We use Driving-OPD as the abbreviation for our proposed framework to highlight our exploration of OPD as a complementary training paradigm beyond mainstream RL-based optimization in autonomous driving. Our intention is to explore complementary training strategies for autonomous driving and provide useful insights for future research, rather than to imply that OPD constitutes the entire contribution of the framework.
Thanks for the reply. I find the new results to be interesting and I will keep my current positive score.
Thank you for your follow-up and for maintaining your positive assessment. We appreciate your constructive feedback, which has helped us strengthen the analysis and experiments. We will incorporate the additional results and clarifications into the revised manuscript.
Response to W1 and W2
We sincerely thank the reviewer for the positive assessment of our work and the thoughtful, constructive suggestions. We greatly appreciate the reviewer’s recognition and believe that these valuable comments have helped us further improve the quality and clarity of our paper.
W1:
We appreciate the reviewer’s suggestion to further clarify the relationship between reward variance, advantage magnitude, and the resulting driving behaviors, particularly the distinct performance characteristics of VGS-GRPO and Dr.GRPO in terms of EP and NC.
Low-variance groups are dominated by fine-grained EP differences.
According to the official NAVSIM PDMS scoring formula, EP and TTC have the same weight (5/12), but differ in granularity: EP is continuous, whereas TTC is binary ({0, 1}). NC and DAC introduce additional discontinuities through multiplicative zeroing, while Comfort contributes little variation since nearly all trajectories satisfy this criterion.
To examine their effects on rollout variance, we analyze VGS-GRPO on-policy rollout groups (K=8).
Groups with mixed TTC states exhibit substantially higher reward variance, whereas groups with uniform TTC states (all 0 or all 1) exhibit similarly low variance. This suggests that although TTC and EP have identical scoring weights, advantage scaling in low-variance groups primarily acts on the fine-grained differences provided by continuous EP. Moreover, NC and TTC both reflect collision-related safety, suggesting a degree of synergy between these two metrics.
These findings also explain the distinct optimization tendencies of the two methods: VGS-GRPO preferentially amplifies low-variance groups dominated by EP differences, whereas Dr.GRPO retains the relative contribution of larger advantage signals from high-variance groups, which are mainly associated with safety-related metrics. Consequently, VGS-GRPO exhibits stronger EP performance, whereas Dr.GRPO achieves higher scores on safety-related metrics, including NC, DAC, and TTC.
W2:
Ablation Study on VGS-GRPO Hyperparameters
We thank the reviewer for this valuable suggestion. In VGS-GRPO, the gating coefficient is defined as:
βq=1+exp(k(σq−τ))1.
Here, τ determines the transition threshold of the variance gate, while k controls the sharpness of the transition. Increasing τ extends the region where the gate retains stronger variance-dependent scaling to groups with higher reward variance, allowing more groups to receive additional scaling for fine-grained reward differences. In contrast, decreasing τ restricts this effect to lower-variance groups. Increasing k makes the transition sharper, strengthening the preferential scaling of low-variance groups, whereas decreasing k produces a smoother transition and weakens this distinction. Since low-variance groups in our training distribution mainly reflect fine-grained differences among otherwise successful trajectories, retaining their scaling sensitivity is important for preserving progress-related optimization signals.
We set τ=0.06 based on the training distribution, where approximately 75.8% of rollout groups have a within-group standard deviation below this threshold. We choose k=40 empirically and set smax=8 to prevent excessive advantage amplification.
Following the reviewer's suggestion, we conduct additional ablation experiments on τ and k, while keeping smax=8. We also include the main experimental configuration for reference.
The results illustrate how the gate parameters regulate the relative emphasis on fine-grained reward differences. Increasing τ expands the range of groups receiving variance-dependent scaling, whereas increasing k sharpens the distinction between low- and high-variance groups. Conversely, decreasing τ narrows this range, while decreasing k produces a smoother transition. The observed changes in EP and safety-related metrics reflect the resulting shift in optimization preference. These ablations provide empirical support for the intended roles of τ and k in controlling variance-dependent scaling.