- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PapoDrive studies the loss of driving efficiency after safety oriented reinforcement learning for VLA policies. It models optimization as probability redistribution over driving patterns and relates efficiency drift to covariance between pattern efficiency and safety induced reweighting. The method clusters planning tokens and uses estimated pattern efficiency to modulate KL regularization, combined with a joint safety and efficiency reward. Experiments with ORION and ReCogDrive on Bench2Drive and NAVSIM report improvements in the balance between safety and efficiency in selected settings.
Critical strengths
- The pattern based formulation offers a useful perspective on the conflict between safety and efficiency. Using pattern efficiency to modulate KL regularization provides methodological novelty beyond directly adjusting reward weights. The implementation reuses planning tokens and requires no additional neural network.
- The NAVSIM results on high risk scenarios provide meaningful controlled evidence. Compared with GRPO, the full method improves NC from 90.4 to 91.8 and reduces the efficiency loss relative to the base model from 10.2% to 2.2%. Comparisons with joint rewards and global adaptive KL support the additional value of modulation at the pattern level.
Critical weaknesses
- The mechanism is described more strongly than the analysis supports. Changing KL coefficients by pattern changes the regularized objective and optimal policy, not merely the optimization step size. Proposition 4 provides a conditional result about probability increases, rather than a general safety guarantee, and the final system also uses an efficiency reward. Moreover, clustering stability does not establish consistent driving semantics. A comparison with KL modulation based directly on each rollout’s efficiency is missing.
- Experimental attribution and scope remain limited. Table 2 supports adding PSM to joint rewards, but omits PSM alone and a sweep of reward weights. Corresponding component ablations are missing on Bench2Drive, and checkpoint selection for NAVSIM needs clarification. Benefits are also uneven. The mean traffic sign score falls from 69.15 to 53.72 on Bench2Drive, while full NAVSIM PDMS remains 90.8 and EP falls from 86.0 to 85.5. These results limit claims of broadly improved driving performance.
Questions for authors
- Please state the exact combined training objective and efficiency reward weight. Under what conditions does safety alone determine which patterns gain probability, and does Proposition 4 apply to the combined objective?
- Can you report PSM without joint rewards, a sweep of reward weights, and the corresponding Bench2Drive ablations? For NAVSIM, how many checkpoints are considered, which metric selects them, and is selection performed on an independent validation set?
- How does the method compare with directly modulating KL using each rollout’s efficiency without clustering? Beyond geometric stability and qualitative examples, what evidence shows that the clusters represent consistent driving strategies?
6 comments
Official Response by Authors to Reviewer BADU (2/6)
Response 1 - Precise expression of the combined training objective and scope of Proposition 4 (W1 & Q1).
(1) Precise expression of the combined training objective.
Q1: Please state the exact combined training objective and efficiency reward weight.
The final objective of PapoDrive combines the joint reward with pattern-level modulation:
where and are safety reward and efficiency reward, respectively. We use efficiency reward weight = 0.1 and KL coefficient = 0.1.
(2) Clarification of the description "Pattern-level Step-size Modulation".
W1: The mechanism is described more strongly than the analysis supports. Changing KL coefficients by pattern changes the regularized objective and optimal policy, not merely the optimization step size.
The pattern-level step-size modulation indeed changes the regularized objective and optimal policy, as we reported in the section 4.2 of paper. The reason for naming this mechanism "step-size modulation" is to describe the scaling of the variation magnitude of each pattern in the policy probability space, during the reweighting process of the optimal RL policy, not referring to the step length of the optimizer's parameter update during training. As we analyzed on lines 281-284 of paper: "the modulated weighting factor will encourage the probability increase of high-efficiency patterns, while suppress the probability increase of low-efficiency patterns."
(3) Extending Proposition 4 to the combined objective.
Q1: Does Proposition 4 apply to the combined objective?
Proposition 4 applies completely to the joint objective. Replacing "" by "" gives:
Corollary. Under the combined objective,
And is up-weighted if and only if
Consequently, if
it cannot happen that is up-weighted while is not. The right-hand side is at most . For = 0, this Corollary reduces to Proposition 4. With our hyperparameters = 0.1,the new item are limited to very small values, which does not affect the conclusion of Proposition 4.
Official Response by Authors to Reviewer BADU (3/6)
Response 2 - Independent evidence that the extracted patterns correspond to distinct driving strategies (W1 & Q3).
W1 & Q3: Beyond geometric stability and qualitative examples, what evidence shows that the clusters represent consistent driving strategies?
Thank you for this valuable suggestion. We have empirically checked the relationship between the pattern clusters and the driving behaviors, and the results demonstrate that the pattern clusters correspond to objectively different driving strategies, rather than arbitrary partitions of the representation space.
Detailed responses:
We designed an experiment to demonstrate that stable clusters correspond to objectively different driving strategies, carried out in the following three steps:
Step 1 (Pattern clustering). On the one hand, we sampled G=16 samples per scenario across 126 scenarios, totaling 2016 samples, which cover a rich variety of long-tail scenarios in the Navsim dataset, and maintain consistency with the actual sampling temperature during model training. Then we performed pattern clustering according to the scheme provided in the paper.
Step 2 (Driving behavior classifying). On the other hand, we designed completely independent labels, classifying the samples into different categories based on multiple objective indicators of driving behavior.
Step 3 (Correlation test). We measured the correlation between pattern clusters and driving behavior categories through permutation tests, to determine whether the pattern clusters match the independent labels of driving behavior. Here, measures the difference degree in driving behavior indicators between pattern clusters, the permutation test then checks whether this difference is statistically significant.
The results are as follows:
- The correlation indicator :
- 0 = pattern clusters are unrelated to driving behavior.
- 1 = pattern clusters are completely related to driving behavior.
- As a baseline, = 0.13 when labels are randomly shuffled.
| Driving Behavior Label | Mean | Significant Scenes (%) |
|---|---|---|
| Average speed | 0.80 | 81% |
| Speed compliance | 0.80 | 81% |
| Trajectory smoothness | 0.65 | 79% |
| Min. obstacle distance | 0.71 | 70% |
| Ego progress | 0.80 | 79% |
| L2 error between GT traj. | 0.69 | 82% |
For all six indicators, the mean ranges from 0.65 to 0.80, far above the shuffled baseline of 0.13, i.e., the cluster membership explains most of the within-scenario variation of each driving-behavior indicator. The association is statistically significant in 70%-82% of the scenarios.
Together with the qualitative examples in Figure 4, in which the CoTs of different clusters reason about the same scenario from different perspectives, these results indicate that the pattern clusters extracted from planning tokens correspond to objectively different driving strategies. Therefore, pattern clusters have actual semantic significance, rather than arbitrary partitions of the representation space.
Official Response by Authors to Reviewer BADU (1/6)
Dear Reviewer BADU,
We thank the reviewer for the constructive suggestions, and for recognizing the methodological novelty and performance of our method. We have added the requested ablations and analyses and included them in the revised paper ( Section 4.2/5.2/5.3 & Table 2 & Appendix D3/D4/D6/G, with blue highlight ). In summary:
The detailed responses are provided as follows.