acceptodds
Under review as a conference paper at ICLR 2027

PapoDrive: Pattern-Aware Policy Optimization for Safe and Efficient Driving Vision-Language-Action Model

Abstract

Reinforcement learning (RL)-based post-training has become a mainstream paradigm for improving the safety of vision-language-action (VLA) models in autonomous driving, yet the safety gains often come at the cost of overly conservative behaviors and significant efficiency degradation. To address this safety-efficiency trade-off, we propose PapoDrive, a Pattern-Aware Policy Optimization framework for the RL-based post-training. Specifically, we model the VLA planning as a two-stage decision, where the large language model (LLM) first selects a macro reasoning pattern, and the planner then decodes a trajectory conditioned on it. Under this view, the conventional KL-regularized RL essentially acts as a reweighting mechanism over the reference reasoning patterns, with the resulting efficiency drift governed by the covariance between the pattern-level efficiency and safety rewards, which is significantly negative in many driving scenarios and thus becomes the source of this trade-off. Guided with this insight, we develop a pattern-level step-size modulation (PSM) mechanism, allowing efficient patterns to move further from the reference policy while keeping inefficient ones close to it. We further design a lightweight pattern extraction scheme, such that PSM can be integrated into GRPO with a negligible overhead. PapoDrive improves the Success Rate by 5.72 and Efficiency by 6.15 over the ORION baseline on Bench2Drive benchmark, and achieves the best safety metrics while limiting efficiency drop to 2.2%, compared with 10.2% for GRPO on high-risk Navsim scenarios.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Vision-Language-Action Model, Reinforcement Learning, Post-Training, Reasoning Pattern

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BADUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PapoDrive studies the loss of driving efficiency after safety oriented reinforcement learning for VLA policies. It models optimization as probability redistribution over driving patterns and relates efficiency drift to covariance between pattern efficiency and safety induced reweighting. The method clusters planning tokens and uses estimated pattern efficiency to modulate KL regularization, combined with a joint safety and efficiency reward. Experiments with ORION and ReCogDrive on Bench2Drive and NAVSIM report improvements in the balance between safety and efficiency in selected settings.

Critical strengths
  1. The pattern based formulation offers a useful perspective on the conflict between safety and efficiency. Using pattern efficiency to modulate KL regularization provides methodological novelty beyond directly adjusting reward weights. The implementation reuses planning tokens and requires no additional neural network.
  2. The NAVSIM results on high risk scenarios provide meaningful controlled evidence. Compared with GRPO, the full method improves NC from 90.4 to 91.8 and reduces the efficiency loss relative to the base model from 10.2% to 2.2%. Comparisons with joint rewards and global adaptive KL support the additional value of modulation at the pattern level.
Critical weaknesses
  1. The mechanism is described more strongly than the analysis supports. Changing KL coefficients by pattern changes the regularized objective and optimal policy, not merely the optimization step size. Proposition 4 provides a conditional result about probability increases, rather than a general safety guarantee, and the final system also uses an efficiency reward. Moreover, clustering stability does not establish consistent driving semantics. A comparison with KL modulation based directly on each rollout’s efficiency is missing.
  2. Experimental attribution and scope remain limited. Table 2 supports adding PSM to joint rewards, but omits PSM alone and a sweep of reward weights. Corresponding component ablations are missing on Bench2Drive, and checkpoint selection for NAVSIM needs clarification. Benefits are also uneven. The mean traffic sign score falls from 69.15 to 53.72 on Bench2Drive, while full NAVSIM PDMS remains 90.8 and EP falls from 86.0 to 85.5. These results limit claims of broadly improved driving performance.
Questions for authors
  1. Please state the exact combined training objective and efficiency reward weight. Under what conditions does safety alone determine which patterns gain probability, and does Proposition 4 apply to the combined objective?
  2. Can you report PSM without joint rewards, a sweep of reward weights, and the corresponding Bench2Drive ablations? For NAVSIM, how many checkpoints are considered, which metric selects them, and is selection performed on an independent validation set?
  3. How does the method compare with directly modulating KL using each rollout’s efficiency without clustering? Beyond geometric stability and qualitative examples, what evidence shows that the clusters represent consistent driving strategies?
6 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer BADU (1/6)

Dear Reviewer BADU,

We thank the reviewer for the constructive suggestions, and for recognizing the methodological novelty and performance of our method. We have added the requested ablations and analyses and included them in the revised paper ( Section 4.2/5.2/5.3 & Table 2 & Appendix D3/D4/D6/G, with blue highlight ). In summary:

  • Combined objective and Proposition 4 (W1/Q1). The objective is rsafety+λ refficiencyr_{\rm safety} + \lambda\, r_{\rm efficiency} with pattern-level KL coefficient β mx(p)\beta\, m_x(p), where λ\lambda = 0.1 and β\beta = 0.1. Proposition 4 extends to this objective: the safety order of two patterns is preserved whenever their safety gap exceeds λ+β(α+γ1+γ)\lambda + \beta(\alpha + \frac{\gamma}{1+\gamma}), i.e., 0.283 with our hyperparameters (0.183 without the efficiency reward). "Step-size" refers to how far each pattern moves from the reference policy in the optimal policy, not to the optimizer's step size.
  • PSM alone and reward-weight sweep (W2/Q2). PSM alone already mitigates the trade-off: with an NC gain similar to GRPO (+10.7 vs. +10.2), it reduces the efficiency loss from -14 to -6. Joint rewards alone cannot achieve both at any λ∈\lambda \in {0.1, 0.5, 1.0}: their NC gain is at most +8.8, while PapoDrive (λ\lambda = 0.1) reaches NC +11.6 with Eff. -3.
  • Bench2Drive ablation (W2/Q2). On Town12, GRPO lowers the driving speed from 3.70 to 3.57 m/s. JRO and PSM each prevent this (3.89 and 3.97 m/s), and their combination is the fastest (3.98 m/s).
  • Rollout-level modulation (W1/Q3). Under the same reward and training budget, pattern-level modulation beats rollout-level modulation on both axes (Eff. -3 vs. -7, NC +11.6 vs. +9.8).
  • Clusters match driving behaviors (W1/Q3). Against six objective driving-behavior indicators, mean η2\eta^2 = 0.65–0.80, significant in 70%–82% of the scenarios.
  • NAVSIM checkpoint selection (W2/Q2) is detailed below.

The detailed responses are provided as follows.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer BADU (2/6)

Response 1 - Precise expression of the combined training objective and scope of Proposition 4 (W1 & Q1).

(1) Precise expression of the combined training objective.

Q1: Please state the exact combined training objective and efficiency reward weight.

The final objective of PapoDrive combines the joint reward with pattern-level modulation:

J(θ)=Ex∼D,p∼πθ(⋅∣x),τ∼πplanner(⋅∣x,p)[rsafety(x,τ)+λ⋅refficiency(x,τ)−βmx(p)log⁡πθ(p∣x)πref(p∣x)],\mathcal{J}(\theta)=\mathbb{E}_{\boldsymbol{x} \sim \mathcal{D}, p \sim \pi_\theta(\cdot|\boldsymbol{x}), \boldsymbol{\tau} \sim \pi_{\text{planner}}(\cdot|\boldsymbol{x},p)}\Big[r_{\rm safety}(\boldsymbol{x}, \boldsymbol{\tau})+ \lambda \cdot r_{\text{efficiency}}(\boldsymbol{x}, \boldsymbol{\tau})-\beta m_x(p)\log\frac{\pi_\theta(p|\boldsymbol{x})}{\pi_{\rm ref}(p|\boldsymbol{x})}\Big],

where rsafety(x,τ)r_{\rm safety}(\boldsymbol{x}, \boldsymbol{\tau}) and refficiency(x,τ)r_{\rm efficiency}(\boldsymbol{x}, \boldsymbol{\tau}) are safety reward and efficiency reward, respectively. We use efficiency reward weight λ\lambda = 0.1 and KL coefficient β\beta = 0.1.

(2) Clarification of the description "Pattern-level Step-size Modulation".

W1: The mechanism is described more strongly than the analysis supports. Changing KL coefficients by pattern changes the regularized objective and optimal policy, not merely the optimization step size.

The pattern-level step-size modulation indeed changes the regularized objective and optimal policy, as we reported in the section 4.2 of paper. The reason for naming this mechanism "step-size modulation" is to describe the scaling of the variation magnitude of each pattern in the policy probability space, during the reweighting process of the optimal RL policy, not referring to the step length of the optimizer's parameter update during training. As we analyzed on lines 281-284 of paper: "the modulated weighting factor will encourage the probability increase of high-efficiency patterns, while suppress the probability increase of low-efficiency patterns."

(3) Extending Proposition 4 to the combined objective.

Q1: Does Proposition 4 apply to the combined objective?

Proposition 4 applies completely to the joint objective. Replacing "sx(p)s_x(p)" by "sx(p)+λ⋅gx(p)s_x(p)+\lambda \cdot g_x(p)" gives:

Corollary. Under the combined objective,

πθ-PSM∗(p∣x)=πref(p∣x)exp⁡(sx(p)+λ⋅gx(p)−μx∗βmx(p)−1).\pi^{\ast}_{\theta\text{-PSM}}(p\mid\boldsymbol{x})=\pi_{\mathrm{ref}}(p\mid\boldsymbol{x}) \exp\Big(\frac{s_x(p)+\lambda \cdot g_x(p)-\mu^{\ast}_x}{\beta m_x(p)}-1\Big).

And pp is up-weighted if and only if

s~x(p)=sx(p)+λ⋅gx(p)−β⋅mx(p)>μx∗.\tilde s_x(p)=s_x(p)+\lambda \cdot g_x(p)-\beta \cdot m_x(p)>\mu^{\ast}_x.

Consequently, if

sx(p)−sx(q)>λ⋅(gx(q)−gx(p))+β⋅(mx(p)−mx(q)),s_x(p)-s_x(q)>\lambda \cdot(g_x(q)-g_x(p))+\beta \cdot (m_x(p)-m_x(q)),

it cannot happen that qq is up-weighted while pp is not. The right-hand side is at most λ+β(α+γ1+γ)\lambda+\beta(\alpha+\tfrac{\gamma}{1+\gamma}). For λ\lambda = 0, this Corollary reduces to Proposition 4. With our hyperparameters λ\lambda = 0.1,the new item λ⋅(gx(q)−gx(p))\lambda \cdot(g_x(q)-g_x(p)) are limited to very small values, which does not affect the conclusion of Proposition 4.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer BADU (3/6)

Response 2 - Independent evidence that the extracted patterns correspond to distinct driving strategies (W1 & Q3).

W1 & Q3: Beyond geometric stability and qualitative examples, what evidence shows that the clusters represent consistent driving strategies?

Thank you for this valuable suggestion. We have empirically checked the relationship between the pattern clusters and the driving behaviors, and the results demonstrate that the pattern clusters correspond to objectively different driving strategies, rather than arbitrary partitions of the representation space.

Detailed responses:

We designed an experiment to demonstrate that stable clusters correspond to objectively different driving strategies, carried out in the following three steps:

Step 1 (Pattern clustering). On the one hand, we sampled G=16 samples per scenario across 126 scenarios, totaling 2016 samples, which cover a rich variety of long-tail scenarios in the Navsim dataset, and maintain consistency with the actual sampling temperature during model training. Then we performed pattern clustering according to the scheme provided in the paper.

Step 2 (Driving behavior classifying). On the other hand, we designed completely independent labels, classifying the samples into different categories based on multiple objective indicators of driving behavior.

Step 3 (Correlation test). We measured the correlation between pattern clusters and driving behavior categories through permutation tests, to determine whether the pattern clusters match the independent labels of driving behavior. Here, η2\eta^2 measures the difference degree in driving behavior indicators between pattern clusters, the permutation test then checks whether this difference is statistically significant.

The results are as follows:

  • The correlation indicator η2∈[0,1]\eta^2 \in [0,1]:
  • 0 = pattern clusters are unrelated to driving behavior.
  • 1 = pattern clusters are completely related to driving behavior.
  • As a baseline, η2\eta^2 = 0.13 when labels are randomly shuffled.
Driving Behavior Labelη2\eta^2 MeanSignificant Scenes (%)
Average speed0.8081%
Speed compliance0.8081%
Trajectory smoothness0.6579%
Min. obstacle distance0.7170%
Ego progress0.8079%
L2 error between GT traj.0.6982%

For all six indicators, the mean η2\eta^2 ranges from 0.65 to 0.80, far above the shuffled baseline of 0.13, i.e., the cluster membership explains most of the within-scenario variation of each driving-behavior indicator. The association is statistically significant in 70%-82% of the scenarios.

Together with the qualitative examples in Figure 4, in which the CoTs of different clusters reason about the same scenario from different perspectives, these results indicate that the pattern clusters extracted from planning tokens correspond to objectively different driving strategies. Therefore, pattern clusters have actual semantic significance, rather than arbitrary partitions of the representation space.

Reviewer_y1PjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the well-known safety-efficiency trade-off that emerges during RL post-training for vision-language-action driving models. The authors theoretically analyse how efficiency drift can arise when safety and efficiency are negatively correlated across reasoning patterns. They propose PapoDrive, a pattern-aware policy optimization framework. Its core Pattern-level Step-size Modulation dynamically adjusts the KL constraint strength according to pattern-level efficiency estimates, together with a lightweight pattern-extraction module that groups reasoning patterns using planning-token representations. Evaluations are conducted across multiple VLA backbones on Bench2Drive and high-risk Navsim scenarios. The results show improvements in both safety and efficiency metrics, while ablation studies investigate the main components of the proposed framework.

Critical strengths
  1. The text part of the paper is well organized. Its motivation, theoretical analysis, proposed method, and experiments are presented in a logical sequence.
  2. The paper provides a clear mathematical formulation of the problem.
  3. The paper focuses on an important problem in autonomous driving. The safety–efficiency trade-off is a meaningful and worthwhile research direction.
Critical weaknesses
  1. The method oversimplifies the safety–efficiency trade-off in autonomous driving. Driving behavior cannot generally be reduced to a distinction between “efficient” and “inefficient” patterns; the appropriateness of a behavior depends on the scenario and involves multiple objectives and constraints. Moreover, the reported efficiency gains may reflect optimization toward the specific benchmark efficiency metric rather than a broader improvement in driving efficiency.
  2. Figures 1–4 are difficult to interpret and provide limited explanatory value. The text and annotations are too small to read clearly, and several figures are presented with little explanation of how they support the paper’s claims. As a result, it is difficult to understand the intended takeaways, particularly the evidence for the proposed reasoning-pattern formulation.
  3. The validity and stability of the extracted reasoning patterns are not established. The method applies PCA and K-Means to planning-token hidden representations and interprets the resulting clusters as semantic reasoning patterns. However, each scenario has only a small number of rollouts, and the paper does not demonstrate that planning-token clusters correspond to stable, semantically meaningful patterns across samples, training runs, or model states.
  4. The efficiency feedback used to score each pattern is highly heuristic and rule-based. It is derived from the ratio of the ego vehicle’s speed to the average speed of nearby vehicles, and then converted into a binary reward based on the mean efficiency of the sampled rollouts. This proxy may not reliably capture actual driving efficiency: it can favor matching or exceeding surrounding traffic speed without accounting for factors such as road conditions, traffic rules, safety, comfort, or route progress.
Questions for authors

Refer to the weakness part above.

7 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer y1Pj (1/6)

Dear Reviewer y1Pj,

We thank the reviewer for the detailed comments, and for recognizing the paper organization and theoretical value. We have added new measurements and analyses and included them in the revised paper ( Section 5.3 & Appendix D4/D5 & Figure1-4, with blue highlight ) . In summary:

  • Efficiency is scenario-dependent, and the gains are not metric-specific (W1). Efficiency is measured within each scenario and relative to the surrounding traffic, so the same behavior can be efficient in one scenario but not in another. On two efficiency metrics that are never optimized during training, our method is also the best: absolute ego speed 3.98 m/s vs. 3.57 m/s for GRPO (Bench2Drive), and EP 57.3 vs. 52.8 for GRPO (NAVSIM high-risk).
  • Patterns are semantically meaningful and stable (W3). The clusters strongly match six objective driving-behavior indicators (mean η2\eta^2 = 0.65–0.80, significant in 70%–82% of the scenarios). They are recovered with a purity of 0.94 from only 8 rollouts (Appendix D.1), and their efficiency ranking is preserved at every training checkpoint under both GRPO and PSM.
  • Efficiency reward (W4). We use the standard Bench2Drive efficiency metric, i.e., ego speed relative to surrounding vehicles. Because the reference moves with the surrounding traffic, it adapts to road and traffic conditions. Since PSM only uses efficiency to scale the KL strength of each pattern, a more comprehensive efficiency reward can be plugged in directly.
  • Figures (W2). We have enlarged all text in Figures 1–4, and added explanation of how they support the paper’s claims in caption. The takeaway of each figure will be given below.

The detailed responses are provided as follows.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer y1Pj (2/6)

Response 1 - Clarification of the efficiency metric (W1).

(1) The appropriateness of efficiency depends on the scenario.

W1: The method oversimplifies the safety–efficiency trade-off in autonomous driving. Driving behavior cannot generally be reduced to a distinction between “efficient” and “inefficient” patterns; the appropriateness of a behavior depends on the scenario and involves multiple objectives and constraints.

Efficiency in our method is measured within each scenario and relative to the surrounding traffic, so its appropriateness already depends on the scenario.

In terms of reward design, we adopted the conventional efficiency reward used in mainstream benchmarks and training (such as CARLA [R3], Bench2Drive [R4], and SimLingo [R5]), utilizing the relative speed between the ego vehicle and surrounding vehicles, which also takes into account the constraints and traffic conditions in each specific scenario.

In our method, efficiency is not a simple fixed label attached to patterns. Our efficiency expectation gx(p)g_x(p) with the subscript xx indicates that the efficiency is measured within each scenario. The same driving behavior may be low-efficient in highway scenarios but could be the appropriate choice in urban scenarios. This exactly corresponds to what you mentioned, "the appropriateness of behavior depends on the scenario".

[R3] Alexey Dosovitskiy, et al. CARLA: An open urban driving simulator. CoRL 2017.

[R4] Xiaosong Jia, et al. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. NeurIPS 2024.

[R5] Katrin Renz, et al. SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment. CVPR 2025.

(2) Is the reported gain reflects optimization toward the specific benchmark efficiency metric?

W1: The reported efficiency gains may reflect optimization toward the specific benchmark efficiency metric rather than a broader improvement in driving efficiency.

In addition to the efficiency metric used in our training, there are two independent metrics that can also reflect driving efficiency, which are:

  • Absolute ego speed: the absolute value of the ego vehicle's speed, calculated with displacement and time.
  • EP (Ego Progress): measures the progress towards the navigation destination within a unit of time.

In response to your concerns about "the reported efficiency gains may reflect optimization toward the specific benchmark efficiency metric", we can report these two independent efficiency metrics that were not directly optimized during the training, to demonstrate that our post-training indeed brought a broad improvement in driving efficiency.

Absolute ego speed:

MethodAbsolute ego speed (m/s)
Baseline3.70
GRPO3.57
GRPO+JRO3.89
GRPO+JRO+PSM3.98

Compare the "absolute ego speed" metric under the premise of achieving the same level of safety: It can be seen that basic GRPO post-training makes driving efficiency conservative, while JRO can prevent efficiency degradation. The combined use of JRO and PSM achieves the optimal effect, which is consistent with the analysis in the paper.

EP:

MethodEP
Baseline50.7
GRPO52.8
GRPO+JRO49.6
GRPO+JRO+PSM57.3

These above experiments demonstrate that our method can also achieve the optimal EP metric.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer y1Pj (3/6)

Response 2 - Small font and explanatory value of Figures 1–4 (W2).

W2: Figures 1–4 are difficult to interpret and provide limited explanatory value. The text and annotations are too small to read clearly, and several figures are presented with little explanation of how they support the paper’s claims.

Thank you for your detailed feedback. We have enlarged the text and annotations in Figs. 1–4 in the revised paper, and added explanation to each caption stating which claim the figure supports. In summary, the figures form the evidence chain for the reasoning-pattern formulation: Fig. 1 shows that RL mainly changes which pattern the LLM selects rather than how each pattern performs. Fig. 2 shows where the patterns come from in the architecture. Fig. 3 shows that a negative safety-efficiency covariance is common and is the origin of the trade-off, and Fig. 4 shows that planning-token clusters correspond to distinct driving strategies.

Here we explain the conclusions of each figure and how they support the paper's claims:

FigureSupports What ClaimHow to Read
Fig. 1-aThe LLM dominates planning, which justifies the two-stage model (Sec. 3.1).Optimizing only the LLM (planner frozen) and jointly optimizing the LLM and the planner give nearly the same results: safety 92.81 → 94.03 vs. 93.99; efficiency 93.8 → 86.3 vs. 86.5. The figure also shows the trade-off itself: safety rises while efficiency drops.
Fig. 1-bRL changes which pattern is selected, not how well each pattern performs.For the same pattern, the safety and efficiency probabilities keep stable before and after RL. The gain of RL therefore comes from redistributing probability among patterns.
Fig. 2Where the two stage modeling and the planning token come from.The LLM’s reasoning is encoded as a planning token, which is then passed to the generative planner as condition. This is why the LLM’s choice determines a macro decision (Sec. 3.1), and why the planning token is a natural proxy for the pattern (Sec. 4.3).
Fig. 3-aThe safety-efficiency covariance is significantly negative in a substantial fraction of scenarios, which is the origin of the trade-off (Sec. 3.3).Negative covariance occurs in 30.0% of Navsim and 47.7% of Bench2Drive scenarios.
Fig. 3-bNegative covariance scenarios tend to involve complex interactions.The negatively correlated scenarios tend to involve more complex traffic interactions, which highlights the necessity to address trade-off during the post-training of reasoning capability.
Fig. 3-cConcrete pattern-level examples of the negative covarianceIn these scenarios, the safest pattern is usually the least efficient one and vice versa. This is exactly the pattern-level examples of the negative covariance.
Fig. 4Planning-token clusters correspond to distinct driving strategies.Rollouts of the same scenario fall into three clusters whose CoTs reason from different perspectives.
Reviewer_X2X6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PapoDrive, which aims to address the trade-off between safety and efficiency encountered when fine-tuning Vision-Language-Action models via reinforcement learning. The approach models the planning process as a two-stage framework: "reasoning pattern selection" and "trajectory generation." By analyzing the covariance between safety and efficiency at the pattern level, the authors investigate the root causes of efficiency degradation. Based on this analysis, the paper proposes a "Pattern-level Step-size Modulation" mechanism, which dynamically adjusts the strength of KL regularization according to the estimated efficiency of each pattern. PCA and K-Means clustering are employed to extract distinct reasoning patterns from the planning tokens. Experimental results on Bench2Drive and NAVSIM demonstrate that this method effectively improves the trade-off between safety and efficiency.

Critical strengths
  1. The paper studies a practical problem: RL can make a driving model safer but also too slow and cautious.

  2. The covariance analysis helps explain why safety training can reduce efficiency. It also gives a clear reason for adjusting the KL penalty for different patterns.

  3. PapoDrive improves both success rate and efficiency on Bench2Drive. On high-risk NAVSIM scenes, it reduces the efficiency drop from 10.2% with GRPO to 2.2%. The paper also tests clustering choices and errors in efficiency estimates.

Critical weaknesses
  1. The two-stage pattern-selection idea comes from earlier work. The covariance analysis is useful, but the main training change is to adjust the KL penalty based on pattern efficiency. The method therefore seems to be an incremental improvement.

  2. Figure 4 gives examples of what the clusters mean, and the appendix checks their stability. However, stable clusters do not necessarily mean different reasoning strategies. More independent evidence is needed.

  3. Figure 1(b) shows that each pattern’s safety and efficiency stay similar before and after RL. However, the paper does not show how pattern probabilities, efficiencies, and KL coefficients change during PSM training. This makes it hard to check how PSM keeps efficient patterns in use.

  4. The full NAVSIM PDMS stays at 90.8, while Traffic-Sign performance drops on Bench2Drive. The authors explain these results, but more detailed tests would help show where the method works and where it fails.

Questions for authors
  1. Is clustering needed for the gains? Please compare PSM with a method that sets the KL penalty for each sample based on its own efficiency, without clustering. Use the same rewards and training budget.

  2. Could the authors check whether the clusters match independent labels of reasoning strategies or driving behaviors, beyond the examples in Figure 4?

  3. Beyond Figure 1(b), please follow the same patterns across training checkpoints. How do their probabilities, efficiencies, and KL coefficients change under GRPO and PSM?

  4. Which types of scenes improve or get worse on full NAVSIM? Could adding traffic-rule rewards reduce the Traffic-Sign drop while keeping the safety and efficiency gains?

7 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer X2X6 (1/6)

Dear Reviewer X2X6,

Thank you for the constructive suggestions to our work, and for recognizing both the theoretical value and method performance. We have added three new analyses and included them in the revised paper ( Section 5.3 & Appendix D3/D4/D5, with blue highlight ).

  • Contributions (W1). The two-stage view is borrowed from LLM work. What is new is verifying that it holds for driving VLA models, using it to locate the source of the trade-off (a negative safety-efficiency covariance), showing that joint rewards can reverse the safety ranking among patterns, and proposing PSM, which changes how far each pattern may move from the reference policy rather than what is rewarded.
  • Pattern-level vs. rollout-level modulation (Q1). Under the same reward and training budget, rollout-level KL modulation already reduces the efficiency loss of GRPO (Eff. -7 vs. -14), but pattern-level modulation is better on both axes: Eff. -3, and NC +11.6 vs. +9.8. Averaging over the rollouts of a pattern reduces the noise of single rollouts.
  • Clusters match driving behaviors (W2/Q2). We tested the clusters against six objective driving-behavior indicators that are not used for clustering (126 scenarios, 2,016 rollouts). Cluster membership explains most of the within-scenario variance (mean η2\eta^2 = 0.65–0.80), and the association is significant in 70%–82% of the scenarios.
  • Pattern dynamics during training (W3/Q3). Under GRPO, the probability of the fast pattern falls from 0.28 to 0.03, and the slow pattern rises to 0.84. Under PSM, the fast pattern always receives the smallest KL coefficient (0.67–0.81 × β\times\,\beta), and its probability is still 0.28 at iteration 45. The efficiency ranking of the patterns stays unchanged throughout training.
  • Traffic-rule rewards and full NAVSIM (W4/Q4) are discussed below.

The detailed responses are provided as follows.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer X2X6 (2/6)

Response 1 - Clarify the actual contributions of our method beyond earlier work (W1).

W1: The two-stage pattern-selection idea comes from earlier work. The covariance analysis is useful, but the main training change is to adjust the KL penalty based on pattern efficiency. The method therefore seems to be an incremental improvement.

We agree that the two-stage view is inspired by prior work on LLMs (discussed in Section 2.2), but it is only our starting point. Beyond this inspiration, our paper i) verifies that this view holds for driving VLA models, ii) uses it to explain where the safety-efficiency trade-off comes from, and iii) derives from it a mechanism that differs from reward re-weighting: PSM changes how strongly each pattern is constrained rather than what is rewarded, which avoids the safety rank reversal of joint reward optimization. Together, these form a complete research loop on the trade-off in the post-training of driving VLA models: (1) mechanism modeling, (2) theoretical explanation, (3) weakness analysis of existing methods, (4) the proposed method, and (5) adaptation to large-scale training. We detail each step below:

(1) Mechanism modeling of VLA decision-making (Section 3.1 & 3.2): The frameworks of Chen et al. (2026b) and related works were proposed for LLMs on math and code reasoning. We did not simply migrate it to autonomous driving VLA, but conducted detailed analysis and empirical experiments from aspects such as model architecture, post-training properties, and trajectory generation process. These analysis itself is valuable for understanding the decision-making mechanisms of autonomous driving VLA.

(2) Theoretical explanation of safety-efficiency trade-off (Section 3.3): Based on the theoretical representation of RL optimal policy and efficiency drift, we conducted covariance analysis on the driving scenarios in mainstream datasets, using actual results to reveal the fundamental reasons for the safety-efficiency trade-off in driving models.

(3) Weakness analysis of existing method (Section 4.1): Guided by the established theoretical framework, we examined the commonly adopted approach (joint reward optimization) from a more fundamental perspective, revealing its inherent weakness of "safety rank reversal" mathematically. Subsequent experiments (Table 2, ID-3) also validated the rationality of mathematical analysis.

(4) Introducing the proposed method (Section 4.2): Motivated by the analysis above, we proposed another approach to tightening the efficiency drift. Instead of adding efficiency into the reward, PSM modulates the KL strength of each pattern according to a efficiency-related adjustment function. Mathematically, PSM has been proven to bound the influence of efficiency on the safety ranking among patterns; Experimentally, it effectively mitigates the trade-off between safety and efficiency across different benchmarks and backbones.

(5) Adaptation to large-scale training (Section 4.3): Existing pattern identification relies on rule-based parsing or auxiliary LLM, which is too costly during the RL loop. We proposed a lightweight pattern extraction scheme, so PSM can be integrated into large-scale training with negligible overhead.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer X2X6 (3/6)

Response 2 - Compare PSM with per-rollout KL modulation without cluster (Q1)

Q1: Is clustering needed for the gains? Please compare PSM with a method that sets the KL penalty for each sample based on its own efficiency, without clustering. Use the same rewards and training budget.

We have conducted this comparison under the same reward and training budget. Both modulation schemes reduce the efficiency drop of vanilla GRPO, confirming that efficiency-dependent KL modulation is beneficial in itself. However, pattern-level modulation is better on both axes: it achieves a smaller efficiency drop (Eff. -3 vs. -7) and a larger safety gain (NC +11.6 vs +9.8). We attribute the gap to two reasons, detailed after the table: averaging over the rollouts of a pattern reduces the variance of the modulation signal, and the pattern is the granularity at which gx(p)g_x(p) is defined in our theory. Overall, these results indicate that the pattern granularity is a substantive component of PSM, rather than only an interpretive device.

Detailed responses:

To demonstrate the effectiveness of clustering, we have supplemented this set of comparisons under the same reward and training budget:

Modulation LevelEff.(Efficiency Metric)Δ\Delta Eff.NC (Safety Metric)Δ\Delta NC
Baseline137080.20
GRPO123-14 ↓\downarrow ↓\downarrow ↓\downarrow90.4+10.2 ↑\uparrow ↑\uparrow
Rollout-level (without cluster)130-7 $ \downarrow$ ↓\downarrow90.0+9.8 $ \uparrow$ ↑\uparrow
Pattern-level (with cluster)134-3 $ \downarrow$91.8+11.6 ↑\uparrow ↑\uparrow ↑\uparrow

The advantage of pattern-level modulation is consistent with our analysis from both the experimental and the theoretical perspectives:

(1) From the experimental perspective: modulating on rollout-level will be more rough, a single high-efficiency / low-efficiency sample will lead to abnormal large modulation strength. Averaging at the pattern-level can reduce variance of the modulation signal and make training more stable.

(2) From the theoretical perspective: The basic idea of our theoretical modeling is to view the VLA decision process as two stages: "macro logical reasoning" and "micro action decoding." The modulation in the theory is based on gx(p)=E[refficiency(x,τ)∣p]g_x(p) = \mathbb{E}[ r_{\text{efficiency}}(\boldsymbol{x}, \boldsymbol{\tau}) \mid p ], which represents "the expected efficiency given a macro pattern". If only the efficiency reward from each single rollout is used for modulation, it places more emphasis on the "micro action decoding" stage and includes sampling noise from specific CoT decoding and trajectory generation, which is essentially closer to a reward shaping method that depends on the result, rather than modulating the reasoning decision itself. Modulating from the pattern dimension is strictly in line with the correct granularity of theoretical derivation.

Overall, these results indicate that the pattern granularity is a substantive component of PSM, rather than only an interpretive device. Removing the clustering step weakens both the safety gain and the efficiency preservation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful evaluations and constructive feedback. We are encouraged from reviewers' recognition of PapoDrive's contributions. The reviewers recognized that the safety-efficiency trade-off in post-training of driving VLA is an important and practical problem (X2X6, y1Pj). The pattern based theoretical analysis are considered offering a useful perspective with clear mathematical formulation (X2X6, y1Pj, BADU). The proposed pattern-level modulation mechanism was evaluated as providing methodological novelty beyond existing method (BADU). The encouraging improvements across different benchmarks and the meaningful ablation analysis are also recognized (X2X6, BADU).

The main concerns are about modulation granularity, pattern semantics and additional ablation comparisons. We have supplemented the experiments and clarification and revised the manuscript accordingly. The main changes are:

  • Comparison on modulation granularity (X2X6, BADU): The comparisons between the pattern-level and the rollout-level modulation have been conducted under the same training budget. Both modulation schemes reduce the efficiency drop of vanilla GRPO, confirming that efficiency-dependent KL modulation is beneficial in itself. However, pattern-level modulation is better on both safety and efficiency, indicating that the pattern granularity is a substantive component both experimentally and theoretically. (in the revised manuscript: Section 5.3 & Appendix D3).
  • Semantic validity of patterns (X2X6, y1Pj, BADU): Independent driving-behavior indicators were analyzed to demonstrate the semantic validity of patterns (in the revised manuscript: Section 5.3 & Appendix D4).
  • Pattern dynamics during training (X2X6, y1Pj): We tracked the pattern-level dynamics across training checkpoints under GRPO and PSM, with the results being consistent with our theoretical modeling (in the revised manuscript: Section 5.3 & Appendix D5).
  • Additional ablation comparisons (X2X6, BADU): Additional ablations including PSM alone, reward-weight sweep and comparison on Bench2Drive have been provided (in the revised manuscript: Section 5.2 & Appendix D6).
  • Further development and clarification of theory (BADU): We have supplemented the precise expression of the combined training objective, and extended Proposition 4, proving that its conclusion holds in a broader context as well (in the revised manuscript: Section 4.2 & Appendix G).
  • Adjustments of figures and captions (y1Pj): We have enlarged the text and annotations in Figures 1–4, and added explanation to each caption stating which claim the figure supports (in the revised manuscript: Figures 1–4).

Point-by-point responses are provided under each review.

We thank the reviewers again for their valuable feedback, which has helped us further strengthen the presentation and validation of PapoDrive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.