- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Run-then-Walk, a two-stage GRPO training strategy for VLM-based autonomous driving that first encourages driving progress under relaxed constraints and subsequently introduces endpoint and safety rewards to refine the policy. The approach aims to overcome overly conservative behavior from safety-first training while avoiding unsafe aggressive driving. Experiments on NAVSIMv1, NAVSIMv2, Navhard, and nuScenes using autoregressive and diffusion-based planners report improved driving performance and 40–50% fewer RL training epochs compared to existing approaches.
Critical strengths
- The paper addresses an important trade-off between driving progress and safety, proposing a simple training strategy that can be integrated into existing VLM planners without architectural modifications.
- Experiments across multiple benchmarks and two different planner architectures demonstrate consistent performance improvements with fewer RL epochs. The recipe ablations, training curves, and multi-seed evaluations provide useful supporting evidence.
Critical weaknesses
- The methodological novelty is somewhat limited, as the approach primarily changes the ordering of existing reward objectives. The theoretical analysis is highly idealized and does not establish that this particular ordering is optimal or guarantees improved GRPO training.
- The claimed training efficiency is not fully validated through controlled comparisons. The proposed method uses 5–6 epochs versus 10 for the original baselines, but equal-budget comparisons, wall-clock measurements, and sensitivity to the stage-switching point are insufficiently explored.
- The improvements are relatively modest on some benchmarks, and the safety trade-off is not uniformly positive. For instance, AutoDrive-P3 achieves higher PDMS on NAVSIMv1 but slightly lower NC, DAC, and TTC than the original baseline. Additionally, changes in learning rate and reference policy between stages are not independently ablated, making it difficult to isolate the effect of reward scheduling.
Questions for authors
- Can the authors provide equal-compute comparisons against the original training strategies and report actual GPU-hours or wall-clock training costs?
- How sensitive is performance to the Run/Walk epoch allocation and switching criterion? Would alternative schedules or a gradual reward transition achieve similar results?
- Can the authors independently evaluate the effects of reward ordering, reference-policy resetting, and stage-specific learning rates, particularly for ReCogDrive?
- How do the authors justify the claimed safety improvements when some safety metrics decrease relative to the original baseline? Are these differences acceptable in safety-critical driving scenarios?
10 comments
Response to W1: Novelty and Theory (2/2)
Table 1. RL recipes on NAVSIMv1 (AutoDrive-P3, same SFT checkpoint)
| Recipe | RL epochs | NC↑ | DAC↑ | EP↑ | TTC↑ | Comf↑ | PDMS↑ |
|---|---|---|---|---|---|---|---|
| Only Run | 5 | 94.8 | 95.0 | 89.0 | 84.7 | 100.0 | 85.8 |
| Only Walk | 5 | 99.1 | 97.7 | 81.5 | 97.3 | 100.0 | 89.7 |
| Walk-and-Run | 5 | 98.0 | 94.2 | 85.4 | 94.0 | 100.0 | 85.9 |
| Linear annealing (Run → Walk) | 5 | 97.1 | 96.3 | 88.5 | 95.3 | 100.0 | 90.5 |
| Walk-then-Run (original AutoDrive-P3) | 5 | 99.2 | 97.8 | 80.5 | 97.4 | 100.0 | 89.2 |
| Walk-then-Run (original AutoDrive-P3) | 10 | 98.9 | 97.7 | 83.7 | 96.6 | 99.9 | 90.2 |
| Run-then-Walk (ours) | 5 | 98.6 | 97.3 | 88.9 | 95.5 | 100.0 | 91.8 |
Table 2. RL recipes on NAVSIMv1 (ReCogDrive, same SFT checkpoint)
| Recipe | RL epochs | NC↑ | DAC↑ | EP↑ | TTC↑ | Comf↑ | PDMS↑ |
|---|---|---|---|---|---|---|---|
| Only Run | 6 | 97.9 | 97.6 | 86.5 | 95.0 | 100.0 | 90.4 |
| Only Walk | 6 | 99.0 | 98.3 | 78.8 | 97.6 | 99.9 | 89.0 |
| Linear annealing (Run → Walk) | 6 | 98.5 | 97.7 | 85.4 | 95.0 | 100.0 | 90.7 |
| Walk-then-Run | 6 | 97.9 | 96.5 | 84.5 | 94.6 | 99.9 | 88.3 |
| Walk-and-Run (original ReCogDrive) | 6 | 98.1 | 98.1 | 85.4 | 95.1 | 100.0 | 90.2 |
| Walk-and-Run (original ReCogDrive) | 10 | 98.1 | 97.7 | 86.5 | 94.9 | 100.0 | 90.6 |
| Run-then-Walk (ours) | 6 | 98.6 | 97.9 | 85.9 | 95.8 | 100.0 | 91.0 |
Table 3. Theoretical conditions vs. observations (AutoDrive-P3, NAVSIMv1)
| Theoretical statement | Implication | Observation |
|---|---|---|
| J ≤ min{E[S], E[Q]} (Eq. 8) | Neither progress nor safety alone suffices | Only Run: EP 89.0 but PDMS 85.8; Only Walk: NC 99.1 but PDMS 89.7 |
| Safety direction of Walk (Eqs. 18–19) | A positive reward gap γ = 1.5·λ_plan − ω raises the probability of passing the gate | γ ≥ 0.85; from the Run checkpoint, Walk raises NC / DAC / TTC by +3.8 / +2.2 / +10.7 |
| Progress retention (Eq. 9) | Walk retains the high-progress mass a_R of its reference policy | Scenes with EP > 0.9: SFT 46.2% → Run 85.1%; EP after Walk 88.9 (reference = Run) vs. 81.5 (reference = SFT, Only Walk) |
Response to W2 & Q1: Equal-compute Comparison and GPU-hours
(1) Equal-compute comparison: Run-then-Walk clearly outperforms the original recipes. All hyperparameters follow the original open-source configurations of AutoDrive-P3 and ReCogDrive without additional tuning, and the original recipes are re-trained with the same infrastructure (bf16, 8×A100), SFT checkpoints and seeds; we only change the planning reward and its schedule. The training code and configurations are available in our anonymous repository: https://anonymous.4open.science/r/Run-then-Walk-065A/. In all recipes, each RL epoch samples G=8 rollouts per query and evaluates every rollout with the same PDM scorer, so the same number of RL epochs corresponds to the same compute. Following your suggestion, we re-train both original recipes with the same 5/6 RL epochs as Run-then-Walk (Table 4; numbers in parentheses: PDMS change relative to the original recipe with the same compute). With the same 1,000 GPU-hours, Run-then-Walk improves the PDMS of AutoDrive-P3 by +2.6 (91.8 vs. 89.2); with the same 96 GPU-hours, it improves the PDMS of ReCogDrive by +0.8 (91.0 vs. 90.2). Even with the full 10-epoch compute, the original recipes (90.2 and 90.6) remain below Run-then-Walk, while extending Run-then-Walk to 10 epochs brings no further gain.
(2) GPU-hours and wall-clock cost. One RL epoch takes about 25 h on 8×A100 for AutoDrive-P3 (803 steps), which decodes its perception, prediction and planning tokens autoregressively, and about 2 h for ReCogDrive (1330 steps), whose diffusion head uses 5 DDIM denoising steps as in the original implementation. Run-then-Walk therefore reduces the RL training time from about 250 h (2,000 GPU-hours) to 125 h (1,000 GPU-hours) for AutoDrive-P3 and from about 20 h (160 GPU-hours) to 12 h (96 GPU-hours) for ReCogDrive: the 40–50% reduction holds for the measured GPU-hours, not only for the epoch count.
(3) Switching point. The sensitivity to the switching point is reported in (Sec. 6, Table 7 of updated manuscript).
Table 4. Equal-budget comparison and training cost on NAVSIMv1 (8×A100)
| Planner | Recipe | RL epochs | RL steps | Wall-clock (h) | GPU-hours | PDMS↑ |
|---|---|---|---|---|---|---|
| AutoDrive-P3 | Walk-then-Run (original) | 5 | 4,015 | 125 | 1,000 | 89.2 |
| AutoDrive-P3 | Run-then-Walk (3 + 2, ours) | 5 | 4,015 | 125 | 1,000 | 91.8 (+2.6) |
| AutoDrive-P3 | Walk-then-Run (original) | 10 | 8,030 | 250 | 2,000 | 90.2 |
| AutoDrive-P3 | Run-then-Walk (3 + 7) | 10 | 8,030 | 250 | 2,000 | 91.8 (+1.6) |
| ReCogDrive | Walk-and-Run (original) | 6 | 7,980 | 12 | 96 | 90.2 |
| ReCogDrive | Run-then-Walk (5 + 1, ours) | 6 | 7,980 | 12 | 96 | 91.0 (+0.8) |
| ReCogDrive | Walk-and-Run (original) | 10 | 13,300 | 20 | 160 | 90.6 |
| ReCogDrive | Run-then-Walk (5 + 5) | 10 | 13,300 | 20 | 160 | 91.0 (+0.4) |
Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.
Response to W1: Novelty and Theory (1/2)
We apologize for the misunderstanding of our contributions arising from the wording of our original manuscript. Run-then-Walk is not a simple reordering of existing rewards: it is derived from an RL training phenomenon that previous VLM-based driving work does not identify, and it is selected by systematic recipe comparisons across planners and benchmarks.
(1) Contributions. First, we identify two RL regimes for VLM-based driving from the same SFT checkpoint (Figs. 3 and 5 of the original manuscript): a progress regime (Run-GRPO) that explores high progress at the cost of safety, and a safety regime (Walk-GRPO) that preserves safety but stays conservative. Second, based on this finding, Run-then-Walk first expands the progress distribution with a PDMS-only Run stage and then repairs safety with a gated endpoint-and-safety Walk stage whose KL reference is reset to the Run checkpoint; this differs from the Walk-and-Run recipe of ReCogDrive [1], the Walk-then-Run recipe of AutoDrive-P3 [2] and generic easy-to-hard curricula [3], which all start from constrained or easy objectives (Figure 1). Third, Run-then-Walk is the best recipe on both planners (Tables 1 and 2), improves performance on four benchmarks and saves 40–50% of RL epochs, and applies to both autoregressive and diffusion VLM planners without changing the planner architecture or the inference cost.
(2) Theory. The role of the theory is to explain why the Run-then-Walk direction works: as stated in Section 4.3, it "serves as a reward-direction explanation rather than a convergence guarantee". Its conditions can be checked against our experiments, and the theory, Figs. 3 and 5, and Table 5 of the original manuscript are consistent with each other (Table 3). The bound J ≤ min{E[S], E[Q]} explains why neither progress alone nor safety alone suffices; the positive reward gap of the Walk gate explains why Walk restores safety from the Run checkpoint; and the progress-retention bound explains why Run must come first: Walk can only retain the high-progress mass that already exists in its reference policy, and Run increases this mass from 46.2% to 85.1% of scenes. We have made this connection explicit in Appendix B of the updated manuscript.
Figure 1. Compared RL training recipes
[1] ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving. (26' ICLR)
[2] AutoDrive-P3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning. (26' ICLR)
[3] Curriculum Learning. (09' ICML)