acceptodds
Under review as a conference paper at ICLR 2027

DriveBeyond: Rethinking Group Advantage for Driving Policy Optimization

Abstract

End-to-end autonomous driving models are typically initialized through imitation learning (IL), after which reinforcement fine-tuning (RFT) is increasingly used to improve driving quality. Group Relative Policy Optimization (GRPO) has been adopted for this stage, estimating each trajectory's advantage from its reward relative to other trajectories sampled for the same scene. We revisit this group-relative construction and characterize two issues in its advantage estimation under driving RFT: advantage shift, where centering within each group discards the scene-level progress over training, and advantage collapse, where candidates receiving identical rewards yield vanishing advantages. Motivated by these findings, we propose DriveBeyond, a driving policy optimization method that incorporates references beyond the current sampling group through a shared memory module, calibrating advantages for groups with sufficient reward variation and helping recover learning signals for groups without. Experiments on NAVSIM v1 and v2 demonstrate that DriveBeyond consistently improves planning performance across both VLA- and WAM-based planners under three evaluation settings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, reinforcement learning, post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_gmQYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveBeyond, a modification to GRPO for reinforcement fine-tuning of autonomous driving policies. It identifies two issues, advantage shift and advantage collapse, and introduces historical reward references and a ghost reward to address them. Experiments on three driving planners show consistent improvements across NAVSIM v1 and v2.

Critical strengths
  1. This work identifies two issues of GRPO, advantage shift and advantage collapse, and introduces historical reward references and a ghost reward to address them.
  2. The paper presents an interesting analysis and reasonably strong experimental results.
Critical weaknesses
  1. My main concern is the motivation behind advantage shift. Receiving different advantages for the same reward across sampling groups is an expected property of relative advantage estimation, rather than necessarily an optimization flaw. Interestingly, Proposition D.2 in Appendix D.3 shows that the historical reference does not affect the expected first-order policy gradient under the stated assumptions. While it may influence gradient variance and PPO clipping, the paper does not provide sufficient evidence that these effects explain the observed improvements.
  2. I also find the methodological novelty somewhat limited. Both modules essentially introduce group-wise offsets to the advantages. More importantly, for AutoVLA and ExploreVLA, the proposed method uses a within-group weight of 10, which may change the effective balance between the policy objective and KL regularization. A comparison with Dr.GRPO under matched advantage scaling would help clarify how much of the improvement actually comes from the proposed historical references.
  3. The results on navhard are encouraging, but there are some concerning trade-offs. For instance, the Extended Comfort score of ReCogDrive drops from 54.5 to 16.1 in Stage 1. Although the authors acknowledge this issue, I am not fully convinced that the improvement in aggregate EPDMS justifies such a substantial degradation in planning consistency.
Questions for authors

Can the authors provide more direct evidence that advantage shift causes harmful policy updates, rather than reflecting the normal behavior of group-relative advantage estimation? How does the method compare against Dr.GRPO with matched advantage scaling and KL regularization? Could the authors further investigate the substantial degradation in Extended Comfort on navhard?

1 comment
Reviewer_gmQYOpenReview ↗

More analysis on Weakness 1 is needed.

Thank you for the detailed response and additional experiments. The Dr.GRPO×10 baseline is helpful and suggests that advantage scaling accounts for part of the gain, with a 0.24 PDMS margin remaining. However, this comparison covers only ExploreVLA on navtest v1, without variance estimates, whereas the largest gains over Dr.GRPO occur on navhard. For W3, the consistency variant substantially alleviates the EC degradation, which is encouraging. However, a GRPO + consistency baseline is missing, and Stage 1 DAC (88.7) falls below GRPO (89.6), indicating that some safety-consistency trade-offs remain. My main concern remains W1. The revised framing appropriately avoids treating advantage shift as inherently harmful. However, Proposition D.2 shows that, under its stated assumptions, the historical reference does not affect the expected first-order policy gradient. Its potential effects through gradient variance, PPO clipping, or group routing remain insufficiently investigated. The Ghost Module is better motivated in this respect. Overall, I appreciate the additional evidence, but the remaining uncertainty about the mechanism behind the gains, together with the limited methodological novelty, leads me to maintain my original score.

Reviewer_9EtqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies GRPO-based reinforcement fine-tuning for end-to-end autonomous driving. It identifies two problems related to advantage calculation: advantage shift and advantage collapse. The first issue occurs when an advantage does not reflect progress across visits to the same scene, while the second issue occurs when similar rewards within a group provide little learning signal. DriveBeyond addresses them using the IL policy and previous scene visits as references. Experiments on NAVSIM v1 and v2 demonstrate that DriveBeyond can improve the performance of several VLA-based models, such as ReCogDrive, AutoVLA, and ExploreVLA.

Critical strengths
  1. The paper discusses a relevant problem in RFT for end-to-end autonomous driving. Advantage collapse is particularly meaningful for NAVSIM, where the safety metrics can assign the same zero reward to many failed trajectories.

  2. Empirical results demonstrate that the proposed method improves open-loop trajectory prediction over various group-based baselines. This improvement is consistent across three evaluation settings: NAVSIM v1 Navtest, NAVSIM v2 Navtest, and NAVSIM v2 Navhard.

  3. DriveBeyond is easy to integrate, works with both VLA- and WAM-based planners, and adds no inference-time overhead.

Critical weaknesses
  1. My major concern is the limited evaluation scope. DriveBeyond is an RFT framework designed for open-loop trajectory prediction, where rewards are computed over groups of trajectories for offline scenes and are based on the same metrics used in NAVSIM. However, end-to-end autonomous driving systems should also demonstrate robustness in closed-loop settings, which are not evaluated in this paper. Evaluations in closed-loop simulators such as HUGSIM [1] would provide stronger evidence of the method's effectiveness.

  2. The improvements over GRPO on the standard NAVSIM v1 Navtest splits are relatively modest, ranging from 0.4 to 1.3 PDMS. Although the gains on Navhard are more substantial, all experiments are still conducted within the NAVSIM benchmark. Comparisons with recent planners, such as DrivoR [2] and ZTRS [3], would also help strengthen the evaluation.

  3. There are also some minor issues with presentation. The term "scene-level progress" is used very early in the abstract and introduction without a clear definition, which makes the paper somehow hard to follow in the beginning. In Figure 1(a), the arrow appears to be pointing at the wrong bar. It should point to the bar with orange borders like Figure 1 (b). Figure 6 is also crowded and hard to follow.

[1] Zhou H, Lin L, Wang J, et al. Hugsim: A real-time, photo-realistic and closed-loop simulator for autonomous driving[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.

[2] Kirby E, Boulch A, Xu Y, et al. Driving on registers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 32058-32069.

[3] Li Z, Yao W, Wang Z, et al. Ztrs: Zero-imitation end-to-end autonomous driving with trajectory scoring[J]. arXiv preprint arXiv:2510.24108, 2025.

Questions for authors

I don't have other questions. Please refer to the Critical Weaknesses section.

Reviewer_ZJXTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify two limitations of conventional GRPO, namely advantage shift and advantage collapse. Memory Module maintains historical reward references, a Progress Module adjusts advantages based on scene-level progress, and a Ghost Module provides additional learning signals for low-variance reward groups.

Critical strengths
  1. Good topic. Autonomous driving's GRPO suffers from advantage collapses problem for a long time.
  2. Connecting advantages of GRPO towards scene-dependent trajectory rewards.
Critical weaknesses
  1. Related work (such as CuriousVLA, who has already even claims 'narrow policy" in their title) has already explored the central ideas underlying the proposed approach, and thus I am not sure about the methodological novelty.

  2. The experimental improvements demonstrate the effectiveness of the combined method but do not fully establish the novelty of the underlying mechanism. Compared with other methods used for solving the advantage collapsing problem such as the ADAS in CuriousVLA, more evidence should be presented to prove the effectiveness of the proposed three module.

Questions for authors
  1. Compare with ADAS in CuriousVLA, what advantage of the proposed methods have ? Why the improvement of the proposed three modules so weak ?

  2. Can the authors provide stronger theoretical or controlled empirical evidence that the proposed advantage modifications address a limitation not already resolved by existing reference-based GRPO variants, such as NGRPO and AVSPO?

Reviewer_DSrSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DriveBeyond, a reinforcement fine-tuning method for autonomous driving based on GRPO. The authors identify two limitations of GRPO: "advantage shift" and "advantage collapse." To address these, the proposed method maintains scene-specific historical rewards and incorporates a "Progress Module" to calibrate advantages based on past performance, while using a "Ghost Module" to recover training signals for low-variance groups. Experiments conducted on three types of VLA/WAM planners, using the NAVSIM v1/v2 datasets, demonstrate consistent performance improvements.

Critical strengths
  1. The paper is well-motivated with thorough problem analysis. Two specific limitations of GRPO are identified with intuitive examples and improved by the proposed methods.

  2. The design is simple and lightweight. The proposed modules refine only the advantage estimation process; they do not require an additional critic, nor do they increase computational overhead during inference.

  3. The experimental results are robust and solid. The method yields performance gains across three different planners and various NAVSIM evaluation settings, supported by comprehensive baseline comparisons and ablation studies.

Critical weaknesses
  1. Incremental methodological novelty.

DriveBeyond builds on existing ideas of historical reward baselines and virtual-reward-based advantage correction. Although the paper clarifies its differences from prior methods, its main innovations concern the choice of reward references and the form of advantage adjustment. These constitute incremental refinements of established mechanisms, limiting the method’s conceptual novelty.

  1. The Ghost Module does not resolve within-group credit assignment.

When all sampled trajectories receive the same reward, the Ghost Module assigns an identical positive or negative advantage value to each candidate trajectory. While this results in a non-zero update, it provides no information regarding which trajectories performed better; performance improvements remain dependent on the exploration behavior of the policy itself.

  1. Reliance on repeated visits to scenarios limits generalization.

The method depends on a memory of rewards specific to individual scenarios throughout the training iterations. Its effectiveness in settings such as streaming reinforcement learning (RL), continuously changing environments, or previously unseen scenarios remains unclear.

Questions for authors
  1. Beyond the existing comparisons and ablations, what methodological contribution distinguishes DriveBeyond from a combination of historical reward baselines and virtual-reward correction? Could simpler running-mean or EMA references achieve comparable performance?

  2. Since the Ghost Module assigns shared advantages to trajectories with identical rewards, how do the authors envision addressing this limitation in future work? Could finer-grained reward signals help distinguish trajectory quality and enable within-group credit assignment?

  3. How does DriveBeyond perform when training scenarios are rarely revisited, particularly in streaming or continually changing environments? How are newly encountered training scenarios handled without historical reward records?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.