- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DriveBeyond, a modification to GRPO for reinforcement fine-tuning of autonomous driving policies. It identifies two issues, advantage shift and advantage collapse, and introduces historical reward references and a ghost reward to address them. Experiments on three driving planners show consistent improvements across NAVSIM v1 and v2.
Critical strengths
- This work identifies two issues of GRPO, advantage shift and advantage collapse, and introduces historical reward references and a ghost reward to address them.
- The paper presents an interesting analysis and reasonably strong experimental results.
Critical weaknesses
- My main concern is the motivation behind advantage shift. Receiving different advantages for the same reward across sampling groups is an expected property of relative advantage estimation, rather than necessarily an optimization flaw. Interestingly, Proposition D.2 in Appendix D.3 shows that the historical reference does not affect the expected first-order policy gradient under the stated assumptions. While it may influence gradient variance and PPO clipping, the paper does not provide sufficient evidence that these effects explain the observed improvements.
- I also find the methodological novelty somewhat limited. Both modules essentially introduce group-wise offsets to the advantages. More importantly, for AutoVLA and ExploreVLA, the proposed method uses a within-group weight of 10, which may change the effective balance between the policy objective and KL regularization. A comparison with Dr.GRPO under matched advantage scaling would help clarify how much of the improvement actually comes from the proposed historical references.
- The results on navhard are encouraging, but there are some concerning trade-offs. For instance, the Extended Comfort score of ReCogDrive drops from 54.5 to 16.1 in Stage 1. Although the authors acknowledge this issue, I am not fully convinced that the improvement in aggregate EPDMS justifies such a substantial degradation in planning consistency.
Questions for authors
Can the authors provide more direct evidence that advantage shift causes harmful policy updates, rather than reflecting the normal behavior of group-relative advantage estimation? How does the method compare against Dr.GRPO with matched advantage scaling and KL regularization? Could the authors further investigate the substantial degradation in Extended Comfort on navhard?
More analysis on Weakness 1 is needed.
Thank you for the detailed response and additional experiments. The Dr.GRPO×10 baseline is helpful and suggests that advantage scaling accounts for part of the gain, with a 0.24 PDMS margin remaining. However, this comparison covers only ExploreVLA on navtest v1, without variance estimates, whereas the largest gains over Dr.GRPO occur on navhard. For W3, the consistency variant substantially alleviates the EC degradation, which is encouraging. However, a GRPO + consistency baseline is missing, and Stage 1 DAC (88.7) falls below GRPO (89.6), indicating that some safety-consistency trade-offs remain. My main concern remains W1. The revised framing appropriately avoids treating advantage shift as inherently harmful. However, Proposition D.2 shows that, under its stated assumptions, the historical reference does not affect the expected first-order policy gradient. Its potential effects through gradient variance, PPO clipping, or group routing remain insufficiently investigated. The Ghost Module is better motivated in this respect. Overall, I appreciate the additional evidence, but the remaining uncertainty about the mechanism behind the gains, together with the limited methodological novelty, leads me to maintain my original score.