- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates context-aware decision-making in end-to-end autonomous driving. The authors introduce PlanContext, a benchmark constructed from NAVSIM that covers six categories of driving contexts. They evaluate 23 representative models across three paradigms and provide extensive comparative analyses. The findings offer useful insights into the limitations of existing autonomous driving models, particularly regarding their ability to make context-appropriate decisions.
Critical strengths
- Clear motivation. The paper identifies an important gap in existing open-loop driving evaluation benchmarks, where context-dependent decision correctness has not been sufficiently assessed. The authors address this limitation by designing a systematic pipeline for benchmark construction and evaluation.
- Extensive and credible experiments. The authors evaluate 23 representative models across different paradigms and provide comprehensive experimental results. The inclusion of uncertainty estimates and confidence intervals further strengthens the credibility of the findings.
Critical weaknesses
- Comparison with closed-loop benchmarks. Although PlanContext extends conventional open-loop benchmarks such as NAVSIM by explicitly evaluating context-aware decision-making, its advantages over existing closed-loop benchmarks remain unclear. Closed-loop evaluation naturally captures contextual dependencies, as previous decisions influence subsequent vehicle states and observations, and it more closely reflects real-world driving. The authors should clarify what additional capabilities or failure modes PlanContext can reveal beyond those captured by closed-loop benchmarks.
- Insufficient explanation of the experimental findings. The authors report extensive results and demonstrate a substantial discrepancy between conventional trajectory-quality metrics and context-aware evaluation metrics. However, the underlying reasons for this discrepancy are not sufficiently explored. Further analysis of why these metrics produce different model rankings, as well as the relationships among the proposed evaluation metrics, would provide deeper insights and strengthen the impact of the benchmark.
- Near-zero LAS across most models. Most evaluated models achieve LAS scores close to zero, which is a striking result. Although the authors briefly discuss the ambiguity of navigation commands in Appendix D.9, it remains unclear whether such low scores primarily reflect limitations in the models' decision-making capabilities, ambiguities in the navigation instructions, or the proposed evaluation criteria. A more detailed analysis of this phenomenon would be valuable.
Questions for authors
None.
Thank you for the answers with extensive results in such a short rebuttal period. The answers have addressed my concerns. I believe this benchmark is valuable to the community. So, I decide to revise my previous assessment to positive.