acceptodds
Under review as a conference paper at ICLR 2027

PlanContext: Benchmarking In-Context Driving Decision-Making

Abstract

Real-world driving requires context-aware decision-making. Existing end-to-end driving benchmarks primarily assess trajectory quality through safety, compliance, comfort, and driving progress, leaving context-dependent decision-making insufficiently characterized. To fill this gap, we present PlanContext, a benchmark for evaluating in-context driving decision-making. PlanContext constructs paired evaluations by varying high-level driving intentions while keeping the observed scene fixed, enabling a natural and controllable assessment of context-dependent decision-making. We further define six representative driving contexts and jointly evaluate contextual decision correctness, road compliance, and dynamic safety with class-balanced metrics. Experiments on 9,561 scene–condition instances from NAVSIM and 23 public end-to-end driving models show that high conventional driving-quality scores do not consistently correspond to strong route-level in-context driving decision-making. Further analysis shows that model behavior varies with decision timing, road structure, and traffic interaction depending on the decision required by the current context. Our results highlight in-context driving decision-making as an important complementary dimension for evaluating end-to-end driving systems.

Primary area:
datasets and benchmarks
Keywords:
End-to-End Autonomous Driving, In-Context Decision-Making, Context-Aware Evaluation, Driving Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YzxcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates context-aware decision-making in end-to-end autonomous driving. The authors introduce PlanContext, a benchmark constructed from NAVSIM that covers six categories of driving contexts. They evaluate 23 representative models across three paradigms and provide extensive comparative analyses. The findings offer useful insights into the limitations of existing autonomous driving models, particularly regarding their ability to make context-appropriate decisions.

Critical strengths
  1. Clear motivation. The paper identifies an important gap in existing open-loop driving evaluation benchmarks, where context-dependent decision correctness has not been sufficiently assessed. The authors address this limitation by designing a systematic pipeline for benchmark construction and evaluation.
  2. Extensive and credible experiments. The authors evaluate 23 representative models across different paradigms and provide comprehensive experimental results. The inclusion of uncertainty estimates and confidence intervals further strengthens the credibility of the findings.
Critical weaknesses
  1. Comparison with closed-loop benchmarks. Although PlanContext extends conventional open-loop benchmarks such as NAVSIM by explicitly evaluating context-aware decision-making, its advantages over existing closed-loop benchmarks remain unclear. Closed-loop evaluation naturally captures contextual dependencies, as previous decisions influence subsequent vehicle states and observations, and it more closely reflects real-world driving. The authors should clarify what additional capabilities or failure modes PlanContext can reveal beyond those captured by closed-loop benchmarks.
  2. Insufficient explanation of the experimental findings. The authors report extensive results and demonstrate a substantial discrepancy between conventional trajectory-quality metrics and context-aware evaluation metrics. However, the underlying reasons for this discrepancy are not sufficiently explored. Further analysis of why these metrics produce different model rankings, as well as the relationships among the proposed evaluation metrics, would provide deeper insights and strengthen the impact of the benchmark.
  3. Near-zero LAS across most models. Most evaluated models achieve LAS scores close to zero, which is a striking result. Although the authors briefly discuss the ambiguity of navigation commands in Appendix D.9, it remains unclear whether such low scores primarily reflect limitations in the models' decision-making capabilities, ambiguities in the navigation instructions, or the proposed evaluation criteria. A more detailed analysis of this phenomenon would be valuable.
Questions for authors

None.

Reviewer_8SvhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

the paper introduces a benchmark for in-context driving decision-making. the goal of this benchmark is to evaluate of end-to-end models make proper decisions considering the driving context and the intention. the paper provides both the comprehensive evaluation metrics for the in-context driving decisions and also the dataset for the evaluation. the paper also conducts evaluation of 23 e2e models and provide the evaluation report on the in-context decision making capability, it shows that conventional driving quality metrics can not fully capture that.

Critical strengths
  1. the paper evaluates the model's proper responsiveness, rather than just trajectory quality or if it is obeying the command or not. A driving model should be able to tell if it should follow the instruction under current context, like the requirements categorized in the paper: follow, wait, reject, preserve. It is useful to evaluate both under-responsive and over-responsive behaviors.
  2. the paper provides a solid empirical study that supports the usefulness of the proposed benchmark, also provide insights on the current e2e models' performance on its own.
Critical weaknesses
  1. i dont quite understand how the paper can determine the requested maneuver is legal and safely achievable within 4 seconds. for example, if there is a red light, if the policy decides to wait, then it could be scored as a failure decision? The potential problem is that a model could correctly wait and fail the required branch-change criterion, while executing the maneuver would fail safety. Such an instance would not have a valid successful response under the scoring rules.
  2. ambiguous interpretation of low scores: unclear command semantics and under-specified per-model navigation interfaces make it difficult to distinguish poor contextual decision-making from training-time command mismatches or integration limitations. A low score may reflect: a mismatch between training-time and benchmark command meanings or limitations of the model’s supported navigation interface or inconsistent handling of navigation-related inputs. how we can better help users to get correct signal from the benchmark scores?
Questions for authors

N/A, please refer to the weakness part

Reviewer_XbKTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PlanContext, a benchmark for evaluating in-context driving decision-making in end-to-end autonomous driving. It aims to go beyond trajectory quality and systematically assess intention–context–decision consistency. PlanContext keeps the observed scene fixed while changing the high-level navigation intention. It defines six driving contexts grouped into four contextual requirements. The benchmark contains 9,561 strictly filtered scene–condition instances built from NAVSIM navtest. Extensive experiments on 23 public end-to-end models show that high scores on conventional driving-quality metrics do not consistently correspond to strong route-level in-context decision-making.

Critical strengths

S1. A protocol for evaluating intention and context consistency

The authors propose a NAVSIM-based protocol for evaluating in-context driving decision-making. They show that end-to-end models with high PDMS can still respond inappropriately when the navigation intention changes [Fig. 1]. The protocol pairs predictions under the original and alternative intentions to define when an intention should be executed, deferred, rejected, or preserved. This shows a gap that trajectory-level metrics do not fully reflect. The blinded expert audit and scene-level bootstrap support the reliability of the automated labels and the statistical comparisons between models [App. D.9, D.3].

Critical weaknesses

W1. Unclear link to context understanding

NAS, CPS, LAS, and MCS may partly reflect how a model responds to the intervention itself rather than whether it understands the context. For example, models with high mPDMS have near-zero NAS but comparable CPS [Table 2], and StateMLP, NAVSIM's ego-status MLP baseline, has the highest LAS. Discussing this point further would help readers interpret the negative mPDMS–NCS correlation.

W2. Limited analysis beyond aggregate scores

The failure decomposition and context-factor analysis are useful [Sec. 4.4, App. D.8], but they combine results from all models, and the examples for individual models are qualitative [App. E.2]. It would be more informative to include a quantitative analysis for each model, such as error types, comparisons across VA/VLA/WA paradigms, or statistical variance beyond NCS.

Questions for authors

Q1. Navigation responsiveness and context dependent selectivity

Models with high mPDMS have near-zero NAS but CPS comparable to other models (e.g., CLOVER, Drive-JEPA, DiffusionDriveV2) [Table 2]. Their CPS may therefore partly reflect a weak response to the navigation intention rather than correct rejection. Could you report the B.4 path-change rate in A/L versus B1/B2/C for each model? This would help separate navigation responsiveness from context-dependent selectivity.

Q2. Performance differences within each paradigm

The paper notes that the paradigm alone does not consistently explain performance [Sec. 4.2]. NAS ranges from near zero to about 15–22 within each paradigm (DrivoR vs. DiffusionDriveV2, OneVL vs. WAM-Flow, SUV vs. Drive-JEPA). Could you discuss what separates strong and weak models within each paradigm, such as the training objective or how the command is used?

Q3. Dependence on ego motion inputs

In the NAVSIM training data, the navigation command is largely predictable from the route and ego motion, so models may learn to rely on ego velocity and acceleration rather than the command [1]. Could you check how NAS changes when only the velocity input is perturbed at inference [2]? As an optional further check, you could retrain a representative model (e.g., TransFuser) with ego-status dropout [3].

Q4. Novelty beyond NavControl

The paper discusses the difference from NavControl [4] in only one sentence (Sec. 2). Since category A largely overlaps with NavControl's feasible-command evaluation, the contribution beyond it appears to be mainly the additional B1/B2/C (and L/LC) cases. Could you explain the novelty beyond NavControl more clearly? For example, what does PlanContext reveal that NavControl's compliance metric does not?

[1] Li et al., "Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?," CVPR 2024. [2] Tang et al., "Decoupling Scene Perception and Ego Status: A Multi-Context Fusion Approach for Enhanced Generalization in End-to-End Autonomous Driving," AAAI 2026. [3] Cheng et al., "Rethinking Imitation-based Planner for Autonomous Driving," ICRA 2024. [4] Wu et al., "Closing the Navigation Compliance Gap in End-to-end Autonomous Driving," arXiv:2512.10660, 2025.

2 comments

Thank you for the answers with extensive results in such a short rebuttal period. The answers have addressed my concerns. I believe this benchmark is valuable to the community. So, I decide to revise my previous assessment to positive.

Dear Reviewer XbKT,

Thank you for reconsidering your assessment and for your encouraging comments on our work. Your detailed questions and constructive suggestions have helped us strengthen the experimental analyses and clarify the interpretation of PlanContext’s results. We are pleased that our responses have addressed your concerns and sincerely appreciate your time and careful evaluation.

Best regards,

The Authors of PlanContext

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.