acceptodds
Under review as a conference paper at ICLR 2027

DriveAltas: A Cross-Country Benchmark for Grounded Driving-Rule Reasoning

Abstract

Vision–language models (VLMs) are becoming the perception-and-reasoning backbone of end-to-end driving systems, but the benchmarks used to assess them are built from one or two regions: they tell whether a model recognizes a scene, not whether it acts on the traffic rule that applies *where the scene was taken*. We introduce DriveAtlas, a cross-country, rule-grounded benchmark for *grounded driving-rule reasoning*, with 10,143 test and 26,539 training questions across eight countries and four tasks (perception, prediction, planning, and region). Each question is tied to a clause of the local traffic code, drawing on 5,962 unique rule texts, and is kept only if its correct answer changes under another country's rules; the country is never revealed to the model. We evaluate 16 open-weight VLMs (2B–9B parameters, three model generations) and report accuracy alongside a sampling-noise-corrected cross-country disparity . All 16 models show significant differences in driving performance across countries: ranges from 5.22 to 10.39, and its 95% interval excludes zero for every model. Scale does not remove these differences: a larger model of the same family is more accurate, but its country-dependent variation remains high, and in three of the four families where size buys accuracy, moves by under half a point. An error analysis of two models finds that naming the wrong country is rarely the failure (under 3% of sampled errors), whereas about one item in five fails on a regional rule gap. We therefore propose DriveOPD, an on-policy self-distillation algorithm whose teacher is the same model reading a jurisdiction-redacted traffic-rule handbook: on both backbones we train, it raises accuracy on all four tasks and lowers by about three points (6.91 → 3.87, 7.07 → 3.88). Used as the backbone of a vision–language–action policy, DriveOPD lowers open-loop L2 error and collision rate in both Boston and Singapore, and narrows the gap between the two cities by 19–26%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Culture, traffic rule, autonomous driving, VLA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PzdiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DriveAtlas evaluates driving-rule reasoning across eight countries using nine public driving corpora. It contains 10,143 test and 26,539 training questions, with country labels withheld and a rule-swap filter intended to retain jurisdiction-dependent items. The paper evaluates 16 generalist VLMs and introduces DriveOPD, which distills a frozen, handbook-conditioned copy of a model into a student without handbook access. Experiments report improved QA accuracy and reduced cross-country disparity on two backbones, with additional open-loop planning results in Boston and Singapore.

Critical strengths

The benchmark addresses a useful evaluation gap with meaningful validity controls. Withholding jurisdiction and filtering for rule-dependent answers offers a more focused test than ordinary multi-country VQA. Sequence-disjoint splits, sequence-level bootstrapping, image controls, and explicit analysis of country–corpus confounding strengthen the evaluation.

The paper provides several complementary checks on its findings. The answer judge is audited against a human expert and two unrelated model lineages. DriveOPD improves all four tasks on two backbones while reducing disparity, and the open-loop VLA results provide an initial connection to trajectory prediction. The benchmark is the strongest contribution in my assessment.

Critical weaknesses

Table 3 compares the trained student only with its unadapted backbone. There is no matched-data, matched-budget QA fine-tuning or handbook-free distillation control. Since the method explicitly builds on existing context and on-policy distillation recipes, these comparisons matter for determining whether the gains come specifically from internalizing regional rules rather than general adaptation to the benchmark. The PDF also provides insufficient training and VLA implementation details to reconstruct the downstream results.

The appendix examples do not show which alternative jurisdiction and corresponding clause change each answer. For example, providing the actual counterfactual for the German no-entry item would clarify the admission criterion. Table 7 also retains 192 overtaking candidates marked undetermined at the geometry-audit stage. Although human reference verification is stated, their final inclusion and adjudication are unclear. This does not establish that these references are wrong, but leaves uncertainty about the benchmark’s most distinctive property and remaining label quality.

Questions for authors

How does DriveOPD compare with matched-budget QA fine-tuning or handbook-free distillation? Can the contributions of handbook conditioning and jurisdiction-name redaction be separated?

Can the authors provide concrete rule-swap clauses and changed answers for the appendix examples, and clarify the final treatment of undetermined audit candidates? Are the main findings stable when unresolved candidates are excluded?

Please specify the training hyperparameters and VLA architecture, inputs, splits, and scoring protocol. Are the nuScenes downstream test sequences excluded from DriveAtlas adaptation training?

4 comments

1. Training Controls and Name Redaction (W1, Q1)

W1/Q1: How does DriveOPD compare with matched-data, matched-budget QA fine-tuning or handbook-free distillation? Can the contributions of handbook conditioning and jurisdiction-name redaction be separated?

We thank the reviewer for these specific suggestions. We additionally evaluate reference-answer SFT, offline teacher-answer distillation, a country-only teacher, and a teacher given another country’s handbook. SFT uses reference answers, while offline distillation uses greedy answers from a frozen handbook-conditioned teacher. The country-only teacher receives the correct jurisdiction without rule text, and the other-country handbook teacher receives another country’s handbook. All adapted students are evaluated under Direct prompting, without an explicit country label or handbook. DriveOPD achieves the highest observed accuracy and lowest DcD_c on both backbones among these configurations: it exceeds reference-answer SFT by 2.8/4.9 accuracy points and offline distillation by 8.7/9.8 points on Qwen/InternVL, respectively. Relative to SFT, DcD_c decreases from 5.75 to 3.87 and 6.42 to 3.88. The DriveOPD values below are unchanged from paper Table 3.

Training methodQwen3.5-2B Acc. ↑ / DcD_c ↓InternVL3.5-2B Acc. ↑ / DcD_c ↓
Base41.0 / 6.9120.9 / 7.07
DriveOPD (submitted Table 3)45.6 / 3.8739.0 / 3.88
Reference-answer SFT42.8 / 5.7534.1 / 6.42
Offline teacher-answer distillation36.9 / 4.6029.2 / 8.31
Country-only teacher42.4 / 7.4934.0 / 8.46
Other-country handbook teacher40.6 / 7.6828.6 / 9.25

Comparison conditions and component attribution. The country-only and other-country handbook KL controls reuse DriveOPD’s data pool, sampler, LoRA configuration, training steps, and twelve/eight sampled answers per update. The country-only teacher is handbook-free but retains jurisdiction information. SFT/offline CE controls use two target answers per update with an untuned learning rate; SFT additionally removes audit-rejected references, leaving 1,003/18,202 items versus DriveOPD’s 1,102/18,992. Handbook versions differ, and the checkpoint-selection procedure is disclosed under Q3. These conditions limit isolated component attribution and fully matched comparisons with CE training. The individual contribution of jurisdiction-name redaction is not established by these results. Whole-Handbook-Given scores of 45.2%/40.1% use the correct country name and complete handbook at inference; they are not a names-retained training ablation. We limit the method claim to the observed advantage of the tested configuration.

2. Counterfactual Examples and Undetermined Candidates (W2, Q2)

W2/Q2: Please provide concrete alternative jurisdictions, clauses, and changed answers for the appendix examples, particularly the German no-entry item. What happened to the 192 undetermined overtaking candidates, and are the findings stable when unresolved candidates are excluded?

We thank the reviewer for the careful reading and agree that the appendix should identify the alternative jurisdiction, applicable clause, and changed answer. The comparisons below distinguish changes in rule interpretation or legal permission from sign-format mismatches and unchanged answers. The German no-entry panel illustrates a sign-grounding failure; its UK comparison preserves “do not enter”. An answer-changing counterpart for the underlying German item is not established by these comparisons, so its item-specific admission remains unsubstantiated here. The UK speed-limit example illustrates a unit change. The Swiss panel asks what the car must do before turning right when the signal is red and no green arrow is shown; its comparison concerns the applicable legal requirements under those signal conditions. These clause comparisons are not recovered original item-level admission records.

Appendix exampleOriginal jurisdiction and applicable provisionSubstituted jurisdiction and answerWhat the comparison establishes
Fig. 7, UK: circular “20” speed-limit signUK: The sign indicates 20 mph, following the UK speed-sign convention.Germany: Under StVO, Zeichen 274, the corresponding sign indicates 20 km/h.Answer changes: the unit and legal speed differ despite the same numeral.
Fig. 7, US: pentagonal school-warning signUS: The pentagonal school sign warns of a school area and children, following MUTCD Part 7.UK: The prescribed school-warning sign is a red-bordered triangle (DfT guidance); the observed pentagon does not match that format, although its warning remains recognizable.Sign-system mismatch: this does not establish a changed precaution or driving action.
Fig. 7, Germany: red no-entry signGermany: StVO, Zeichen 267 prohibits entry from the indicated direction.UK: The no-entry sign imposes the same obligation: do not enter.Unchanged comparison: this figure illustrates sign grounding, not an established answer-changing substitution.
Fig. 8, Switzerland: what must the car do before turning right at red, with no green arrow?Switzerland: The car must stop and remain stopped until a green light or an authorizing green arrow permits the turn; stopping alone does not authorize an ordinary right turn on red (SSV Art. 68).China: Art. 38 permits a right turn at a circular red signal without impeding traffic permitted to proceed or pedestrians; this does not override a prohibiting directional signal. US—Massachusetts: Ch. 89 §8 permits a right turn after stopping and yielding unless prohibited by a posted restriction.Legal permission changes, conditional on the signal and intersection restrictions.

We thank the reviewer for pointing out the overtaking ambiguity. These 192 cases involve two- or multi-lane roads where overtaking depends on traffic direction and the ego vehicle’s lane. For example, when ego already occupies the leftmost lane of a multi-lane one-way road, whether it may pass on the right or must remain behind depends on local rules and scene geometry. We manually reviewed these cases and corrected their reference answers where necessary. The automated “undetermined” flags therefore do not indicate unresolved reference labels in the benchmark.

3. Training Details, VLA Protocol, and Split Integrity (W1, Q3)

Q3: Please specify the training hyperparameters, VLA architecture, inputs, splits, and scoring protocol. Were the downstream nuScenes evaluation sequences excluded from DriveAtlas adaptation training?

We thank the reviewer for identifying these reproducibility gaps. DriveOPD uses the initial backbone as a frozen teacher with adapters disabled. The teacher receives the frames, question, and country-name-redacted handbook; the student receives the Direct prompt. Reverse KL is computed on student-sampled answers, with both distributions renormalized over the teacher’s top-64 tokens, averaged within each answer and then across answers; the end-of-turn token is excluded. Language-model LoRA targets MLP and q/k/v/o projections where present, with vision components frozen. The implementation details below correct the uniform rank-32/four-rollout description without changing submitted Table 3 results. The actual image-resolvable pools exclude Japanese items and part of the UK data; Qwen further selects teacher-aligned training questions. The reported Qwen step-450 checkpoint was selected among variants and checkpoints using test accuracy; its later validation ranking is a post-hoc check. InternVL step 400 was the only checkpoint scored from its run. These selection conditions limit the comparison and will be stated explicitly.

DriveOPD configurationQwen3.5-2BInternVL3.5-2B
Teacher handbook in the reported Table 3 runEarlier base handbook plus sections generated from training itemsBase handbook
LoRA rank / alpha / dropout64 / 64 / 0.0532 / 64 / 0.05
Trainable parameters43.6M34.9M
Rollouts per prompt / prompts per update6 / 2; 12 answers per update4 / 2; 8 answers per update
OptimizerAdamW, beta = (0.9, 0.95), weight decay 0Same
Learning rate / warm-up1.5×10−51.5 \times 10^{-5} / 20 steps1×10−51 \times 10^{-5} / 20 steps
Cosine schedule / reported checkpoint900 steps / step 4501,200 steps / step 400
Effective adaptation pool1,102 teacher-aligned items after sequence holdout18,992 items after sequence holdout
Country samplingFixed country weights from a training-split probe, then an item within countryUniform over available countries, then an item within country
Adaptation imagesLast at most two frames; each ≤200,704 pixelsLast at most two frames; each ≤6 tiles of 448 × 448 plus thumbnail
Rollout generationTemperature 1.0, top-p 1.0, ≤128 new tokensSame
Other settingsGradient clip 1.0, bf16, gradient checkpointing, A100-80GB, seed 17Same
Benchmark evaluationMerged adapter, all available frames up to three, greedy decoding, ≤128 tokens, all 10,143 test items and the same judge; Overall is the equal-weighted mean of 32 country–task cellsSame

VLA implementation. Each preceding backbone receives a new planner LoRA under the common within-backbone training recipe below. The single-run results in submitted Table 4 remain unchanged; the additional three-seed planner runs are reported separately from those original values. The manifests contain 23,930 training keyframes from 700 scenes and 5,119 evaluation keyframes from 150 scenes. Every record requires six future targets in the same scene, excluding its final six keyframes. There is one record per retained keyframe, and the planner’s training and evaluation scenes are disjoint. The 2 s input history and 3 s output horizon are distinct.

VLA componentActual configuration
Image inputOne CAM_FRONT keyframe; Qwen ≤200,704 pixels; InternVL ≤6 tiles of 448 × 448 plus thumbnail
Vehicle stateCAN-bus signed forward speed, three-axis acceleration and angular rate; raw vertical acceleration includes gravity
HistoryFour positions at −2.0, −1.5, −1.0, −0.5 s in the current ego frame; fewer at scene start
High-level commandStraight / left / right derived from recorded future lateral displacement at 3 s, using ±2 m thresholds
OutputSix textual waypoints at 0.5–3 s; metres, x forward and y left
Trainable architectureNew language-model LoRA, rank 64, alpha 64, dropout 0.05; Qwen 43.6M / InternVL 69.7M trainable parameters; no separate action head
Frozen parametersPreceding backbone, including merged DriveOPD weights, vision encoder, and projector
Loss and optimizerWaypoint-token cross-entropy; prompt and end-of-turn masked; AdamW (0.9, 0.999), weight decay 0
Training scheduleLearning rate 10−410^{-4}, batch 8, 50 warm-up steps, cosine decay, 3,000 updates; clip 1.0, bf16, gradient checkpointing
Checkpoint and restartsFinal step 3,000 in every arm; Qwen restarts AdamW at steps 900/1,800/2,700 identically across its arms
Additional repeated runsSeeds 17/18/19 vary planner LoRA initialization and sample order; preceding backbone checkpoint stays fixed
Missing CAN state507 training keyframes retained with zero-filled state; every evaluation keyframe has CAN state
DecodingGreedy, ≤96 new tokens; parse first six coordinate pairs; no malformed outputs occurred
L2 scoringAt 1/2/3 s, mean Euclidean error over the first 2/4/6 waypoints; Avg. is the mean of those three metrics; aggregate over keyframes, overall and per city
Collision scoring4.084 m × 1.730 m rectangle centered at each predicted ego-pose point, oriented along the predicted segment; separating-axis intersection with future annotated vehicles, cyclists, and pedestrians; cumulative collision within each horizon
Planner splitBoston keyframes (scenes)Singapore keyframes (scenes)Total keyframes (scenes)
Training13,355 (390)10,575 (310)23,930 (700)
Evaluation2,628 (77)2,491 (73)5,119 (150)

Adaptation-to-evaluation separation. Downstream planner train/evaluation disjointness does not establish disjointness from preceding DriveAtlas adaptation. Exclusion was not enforced at the parent adaptation-pool level: that pool contains 204 images from eight Singapore nuScenes validation scenes, covering 273 evaluation keyframes; no Boston scene overlaps. Since the actual DriveOPD pools are filtered subsets, these counts do not establish checkpoint-specific exposure, which has not been verified. The current results therefore do not establish scene-disjoint adaptation transfer. We interpret them as observed open-loop planning improvements under the reported setup, with possible scene exposure and visual familiarity remaining limitations of any regional-rule attribution.

Reviewer_F2mUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DriveAtlas, a benchmark for driving rule reasoning across eight countries, comprising perception, prediction, planning, and regional knowledge tasks. Questions are grounded in traffic regulations and filtered through counterfactual rule substitution, with country labels withheld. Evaluations reveal substantial performance disparities across countries. The paper also proposes DriveOPD, which distills a handbook conditioned teacher into a model without handbook access. Experiments demonstrate improved benchmark accuracy, reduced country disparities, and better open loop trajectory prediction on two nuScenes city subsets.

Critical strengths
  1. The benchmark extends driving evaluation to regional traffic rules across eight countries. Grounding questions in regulatory provisions and filtering them through counterfactual rule substitution provides a useful foundation for studying how models combine scene understanding with jurisdiction-dependent knowledge. 
  2. The evaluation includes sequence-disjoint splits, sequence-level bootstrapping, text-only and mismatched-image controls, and human and independent model audits of the answer judge. These checks strengthen the benchmark assessment and help characterize its limitations.
Critical weaknesses
  1. The authors make withheld jurisdiction information and internalized rule knowledge central to the evaluation, primarily to test their defined “visual–cultural reasoning” capability. However, in autonomous driving, where localization and regulatory retrieval are available, the paper does not sufficiently justify the need for this restricted setting.

  2. Country and source corpus are largely confounded. Cross-country disparity therefore mixes differences in regulatory knowledge, visual conditions, and question difficulty. The within-country controls do not resolve this issue, limiting the metric’s interpretation as an indicator of regional rule reasoning.

  3. DriveOPD is compared only with unadapted backbones, without component ablations. These comparisons establish improvement after training but provide insufficient evidence for the advantages of the proposed training design over simpler alternatives, such as supervised fine-tuning on reference answers or offline distillation using teacher-generated answers.

Questions for authors
  1. Why is visual–cultural reasoning necessary for autonomous driving? What driving scenarios require withholding explicit region labels and preventing models from looking up local traffic rules?
4 comments

1. Motivation and Evaluation Setting (W1, Q1)

W1/Q1: Why is visual–cultural reasoning necessary when localization and regulatory retrieval are available? What driving scenarios require withholding explicit region labels and preventing rule lookup?

We thank the reviewer for asking us to clarify the practical motivation. Prior work provides evidence that driving-oriented language supervision and access to regional rules can support planning. DriveCombo studies rules from five countries and reports planning benefits from rule-oriented QA fine-tuning. Impromptu VLA combines planning-oriented QA and trajectory supervision across multiple source datasets and improves downstream planning. Driving Everywhere (LLaDA) demonstrates handbook-assisted adaptation of existing plans, while Alpamayo 2 Super reports strong performance across driving tasks, including leading LingoQA performance in NVIDIA’s evaluation. These complementary results motivate a more targeted assessment of what driving models understand across geographic environments and regulatory systems.

Prior workEvidence relevant to our motivation
DriveComboBuilds CARLA QA scenarios from five countries’ rules and reports improved nuScenes trajectory prediction after rule-oriented fine-tuning.
Impromptu VLACurates approximately 80K clips from eight source datasets with planning-oriented QA and trajectory supervision. Adding the Impromptu training stage before nuScenes training improves the 3B model’s average L2 from 0.34 to 0.30 m.
Driving Everywhere / LLaDAUses local-handbook rules for training-free plan adaptation and reports improvements in nuScenes planning. This supports the usefulness of explicit regional knowledge.
Alpamayo 2 SuperSupports planning, reasoning, and VQA; reports a LingoQA score of 79.2, ranking first among the 37 models evaluated by NVIDIA.

DriveAtlas builds on this evidence through a focused diagnostic evaluation of geographic context, general rules, and jurisdiction-dependent rules. Strong aggregate driving-QA or planning performance leaves open whether models recognize geographic cues, follow shared rules, and correctly apply requirements that vary by region. Knowing the country or retrieving a handbook still leaves the challenge of grounding the applicable provision in the observed scene. Our error taxonomy distinguishes geographic misclassification, cultural rule gaps, general rule violations, visual failures, and reasoning errors. Withholding country labels is a diagnostic choice for examining the model’s existing knowledge; reliable localization and retrieval should be used when available. We also evaluate explicit support: Country-Given supplies the correct country name, while Whole-Handbook-Given supplies both that name and its complete handbook. Country-Given changes accuracy by −0.4 to +2.0 points, whereas handbook access improves all three models by 10.3–18.2 points over Direct. These settings characterize model behavior with different levels of regional support. We will clarify this evaluation goal and its distinction from deployment requirements. Values use the paper’s macro accuracy (%) ↑ (equal-weighted over the 32 country–task cells); the Direct values match submitted Table 2.

SettingQwen3.5-9BQwen3-VL-8BQwen2.5-VL-7B
Direct (submitted Table 2)50.442.539.0
Country-Given52.442.140.0
Whole-Handbook-Given68.657.149.3

2. Country–Corpus Confounding and Disparity (W2)

W2: Country and source corpus are largely confounded, so cross-country disparity mixes regulatory knowledge, visual conditions, and question difficulty. The within-country controls do not resolve this issue.

We thank the reviewer for this distinction and agree that the controls do not fully disentangle these factors. DcD_c describes performance dispersion across country–corpus groups after correcting for sampling noise; it does not identify the causal contribution of regional rules. Country differences can reflect image conditions, question composition and difficulty, and models’ existing knowledge. Figure 4c and Appendix Table 11 explicitly distinguish visual misperception, geographic misclassification, cultural rule gaps, general rule violations, and reasoning errors, with country-specific breakdowns. A cultural rule gap concerns a jurisdiction-dependent requirement, such as turn-on-red permissions, whereas a general rule violation concerns a shared obligation, such as respecting a marking that prohibits crossing. The table below reports country-averaged shares of sampled errors under the Reasoning prompt: cultural rule gaps account for 25.6–44.7%, separately from visual failures and general rule violations.

ModelVisual misperceptionGeographic misclassificationCultural rule gapGeneral rule violationReasoning error
Qwen3.5-9B43.1%2.2%44.7%5.0%5.0%
Qwen3-VL-8B46.6%3.4%37.2%10.3%2.5%
Qwen2.5-VL-7B45.6%2.5%36.2%12.8%2.8%
InternVL3.5-8B61.9%1.9%25.6%7.2%3.4%

Complementary evidence and its limits. Figure 4b localizes disparity by regulatory family: signalized intersection control has Dc=16.94D_c=16.94, compared with 1.88 for speed regulation. Table 13a shows mean accuracy falling from 45.7% to 20.6% without images on Qwen3.5-2B/9B; the residual text-only disparity of 4.76 does not identify its source. We also acknowledge the overlapping within-country and cross-country intervals in Table 13b. Separately, a text-conditioned audit of all 10,143 questions identifies 9,566 (94.3%) with an answer change under at least one alternative jurisdiction, 5,006 (49.4%) under at least two, and 2,960 (29.2%) under at least three. The verifier compares seven alternative jurisdictions using the question, image-derived scene description, reference answer, and regulatory context, without inspecting the original image. These are model-audited judgments of item-level rule sensitivity, not independent image-based validation. Neither this audit nor the error taxonomy removes country–corpus confounding. We will make the distinction explicit: the benchmark diagnoses performance across real geographic environments and regional rules, while aggregate disparity remains a descriptive measure.

3. Comparisons with Simpler Training Alternatives (W3)

W3: Comparison only with unadapted backbones does not establish the benefits of DriveOPD over reference-answer supervised fine-tuning or offline distillation using teacher-generated answers.

We thank the reviewer for requesting these comparisons. We additionally evaluate reference-answer SFT and offline teacher-answer distillation, together with country-only and other-country handbook teacher controls. Reference-answer SFT trains directly on the benchmark references; offline distillation trains on greedy answers from a frozen handbook-conditioned teacher. The country-only teacher supplies the correct jurisdiction without rule text, while the other-country handbook teacher receives another country’s handbook. DriveOPD supplies a jurisdiction-name-redacted correct handbook to the frozen teacher and distills its distributions on student-generated answers. All adapted students are evaluated under Direct prompting, without an explicit country label or handbook. The submitted DriveOPD results are preserved below; values are accuracy (%) and DcD_c.

Training methodQwen3.5-2B Acc. ↑ / DcD_c ↓InternVL3.5-2B Acc. ↑ / DcD_c ↓
Base41.0 / 6.9120.9 / 7.07
DriveOPD (submitted Table 3)45.6 / 3.8739.0 / 3.88
Reference-answer SFT42.8 / 5.7534.1 / 6.42
Offline teacher-answer distillation36.9 / 4.6029.2 / 8.31
Country-only teacher42.4 / 7.4934.0 / 8.46
Other-country handbook teacher40.6 / 7.6828.6 / 9.25

DriveOPD has the highest observed accuracy and lowest DcD_c on both backbones among these configurations. It exceeds SFT by 2.8/4.9 accuracy points and offline distillation by 8.7/9.8 points on Qwen/InternVL; relative to SFT, DcD_c falls from 5.75 to 3.87 and 6.42 to 3.88. The two KL controls reuse DriveOPD’s pool, sampler, LoRA configuration, training steps, and twelve/eight sampled answers per update. SFT/offline CE controls use two target answers per update with an untuned learning rate; SFT also excludes audit-rejected references, leaving 1,003/18,202 items versus DriveOPD’s 1,102/18,992. Historical and current handbook versions differ, and the reported Qwen checkpoint was selected using test accuracy. We therefore report the observed configuration-level advantage while acknowledging that the CE comparisons are not fully data- or compute-matched and that the results do not isolate each component’s contribution.

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We understand this is a busy period, and we truly appreciate your efforts.

As the discussion period will close in less than two days, we wanted to kindly check whether our response has addressed your concerns. If there are any remaining questions or points that would benefit from further clarification, we would be very happy to discuss them before the deadline.

Thank you very much for your consideration.

Best regards, The Authors

Reviewer_e76HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DriveAtlas, a benchmark containing 26,539 training and 10,143 test questions built from nine driving datasets and eight countries. The authors try to highlight the geographic relevance of these Q&As, and report uneven accuracy across countries on 16 VLMs. They also propose DriveOPD, which self-distills a handbook-conditioned teacher into a handbook-free student. It slightly improves open-loop planning on nuScenes.

Critical strengths

Detailed diagnostic analysis. As a benchmark paper, beyond reporting aggregate accuracy, the paper breaks down cross-country disparities and rule families. The finding that signalized intersection rules exhibit larger disparities than speed regulation, with the distinction between visual misperception and cultural rule gaps, provides useful insight into where current driving VLMs struggle.

Critical weaknesses
  1. The cross-country findings are difficult to disentangle from current experiments. Country identity is almost confounded with the source dataset, so differences in accuracy may reflect camera characteristics, visual domains, scene composition, or question difficulty rather than regional traffic rules. The paper's own controls reinforce this concern: text-only inputs still yield substantial disparity (D_c = 4.76, Table 13a), and the spread across cities within Cityscapes is not clearly distinguishable from the spread across countries (Table 13b). Also, the rule-swap admission criterion requires an answer change under only one alternative country's rules, so an item may be distinctive to only one of the eight countries. While the results establish performance differences across country–dataset groups, as a benchmark targeting cross-country tasks, they do not establish how much of these differences is attributable to regional-rule reasoning. This weakens the interpretation of the paper's central findings.
  2. The incremental contribution and downstream benefits are unclear. The authors do not include DriveCombo (CVPR 2026) as an important prior work. DriveCombo already studies traffic-rule reasoning across five countries, reports cross-country performance differences, and shows that fine-tuning on traffic-rule data improves downstream open-loop planning. It also provides closed-loop evaluation scenarios. DriveAtlas is different in evaluating real world without explicit country labels and in targeting cross-country consistency through rule-aware distillation. However, it remains unclear whether these differences lead to benefits beyond standard traffic-rule training. In particular, DriveOPD lacks comparisons with standard SFT, no-handbook self-distillation, and incorrect-handbook distillation, making it difficult to attribute its gains specifically to regional-rule internalization. The downstream VLA evaluation in Boston and Singapore is also limited to small improvements and insufficient training details. While the reported reduction in the inter-city planning gap is interesting, stronger controls are still needed to prove that it results from learning region-specific rules rather than general adaptation. Simple example: DriveAtlas’s training data includes nuScenes-Singapore images but no nuScenes-Boston images, which could easily explain the larger gain in Singapore. As DriveOPD and VLA results form one of the paper’s main contributions, this needs much more verification.
Questions for authors
  1. Benchmark validity. How do the authors verify that the questions truly require country-specific traffic rules rather than general visual understanding? It would also be helpful to report how much accuracy improves when the correct country is explicitly given to the model.

  2. DriveOPD effectiveness. How do we know that DriveOPD's improvements come from learning regional traffic rules rather than simply from additional training? Could the authors compare it with standard SFT and self-distillation without a handbook? DriveCombo has already shown that training on traffic-rule data can improve planning, so it would be useful to show what DriveOPD adds beyond standard training.

  3. Downstream evaluation. The VLA results are based on a single run in Boston and Singapore. How consistent are these improvements across different training runs? How to separate the general gains from the rule-related planning? In particular, the larger improvement in Singapore may simply come from the model becoming more familiar with that dataset.

  4. Can we spend more effort on the presentation? The abstract and introduction contain too many details and repeated points that belong in later sections. The example frames in the main Figure 2 are very low-resolution.

5 comments

1. Benchmark Validity and Regional-Rule Reasoning (W1, Q1)

W1/Q1: Cross-country differences may reflect visual domains or question difficulty rather than regional rules. How do the authors verify that questions require country-specific traffic rules, and how much does accuracy improve when the correct country is explicitly given?

We thank the reviewer for the thoughtful feedback and the suggestion to evaluate Country-Given prompting. The paper distinguishes regional-rule errors from other failures. Figure 4c and Appendix Table 11 separate visual misperception, geographic misclassification, cultural rule gaps, general rule violations, and reasoning errors, with country-specific breakdowns. The table below reports country-averaged shares of sampled errors under the Reasoning prompt, assigning each error to its earliest failing stage. A cultural rule gap concerns a jurisdiction-dependent requirement, such as turn-on-red permissions; a general rule violation concerns a shared obligation, such as respecting a marking that prohibits crossing. Cultural rule gaps account for 25.6–44.7% of sampled errors. Appendix F includes a model that identifies Switzerland but applies another jurisdiction’s turn-on-red rule. Performance also depends on visual conditions and question difficulty: removing images reduces mean accuracy from 45.7% to 20.6% on Qwen3.5-2B/9B, while text-only disparity remains 4.76. This control does not identify its source. Figure 4b reports greater disparity for signalized intersection control (16.94) than speed regulation (1.88). Country–corpus confounding and overlapping control intervals therefore limit DcD_c to a descriptive measure, as acknowledged in §3.1.

ModelVisual misperceptionGeographic misclassificationCultural rule gapGeneral rule violationReasoning error
Qwen3.5-9B43.1%2.2%44.7%5.0%5.0%
Qwen3-VL-8B46.6%3.4%37.2%10.3%2.5%
Qwen2.5-VL-7B45.6%2.5%36.2%12.8%2.8%
InternVL3.5-8B61.9%1.9%25.6%7.2%3.4%

Verifying question-level dependence on regional rules. Beyond analyzing model errors, we conduct a text-conditioned counterfactual audit of all 10,143 test questions. Holding the question and image-derived scene description fixed, a separate model verifier assesses whether the correct answer would change under each of the seven alternative jurisdictions. The audit identifies 9,566 questions (94.3%) with at least one answer-changing substitution, 5,006 (49.4%) with at least two, and 2,960 (29.2%) with at least three. Among questions with an identified change, 52.3% change under multiple alternative jurisdictions. A valid answer-changing substitution is sufficient to establish jurisdiction dependence for an item; requiring changes under additional jurisdictions characterizes the breadth of that dependence. These results provide model-audited evidence of rule sensitivity under a fixed scene description, rather than an independent image-based validation of every item. Providing the correct country and regional handbook. Following the reviewer’s suggestion, Country-Given adds only “this scene was recorded in X; answer under X’s rules,” while preserving the Direct answer format. Whole-Handbook-Given supplies the correct country name and its complete handbook. Values below use the paper’s macro accuracy (%) ↑ (equal-weighted over the 32 country–task cells); the Direct values match submitted Table 2. Providing the country alone yields limited, model-dependent changes of −0.4 to +2.0 percentage points. Supplying the complete regional handbook improves all three models, with gains of 10.3–18.2 points over Direct. This contrast supports examining access to regulatory knowledge separately from access to the country label.

SettingQwen3.5-9BQwen3-VL-8BQwen2.5-VL-7B
Direct (submitted Table 2)50.442.539.0
Country-Given52.442.140.0
Whole-Handbook-Given68.657.149.3

Supplying regulations versus DriveOPD adaptation. For the two 2B backbones, we additionally compare Rule-Given, which supplies the correct country and relevant clause, with full-handbook inference and DriveOPD. The DriveOPD student receives only the image and question at inference. Providing the relevant clause raises accuracy from 41.0% to 49.9% and 20.9% to 42.8%, respectively. Full-handbook access also improves both models. DriveOPD approaches the accuracy of full-handbook inference without requiring that context at test time: 45.6% versus 45.2% on Qwen3.5-2B and 39.0% versus 40.1% on InternVL3.5-2B. These results support the usefulness of learning handbook-informed behavior for evaluation without external regulatory text.

SettingRegional information supplied at inferenceQwen3.5-2B Acc. ↑InternVL3.5-2B Acc. ↑
DirectNone41.020.9
Rule-GivenCorrect country and relevant clause49.942.8
Whole-Handbook-GivenCorrect country name and complete handbook45.240.1
DriveOPDNone45.639.0

2. Relationship to DriveCombo and DriveOPD’s Incremental Contribution (W2, Q2)

W2/Q2: How does DriveAtlas differ from DriveCombo? Do DriveOPD’s gains come from learning regional rules rather than additional training? Please compare against standard SFT, handbook-free self-distillation, and incorrect-handbook distillation.

We thank the reviewer for highlighting DriveCombo, an important related work that we will cite and discuss in the revision. DriveCombo focuses on compositional traffic-rule reasoning: it extracts rules from five countries and constructs CARLA scenarios through Rule2Scene. Its L1–L5 hierarchy evaluates individual rules, static and dynamic rule composition, and conflict resolution; Appendix §7.3 explicitly supplies jurisdiction in the question prompt. We acknowledge its cross-country analysis, open-loop planning benefits, and closed-loop scenarios. DriveCombo, §§3–5 and Appendix §§7.3, 9.1–9.2. DriveAtlas evaluates geographic and regulatory diversity together in real imagery. Naturally occurring combinations of signage, road markings, infrastructure, and visual conditions let us test whether models connect geographic context with applicable rules, complementing CARLA’s controlled rule compositions. Withholding country labels in the default setting tests the model’s existing regional knowledge alongside scene understanding. Open-ended questions require models to formulate their own interpretations and decisions, removing answer-option and elimination cues while exposing visual and rule-application errors for the analysis in Figure 4 and Table 11.

DimensionDriveComboDriveAtlas
Visual observationsCARLA-rendered scenariosReal driving imagery from eight countries
Jurisdiction informationExplicitly supplied in the questionWithheld in the default setting
Main evaluation focusL1–L5 compositional rule reasoningCombining geographic cues, scene understanding, and regional-rule knowledge
Answer formatMultiple-choice questionsOpen-ended perception, prediction, planning, and region questions

DriveOPD and adaptation controls. We additionally evaluate reference-answer SFT, offline teacher-answer distillation, a country-only teacher, and a teacher supplied with another country’s handbook. SFT uses the reference answers; offline distillation uses greedy answers from a frozen handbook-conditioned teacher. The country-only teacher receives the correct jurisdiction without rule text. All adapted students are evaluated under Direct prompting without a country label or handbook. DriveOPD has the highest observed accuracy and lowest DcD_c among these configurations on both backbones. It exceeds SFT by 2.8/4.9 accuracy points and offline distillation by 8.7/9.8 points on Qwen/InternVL, respectively. The DriveOPD results below are unchanged from submitted Table 3.

Training methodQwen3.5-2B Acc. ↑ / DcD_c ↓InternVL3.5-2B Acc. ↑ / DcD_c ↓
Base41.0 / 6.9120.9 / 7.07
DriveOPD (submitted Table 3)45.6 / 3.8739.0 / 3.88
Reference-answer SFT42.8 / 5.7534.1 / 6.42
Offline teacher-answer distillation36.9 / 4.6029.2 / 8.31
Country-only teacher42.4 / 7.4934.0 / 8.46
Other-country handbook teacher40.6 / 7.6828.6 / 9.25

Comparison conditions. The country-only and other-country handbook KL controls reuse DriveOPD’s data pool, sampler, LoRA configuration, training steps, and twelve/eight sampled answers per update. DriveOPD exceeds these controls by 3.2/5.0 and 5.0/10.4 accuracy points, respectively, on Qwen/InternVL. The SFT/offline CE comparisons have additional differences: SFT uses 1,003/18,202 items versus DriveOPD’s 1,102/18,992, and CE controls use two target answers per update with an untuned learning rate. Historical and current handbook versions differ, and the reported Qwen checkpoint was selected using test accuracy. The country-only teacher retains jurisdiction information, and another country’s handbook may share the applicable rule. These results support the observed advantage of the full configuration among the tested alternatives; they do not establish fully data- and compute-matched superiority over CE training or isolate name redaction and regional-rule internalization.

3. Downstream Reliability and Transfer (W2, Q3)

Q3: The VLA results are based on a single run in Boston and Singapore. How consistent are these improvements across different training runs? How can general adaptation be separated from rule-related planning gains, particularly given the additional Singapore exposure during DriveAtlas training?

We thank the reviewer for requesting repeated runs and a clearer account of the downstream protocol. We report the submitted Table 4 results alongside additional planner-training runs with seeds 17, 18, and 19. These seeds vary planner LoRA initialization and sample order while keeping each preceding backbone checkpoint fixed. In the additional runs, the worst observed DriveOPD result has lower average L2 than the best baseline result in all four backbone–city comparisons. Submitted single-run values and additional-run summaries are distinguished below. Additional-run values are midpoint ± half-range, not mean ± standard deviation or confidence intervals. All L2 values are in metres; average L2 is the mean of the 1/2/3 s metrics. In submitted Table 4, the Singapore–Boston gap falls from 0.134 to 0.099 m for Qwen and 0.135 to 0.110 m for InternVL. The additional-run midpoints give gaps of 0.1310 to 0.1010 m and 0.1300 to 0.1105 m, respectively. The repeated-run assessment covers L2.

BackboneMethodSubmitted Table 4: Boston L2 ↓Submitted Table 4: Singapore L2 ↓Additional runs: Boston L2 ↓Additional runs: Singapore L2 ↓
Qwen3.5-2BBase0.2880.4220.2890 ± 0.00100.4200 ± 0.0020
Qwen3.5-2BDriveOPD0.2800.3790.2805 ± 0.00250.3815 ± 0.0025
InternVL3.5-2BBase0.3020.4370.3060 ± 0.00300.4360 ± 0.0010
InternVL3.5-2BDriveOPD0.2910.4010.2875 ± 0.00350.3980 ± 0.0030

Common downstream protocol within each backbone. We use a command-conditioned, Impromptu-VLA-style open-loop planner. In the additional runs, both initializations receive the same prompts, planner training configuration, and data. Inputs comprise one CAM_FRONT image, CAN-bus state, four past ego positions at −2.0, −1.5, −1.0, and −0.5 s, and a high-level command derived from recorded future motion. The output is six textual ego-frame waypoints over the next 3 s. A new planner LoRA is trained while the preceding backbone, including merged DriveOPD weights where applicable, remains frozen. The manifests contain 23,930 training keyframes from 700 scenes and 5,119 evaluation keyframes from 150 scenes, with one record per keyframe. Each scene’s last six keyframes are excluded because complete future targets are required. Both cities supply downstream training data, and planner training and evaluation scenes are disjoint. Here, Base denotes a backbone that also receives downstream planner training.

Planner splitBoston keyframes (scenes)Singapore keyframes (scenes)Total keyframes (scenes)
Training13,355 (390)10,575 (310)23,930 (700)
Evaluation2,628 (77)2,491 (73)5,119 (150)

Attribution and adaptation exposure. Common planner supervision does not eliminate differences in preceding adaptation. Exclusion of downstream evaluation scenes was not enforced at the parent DriveAtlas adaptation-pool level: the pool contains 204 images from eight Singapore nuScenes validation scenes, covering 273 of the 2,491 Singapore evaluation keyframes; no Boston scene overlaps. These are parent-pool counts, and do not establish which images entered each filtered DriveOPD checkpoint. Checkpoint-specific exposure has not been verified. The additional runs show consistent reductions in open-loop L2 across the tested planner seeds, while the contributions of regional-rule learning, visual familiarity, and possible scene exposure remain inseparable. We limit the downstream claim accordingly; the results do not establish scene-disjoint adaptation transfer or an exclusively rule-related explanation of the larger Singapore gain.

4. Presentation Quality (Q4)

Q4: The abstract and introduction contain too many details and repeated points that belong in later sections. The example frames in Figure 2 are very low-resolution.

We thank the reviewer for these helpful suggestions. Abstract and introduction. We have shortened the abstract to focus on the motivation, benchmark, main findings, and DriveOPD. The introduction now presents the problem setting, benchmark overview, and contributions more concisely. We removed repeated descriptions of scene mining, coverage statistics, audit percentages, and detailed result breakdowns from these opening sections; the corresponding details remain in the Method and Experiments sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.