- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DriveAtlas evaluates driving-rule reasoning across eight countries using nine public driving corpora. It contains 10,143 test and 26,539 training questions, with country labels withheld and a rule-swap filter intended to retain jurisdiction-dependent items. The paper evaluates 16 generalist VLMs and introduces DriveOPD, which distills a frozen, handbook-conditioned copy of a model into a student without handbook access. Experiments report improved QA accuracy and reduced cross-country disparity on two backbones, with additional open-loop planning results in Boston and Singapore.
Critical strengths
The benchmark addresses a useful evaluation gap with meaningful validity controls. Withholding jurisdiction and filtering for rule-dependent answers offers a more focused test than ordinary multi-country VQA. Sequence-disjoint splits, sequence-level bootstrapping, image controls, and explicit analysis of country–corpus confounding strengthen the evaluation.
The paper provides several complementary checks on its findings. The answer judge is audited against a human expert and two unrelated model lineages. DriveOPD improves all four tasks on two backbones while reducing disparity, and the open-loop VLA results provide an initial connection to trajectory prediction. The benchmark is the strongest contribution in my assessment.
Critical weaknesses
Table 3 compares the trained student only with its unadapted backbone. There is no matched-data, matched-budget QA fine-tuning or handbook-free distillation control. Since the method explicitly builds on existing context and on-policy distillation recipes, these comparisons matter for determining whether the gains come specifically from internalizing regional rules rather than general adaptation to the benchmark. The PDF also provides insufficient training and VLA implementation details to reconstruct the downstream results.
The appendix examples do not show which alternative jurisdiction and corresponding clause change each answer. For example, providing the actual counterfactual for the German no-entry item would clarify the admission criterion. Table 7 also retains 192 overtaking candidates marked undetermined at the geometry-audit stage. Although human reference verification is stated, their final inclusion and adjudication are unclear. This does not establish that these references are wrong, but leaves uncertainty about the benchmark’s most distinctive property and remaining label quality.
Questions for authors
How does DriveOPD compare with matched-budget QA fine-tuning or handbook-free distillation? Can the contributions of handbook conditioning and jurisdiction-name redaction be separated?
Can the authors provide concrete rule-swap clauses and changed answers for the appendix examples, and clarify the final treatment of undetermined audit candidates? Are the main findings stable when unresolved candidates are excluded?
Please specify the training hyperparameters and VLA architecture, inputs, splits, and scoring protocol. Are the nuScenes downstream test sequences excluded from DriveAtlas adaptation training?
4 comments
2. Counterfactual Examples and Undetermined Candidates (W2, Q2)
W2/Q2: Please provide concrete alternative jurisdictions, clauses, and changed answers for the appendix examples, particularly the German no-entry item. What happened to the 192 undetermined overtaking candidates, and are the findings stable when unresolved candidates are excluded?
We thank the reviewer for the careful reading and agree that the appendix should identify the alternative jurisdiction, applicable clause, and changed answer. The comparisons below distinguish changes in rule interpretation or legal permission from sign-format mismatches and unchanged answers. The German no-entry panel illustrates a sign-grounding failure; its UK comparison preserves “do not enter”. An answer-changing counterpart for the underlying German item is not established by these comparisons, so its item-specific admission remains unsubstantiated here. The UK speed-limit example illustrates a unit change. The Swiss panel asks what the car must do before turning right when the signal is red and no green arrow is shown; its comparison concerns the applicable legal requirements under those signal conditions. These clause comparisons are not recovered original item-level admission records.
| Appendix example | Original jurisdiction and applicable provision | Substituted jurisdiction and answer | What the comparison establishes |
|---|---|---|---|
| Fig. 7, UK: circular “20” speed-limit sign | UK: The sign indicates 20 mph, following the UK speed-sign convention. | Germany: Under StVO, Zeichen 274, the corresponding sign indicates 20 km/h. | Answer changes: the unit and legal speed differ despite the same numeral. |
| Fig. 7, US: pentagonal school-warning sign | US: The pentagonal school sign warns of a school area and children, following MUTCD Part 7. | UK: The prescribed school-warning sign is a red-bordered triangle (DfT guidance); the observed pentagon does not match that format, although its warning remains recognizable. | Sign-system mismatch: this does not establish a changed precaution or driving action. |
| Fig. 7, Germany: red no-entry sign | Germany: StVO, Zeichen 267 prohibits entry from the indicated direction. | UK: The no-entry sign imposes the same obligation: do not enter. | Unchanged comparison: this figure illustrates sign grounding, not an established answer-changing substitution. |
| Fig. 8, Switzerland: what must the car do before turning right at red, with no green arrow? | Switzerland: The car must stop and remain stopped until a green light or an authorizing green arrow permits the turn; stopping alone does not authorize an ordinary right turn on red (SSV Art. 68). | China: Art. 38 permits a right turn at a circular red signal without impeding traffic permitted to proceed or pedestrians; this does not override a prohibiting directional signal. US—Massachusetts: Ch. 89 §8 permits a right turn after stopping and yielding unless prohibited by a posted restriction. | Legal permission changes, conditional on the signal and intersection restrictions. |
We thank the reviewer for pointing out the overtaking ambiguity. These 192 cases involve two- or multi-lane roads where overtaking depends on traffic direction and the ego vehicle’s lane. For example, when ego already occupies the leftmost lane of a multi-lane one-way road, whether it may pass on the right or must remain behind depends on local rules and scene geometry. We manually reviewed these cases and corrected their reference answers where necessary. The automated “undetermined” flags therefore do not indicate unresolved reference labels in the benchmark.
3. Training Details, VLA Protocol, and Split Integrity (W1, Q3)
Q3: Please specify the training hyperparameters, VLA architecture, inputs, splits, and scoring protocol. Were the downstream nuScenes evaluation sequences excluded from DriveAtlas adaptation training?
We thank the reviewer for identifying these reproducibility gaps. DriveOPD uses the initial backbone as a frozen teacher with adapters disabled. The teacher receives the frames, question, and country-name-redacted handbook; the student receives the Direct prompt. Reverse KL is computed on student-sampled answers, with both distributions renormalized over the teacher’s top-64 tokens, averaged within each answer and then across answers; the end-of-turn token is excluded. Language-model LoRA targets MLP and q/k/v/o projections where present, with vision components frozen. The implementation details below correct the uniform rank-32/four-rollout description without changing submitted Table 3 results. The actual image-resolvable pools exclude Japanese items and part of the UK data; Qwen further selects teacher-aligned training questions. The reported Qwen step-450 checkpoint was selected among variants and checkpoints using test accuracy; its later validation ranking is a post-hoc check. InternVL step 400 was the only checkpoint scored from its run. These selection conditions limit the comparison and will be stated explicitly.
| DriveOPD configuration | Qwen3.5-2B | InternVL3.5-2B |
|---|---|---|
| Teacher handbook in the reported Table 3 run | Earlier base handbook plus sections generated from training items | Base handbook |
| LoRA rank / alpha / dropout | 64 / 64 / 0.05 | 32 / 64 / 0.05 |
| Trainable parameters | 43.6M | 34.9M |
| Rollouts per prompt / prompts per update | 6 / 2; 12 answers per update | 4 / 2; 8 answers per update |
| Optimizer | AdamW, beta = (0.9, 0.95), weight decay 0 | Same |
| Learning rate / warm-up | / 20 steps | / 20 steps |
| Cosine schedule / reported checkpoint | 900 steps / step 450 | 1,200 steps / step 400 |
| Effective adaptation pool | 1,102 teacher-aligned items after sequence holdout | 18,992 items after sequence holdout |
| Country sampling | Fixed country weights from a training-split probe, then an item within country | Uniform over available countries, then an item within country |
| Adaptation images | Last at most two frames; each ≤200,704 pixels | Last at most two frames; each ≤6 tiles of 448 × 448 plus thumbnail |
| Rollout generation | Temperature 1.0, top-p 1.0, ≤128 new tokens | Same |
| Other settings | Gradient clip 1.0, bf16, gradient checkpointing, A100-80GB, seed 17 | Same |
| Benchmark evaluation | Merged adapter, all available frames up to three, greedy decoding, ≤128 tokens, all 10,143 test items and the same judge; Overall is the equal-weighted mean of 32 country–task cells | Same |
VLA implementation. Each preceding backbone receives a new planner LoRA under the common within-backbone training recipe below. The single-run results in submitted Table 4 remain unchanged; the additional three-seed planner runs are reported separately from those original values. The manifests contain 23,930 training keyframes from 700 scenes and 5,119 evaluation keyframes from 150 scenes. Every record requires six future targets in the same scene, excluding its final six keyframes. There is one record per retained keyframe, and the planner’s training and evaluation scenes are disjoint. The 2 s input history and 3 s output horizon are distinct.
| VLA component | Actual configuration |
|---|---|
| Image input | One CAM_FRONT keyframe; Qwen ≤200,704 pixels; InternVL ≤6 tiles of 448 × 448 plus thumbnail |
| Vehicle state | CAN-bus signed forward speed, three-axis acceleration and angular rate; raw vertical acceleration includes gravity |
| History | Four positions at −2.0, −1.5, −1.0, −0.5 s in the current ego frame; fewer at scene start |
| High-level command | Straight / left / right derived from recorded future lateral displacement at 3 s, using ±2 m thresholds |
| Output | Six textual waypoints at 0.5–3 s; metres, x forward and y left |
| Trainable architecture | New language-model LoRA, rank 64, alpha 64, dropout 0.05; Qwen 43.6M / InternVL 69.7M trainable parameters; no separate action head |
| Frozen parameters | Preceding backbone, including merged DriveOPD weights, vision encoder, and projector |
| Loss and optimizer | Waypoint-token cross-entropy; prompt and end-of-turn masked; AdamW (0.9, 0.999), weight decay 0 |
| Training schedule | Learning rate , batch 8, 50 warm-up steps, cosine decay, 3,000 updates; clip 1.0, bf16, gradient checkpointing |
| Checkpoint and restarts | Final step 3,000 in every arm; Qwen restarts AdamW at steps 900/1,800/2,700 identically across its arms |
| Additional repeated runs | Seeds 17/18/19 vary planner LoRA initialization and sample order; preceding backbone checkpoint stays fixed |
| Missing CAN state | 507 training keyframes retained with zero-filled state; every evaluation keyframe has CAN state |
| Decoding | Greedy, ≤96 new tokens; parse first six coordinate pairs; no malformed outputs occurred |
| L2 scoring | At 1/2/3 s, mean Euclidean error over the first 2/4/6 waypoints; Avg. is the mean of those three metrics; aggregate over keyframes, overall and per city |
| Collision scoring | 4.084 m × 1.730 m rectangle centered at each predicted ego-pose point, oriented along the predicted segment; separating-axis intersection with future annotated vehicles, cyclists, and pedestrians; cumulative collision within each horizon |
| Planner split | Boston keyframes (scenes) | Singapore keyframes (scenes) | Total keyframes (scenes) |
|---|---|---|---|
| Training | 13,355 (390) | 10,575 (310) | 23,930 (700) |
| Evaluation | 2,628 (77) | 2,491 (73) | 5,119 (150) |
Adaptation-to-evaluation separation. Downstream planner train/evaluation disjointness does not establish disjointness from preceding DriveAtlas adaptation. Exclusion was not enforced at the parent adaptation-pool level: that pool contains 204 images from eight Singapore nuScenes validation scenes, covering 273 evaluation keyframes; no Boston scene overlaps. Since the actual DriveOPD pools are filtered subsets, these counts do not establish checkpoint-specific exposure, which has not been verified. The current results therefore do not establish scene-disjoint adaptation transfer. We interpret them as observed open-loop planning improvements under the reported setup, with possible scene exposure and visual familiarity remaining limitations of any regional-rule attribution.
1. Training Controls and Name Redaction (W1, Q1)
We thank the reviewer for these specific suggestions. We additionally evaluate reference-answer SFT, offline teacher-answer distillation, a country-only teacher, and a teacher given another country’s handbook. SFT uses reference answers, while offline distillation uses greedy answers from a frozen handbook-conditioned teacher. The country-only teacher receives the correct jurisdiction without rule text, and the other-country handbook teacher receives another country’s handbook. All adapted students are evaluated under Direct prompting, without an explicit country label or handbook. DriveOPD achieves the highest observed accuracy and lowest Dc on both backbones among these configurations: it exceeds reference-answer SFT by 2.8/4.9 accuracy points and offline distillation by 8.7/9.8 points on Qwen/InternVL, respectively. Relative to SFT, Dc decreases from 5.75 to 3.87 and 6.42 to 3.88. The DriveOPD values below are unchanged from paper Table 3.
Comparison conditions and component attribution. The country-only and other-country handbook KL controls reuse DriveOPD’s data pool, sampler, LoRA configuration, training steps, and twelve/eight sampled answers per update. The country-only teacher is handbook-free but retains jurisdiction information. SFT/offline CE controls use two target answers per update with an untuned learning rate; SFT additionally removes audit-rejected references, leaving 1,003/18,202 items versus DriveOPD’s 1,102/18,992. Handbook versions differ, and the checkpoint-selection procedure is disclosed under Q3. These conditions limit isolated component attribution and fully matched comparisons with CE training. The individual contribution of jurisdiction-name redaction is not established by these results. Whole-Handbook-Given scores of 45.2%/40.1% use the correct country name and complete handbook at inference; they are not a names-retained training ablation. We limit the method claim to the observed advantage of the tested configuration.