acceptodds
Under review as a conference paper at ICLR 2027

AnchorReasoning: A Visual Grounding and Causal Reasoning Dataset in Long-Tail Autonomous Driving Scenarios

Abstract

Vision-language models (VLMs) offer a promising approach to long-tail autonomous driving, but existing driving datasets provide limited supervision for connecting decision-critical visual evidence with reasoning and planning. We introduce AnchorReasoning, a visually grounded reasoning dataset built on WOD-E2E, containing 416,119 annotated frames and 395,379 decision-critical elements across four major categories and 19 fine-grained types. Each frame is organized as a visually grounded chain-of-thought (VG-CoT) that links decision-critical element identification and localization, element attributes and implications, driving-action rationale, and action and trajectory planning. We further develop a curriculum supervised fine-tuning strategy that progressively learns these hierarchical capabilities, together with an object-size-aware grounding metric for evaluating localization quality. Experiments across eight general-purpose, embodied-AI, and AV-specific backbones show that VG-CoT supervision improves grounded reasoning and trajectory prediction. Across models, 5-s ADE and FDE decrease by 71.4% and 60.6%, while RFS Frame and Cluster improve by 27.0% and 27.5%. These gains are achieved with 18.5 fewer reasoning tokens and 0.32 s/frame lower inference latency on average, demonstrating the value of visually grounded, decision-focused supervision for VLM reasoning and planning in long-tail autonomous driving.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Decision-Critical Elements, Perception-to-Planning, Curriculum Learning, Trajectory Prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y2uPOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AnchorReasoning, a driving dataset built on WOD-E2E with annotations for decision-critical elements, visual grounding, impact ranking, element-level implications, driving rationales, and action/trajectory planning. It also proposes a curriculum supervised fine-tuning strategy and evaluates the resulting supervision across eight VLM/VLA backbones. The main contribution is a structured annotation resource connecting object-level evidence with driving explanations and planning.

Critical strengths

1.The annotation pipeline includes substantial human involvement, including critical-element selection, cross-review, and correction of generated implications, rationales, and action plans.

2.Experiments cover a broad set of backbones and show improvements in both trajectory prediction and the official RFS metrics.

3.The grounding ablation in Table 4 provides useful evidence that grounded supervision improves planning beyond the corresponding nongrounded CoT variant.

Critical weaknesses

1.Table 4 shows relatively modest RFS improvements for some models, such as Qwen3-VL (+0.10) and Cosmos-R2-2B (+0.21). Could the authors provide one or two additional experiments, such as challenging-scenario evaluations or impact-ranking ablations, to further demonstrate the benefits of visual grounding?

2.The proposed annotation pipeline appears similar to DriveLM, nuReasoning. In particular, Table 1 marks DriveLM as lacking visual grounding, although it uses coordinate-based object references, while nuReasoning also incorporates spatial grounding and decision reasoning. Could the authors clarify the key differences, especially how AnchorReasoning uniquely connects visual grounding, object-level reasoning, and trajectory planning?

3.Appendix A.1 states that 456 validation frames were annotated, but their representativeness remains unclear. Could the authors provide a comparison of training and validation distributions, particularly regarding critical object categories, driving scenarios, and action types?

Questions for authors

1.How many of the 456 validation frames contain rare events or interactions involving multiple critical elements, and what is the difficulty distribution of these scenarios?

2.Table 1 marks DriveLM as lacking visual grounding, although it uses coordinate-based object references in perception, prediction, and planning QA. nuReasoning also includes spatial grounding and decision reasoning. Rather than treating visual grounding as a binary distinction, the authors should clarify what AnchorReasoning adds beyond these datasets, particularly in how grounding, object-level reasoning, and trajectory planning are connected.

5 comments
Reviewer_Y2uPOpenReview ↗

Suggestions for the Rebuttal

I apologize for revising some of my review comments, as a few of my previous questions may have been difficult to address. I hope the authors can focus on the first two weaknesses during the rebuttal. If these concerns are adequately addressed, I would be happy to raise my score.

AuthorsOpenReview ↗

Response to Suggestions for the Rebuttal

Dear Reviewer Y2uP,

Thank you for the clarification, and no apology is needed. As suggested, we focused on the first two weaknesses and have revised the paper accordingly. Our detailed responses to each point are provided below, together with brief answers to the remaining questions.

Best regards,

The Authors of AnchorReasoning

AuthorsOpenReview ↗

Response to Reviewer Y2uP part 1

We thank the reviewer for the constructive questions on the benefit of visual grounding, the relation to prior datasets, and the composition of our validation set. We have added challenging-scenario evaluations and distribution analyses in response, as detailed below.

W1: further evidence for the benefit of visual grounding

Visual grounding brings the most benefit in challenging long-tail scenes, where the driving decision depends on identifying the right decision-critical elements. The modest overall gain averages these scenes with simpler ones, where the correct action is often evident without grounding. On five challenging subsets of the validation set, VG-CoT improves RFS over its no-grounding ablation by +0.22 to +0.43 on average across three backbones.

Each subset targets a situation in which finding and interpreting the right elements matters: longitudinal transitions (the ego vehicle starts, stops, or decelerates); rare element states or intentions (e.g., turning, lane changing, backing, pulling out); multiple element categories (critical elements from at least two categories); the highest difficulty level (multiple critical elements, rare content, and a longitudinal transition together); and single-lane maneuvers (an official WOD-E2E cluster). We compare VG-CoT with its no-grounding ablation (text CoT without visual grounding) under the same training recipe on Alpamayo-1.5, Qwen3-VL, and Cosmos-R2-2B.

RFS improvement of VG-CoT over the no-grounding ablation.

ScenarioFramesAlpamayo-1.5Qwen3-VLCosmos-R2-2BMean Δ\DeltaRFS [95% CI]
Longitudinal transitions224+0.21+0.20+0.42+0.28 [0.11, 0.44]
Rare element states/intentions100+0.34+0.32+0.20+0.29 [0.08, 0.52]
Multiple element categories112+0.17+0.25+0.23+0.22 [0.05, 0.41]
Highest difficulty level46+0.24+0.47+0.36+0.36 [0.08, 0.69]
Single-lane maneuvers34+0.44+0.53+0.33+0.43 [0.10, 0.80]

The benefit also grows with scene difficulty. Counting three difficulty factors (multiple critical elements, rare content, longitudinal transition), the mean RFS improvement rises from $-$0.04 for frames with none of them to +0.36 for frames with all three. In simple scenes, the correct action is often to keep the current motion, which the no-grounding model already predicts. Some baselines also already reach the maximum rater score, and grounding sometimes yields more cautious trajectories that raters score lower when they prefer continued motion.

These gains are meaningful for WOD-E2E. Published improvements include +0.08 RFS from WOD-E2E fine-tuning, +0.08 from multi-camera inputs, and +0.09 from test-time scaling (Xu et al., Table 3), and +0.08 from RL post-training (Poutine-Base 7.91 vs. Poutine 7.99; Rowe et al.). The gaps between AutoVLA and HMVLM and between HMVLM and Poutine are +0.18 and +0.25 (Xu et al., Table 2). These results use different splits and are not directly comparable, but they put our +0.22 to +0.43 in context. The results are added to Appendix C.8 (highlighted in red).

Reference:

Xu et al. (2026). WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios. CVPR 2026.

Rowe et al. (2025). Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving. arXiv:2506.11234.

Reviewer_KnRLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AnchorReasoning, a vision-language dataset built on WOD-E2E for reasoning and planning in long-tail autonomous-driving scenarios. It contains 416K annotated frames and 395K decision-critical elements, with annotations covering visual grounding, element attributes and implications, impact ranking, driving rationales, actions, and future trajectories. These annotations are organized into a visually grounded chain-of-thought (VG-CoT) that connects scene perception to reasoning and planning. The paper further proposes a two-stage curriculum fine-tuning strategy and an object-size-aware grounding metric. Experiments on eight general-purpose, embodied, and driving-specific VLM/VLA backbones report improvements in grounding, reasoning, and open-loop trajectory prediction.

Critical strengths
  1. The paper is generally well written and easy to follow, with a clear presentation of the dataset construction, annotation schema, training strategy, and evaluation protocol.

  2. Building large-scale, decision-focused annotations on WOD-E2E is meaningful, as long-tail driving scenarios are particularly important for evaluating and improving the robustness of autonomous-driving systems.

  3. The evaluation goes beyond language-based QA metrics by including open-loop trajectory prediction results such as ADE and FDE, providing a more direct assessment of planning-related performance.

Critical weaknesses
  1. Limited novelty and incomplete related work (major). The core idea of linking grounded critical objects, reasoning, decisions, and trajectories has already been explored by DriveLM, EMMA, DriveCoT, OmniDrive, and especially AutoDrive-P³ (ICLR 2026), which is not cited. Table 1 also appears to mischaracterize DriveLM as lacking visual grounding. The main distinction is therefore the application of an established formulation to WOD-E2E with larger-scale annotations.

  2. The experiments do not isolate the value of VG-CoT. The Base models retain their model-specific objectives and output formats, whereas the VG-CoT variants use additional annotations and a different structured output. A controlled comparison using the same backbone, training data, and optimization settings with trajectory-only supervision is missing. Moreover, the proposed curriculum is not compared with standard joint SFT, while Table 4 only ablates visual grounding. It is therefore unclear whether the gains arise from the reasoning supervision, the curriculum strategy, or simply additional fine-tuning.

  3. The headline average improvements may be misleading. The reported 71.4% ADE@5s reduction is computed from the percentage change between the cross-model means and is disproportionately influenced by the large baseline errors of the general-purpose and embodied-AI backbones. In contrast, the ADE@5s improvements on the driving-specialized Alpamayo models are only 11.9% and 15.8%. The paper should report and emphasize per-model relative improvements rather than using a single aggregate number.

Questions for authors

See weaknesses. If the authors appropriately temper their claims to better reflect the scope of the current evidence, I would be willing to raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KnRL part 1

We thank the reviewer for the detailed feedback, in particular for pointing out the missing related work and the problem with our aggregate metric. These comments led us to revise how we position our contribution and report our results. Our responses follow.

W1: novelty and missing related work

We agree that DriveLM, EMMA, DriveCoT, OmniDrive, and AutoDrive-P$^3$ already connect grounding, reasoning, and planning, and we no longer claim this connection as our contribution. We now credit these works and have corrected Table 1, which wrongly marked DriveLM as lacking visual grounding. As the revised Table 1 shows, AnchorReasoning differs in combining three properties at a scale of 416K frames: it is built on real-world long-tail scenarios; it provides element-level supervision, with mask-level grounding, an impact rank, and a driving implication for each decision-critical element; and it links this supervision to action and trajectory planning.

The closest datasets each cover part of this combination. DriveLM provides element-level ranking and implications, but not for long-tail scenarios and on 4,871 frames. nuReasoning covers long-tail scenarios, but without impact ranks or mask-level grounding. Our starting point is that real-world long-tail data are scarce: WOD-E2E records such scenarios but lacks the fine-grained supervision that VLA models need to ground and reason about them. AnchorReasoning adds this supervision to WOD-E2E and evaluates it across eight VLM backbones.

In the revision (highlighted in green), the Introduction acknowledges these prior works and positions AnchorReasoning as building on them. Related Work now discusses EMMA, DriveCoT, OmniDrive, and AutoDrive-P$^3$, and describes our contribution as large-scale, fine-grained supervision of grounded critical elements, their driving implications, and reasoning-to-planning relationships in real-world long-tail scenarios. Table 1 adds the missing works, corrects DriveLM’s entry, and replaces the binary grounding label with separate columns for grounding format, impact rank, element-level implication, and reasoning format.

AuthorsOpenReview ↗

Response to Reviewer KnRL part 2

W2: isolating VG-CoT and comparing the curriculum with standard SFT

We added this controlled comparison. With the same backbone, data, and optimization settings, trajectory-only SFT, textual CoT, and VG-CoT trained with standard SFT (no curriculum) perform similarly, and the last two are often worse. Gains appear only when VG-CoT is combined with the curriculum, so they do not come from additional fine-tuning alone. The ablation covers three backbones and five training settings, now in the expanded Table 4.

Table 4: Controlled component ablations.

BackboneTraining settingADE@1s ↓\downarrowADE@3s ↓\downarrowADE@5s ↓\downarrowFDE@5s ↓\downarrowRFS Frame ↑\uparrowRFS Cluster ↑\uparrow
Alpamayo-1.5-10BTraj-only SFT0.251.393.027.377.617.66
Text CoT, no grounding0.261.473.267.977.397.42
VG-CoT, no curriculum0.261.493.368.147.317.35
VG-CoT, no LT oversampling0.231.352.987.117.777.79
VG-CoT, full0.231.342.947.067.817.83
Qwen3-VL-8BTraj-only SFT0.261.413.117.637.487.49
Text CoT, no grounding0.251.393.137.677.477.49
VG-CoT, no curriculum0.251.423.257.917.347.38
VG-CoT, no LT oversampling0.231.403.107.617.497.52
VG-CoT, full0.221.343.047.547.547.56
Cosmos-R2-2BTraj-only SFT0.231.393.127.667.497.52
Text CoT, no grounding0.231.373.117.607.517.55
VG-CoT, no curriculum0.241.383.117.647.507.53
VG-CoT, no LT oversampling0.221.393.107.607.497.53
VG-CoT, full0.221.353.067.587.567.59

Textual CoT does not consistently beat trajectory-only SFT, and adding grounding under standard SFT does not improve RFS either. The curriculum lets the models use VG-CoT supervision: frame-level RFS rises from 7.31 to 7.77 for Alpamayo-1.5 and from 7.34 to 7.49 for Qwen3-VL, while Cosmos-R2-2B changes little (7.50 vs. 7.49). Long-tail oversampling adds a further +0.04, +0.05, and +0.07 RFS. Compared with trajectory-only SFT, the full method lowers ADE@5s from 3.02 to 2.94 m, 3.11 to 3.04 m, and 3.12 to 3.06 m, and raises frame-level RFS by +0.20, +0.06, and +0.07. The results and analysis are added to Table 4 and Section 5.3.

W3: aggregate ADE@5s improvement

We agree and have replaced the 71.4% aggregate figure with per-model results. That figure was computed from cross-model means and overweights models with large baseline errors.

Table 3 shows that VG-CoT improves trajectory prediction on all eight backbones, with varying magnitude. General-purpose and embodied-AI models gain more, partly because of their higher baseline errors, and driving-specialized models also improve: ADE@5s decreases from 3.37 to 2.97 for Alpamayo-R1 and from 3.49 to 2.94 for Alpamayo-1.5, with corresponding FDE and RFS gains. The Abstract, Introduction, Results, and Conclusion are revised accordingly (highlighted in green).

Q1: tempering the claims

We have tempered the claims as suggested. We now claim novelty for the combination shown in the revised Table 1, not for linking grounding and reasoning itself: AnchorReasoning provides element-level supervision (mask-level grounding, an impact rank, and a driving implication for each decision-critical element), links it to action and trajectory planning, and does so on 416K frames of real-world long-tail scenarios (W1). We attribute the planning gains to specific components through controlled ablations (W2), and we report per-model improvements instead of a single aggregate (W3).

We sincerely thank the reviewer again for the valuable feedback, which has helped us position our contribution more accurately and report our results more transparently. We hope these revisions and additional experiments adequately address the concerns and clarify the contribution of AnchorReasoning.

Best regards,

The Authors of AnchorReasoning

Sorry for my late reply, and thank you for the detailed response and additional experiments. My concerns have been addressed, and I raise my score to weak accept.

We sincerely thank the reviewer for your valuable feedback and constructive suggestions. We greatly appreciate the time and effort you have devoted to evaluating our work. Your insightful comments have been instrumental in improving the quality and clarity of our manuscript.

Reviewer_DEcEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AnchorReasoning and VG-CoT curriculum-based SFT, which allows autonomous driving VLM to first locate key decision-making elements and then reason and plan based on visual evidence. Experiments show that reasoning, trajectory and efficiency are improved. The main contribution is to redefine the supervision objective of driving CoT.

Critical strengths
  1. The research question is clearly defined and has practical value. Addressing the lack of explicit correlation between visual evidence and driving decisions in existing autonomous driving VLMs, the research objective is clear, with a particular focus on complex, long-tail scenarios.
  2. The experiments have broad coverage. They are validated on 8 different types of models and examine visual localization, reasoning, trajectory prediction, and reasoning efficiency simultaneously, rather than just reporting a single metric.
  3. The data annotation system is relatively complete. VG-CoT connects key element location, attribute understanding, impact analysis, decision rationale, and trajectory planning, providing a more systematic supervision compared to simple scene descriptions or question-and-answer sessions.
Critical weaknesses
  1. The data annotations in this paper largely rely on automatic generation by VLM. Although the authors conducted manual review, annotation biases may still exist. Furthermore, the paper uses GPT-5.5 to evaluate the model's inference quality; the lack of sufficient independent evaluation may affect the reliability of the results. It is recommended that the authors further demonstrate the accuracy of the data annotations and the reliability of the inference evaluation through independent manual evaluation or cross-validation.
  2. The paper lacks direct verification of closed-loop driving safety. It mainly verifies the effect through indicators such as trajectory prediction error and RFS, but does not fully demonstrate whether the performance improvement can be translated into a reduction in collision rate, an improvement in driving safety, or an improvement in the pass rate in complex scenarios in actual closed-loop driving.
  3. The paper also adds richer annotation information and new training strategies, so it is not yet possible to determine whether the performance improvement is due to the effectiveness of visual anchoring itself, or because the model has obtained more supervision information and the training method has changed.
  4. The paper primarily explains the potential shortcomings of other methods but does not directly demonstrate, through rigorous controlled experiments, that models trained with existing CoT annotations perform worse. For example, the paper does not adequately answer the question: if the same base model, the same training data, and the same training budget are used, how much difference will there be in the final results when training with standard CoT, visually anchored CoT, and CoT on other existing datasets respectively?
Questions for authors
  1. What are the benefits of the proposed method for closed-loop driving? Can the authors provide closed-loop simulation results to demonstrate improvements in driving safety, success rate, and planning performance?
  2. What are the main sources of performance improvements? Can the authors conduct additional ablation studies to disentangle the contributions of visual grounding, structured CoT supervision, curriculum learning, and long-tail sample reweighting?
  3. How does the proposed VG-CoT compare with existing CoT annotation methods? Can the authors conduct controlled experiments under comparable training settings to demonstrate the advantages of VG-CoT over conventional textual CoT and other existing supervision strategies in reasoning quality, visual grounding, and trajectory prediction?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DEcE part 1

W1: reliability of the annotations and the GPT-5.5 judge

Both are supported by independent checks. All decision-critical annotations are produced and cross-checked by humans. The GPT-5.5 judge passes three tests: it agrees with a blinded human rater (87.1% for cause, κ\kappa = 0.70; 89.3% for effect, κ\kappa = 0.75). It ranks the models consistently with an independent open-weight judge, GLM-4-32B, and when we deliberately make a correct explanation wrong, it rejects 89–95% of these wrong versions while accepting all paraphrases of the correct one. The judge therefore checks meaning rather than wording.

On the annotations, AnchorReasoning does not rely solely on VLM outputs. Humans identify and rank decision-critical elements, provide grounding prompts, and annotate their states and intentions (Section 3.3). Each selection and ranking is reviewed by a second annotator, and a third annotator resolves disagreements. Human annotators also inspect and correct the VLM-generated implications, rationales, and action plans (Appendix B.5), and action plans are checked against future trajectories for geometric and kinematic consistency (Appendix B.6). The automatic components agree closely with human labels (Appendix B): scene context on 7,450 frames (76.51–100% across attributes), traffic lights on 11,318 instances (97.81%), object location on 4,439 instances (98.42%), and vehicle type on 3,720 instances (98.95%).

On the evaluation, we ran three additional checks. First, a human rater, blinded to model identity and to the GPT-5.5 judgments, scored 140 prediction–reference pairs; GPT-5.5 agreed on 87.1% of cause judgments (κ\kappa = 0.70) and 89.3% of effect judgments (κ\kappa = 0.75). Second, we re-scored 12,935 items with an independent open-weight judge, GLM-4-32B. Although the two judges differ in strictness, they rank the models consistently (Spearman ρ\rho = 1.00 for Element Implication and 0.88 for Driving Action Rationale). Both judges find Element Implication gains for all six comparable models. For Driving Action Rationale, GPT-5.5 finds gains for all six and GLM-4-32B for five; the exception is Alpamayo-R1, which changes negligibly under either judge. Third, in perturbation tests on 150 implications and 150 rationales, GPT-5.5 accepted all identical and paraphrased references, while rejecting 93–95% of reversed effects and 89–95% of wrong referents in cause judgments. The judge therefore responds to semantic errors rather than to textual overlap. Protocols and results are added to Appendix B.7 and Appendix C.5.

W2 + Q1: closed-loop driving safety

AnchorReasoning is a dataset for teaching VLA models to identify and reason about the elements that drive decisions in real-world long-tail scenarios. Open-loop evaluation on recorded scenes measures these capabilities directly, so it fits this goal. Closed-loop evaluation is an important next step, which we now discuss in the paper.

Long-tail scenarios remain a central challenge for autonomous driving: they are rare, costly to collect, and hard to reproduce realistically in simulation. WOD-E2E is valuable because it records such scenarios from real driving. However, it provides only camera observations and trajectories, without supervision on which elements matter, where they are, and how they should shape the ego vehicle’s action. AnchorReasoning fills this gap. It annotates decision-critical elements with their visual grounding, states, intentions, and driving implications, and links them to frame-level rationales, action plans, and the recorded future trajectory.

Because the goal is better grounding, reasoning, and planning on real long-tail data, we evaluate exactly these on the recorded scenes, across eight backbones. In addition to ADE and FDE, we report the official WOD-E2E RFS, which scores predicted trajectories against human-rated trajectories and therefore reflects whether a trajectory is acceptable, not only how close it is to the logged one.

We agree that open-loop gains do not guarantee closed-loop safety. Testing this requires an interactive simulator that reacts to the model’s actions, which the fixed WOD-E2E recordings cannot provide. We have added this to the Limitations and Future Work section (highlighted in blue).

AuthorsOpenReview ↗

Response to Reviewer DEcE part 2

W3 + Q2: sources of the performance gains

The gains do not come from extra supervision or extra fine-tuning alone. Under identical training settings, adding textual CoT, or VG-CoT without the curriculum, does not consistently improve planning over trajectory-only SFT. Improvements appear when VG-CoT is trained with the curriculum, and removing visual grounding from that setting reduces performance on all three backbones.

Two ablations support this. Without grounding, ADE@5s rises from 2.94 to 3.26 for Alpamayo-1.5, from 3.04 to 3.13 for Qwen3-VL, and from 3.06 to 3.11 for Cosmos-R2-2B, with consistent changes in FDE and RFS. These implementation details are added to the ablation description (highlighted in blue). We also added a component ablation on the same three backbones, now included in Table 4.

Table 4: Controlled component ablations.

BackboneTraining settingADE@1s ↓\downarrowADE@3s ↓\downarrowADE@5s ↓\downarrowFDE@5s ↓\downarrowRFS Frame ↑\uparrowRFS Cluster ↑\uparrow
Alpamayo-1.5-10BTraj-only SFT0.251.393.027.377.617.66
Text CoT, no grounding0.261.473.267.977.397.42
VG-CoT, no curriculum0.261.493.368.147.317.35
VG-CoT, no LT oversampling0.231.352.987.117.777.79
VG-CoT, full0.231.342.947.067.817.83
Qwen3-VL-8BTraj-only SFT0.261.413.117.637.487.49
Text CoT, no grounding0.251.393.137.677.477.49
VG-CoT, no curriculum0.251.423.257.917.347.38
VG-CoT, no LT oversampling0.231.403.107.617.497.52
VG-CoT, full0.221.343.047.547.547.56
Cosmos-R2-2BTraj-only SFT0.231.393.127.667.497.52
Text CoT, no grounding0.231.373.117.607.517.55
VG-CoT, no curriculum0.241.383.117.647.507.53
VG-CoT, no LT oversampling0.221.393.107.607.497.53
VG-CoT, full0.221.353.067.587.567.59

Textual CoT does not consistently beat trajectory-only SFT, and adding grounding without the curriculum does not improve RFS either. The curriculum lets the models use VG-CoT supervision: frame-level RFS rises from 7.31 to 7.77 for Alpamayo-1.5 and from 7.34 to 7.49 for Qwen3-VL, while Cosmos-R2-2B changes little (7.50 vs. 7.49). Long-tail oversampling adds a further +0.04, +0.05, and +0.07 RFS. Compared with trajectory-only SFT, the full method lowers ADE@5s from 3.02 to 2.94 m, 3.11 to 3.04 m, and 3.12 to 3.06 m, and raises frame-level RFS by +0.20, +0.06, and +0.07. The analysis is added to Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer DEcE part 3

W4 + Q3: comparison with existing CoT supervision

Tables 2–3 already include this comparison for three existing CoT-based VLA models: Alpamayo, AutoVLA, and Impromptu-VLA. For each, we compare the model trained with its own original CoT supervision against the same backbone trained with VG-CoT. In all three cases, VG-CoT improves both reasoning and planning. A fully matched comparison, which would re-annotate the same WOD-E2E samples following each method’s CoT scheme, is not feasible within the rebuttal period, so we present these results as preliminary evidence.

The three methods use different forms of reasoning supervision. Alpamayo is trained with Chain-of-Causation reasoning traces, which explain the driving decision through its causal factors in text. AutoVLA uses CoT generated by a large VLM that describes the scene, the critical objects, the driving intention, and the meta-action, and learns to switch between answering directly and reasoning first. Impromptu-VLA uses planning-oriented question–answer pairs on scene description, traffic signals, vulnerable road users, motion intentions, and meta-actions. All three express reasoning mainly as text. VG-CoT differs in that each decision-critical element is linked to its visual location and its own implication for the ego action, and these implications are then combined into the rationale, action plan, and trajectory.

In Tables 2–3, the value before the slash is the Base model with its original supervision, and the value after the slash is the same backbone trained with VG-CoT. For Alpamayo and Impromptu-VLA, we start from the released checkpoints and adapt them to WOD-E2E with their original training recipes, keeping their native reasoning supervision and output formats and using none of our VG-CoT annotations. AutoVLA was already trained on Waymo data with its own CoT supervision, so its released model serves as the Base directly. The two settings therefore differ mainly in the reasoning supervision.

With VG-CoT, ADE@5s decreases from 3.49 to 2.94 for Alpamayo-1.5, from 5.01 to 3.60 for AutoVLA, and from 5.96 to 3.17 for Impromptu-VLA, with corresponding gains in reasoning scores and RFS (Table 3). On Alpamayo-1.5, for example, Element Implication improves from 0.69 to 0.84 and Driving Action Rationale from 0.61 to 0.80. VG-CoT also adds explicit visual grounding, which several of the original models lack (Table 2).

A stricter comparison would build Alpamayo- and Impromptu-VLA-style CoT annotations on the same WOD-E2E samples and train each under matched settings, which requires annotation effort and compute beyond the rebuttal period. Within our own method, the ablations in W3 + Q2 isolate the contribution of grounding under identical training settings.

We sincerely thank the reviewer again for the valuable feedback, which has helped us further strengthen the reliability of our annotations and evaluation. We hope these additional experiments and clarifications adequately address the concerns and provide stronger evidence for the effectiveness of AnchorReasoning.

Best regards,

The Authors of AnchorReasoning

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.