acceptodds
Withdrawn from ICLR 2027

FocusDrive: Reasoning with Visual Focus for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Driving decisions depend on both where to focus and how to act on what is seen. Effective driving reasoning must establish which objects matter, where they are, and how they inform the intended action. Text-based rationales can describe a driving response while leaving its correspondence to specific visual evidence implicit. Visual focus provides a concrete starting point for this connection by identifying what matters in the scene and where it is. We propose FocusDrive, a structured multimodal reasoning framework that organizes end-to-end planning around explicit visual focus. It pairs descriptions of decision-relevant objects with image-patch references, bringing explicit visual focus into the reasoning that generates driving plans and trajectories. We first assess this focus representation through driver gaze prediction, then investigate its role in planning reasoning using driving-focus annotations within existing NAVSIM training scenes. Experiments on W3DA and NAVSIM demonstrate competitive gaze prediction and end-to-end planning performance, with FocusDrive improving over text-based chain-of-thought. These results support visual focus as an effective link between scene understanding and driving action.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_8VB8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FocusDrive, a structured multimodal reasoning framework for autonomous-driving planning. Instead of generating only a textual rationale before a trajectory, the model generates: (i) a description of a decision-relevant object; (ii) a reference to an image patch corresponding to that object; (iii) an action plan; and (iv) eight future waypoints. The patch-reference interface is explicitly adapted from PaDT’s dynamic visual-reference mechanism: current-image patch prototypes are added as autoregressive output choices and subsequently used as input embeddings when selected. The planning model is based on Qwen3-VL-2B and is trained through visual-reference warm-up, planning SFT, and PDMS-rewarded GRPO.

Critical strengths
  1. A coherent and practically meaningful integration of spatial references into driving reasoning

The central design is intuitive: a generated focus object is explicitly linked to a current-image patch before the system generates a driving response. The implementation is technically clean because the same image-conditioned prototype functions as both an output candidate and a subsequent input embedding. This is more structured than merely appending text coordinates, and the Text-2D/reference-ID/patch-prototype ablations are directionally consistent with the intended interpretation.

  1. Strong empirical breadth and unusually detailed implementation reporting

The paper evaluates both dense driver-attention prediction and end-to-end planning, reports component-level NAVSIM metrics, includes matched internal baselines at both SFT and RL stages, and documents prompts and optimization settings. FocusDrive leads most reported W3DA gaze metrics, while the RL planner reaches 91.1 PDMS/EPDMS with deterministic single-trajectory decoding rather than test-time reranking.

Critical weaknesses
  1. The experiments do not isolate the causal contribution of visual patch-reference reasoning

The headline comparison against text CoT conflates several changes: additional decision-relevant object descriptions; spatial focus labels on 13.1% of Navtrain; a new dynamic patch-reference vocabulary; separate object-reference warm-up data and training; image-conditioned selected-region embeddings; and, for the final model, outcome-based RL. Thus, the 1.3/1.4-point SFT improvement over text CoT does not establish that patch references themselves improve planning. Table 5 is more diagnostic: Text-2D already improves over text CoT; replacing coordinates with patch references yields only 0.4 points without warm-up, while the full warmed reference model gains 0.6 points over warmed Text-2D. The reported image-conditioned prototype advantage over the learned-ID variant is 0.3 points. These are potentially meaningful, but no seeds, variance, confidence interval, or paired-scene significance test is provided.

  1. Annotation bias, baseline comparability, and statistical robustness are insufficiently controlled

The focus labels are produced after candidate screening using risk cues and Qwen3.7-Plus with the front image, candidate boxes, and recorded future ego path. Two annotators then revise/select a single patch. The resulting set is highly concentrated: 84.43% of focus objects are vehicles and 89.2% of focus-annotated examples involve deceleration; no inter-annotator agreement, independent duplicate-label rate, rejection rate, or ambiguity analysis is reported. Using future ego motion for training-label construction is not Navtest leakage as described, and the policy does not receive it at inference. Nevertheless, it is privileged offline supervision correlated with future behavior. Since the text-CoT baseline does not receive an equivalently rich focus-label channel, some gain may come from the annotation pipeline rather than the patch-reference representation. The strong deceleration skew is especially relevant because the reported RL model improves collision/TTC-related components while EPDMS progress falls relative to the SFT model. This could represent a safety–progress tradeoff or conservative bias rather than uniformly better planning. External comparisons should also be interpreted cautiously. The paper itself notes that some earlier EPDMS results may not align with the updated NAVSIM implementation and that its results use human-penalty filtering plus an official aggregation correction. Consequently, the external leaderboard tables are supportive context, not clean evidence of superiority. The internal matched comparisons should carry the main burden, but those comparisons currently lack repeated runs and uncertainty analysis.

  1. novelty and causal attribution.

The paper’s high-level premise that decision-relevant visual focus should guide autonomous-driving action is substantially preceded by AEGIS (CHI 2025) [1], which uses human-attention supervision to align policy attention with task-relevant objects and directly maps the resulting attended representation to vehicle control. Meanwhile, FocusDrive’s defining patch-reference interface, including decoding an image-dependent patch token and feeding its prototype back into subsequent reasoning, is adopted from PaDT (ICLR 2026) [2], while FSDrive[3], LLada[4], CoT-VLA[5], AutoVLA[6], and DriveTeach-VLA[7] already cover complementary parts of visual driving reasoning, attention-object explanation, and spatial policy guidance. The remaining contribution therefore appears to be a useful driving-specific composition of these ideas rather than a substantially new reasoning mechanism. Moreover, the current ablations conflate focus annotation, sequence restructuring, reference warm-up, spatial indexing, and image-conditioned feedback; the isolated gain of patch prototypes over learned reference IDs is only 0.3 PDMS/EPDMS. Random/wrong/shuffled-focus and no-feedback controls are needed to establish that the selected visual evidence causally drives the planning improvement rather than functioning as auxiliary supervision or a vehicle/deceleration shortcut.

[1] AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systems. CHI 2025. [2] Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (PaDT). ICLR 2026. [3] FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving (FSDrive). NeurIPS 2025. [4] Where, What, Why: Towards Explainable Driver Attention Prediction (LLada). ICCV 2025. [5] CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models. 2025. [6] AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning. NeurIPS 2025. [7] Teaching Vision-Language-Action Models What to See and Where to Look (DriveTeach-VLA). ECCV 2026.

Questions for authors
  1. Repeated-run robustness: Can the authors report at least three seeds, or paired bootstrap confidence intervals over Navtest, for text CoT, Text-2D, reference-ID, and patch-reference models? Are the 0.3–0.6-point component gains statistically distinguishable from training/evaluation variance?

  2. Matched-supervision control: What happens if the text-CoT baseline receives the same focus-object descriptions and the same object-reference warm-up examples, but patch tokens are removed or masked? This would determine whether the gain is attributable to explicit patch references rather than extra semantic supervision or training.

  3. Focus intervention: How does planning change when the generated/reference patch is replaced at inference with a random patch, a nearby patch, or a plausible wrong-object patch while preserving all text? If performance is unchanged, the claim that selected visual content guides subsequent reasoning would be weakened substantially.

  4. Annotation reliability and privileged information: How many scenes were independently labeled by both annotators, what was agreement on object/category/patch, and how often were VLM candidates rejected or revised? Please also provide an ablation in which candidate construction does not use the recorded future ego path.

  5. Peripheral vision and global scene awareness: How does FocusDrive preserve and utilize peripheral visual information when reasoning is explicitly anchored to a selected image patch? In driving, safety-critical cues may arise outside the primary focus region, such as pedestrians entering from the image periphery or vehicles approaching from adjacent lanes. Does the patch-reference mechanism risk over-concentrating attention on a single object while overlooking other relevant hazards? Please clarify whether global visual context remains accessible during subsequent reasoning, and provide ablations or failure-case analyses involving multiple simultaneous hazards, peripheral objects, and unexpected events outside the selected focus region.

Reviewer_AmJ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FocusDrive, a multimodal reasoning framework for autonomous driving that explicitly connects decision-relevant objects to image patches. By incorporating visual focus into the reasoning sequence, the method aims to better ground driving decisions in visual evidence. The authors evaluate FocusDrive on W3DA for gaze prediction and NAVSIM for end-to-end planning, using visual-reference warm-up, supervised fine-tuning, and GRPO-based reinforcement fine-tuning. Results show strong gaze-prediction performance and improvements over a text-based chain-of-thought (CoT) baseline, suggesting that explicit visual grounding can benefit driving planning.

Critical strengths
  • The paper explores an interesting direction by explicitly linking object descriptions to image patches in driving reasoning. This structured representation potentially improves the connection between scene understanding and action generation, and the results suggest that it can benefit planning beyond conventional text-based CoT.

  • The proposed method achieves strong results on the established W3DA and NAVSIM benchmarks. In particular, it achieves 91.1 on both PDMS and EPDMS after reinforcement fine-tuning, outperforming the text-based CoT baseline. These findings support the potential value of incorporating explicit visual focus into driving VLA models.

Critical weaknesses
  • Some technical details are difficult to follow, particularly how image-patch references are generated and used in subsequent reasoning, and how the GRPO training process operates. A more intuitive explanation with concrete examples would improve clarity. Additionally, the related-work section could better distinguish FocusDrive from existing approaches to gaze prediction, visual grounding, and driving CoT, making the paper's main contributions and novelty clearer.

  • Although the paper demonstrates strong gaze-prediction results, it is unclear how gaze prediction itself contributes to planning, since the two tasks use different datasets and supervision. The planning gains may instead come primarily from the separately constructed driving-focus annotations. Controlled experiments comparing predicted gaze, human-selected focus, and planning without focus supervision would help clarify their respective contributions. It would also be helpful to explain how gaze targets are collected and how they capture decision-relevant information, given that human gaze is often distributed across multiple regions and does not necessarily identify the most causally relevant object.

  • The result tables should clarify the backbones and model sizes used by the VLM/VLA baselines, as performance can vary substantially across models. Reporting inference latency and computational costs would also help assess practical applicability.

  • The lack of a stated plan to release the code.

Questions for authors
  • Please clarify the GRPO formulation, particularly the group size (G), sampled completions (Y_i), reward normalization, and how the resulting advantages update the policy. Further details on how annotators select the final focus and ensure annotation consistency would be helpful.

  • Figure 3 mainly shows relatively obvious cases, such as a pedestrian or a vehicle ahead. More challenging examples involving ambiguous situations or multiple relevant objects would help demonstrate the benefits of the proposed approach.

Reviewer_jtiqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FocusDrive incorporates explicit visual focus into driving vision–language–action reasoning by pairing object descriptions with image-patch references before generating action plans and trajectories. Experiments on W3DA and NAVSIM demonstrate competitive gaze prediction and improved planning over the authors’ text-based CoT baseline. Overall, the paper presents a well-motivated and promising direction for visually grounded driving reasoning.

Critical strengths
  1. Clear motivation. Linking decision-relevant objects to image regions provides a concrete interface between scene understanding and action generation.

  2. The method improves over the text CoT baseline at both the SFT and RL stages, supporting the practical value of the overall framework.

  3. Comparisons with textual coordinates and learned reference-ID embeddings help assess the contributions of spatial grounding and image-conditioned reference inputs.

Critical weaknesses
  1. Limited evidence for the causal role of visual focus. Identifying a lead vehicle and subsequently decelerating does not establish that the action depends on the generated reference.

  2. With the image, ego state, and textual description held fixed, how do trajectories change when the selected reference is replaced with an irrelevant or spatially shifted reference? Such tests should distinguish meaningful decision sensitivity from degradation caused by out-of-distribution perturbations.

  3. Could the authors evaluate focus selection and action consistency on an independently reviewed subset, including cases with correct focus but inappropriate actions? This would help clarify when explicit focus contributes to successful planning.

  4. The authors’ additional driving-focus annotations on existing Navtrain scenes are reviewed and refined by a single annotator. An independent quality audit would strengthen confidence in these annotations.

Questions for authors

Please refer to the weaknesses listed above. I will conduct further evaluation based on the authors' feedback.

Reviewer_D3zLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The study proposes a multimodal reasoning framework that predicts driver’s gazes and uses the focused visual object information to enhance end-to-end reasoning by utilizing both spatial and visual contexts in addition to language-based CoT. A new dataset is created to support the algorithm development and evaluation.

Critical strengths

A new dataset is annotated that can support driving scene understanding tasks.

Critical weaknesses

The whole idea of predicting visual attention and combining targeted visual embeddings with LLM outputs to improve prediction accuracy has been studied in earlier research. Although the network structure and training strategy is not exactly the same, the contribution of the proposed model is marginal.

Driving decision based on static image may not be reliable because temporal information is not fully utilized. This matters not only for vehicle state changes and also gaze searching and fixation dynamics. The current model is under-informed to support real-world driving.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.