acceptodds
Withdrawn from ICLR 2027

TrackForesight: Track-based Visual Foresight for Automotive VLA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive vision-language-action (VLA) models have recently attracted growing interest in autonomous driving, owing to their inherited reasoning capabilities, strong pretrained initializations, and well-established architectures. Despite the elegance of this design, standard end-to-end training asks the model to predict sparse action targets directly from raw sensor inputs, which is a difficult supervision problem that leaves little direct supervision for scene structure and agent behavior. We propose TrackForesight, which guides the policy by tracking driving relevant objects in the context via a point tracker, and training the model to predict their future positions as visual chain-of-thought before emitting the final trajectory. We compare point tracks against dense RGB and semantic foresight modalities and show that they offer an excellent balance between predictability, richness, and efficiency, preserving short-horizon motion with far fewer intermediate tokens. Relative to standard action-only training, After GRPO, TrackForesight improves PDMS on NAVSIM navtest by , reaching a state-of-the-art among VLA-based planners on the NAVSIM v1 benchmark, with only generated foresight tokens and no additional training stage. The results suggest that effective foresight for driving VLAs hinges on the information content of the prediction target, the training balance between foresight and action objectives, and the cost of generating intermediate tokens before action.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Autonomouse driving

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_xe4oOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrackForesight, a VLA with tracking-based visual foresight CoT. TrackForesight analyses different CoT representation, comparing point tracks against dense RGB and semantic segments and demonstrates sparse point tracks offer a favorable balance of richness, predictability, and efficiency and this CoT helps to decision making for planner. In order to improve the policy results, it intriduces the GRPO algorithm, After GRPO training, TrackForesight reaches a state-of-the-art 92.1among VLA-based planners on the NAVSIM v1 benchmark, with only 60 generated foresight. In addition, it achieves a state-of-the-art PDMS 31.7 on Navsim v2-hard. This competitive benchmark results show the effective of tracking-based CoT.

Critical strengths
  1. TrackForesight analyses different CoT representation, comparing point tracks against dense RGB and semantic segments and demonstrates sparse point tracks offer a favorable balance of richness, predictability, and efficiency and this CoT helps to decision making for planner.
  2. In order to improve the policy results, it intriduces the GRPO algorithm, After GRPO training, TrackForesight reaches a state-of-the-art 92.1among VLA-based planners on the NAVSIM v1 benchmark, with only 60 generated foresight.
Critical weaknesses
  1. Lack of close-loop benchmark, such as bench2drive. Navsim is a easy open loop benchmark, it's hard to validate the ability of proposal CoT. It would be better to add a close loop validation, close loop is more similar to real driving environment.
  2. This paper introduces the grpo to improve the policy results. But the reward is pdm score, which is the same as navsim evalution metrics. It's easy to reward hacking. And lack some analysis about the effectiveness of grpo and tracking-base CoT's quality.
Questions for authors
  1. It would be better to add close loop experiments, such as bench2drive. Navsim is a easy open loop benchmark, it's hard to validate the ability of proposal CoT.
  2. Add some analysis about grpo, such as reward hacking. The effectiveness of grpo and CoT should be analysed. Dose the pdm rewad help improve the CoT precision. Table 3 only validates the effectiveness of CoT after SFT by comparing it with the action-only setting. It would be better to also include a comparison after GRPO training.
  3. Add some comparison baselines, such Recogdrive, wam-diff, wam-flow and so on.
  4. I'm confused about "early selection, motion-ranked, smooth motion-ranked, final tokenizer" in Table 4, hoping for more explanation.
  5. It's necessary to add a runtime overhead analysis after introducing the tracking-based visual foresight CoT.
Reviewer_BjsnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TrackForesight, an autoregressive vision-language-action (VLA) model for autonomous driving. Using point-track supervision constructed with SAM3 and CoTracker3, the model predicts the future positions of sparse scene points as visual foresight chain-of-thought before generating the ego trajectory. The paper compares RGB, semantic-segmentation, and point-track intermediate targets, and trains the policy through supervised fine-tuning followed by GRPO. The final configuration achieves 92.1 PDMS on NAVSIM v1 while generating only 60 foresight tokens.

Critical strengths
  • Using sparse point tracks to represent short-horizon scene motion is well motivated. Compared with dense future-image prediction, this representation substantially reduces the intermediate generation length.
  • The planning results are competitive. TrackForesight outperforms the authors’ reproduced DynVLA baseline on both NAVSIM v1 and v2-hard. Evaluating point-prediction quality separately from downstream planning performance also provides useful additional analysis.
Critical weaknesses
  • The methodological novelty is limited, and the positioning against related work is insufficient. Predicting future point trajectories as an intermediate representation between visual prediction and action control has clear precedents. ATM (Wen et al., 2024) uses predicted point trajectories to guide policy learning, while Track2Act (Bharadhwaj et al., 2024) uses future point tracks as interaction plans from which robot actions are derived. These directly relevant works are not discussed. Although discretizing scene-point trajectories and incorporating them into an autoregressive driving VLA introduces a distinction, the current contribution largely concerns representation selection and the combination of existing components. The authors should clarify the additional methodological contribution over prior track-guided control approaches and establish why it is necessary.

  • The experimental evidence is insufficient, with important ablations and cross-benchmark validation missing. Evaluation is restricted to NAVSIM v1 and v2-hard, leaving generalization across data sources and driving environments unclear. Evaluation on an additional independent driving benchmark would strengthen the paper.

  • The efficiency claim is supported mainly by token counts. Generating 60 foresight tokens is attractive, but the complete pipeline also requires segmentation and context tracking. End-to-end latency and computational cost, including these auxiliary models, should be reported. The current evidence establishes a shorter generated sequence, but does not quantify the overall efficiency advantage.

Questions for authors
  • What is the additional methodological contribution over track-guided control approaches such as ATM and Track2Act? Beyond the application domain and autoregressive sequence formulation, which design choices are essential to the gains?

  • Which action-only baseline supports the claimed 1.9-point improvement after GRPO? Does it use identical context tracks, training budgets, and scoring procedures? Are there existing results for a baseline receiving the same track inputs without generating future tracks?

  • Do existing experiments separately establish the effects of point count, selection strategy, loss weighting, and GRPO masking? In particular, what quantitative results support Section 3.2’s claim that action-only masking outperforms the two alternatives?

  • Is the efficiency claim supported by end-to-end measurements covering segmentation, context tracking, and autoregressive generation? If available, what are the measured latency and computational costs relative to the baselines?

Reviewer_mWuyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TrackForesight, a driving VLA model that predicts future 2D point tracks before generating the vehicle’s trajectory. The model is trained with supervised learning followed by GRPO, which applies the reinforcement-learning loss only to action tokens. Compared with predicting future RGB images or segmentation maps, point tracks require fewer tokens and yield better planning results in the reported comparisons. The final model uses 60 foresight tokens and achieves a PDMS of 92.1 on NAVSIM v1.

Critical strengths

The method focuses future prediction on scene motion, using fewer tokens to capture information relevant to driving. This is a practical choice of intermediate representation for driving VLAs. The comparison between ground-truth and generated futures is useful. It shows why a richer prediction target can help planning when provided correctly, yet offer little benefit when the model has to predict it. The paper clearly explains the trade-offs between information content, prediction difficulty, and generation cost. This makes the motivation for using point tracks easy to follow.

Critical weaknesses
  1. Point trajectories are predicted in 2D image space, but the supervision does not remove the effect of ego motion. I am concerned that this could make training unstable. Consider a counterfactual example (this is a common scenario in world model): with exactly the same observation history, if the ego vehicle follows a different future trajectory, other agents will appear at different pixel locations even if their actual motion remains unchanged. The same input can therefore correspond to very different but equally valid targets, while the future ego trajectory that determines these differences is not provided to the model.
  2. GRPO applies the loss only to action tokens. It is therefore unclear whether the planning gains come from better use of the predicted point tracks or improvements in action generation alone. The paper does not report point prediction accuracy before and after GRPO, or test whether the planner actually relies on the generated tracks.
  3. The efficiency claim is mainly supported by fewer generated tokens. However, the method also requires segmentation and point tracking. An end-to-end latency comparison, including these steps, is needed to establish the practical speed advantage.
Questions for authors

Could you please address the first weaknesses? I am really concerned about it.

Reviewer_ksy8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TrackForesight uses SAM3 and CoTracker3 to extract driving-relevant point tracks. Given images, driving context, and historical point locations, the VLA predicts future point locations before generating the ego trajectory. Comparisons using the same backbone and action tokenizer show shorter output sequences and smaller planning performance gaps between ground-truth and generated futures than RGB or semantic targets.

Critical strengths
  1. Point motion is a reasonable target for short-horizon planning. The representation is compact: 20 location tokens per future time step, compared with 576 for RGB or semantic targets.
  2. The oracle comparison helps explain the results. At 2 seconds, RGB PDMS drops from 89.11 to 84.10 when using generated rather than ground-truth futures; point tracks drop from 87.33 to 86.62. This suggests that point tracks retain more of their planning value during generation.
Critical weaknesses
  1. Fewer output tokens do not establish the efficiency of the full system. Inference also involves SAM3, CoTracker3, input processing, and VLA decoding. The paper acknowledges this overhead but provides no end-to-end latency, per-module timings, or comparisons on the same hardware.
Questions for authors
  1. Do all Table 3 baselines receive the same historical and current point locations? Please report an action-only baseline retaining these inputs but omitting future-track generation.
  2. Are history tracks computed from past and current frames only, or extracted after tracking a clip containing future frames? What time window is used for motion-ranked selection in Table 4?
  3. What foresight/action weights are used in Eq. (4)? Do the comparisons hold when the losses are normalized separately or action supervision weights are matched across methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.