acceptodds
Withdrawn from ICLR 2027

TrajWAM: Trajectory-Conditioned World Action Modeling for Vision–Language Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World Action Models (WAMs) connect future scene prediction with action planning for autonomous driving. Multi-view video prediction requires camera-wise image decoding, while trajectory-agnostic bird's-eye-view (BEV) prediction leaves the influence of future ego motion implicit. Thus, we present TrajWAM, a trajectory-conditioned WAM that jointly learns BEV future representations and a vision-language driving policy. TrajWAM links world prediction and planning through a shared backbone by conditioning DINOv3-aligned BEV predictions on ego trajectories. Training combines reconstruction of observed futures with a feature-separation objective that encourages predictions to respond to trajectory changes, without requiring future labels for perturbed trajectories. In inference, the planner directly generates waypoints while bypassing the auxiliary future-prediction branch. On all 220 Bench2Drive routes, TrajWAM achieves 91.53 Driving Score and 75.45% Success Rate, exceeding a schedule-matched planning-only fine-tuning baseline by 5.31 Driving Score points and eight successful routes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, world models, vision-language models, trajectory-conditioned prediction, bird's-eye-view representations, Multimodal fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xC7YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TrajWAM, a trajectory-conditioned auxiliary training framework for vision–language driving. Building on a DeepSight-style planner with DINOv3-aligned BEV future supervision, the method uses an ActionEncoder and FiLM to condition future-query states on logged or perturbed ego trajectories. Logged trajectories receive factual feature-reconstruction supervision, while perturbed trajectories receive a one-sided feature-separation loss. These objectives update the shared planner during training, and the additional conditioned readout is bypassed at inference. The method is evaluated on all 220 Bench2Drive routes, with schedule-matched continuation and component ablations.

Critical strengths

1.All Phase-II variants use the same initial checkpoint and training schedule. The full method improves over planning-only fine-tuning by 5.31 DS and eight successful routes.

2.The method distinguishes factual future supervision from perturbed trajectories, without assigning unavailable future targets to the latter.

3.The authors clearly distinguish trajectory sensitivity from counterfactual accuracy, acknowledge the single-seed limitation, and provide detailed route-level results.

Critical weaknesses

1.The experiments do not sufficiently validate the main motivation. The paper aims to learn trajectory-conditioned future representations, but the experiments mainly show improvements in planning performance. Figures 4–5 only demonstrate feature sensitivity to trajectory perturbations on training samples, without showing whether these changes reflect meaningful future scene dynamics. Table 7 also lacks comparisons with unconditioned and no-separation baselines. It remains unclear whether the planning gains come from better action-conditioned world modeling or simply from additional auxiliary supervision.

2.The evaluation of future representations is insufficient. Figures 4–5 use training samples, while Table 7 only reports reconstruction metrics for the full model. There is no comparison with the unconditioned or no-separation variants, or even a simple feature-persistence baseline. Without evaluation on held-out data, it is difficult to tell whether trajectory conditioning actually improves future prediction.

Questions for authors

1.Could the authors compare the proposed FiLM with bias-only conditioning or shuffled trajectory inputs? What happens if the current-time feature is excluded from the separation loss?

2.Could the authors report held-out future prediction results for the full model, no-separation variant, and unconditioned baseline? Please also clarify the sample count and split used in Table 7.

3.Could the authors provide paired route-level DS differences between the variants, ideally with confidence intervals? Results from additional training seeds or simulator sweeps would also be helpful.

Reviewer_tRPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TrajWAM, a vision–language driving model trained with trajectory-conditioned auxiliary world prediction. Starting from a DeepSight-style base planner, it uses an ActionEncoder and FiLM to condition future-query hidden states on logged or perturbed waypoint trajectories. Training combines DINOv3-aligned BEV reconstruction for the logged trajectory with a one-sided feature-separation loss for the perturbed trajectory. The auxiliary branch is bypassed at inference. Experiments on 220 Bench2Drive routes report improvements over a schedule-matched planning-only baseline.

Critical strengths
  • Building on the DeepSight paradigm, the paper introduces explicit trajectory-conditioned BEV prediction through an ActionEncoder and FiLM, together with a one-sided feature-separation objective for logged and perturbed trajectories. By combining factual future reconstruction with trajectory-sensitive representation learning, TrajWAM achieves state-of-the-art results among the compared methods on all 220 Bench2Drive Base Set routes, with a Driving Score of 91.53 and a Success Rate of 75.45%, while bypassing the auxiliary prediction branch at inference.
  • The component ablations are informative. All Phase-II variants start from the same checkpoint and use the same training duration. Trajectory conditioning and feature separation each improve both DS and SR in the reported comparisons.
Critical weaknesses
  • The connection between the conditioning mechanism and action-dependent world modeling needs stronger evidence. The LLM first computes future-query states independently of the external trajectory, which subsequently modulates them through FiLM. The model could satisfy the separation objective through trajectory-dependent feature shifts without learning physically meaningful scene evolution. Comparisons with simpler conditioning mechanisms or alternative auxiliary regularizers would help clarify its contribution.
  • Future prediction lacks counterfactual correctness and generalization validation. Perturbed trajectories have no corresponding future targets, and the visualizations use only training samples. Held-out prediction results and simulator rollouts under alternative trajectories would help assess whether the predictions match actual outcomes.
  • The conceptual novelty relative to prior work is limited. DeepSight already uses DINOv3-aligned BEV supervision, LAW studies trajectory-conditioned prediction, and Latent-WAM and other methods omit auxiliary modules at inference. The main addition is the combination of FiLM conditioning and one-sided separation; more direct comparisons are needed to establish its distinctive value.
Questions for authors
  • What evidence distinguishes meaningful trajectory-dependent scene prediction from trajectory-dependent feature shifts that merely satisfy the separation margin? Since FiLM is applied after the LLM computes its hidden states, this distinction is central to assessing the world-modeling contribution.

  • What samples and sample count were used for the aggregate DINO prediction metrics in Table 7? Were these samples used for training or hyperparameter selection? This would clarify the evidential scope of the reported reconstruction accuracy.

  • Could the authors provide a matched comparison against a simpler trajectory-dependent additive offset or alternative auxiliary regularizer? Such a comparison would materially help determine whether the proposed FiLM-plus-separation design contributes beyond generic auxiliary supervision.

Reviewer_6DPbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TrajWAM, a trajectory-conditioned World Action Model for vision-language autonomous driving. Built on a shared Qwen2.5-VL-3B backbone, TrajWAM uses an explicit query consisting of four future ego waypoints and conditions future bird’s-eye-view representations through an Action Encoder and latent FiLM. The model uses a frozen DINOv3 teacher to supervise future BEV feature reconstruction. Its training objective combines factual reconstruction for the logged trajectory with a one-sided feature-separation loss for a perturbed trajectory query, encouraging the predicted representation to respond to trajectory changes without requiring future labels for unobserved trajectories. During inference, the auxiliary future-prediction branch is bypassed, so the planner directly generates waypoints without online future rollout or candidate ranking. The main contributions are:

  1. A unified framework that introduces explicit trajectory conditioning into world representation learning and driving policy generation.
  2. A supervision strategy for single-outcome driving logs that combines factual future reconstruction with separation of perturbed trajectory queries.
  3. A closed-loop evaluation and component ablation study examining the effects of trajectory conditioning and the separation objective. The method is evaluated on all 220 routes of the CARLA/Bench2Drive Base Set. TrajWAM achieves a Driving Score of 91.53 and a strict Success Rate of 75.45%, improving over the schedule-matched planning-only fine-tuning baseline by 5.31 Driving Score points and eight additional successful routes. In the ablation study, adding trajectory-conditioned factual reconstruction improves performance to 89.34 DS and 74.55% SR, while adding the separation loss further increases performance to 91.53 DS and 75.45% SR. The model performs strongly across several driving abilities, although performance is uneven, with emergency-braking performance below some existing methods. Overall, the results suggest that trajectory-conditioned future representation learning and perturbed-query separation can improve closed-loop planning. However, the validation is limited to a single training seed, one simulator sweep, and the CARLA/Bench2Drive environment; robustness across seeds, real-world transfer, and actual inference cost remain unverified.
Critical strengths
  1. TrajWAM combines supervised future-feature reconstruction for the logged trajectory with a separation objective for perturbed trajectory queries, without assigning unobserved futures to those queries. This addresses a relevant supervision limitation and provides a concrete way to incorporate trajectory-sensitive auxiliary learning into a VLM planner. The auxiliary branch is bypassed during inference, making the approach compatible with direct waypoint generation.
  2. On all 220 Bench2Drive routes, TrajWAM achieves 91.53 Driving Score and 75.45% Success Rate, exceeding the schedule-matched planning-only baseline by 5.31 DS points and eight successful routes. The ablations show additional improvements from trajectory conditioning and the separation loss. These comparisons strengthen the evidence that the proposed training objectives benefit downstream driving performance, making the empirical contribution consequential to my assessment.
Critical weaknesses

The proposed world-action modeling is not actually used for autonomous-driving decision-making at inference. Trajectory conditioning, future BEV prediction, and the DINOv3 readout are all bypassed during deployment; the system simply generates waypoints with the underlying VLM and PID controller. Thus, the method functions primarily as an auxiliary training regularizer rather than an action-conditioned world model that supports planning or counterfactual reasoning. This creates a conceptual gap between the claimed contribution and the deployed system, and it remains unclear whether the proposed world modeling addresses the core autonomous-driving problem. The perturbed-trajectory objective does not establish physically meaningful counterfactual prediction. Since perturbed queries have no observed future targets, the separation loss only encourages representation differences. It does not show that the predicted future changes are correct, consistent with vehicle dynamics, or useful for evaluating alternative actions. Consequently, the causal link between trajectory-conditioned representation learning and improved driving behavior remains insufficiently demonstrated.

Questions for authors
  1. Please clarify what specific autonomous-driving decision problem is solved by the proposed trajectory-conditioned world model beyond providing an auxiliary training signal. During native inference, the external trajectory query, trajectory-conditioning module, DINOv3 readout, and future-prediction branch are bypassed. What evidence shows that the performance gain comes from learning action-dependent future dynamics rather than from generic auxiliary representation learning?
  2. Please explain how the one-sided separation loss prevents a trivial solution in which the model simply encodes the perturbed trajectory as a query-dependent feature offset. Since the perturbed trajectory has no observed future target, the loss only requires the two predictions to differ. How do the authors verify that these differences correspond to physically meaningful changes in future scene evolution?
  3. This paper states that the trajectory is introduced after planner-logit computation, while the auxiliary losses still update the shared planner layers. Please provide a precise description of the gradient path from the DINOv3 prediction loss to the native waypoint generator. Which planner representations are modified, and why should these modifications improve waypoint generation when the trajectory-conditioned branch is removed at inference?
  4. Please clarify whether the improvements in Tables 3 and 4 can be attributed specifically to trajectory conditioning and separation, rather than to an additional optimization or regularization effect. Do all variants use exactly the same trainable parameters, learning-rate schedule, number of updates, training data, and checkpoint-selection procedure? How is the contribution of the proposed objectives isolated from the effect of additional gradient signals?
  5. The future BEV targets are generated from logged future observations, while the model is conditioned on a trajectory query. Please clarify how ego-motion and coordinate-frame transformations are handled when constructing the DINOv3 targets. In particular, are the targets aligned according to the logged vehicle motion, and how should a representation conditioned on a different trajectory be interpreted when its corresponding future observation is unavailable?
  6. Please justify the specific perturbation distribution used for the alternative trajectory, including the Gaussian noise on the first waypoint and the independent scaling of the second waypoint. How were the perturbation magnitude, the DINO feature margin, and the separation-loss weight selected? Could the reported gains depend strongly on these choices rather than on the general trajectory-conditioning principle?
  7. The full model achieves the best overall Driving Score and Success Rate, but its emergency-braking performance is lower than that of several comparison methods. Please clarify whether the proposed training objective causes a trade-off between progress-oriented planning and safety-critical behavior. Which failure categories improve or deteriorate relative to the planning-only baseline, and how do the authors assess the safety implications of the reported aggregate gains?
  8. This paper claims that the auxiliary branch is bypassed at inference and therefore does not add online computation. Please clarify the actual native inference graph, including the parameters, memory, FLOPs, and latency used by the deployed planner and PID controller. Does TrajWAM have exactly the same inference cost as the planning-only baseline, or are any trajectory-related tokens or adapted layers still active during deployment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.