acceptodds
Withdrawn from ICLR 2027

CoDrive: Cross-Vehicle World-Consistent Video Generation with Precise Trajectory Control for Cooperative Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world driving is inherently multi-agent, yet most existing driving world models generate observations from a single ego vehicle. Independently extending them to multiple vehicles does not ensure that different agents observe a consistent shared world. We present CoDrive, a cross-vehicle, multi-view driving video generation framework that jointly generates observations of vehicles sharing the same dynamic scene with precise camera-trajectory control. CoDrive interleaves local self-attention, which models spatiotemporal dependencies among the views of each vehicle, with global self-attention, which enables information exchange and consistency modeling across vehicles. To explicitly encode their spatial relationships, all camera trajectories are represented in a shared world coordinate system and injected into the attention layers through projective relative positional encoding. We further adopt a progressive mixed-task training strategy that combines large-scale real-world single-agent data with synthetic cross-agent interaction data, allowing the model to benefit from real-world appearance distributions while learning cross-agent consistency from simulation. For systematic evaluation, we introduce CoDrive-Bench, a benchmark covering real and synthetic multi-vehicle scenarios and evaluating trajectory controllability, scene geometry consistency, and instance-level consistency. Experiments show that CoDrive improves trajectory controllability and cross-agent geometric and instance consistency while maintaining competitive visual quality.

Primary area:
generative models
Keywords:
World Model, Video Generation, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qHo6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work studies multi-agent multi-view driving video generation. The contributions of this work are two-fold. First, the authors propose a video generation framework named CoDrive that jointly models cross-vehicle multi-view videos. CoDrive is built on top of HunyuanVideo-1.5 with local attention for intra-agent modeling and global attention for cross-agent modeling. Second, the authors present CoDrive-Bench, a benchmark for cross-vehicle multi-view driving world models. Experiments on the CoDrive-Bench benchmark show the effectiveness of the proposed CoDrive.

Critical strengths
  1. The experimental results on the proposed CoDrive-Bench benchmark show the superiority of the proposed CoDrive over some state-of-the-art methods.
Critical weaknesses
  1. The technical contribution of the proposed CoDrive method is limited. For multi-agent video generation, combining the diffusion transformer with intra-agent attention and cross-agent attention is a straightforward idea. Intra-agent multi-view modeling is common in driving scene video generation, while cross-agent attention can be considered as an extension of cross-camera attention.

  2. The contribution of the proposed CoDrive-Bench benchmark is limited. CoDrive-Bench uses CARLA to generate synthetic video data, while the proposed metrics of controllability, scene consistency, and instance consistency are common in driving scene generation.

  3. Writing needs improvement, especially the introduction section. The limitations of existing methods should be clearly discussed, with appropriate citations provided for the relevant prior work. In addition, citations should be added for the baseline methods mentioned in Lines 364-366.

  4. Additional experiments are needed. The proposed CoDrive method is only evaluated on the proposed CoDrive-Bench benchmark. However, to validate the effectiveness and robustness of the proposed CoDrive method, results on additional datasets are needed.

Questions for authors
  1. Please address the weaknesses.

  2. For a fair comparison, can the authors compare the proposed CoDrive method with more state-of-the-art driving video generation methods, such as UniMLVG, by adding cross-agent attention layers to the diffusion transformer?

  3. The proposed CoDrive method uses HunyuanVideo-1.5 as the backbone diffusion transformer model, while the compared baseline methods uses other models. To ensure a fair comparison, can the authors replace the backbone with other models, such as CogVideoX?

[1] UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving. Chen et al. ICCV 2025.

[2] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. Yang et al. ICLR 2025.

Reviewer_7j3pOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoDrive, a multi-vehicle video generation framework that produces consistent observations across vehicles with precise trajectory control. It introduces interleaved global-local attention for cross-vehicle interaction and uses PRoPE to incorporate relative camera geometry. The authors also develop a progressive training strategy combining real and synthetic data and introduce CoDrive-Bench to evaluate trajectory controllability and cross-vehicle consistency. Experiments demonstrate improvements in controllability and consistency while maintaining competitive visual quality.

Critical strengths
  1. The training recipe and architecture design are complete and informative. The paper presents an interleaved global-local self-attention architecture and a progressive training strategy that effectively combines real-world and synthetic data.
  2. The detailed benchmark for testing the two agent models is good. CoDrive-Bench provides a comprehensive evaluation of two-agent video generation, covering trajectory controllability, scene consistency, and instance-level consistency.
Critical weaknesses
  1. Attention design: IGLA vs. Full Global Attention. The paper lacks a comparison and discussion between the proposed interleaved global-local attention (IGLA) and standard full global attention. It remains unclear whether IGLA provides advantages over simply applying global attention across all views.
  2. Evaluation of Conditional Generation. The paper lacks sufficient evaluation and visualization of cross-agent conditional generation. For example, the authors could fix one agent's trajectory while varying the other agent's behavior to demonstrate whether the generated videos accurately reflect these changes while maintaining cross-agent consistency.
  3. Fig.1 seems like a noticeably AI-generated picture. A cleaner and more carefully designed illustration would improve the overall presentation.
Questions for authors

See weaknesses.

Reviewer_R8PoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CoDrive jointly generates videos for two observing cars, given reference images and camera trajectories. It combines local/global attention with relative camera geometry, and mixes real single-car footage with synthetic paired footage for training.

Critical strengths

Individually plausible videos don't necessarily describe the same world. That's a real issue. Learning cross-car consistency from simulation while getting realistic appearance from single-car footage is also a sensible way around the data bottleneck

Critical weaknesses
  1. Both cars' trajectories are supplied, so this is conditional rendering, not a model of how drivers interact. That's a valid component, but the cooperative-driving motivation doesn't establish why rendering a fixed pair is the important bottleneck. I'd start with shared traffic simulation, a policy per car, and a video model producing observations. Cross-car visual consistency is still hard there. The paper should locate its contribution in that system.
  2. The fixed pair also sidesteps the interesting part of a shared world. With a third observing car, several consistency constraints have to hold at once. Some cars may only share information through another car; observers can enter and leave. A model that makes a pair agree hasn't yet told us how to handle any of that. Writing the formulation for N cars doesn't answer the modeling question. This isn't a request for one more scaling plot; it's about whether the proposed abstraction extends beyond the particular pair it was trained on.
  3. ShareVerse already has the basic two-car setup, camera control, and cross-car attention. PRoPE supplies the relative geometry. Real-world transfer could still be a worthwhile contribution, but I don't see a new modeling principle here, or a new capability strong enough to carry the paper on its own.
Questions for authors
  1. Where does CoDrive fit between a traffic simulator, individual driving policies, and visual rendering? What specifically requires the proposed joint generation?
  2. What changes when a third observing car is added, especially when some views don't overlap directly? Does each number of observers need its own training?
  3. Beyond the existing two-car formulation and relative camera conditioning, what can we now do that we couldn't do before?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.