acceptodds
Under review as a conference paper at ICLR 2027

WM-V2X: Cooperative World Models via V2X Communication for Consistent Occupancy Prediction

Abstract

World models have demonstrated promising capability for 4D occupancy prediction and motion planning in autonomous driving. However, existing methods are designed for individual vehicles and are inherently limited by occlusions and restricted sensing ranges, which leads to inconsistent scene predictions across agents. In this paper, we propose WM-V2X, a cooperative world model framework that addresses this challenge by leveraging Vehicle-to-Everything (V2X) communication to share sensing information across connected agents for 4D forecasting on real multi-agent data. First, we design a cooperative scene tokenizer that fuses multi-agent sensory data into compact latent representations via early or intermediate fusion pipelines to facilitate 4D occupancy prediction. To further improve prediction performance, we introduce satellite imagery as an auxiliary conditioning signal that provides static scene context and visual trajectory information to a spatio-temporal transformer, which auto-regressively predicts future occupancy scenes and ego-trajectories. To evaluate cooperative world models, we reprocess the V2X-Real dataset to generate 3D voxel occupancy ground truth and introduce new cross-agent consistency metrics based on connected component analysis. Extensive experiments demonstrate that WM-V2X achieves the best 4D occupancy prediction with average mIoU of and under early and intermediate cooperation, and the lowest position error of  m with a collision rate in motion planning. Cooperation further enhances cross-agent prediction consistency, raising the mean detection rate for dynamic objects by up to over the single-agent baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cooperative Perception, World Model, Occupancy Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yd8HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WM-V2X, a framework for cooperative occupancy forecasting and motion planning using multi-agent sensory observations and satellite imagery. It combines an early- or intermediate-fusion scene tokenizer with a spatial and temporal forecasting module that predicts future latent states autoregressively. The paper extends V2X-Real with occupancy annotations and aligned satellite images, introduces cross-agent consistency metrics, and provides extensive empirical evaluations.

Critical strengths
  1. This paper proposes and formulates a novel cooperative world model setting.
  2. This paper extends V2X-Real with occupancy annotations and aligned satellite imagery, providing a useful foundation for further research under this setting.
Critical weaknesses
  1. V2X-Real with 5772/1253 train/test frames are small in scale, which is reflect in Table 6 where five methods have exact same collision rate for all horizons (three in Table 2 are all 0). 5 semantic categories are also far fewer than 17 in Occ3D-nuScenes.
  2. WM-V2X performs close to OccProphet (Table 1 & 6) and worse than OccProphet without satellite embedding (Table 4). The performance gain might come from additional satellite input rather than forecasting module.
Questions for authors
  1. V2X-Real includes two infrastructures and two vehicles, totaling four agents. Could the authors report how performance scales as the total number of participating agents increases from one to four?
  2. Could the authors provide camera-only WM-V2X performance w/o satellite inputs?
2 comments

Response to Reviewer Yd8H

We thank the reviewer for the concise review and for recognizing the value of the cooperative world model setting. Below we address the dataset scale, the collision rates, and the classes (A1), give an input-matched comparison with the baselines (A2), and report the two results you asked for, namely the performance from one to four agents (A3) and the camera-only model with and without satellite input (A4).


[W1] Dataset scale, identical collision rates, and the number of classes

[A1]

The scale is that of V2X-Real itself, and our labels cover all of its training and test frames. Cooperative forecasting needs synchronized recordings from several connected agents, which a larger single-vehicle dataset cannot replace. V2X-Real is one of the few real datasets with two vehicles and two roadside units. We label every agent in its 5,772 training and 1,253 test frames. This gives 21,917 and 5,012 agent frames, each with a dense label of about one million voxels (256 × 256 × 16), together with fused labels in the ego frame. The effects we report are consistent at this scale. For example, every added agent improves the forecast at every horizon and for both fusion types (see the table in A3).

The identical collision rates come from how the collision rate is computed. The collision rate counts the test samples in which the predicted ego box overlaps the box of another object. When the predicted ego positions are within 0.2 m of the ground truth, far below the size of a vehicle, the ego boxes overlap the same objects and the rates coincide. In Table 6, for example, OccWorld and WM-V2X both have a rate of 0.052%, while their position errors are 0.166 m and 0.036 m. The methods with larger position errors have different rates. We therefore compare the methods by position error.

Classes. V2X-Real provides 3D boxes rather than point-level semantic labels. We group the box categories into vehicle, pedestrian, and truck, and split the static scene into ground and a remaining class that we call building. These classes keep the distinction that matters for cooperative forecasting, between static structure and the road users that move and get occluded. Finer static classes would need new annotation. We will discuss this choice and the difference from Occ3D-nuScenes explicitly.


[W2] Close to OccProphet, and worse without satellite

[A2]

With matched inputs and no satellite input for any model, WM-V2X leads OccProphet by +2.58. The comparison in the submission gave satellite input to WM-V2X only. In the revision, we treat the satellite crops as a dataset resource, not as a contribution of the model, and compare all forecasters on the same inputs.

We retrained every model without satellite input, so all forecasters on a tokenizer, ours and the external ones, now share the same latent history and past ego poses. Forecasts use 4 history frames (t−1.5 s to t) and 6 future frames (0.5–3 s), matching the 3 s horizon of OccWorld and DOME on nuScenes. All methods are trained and evaluated this way, and the revision will report these new results.

MethodFusion0.5 s1.5 s3 sAvg.
No cooperation–25.4420.0016.5019.97
OccWorldInter.39.7829.2322.7429.32
OccProphetInter.40.0429.0919.4028.33
DOMEInter.32.7827.1222.9726.97
COMEInter.32.5027.2122.6026.81
I2-WorldInter.33.5324.7619.4824.85
EfficientOCFInter.33.2213.343.2114.17
WM-V2XInter.39.6930.7725.5030.91
WM-V2XEarly40.2431.0725.5931.21

Values are the mean IoU (mIoU, in %) over the five semantic classes. Avg. is the mean over all six horizons (0.5–3 s). No cooperation feeds ego-only input to components trained with collaborators.

With intermediate fusion, WM-V2X leads OccProphet by +2.58 and OccWorld by +1.60. This lead comes from how WM-V2X uses the shared past poses, by conditioning each rollout step on a constant-velocity extrapolation of the last two observed poses.

[Q1] Performance as the number of agents grows from one to four

[A3]

To avoid mixing scene effects into this curve, we use 157 windows from 5 scenarios in which all three collaborators are in range, and evaluate every subset of collaborators on the same samples. The target is always the fused ground truth of all agents. With 2 agents, the ego is joined by one of the three collaborators, and with 3 agents by two of them. Each case has three possible choices, and we report their mean. The 1-agent row feeds ego-only input to a model trained with collaborators. The table uses the models compared in A2. Avg. is the mean over all six horizons (0.5–3 s).

FusionAgents0.5 s1.5 s3 sAvg.
Inter.1 (ego only)22.2718.0214.5917.82
Inter.232.4326.3721.8526.20
Inter.338.4531.0525.9930.96
Inter.441.9533.2827.5133.22
Early1 (ego only)28.9022.9718.6122.83
Early236.4229.1424.0829.06
Early340.8732.4026.6232.33
Early443.3734.2027.9934.15

Each added agent improves the forecast. On average, the second, third, and fourth agents add 8.4, 4.8, and 2.3 mIoU points under intermediate fusion and 6.2, 3.3, and 1.8 under early fusion.


[Q2] Camera-only performance without satellite input

[A4]

We retrained the camera-only WM-V2X with and without satellite input, both under the protocol described in A2, and the revision will report these results. The model with satellite input receives crops of the observed frames only and keeps the last one during the rollout.

Method0.5 s1.5 s3 sAvg.
No cooperation10.9210.149.4110.07
WM-V2X, with satellite27.3823.9221.1023.74
WM-V2X, without satellite27.4823.8720.1623.47

Averaged over the six horizons, the camera-only model scores 23.47 without satellite input and 23.74 with it. The difference is 0.27 points, so the camera-only model performs nearly the same without satellite input.


Revision. The revised PDF will follow this response, with the dataset statistics, the input-matched comparison, and the new results on the number of agents and on the camera-only model. We will upload it before the end of the discussion period, after taking your feedback into account.

Reviewer_68T2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents WM-V2X, a cooperative world model for future 3D semantic occupancy and ego-trajectory prediction. A cooperative tokenizer fuses multi-agent LiDAR observations through either early or intermediate fusion. A spatiotemporal predictor uses the fused latent representations, historical ego trajectories, and satellite imagery to generate future scenes and trajectories autoregressively. The authors extend V2X-Real with occupancy labels and aligned satellite images, and introduce metrics for cross-agent prediction consistency. Experiments evaluate occupancy forecasting, trajectory prediction, cross-agent consistency, and robustness to communication and localization errors.

Critical strengths

The paper makes a useful contribution by extending the real-world V2X-Real dataset with 3D semantic occupancy labels and aligned satellite images. The occupancy labels are generated from multi-frame LiDAR observations and existing object annotations. This provides a real-world setting for evaluating cooperative occupancy forecasting. The inclusion of both vehicle and roadside observations supports the study of complementary views under occlusion and limited sensing coverage. The added cross-agent consistency metrics also help assess agreement between predictions from different agents.

Critical weaknesses
  • The proposed intermediate-fusion module uses an incorrect scale when converting relative translations into feature-grid shifts. Appendix E.3 states that collaborator features are shifted by only 64% of the required displacement during both training and evaluation. This error undermines the reliability of the reported intermediate-fusion results. The experiments therefore cannot establish the relative performance of early and intermediate fusion under correct alignment. The evaluation remains incomplete without correcting the scale and retraining the affected models.

  • The proposed model has limited architectural novelty. Early fusion is a common approach in cooperative perception, and related fusion strategies have already been used in multi-vehicle 3D occupancy methods. The paper extends current-frame occupancy estimation to 4D occupancy forecasting. However, it does not introduce a cooperative module that addresses the additional temporal challenges of predicting future scenes. The contribution therefore lies mainly in extending the task and combining existing components, with limited new design for temporal cooperation.

Questions for authors
  • Could the authors correct the translation scale in the intermediate-fusion pipeline, retrain the affected models, and report the updated accuracy and pose-robustness results alongside early fusion?
  • Without satellite images, WM-V2X achieves 44.65% mIoU under early fusion, compared with 45.54% for OccProphet. Could the authors provide a comparison in which the strongest baselines also receive the same satellite information?
  • During autoregressive rollouts, are satellite crops for future steps selected using predicted ego poses or ground-truth future poses?
  • How are unobserved voxels distinguished from free space during occupancy-label generation, and how is the quality of the generated labels checked?
3 comments

Response to Reviewer 68T2

We thank the reviewer for the careful reading and for recognizing the value of the occupancy labels added to V2X-Real. Below we report the corrected tokenizer with its accuracy and pose robustness (A1), explain how the design handles temporal cooperation (A2), give an input-matched comparison of the forecasters (A3), answer the question on the satellite crops (A4), and quantify the effect of unobserved voxels on the evaluation (A5).


[W1, Q1] Correcting the translation scale, retraining, and updated accuracy and pose robustness

[A1]

We corrected the warp and retrained the tokenizer with the same configuration and schedule, and we report the updated accuracy and pose robustness below, alongside early fusion. All forecasters on this tokenizer, ours and the external ones, were then trained on its latents.

Reconstruction. We report the foreground mIoU against the fused ground truth on the 2,342 test frames.

TokenizermIoU
Early fusion52.47
Intermediate fusion, original STTF44.98
Intermediate fusion, corrected51.22

The correction raises the reconstruction by 6.23 points, to within 1.25 points of early fusion.

Pose robustness of the tokenizer. We use 781 test frames (every third frame with a collaborator in range) and add noise to the collaborator poses, with translation N(0, σ²) m and heading N(0, σ²)°.

TokenizerCleanEgo onlyσ = 0.2σ = 0.5σ = 1σ = 2
Early fusion52.4536.9449.5643.6637.1230.78
Intermediate, original45.0439.3745.0545.0144.9444.85
Intermediate, corrected51.2631.6649.6244.3637.7930.38

With the collaborator features now placed at their true positions, the reconstruction follows the pose accuracy, as it does for early fusion (51.26 → 30.38 at σ = 2, against 52.45 → 30.78). The original model was insensitive because its collaborator features were already misplaced. In the revision, we update the pose-robustness analysis with these results.

Forecasting. We report the mIoU of WM-V2X averaged over 0.5–3 s, under the same pose noise on collaborator messages.

FusionCleanσ = 0.2σ = 0.5σ = 1
Intermediate, corrected30.9130.2727.7323.77
Early31.2130.1627.3123.58

Forecasts use 4 history frames (t−1.5 s to t) and 6 future frames (0.5–3 s), matching the 3 s horizon of OccWorld and DOME on nuScenes. All methods are trained and evaluated this way, and the revision will report these updated results. A3 compares all forecasters on the same samples.


[W2] Architectural novelty and temporal cooperation

[A2]

WM-V2X handles temporal cooperation by applying cooperation at every history frame and rolling out the resulting cooperative scene state. It fuses the agents with established operators (early fusion and CoBEVT's FAX), and in the revision we describe them as such. The design addresses the temporal challenge in two steps:

  1. A cooperative state at every history frame. At each frame, the tokenizer brings the agents' LiDAR observations into the ego frame, fuses them, and is trained to reconstruct the occupancy fused over all agents. The predictor therefore receives a sequence of scene states that include objects and regions outside the ego's view.
  2. Rollout of the cooperative state. The predictor applies temporal attention over this fused history, conditions each step on the ego motion, and feeds each predicted latent back into its context. What collaborators contribute is thus carried through the rollout. Collaborator input raises the forecast by 14.25 points at 0.5 s and by 9.00 points at 3 s (see the table in A3). Objects that the ego does not observe during the history but a collaborator does reach an IoU of 5.92 (moving) and 32.59 (stationary) in the forecast, against below 1 without cooperation. The revision reports this breakdown.

Difference from prior work. Cooperative occupancy work estimates the current frame, e.g., the CoHFF method [1] on simulated data and the synthetic COPBench benchmark [2]. Cooperative forecasting methods predict the trajectories of detected objects, e.g., V2XPnP [3], which fuses agents over time on real data, and UniV2X [4] predicts occupancy as an intermediate for planning in vehicle–infrastructure cooperation. Occupancy world models such as OccWorld [5] forecast dense multi-class scenes, but from a single vehicle. To our knowledge, no prior benchmark provides multi-class 3D occupancy forecasting from several connected agents on real data. As you note in the strengths, ours provides a real-world setting for evaluating cooperative occupancy forecasting.

[Q2] Strongest baselines with the same satellite information

[A3]

In the revision, we match the inputs by removing satellite input from WM-V2X, so that all forecasters receive the same information. None of the external forecasters takes satellite input, so adding it would require a new conditioning module of our own design for each of them. There, we also treat the satellite crops as a dataset resource, not as a contribution of the model. We therefore retrained every model without satellite input, WM-V2X and the external forecasters alike, so that all forecasters on a tokenizer share the same latent history and past ego poses.

MethodFusion0.5 s1.5 s3 sAvg.
No cooperation–25.4420.0016.5019.97
OccWorldInter.39.7829.2322.7429.32
OccProphetInter.40.0429.0919.4028.33
DOMEInter.32.7827.1222.9726.97
COMEInter.32.5027.2122.6026.81
I2-WorldInter.33.5324.7619.4824.85
EfficientOCFInter.33.2213.343.2114.17
WM-V2XInter.39.6930.7725.5030.91
WM-V2XEarly40.2431.0725.5931.21

Values are the mean IoU (mIoU, in %) over the five semantic classes. Avg. is the mean over all six horizons (0.5–3 s). No cooperation feeds ego-only input to components trained with collaborators.

On the corrected intermediate tokenizer, WM-V2X leads OccProphet by +2.58 and OccWorld by +1.60. This lead comes from how WM-V2X uses the shared past poses, by conditioning each rollout step on a constant-velocity extrapolation of the last two observed poses (in training, on the logged next-step ego motion). In the revision, this table replaces the comparison in the submission, in which only WM-V2X received satellite input.


[Q3] Satellite crops for future steps: predicted or ground-truth poses?

[A4]

For rollout steps 2–4, the crops were selected with ground-truth future poses. This was not the intended design, and correcting it does not change the results. Satellite crops are cached per frame and centered at the ego pose of that frame. The intended design held the last observed crop during the rollout, but the inference code passed the crops of all frames, and the rollout window slid over them. Step 1 used only observed crops. Steps 2–4 included crops of future frames at ground-truth poses. Only WM-V2X was affected, since no baseline used satellite input.

Thank you for raising this point. We re-ran the same checkpoint with the last observed crop. It gives 50.18 / 47.19 / 44.51 / 42.28 at the four horizons, 46.04 on average, compared with 50.18 / 47.19 / 44.50 / 42.27 and 46.04 as reported, i.e., differences of at most 0.012 points.

[Q4] Unobserved voxels vs. free space, and label quality

[A5]

Unobserved voxels are not separated from free space. A voxel without aggregated LiDAR points is labeled free, in training and evaluation alike. We measured the effect of this choice, and it does not change the comparison.

Effect on evaluation. We measured which voxels the LiDARs actually observed, by ray casting. For every agent in range, we traced a ray from the sensor to each LiDAR point of the five sweeps that a label is built from. A voxel counts as observed if a ray passes through it or if it is occupied in the ground truth. On every tenth target frame (201 frames), the rays pass through 37.7% of the voxels in the evaluation volume. This is a lower bound on the observed space, because laser beams that hit nothing are not stored in the data. The table compares the scores on all voxels with the scores on the observed voxels only (the models compared in A3, 0.5–3 s, on these 201 frames).

MethodAll voxelsObserved voxels only
No cooperation19.9120.59
EfficientOCF14.1515.63
I2-World24.8826.60
COME26.8428.13
DOME26.9828.33
OccProphet28.3230.01
OccWorld29.3331.04
WM-V2X (inter.)30.9032.60
WM-V2X (early)31.2232.76

All scores rise by 0.7 to 1.7 points and the ranking does not change. Treating unobserved space as free therefore makes absolute scores conservative but does not affect the comparison.

Label quality. The labels are generated automatically from the LiDAR sweeps and V2X-Real's 3D box annotations and were not verified manually, so dynamic voxels inherit the quality of the box annotations. When the agents' labels disagree on a voxel, the fused label keeps the dynamic class over a static one. We checked the voxelization and fusion on the test set. All dynamic voxels of an agent's own label lie within 0.5 m (one voxel) of its boxes, and 98.7% in the fused target, consistent with re-gridding collaborator voxels into the ego frame. In addition, 96.2% of the occupied voxels inside boxes (above their lowest 0.5 m) are dynamic in the fused target.

We will describe both points in the revision.


Revision. The revised PDF will follow this response, with the corrected tokenizer and its results, the revised description of the design, the input-matched comparison, and the analysis of unobserved voxels. We will upload it before the end of the discussion period, after taking your feedback into account.

References

[1] Song, Rui, et al. "Collaborative semantic occupancy prediction with hybrid feature fusion in connected automated vehicles." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Wu, Hanlin, et al. "A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

[3] Zhou, Zewei, et al. "V2xpnp: Vehicle-to-everything spatio-temporal fusion for multi-agent perception and prediction." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[4] Yu, Haibao, et al. "End-to-end autonomous driving through v2x cooperation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025.

[5] Zheng, Wenzhao, et al. "Occworld: Learning a 3d occupancy world model for autonomous driving." European conference on computer vision. Cham: Springer Nature Switzerland, 2024.

Reviewer_o4NVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WM-V2X, a framework for cooperative autoregressive 4D occupancy and ego-trajectory forecasting. It combines early or intermediate multi-agent fusion within a scene tokenizer with a spatio-temporal predictor conditioned on historical trajectories and satellite imagery. The authors extend V2X-Real with occupancy annotations and aligned satellite images, introduce cross-agent consistency metrics, and evaluate forecasting accuracy, trajectory prediction, and robustness to imperfect communication.

Critical strengths
  1. Compelling problem formulation. Extending occupancy world models to cooperative settings is a valuable direction. Complementary observations from connected agents can help address occlusions and limited sensing coverage, potentially improving both future scene predictions and agreement across agents.

  2. Useful benchmark extension. Adding sequential occupancy annotations and aligned satellite imagery to real multi-agent data provides a potentially useful resource for cooperative forecasting research. Evaluating cross-agent agreement alongside conventional occupancy accuracy also offers an additional perspective on prediction quality.

  3. Broad experimental analysis. The paper compares several forecasting models using a shared frozen tokenizer and includes component ablations, longer-horizon predictions, communication-cost analysis, and robustness tests. The native-tokenizer checks and results without satellite conditioning are useful efforts to clarify the comparison protocol.

Critical weaknesses
  1. Limited methodological novelty beyond integrating established components. Early fusion follows standard point-cloud aggregation, while intermediate fusion directly adopts CoBEVT’s FAX module. Cross-agent interaction is confined to the tokenizer, and the forecasting model operates on an already fused ego-centric latent sequence. The remaining architecture largely follows established occupancy forecasting approaches. The paper should more clearly identify the technical challenges specific to cooperative forecasting and explain how its design addresses them beyond this integration. Its relationship to cooperative end-to-end driving methods that already incorporate occupancy prediction and planning also needs stronger discussion.

  2. The experiments provide limited evidence for the independent benefit of the proposed forecasting architecture. Under early fusion, WM-V2X achieves 46.04 mIoU versus 45.54 for OccProphet, while removing satellite conditioning reduces its performance to 44.65. Thus, the advantage over the strongest baseline in this setting depends on an additional input modality. The reported input-matched control is helpful, but it does not establish a consistent architectural advantage. Providing comparable satellite conditioning to a strong baseline would better isolate the contribution of the proposed predictor.

  3. The evaluation does not sufficiently distinguish improved observations from improved cooperative future modeling. The consistency metrics measure agreement between agents’ predicted objects, which can improve simply because their inputs become more complete and overlapping. Although occupancy accuracy is also reported, the analysis does not clearly establish how cooperation changes future prediction errors beyond improving the observed scene representation. Horizon-dependent results focused on occluded and moving objects would strengthen this claim. Furthermore, the low-speed intersection setting limits how much the very low trajectory errors and collision rates demonstrate benefits for challenging driving decisions.

Overall assessment

Borderline Reject The cooperative world-model formulation is appealing, and the benchmark extension has potential value. However, the methodological contribution remains incremental, and the experiments do not sufficiently isolate benefits beyond established fusion mechanisms and additional satellite information. I therefore lean toward rejection, while remaining open to clarification and stronger evidence of the contribution’s distinct value.

Questions for authors
  1. Which technical contribution is specific to cooperative future modeling beyond the integration of established fusion modules, an occupancy predictor, and satellite conditioning? Please clarify the distinction from prior cooperative end-to-end systems incorporating occupancy prediction or planning.

  2. Under early fusion, how should the 46.04 versus 45.54 mIoU comparison be interpreted given that performance without satellite conditioning is 44.65? Is there existing evidence from a strong baseline with comparable satellite conditioning that isolates the predictor's contribution?

  3. Do the available results distinguish improvements in future prediction from improvements in the observed scene representation? Horizon-dependent results for occluded and moving objects would be particularly informative if available.

3 comments

Response to Reviewer o4NV

We thank the reviewer for the careful review and for recognizing the value of the benchmark extension. Below we explain how the design addresses the challenge specific to cooperative forecasting (A1), give an input-matched comparison of the forecasters (A2), and separate improved observation from improved prediction with a breakdown by object visibility and motion (A3).


[W1, Q1] Technical contribution specific to cooperative forecasting, and relation to end-to-end systems

[A1]

Cooperative forecasting has to carry what collaborators observe through the rollout of the future, which neither cooperative perception nor single-vehicle forecasting requires. WM-V2X is designed for this challenge. It builds on established fusion operators, and in the revision we describe them as such. Two design choices address this challenge:

  1. Cooperation enters at every history frame, in a state supervised by the fused scene. At each history frame, the tokenizer brings the agents' LiDAR observations into the ego frame, fuses them, and is trained to reconstruct the occupancy fused over all agents. The predictor therefore rolls out a sequence of cooperative scene states. Objects hidden from the ego reach an IoU of 5.92 (moving) and 32.59 (stationary) in the forecast, against below 1 without cooperation (see the breakdown in A3).
  2. The predictor evolves one ego-centric cooperative state. The tokenizer outputs a single latent per frame for any number of agents and for both fusion types, so the temporal model does not depend on who is communicating. Because this state is in the ego frame, WM-V2X conditions each rollout step on the ego motion. With identical inputs, this gives +2.58 over OccProphet and +1.60 over OccWorld (see the comparison in A2).

Benchmark. We also provide sequential multi-class 3D occupancy labels for the two vehicles and two roadside units of V2X-Real, with a fused target in the ego frame. Prior cooperative occupancy work estimates the current frame on simulated data, e.g., CoHFF [1] and COPBench [2]. To our knowledge, no prior benchmark supports multi-class 3D occupancy forecasting from several connected agents on real data.

Relation to end-to-end systems. UniV2X [3] jointly learns perception, mapping, occupancy prediction, and planning for vehicle-infrastructure cooperation, and V2X-WAM [4] predicts future cooperative occupancy and flow conditioned on planned actions. In both, occupancy serves planning. WM-V2X targets the forecast itself, as an autoregressive rollout of multi-class 3D occupancy on real multi-agent data. We will add this discussion to the related work.

[W2, Q2] Interpreting 46.04 vs. 45.54, and a baseline with comparable satellite conditioning

[A2]

In the revision, we isolate the predictor with an input-matched comparison, in which WM-V2X leads OccProphet by +2.58. The 46.04 in the submission was obtained with satellite input, which OccProphet did not have, so that comparison reflects the extra input as well as the predictor. In the revision, we treat the satellite crops as a dataset resource, not as a contribution of the model.

Isolating the predictor. We do not have a satellite-conditioned OccProphet. The published model takes no map input, so adding satellite crops would require a conditioning module of our own design, and the result would no longer be the original method. Instead, we retrained every model without satellite input, so that no forecaster has a map input. All forecasters on a tokenizer, ours and the external ones, receive the same latent history and past ego poses. Forecasts use 4 history frames (t−1.5 s to t) and 6 future frames (0.5–3 s), matching the 3 s horizon of OccWorld and DOME on nuScenes. All methods are trained and evaluated this way, and the revision will report these new results.

MethodFusion0.5 s1.5 s3 sAvg.
No cooperation–25.4420.0016.5019.97
OccWorldInter.39.7829.2322.7429.32
OccProphetInter.40.0429.0919.4028.33
DOMEInter.32.7827.1222.9726.97
COMEInter.32.5027.2122.6026.81
I2-WorldInter.33.5324.7619.4824.85
EfficientOCFInter.33.2213.343.2114.17
WM-V2XInter.39.6930.7725.5030.91
WM-V2XEarly40.2431.0725.5931.21

Values are the mean IoU (mIoU, in %) over the five semantic classes. Avg. is the mean over all six horizons (0.5–3 s). No cooperation feeds ego-only input to components trained with collaborators.

With intermediate fusion, WM-V2X leads OccProphet by +2.58 and OccWorld by +1.60. This lead comes from how WM-V2X uses the past ego poses that all forecasters receive. It conditions each rollout step on a constant-velocity extrapolation of the last two observed poses. Among the forecasters we tested, WM-V2X with ego-motion conditioning has the highest mIoU averaged over 0.5–3 s.

[W3, Q3] Improved future prediction vs. improved observation, by horizon, for occluded and moving objects

[A3]

The breakdown below separates improved observation from improved future prediction. Cooperation improves the forecast mainly through what it adds to the observed scene. For each object at the target frame, we count the ego's LiDAR points inside its box over the four history frames. An object is hidden if the ego has fewer than 5 points in all four frames while a collaborator in range has at least 5, and seen if the ego has at least 5. An object is moving if it travels more than 0.8 m in the 1 s before the target frame, and stationary otherwise. We exclude the ego's own box, and objects that no agent observed. IoU is computed within object footprints against the fused ground truth for the models compared in A2. Avg. is the mean over all six horizons (0.5–3 s).

Objects (share of dynamic voxels)Method0.5 s1.5 s3 sAvg.
Hidden, moving (2.1%)No cooperation0.040.010.670.19
WM-V2X (inter.)12.135.143.665.92
WM-V2X (early)12.965.142.425.77
Hidden, stationary (2.8%)No cooperation0.020.060.060.05
WM-V2X (inter.)44.3632.8025.0632.59
WM-V2X (early)46.1938.3131.1237.60
Seen, moving (37.8%)No cooperation18.097.482.778.06
WM-V2X (inter.)23.837.754.129.80
WM-V2X (early)25.018.924.1410.59
Seen, stationary (56.0%)No cooperation28.4723.3719.1523.07
WM-V2X (inter.)46.9038.0630.5337.51
WM-V2X (early)49.0939.3231.1138.78

Two observations:

  • Cooperation brings occluded objects into the forecast. This is the limitation of single-vehicle world models that the paper addresses. Without cooperation, hidden objects are almost absent from the forecast at every horizon, and with it they reach 5.92 (moving) and 32.59 (stationary).
  • For objects that the ego already sees, cooperation also improves the forecast. The gain is +14.44 and +15.71 points with intermediate and early fusion on stationary objects, and +1.74 and +2.53 on moving objects, where it is largest at 0.5 s (+5.74 and +6.92).

So cooperation mainly improves the forecast by completing the observed scene, including objects the ego cannot see, which the forecaster then carries through the rollout. In the revision, we add this breakdown and state this conclusion.

Scope of the claims. Cross-agent consistency also rises when the agents' inputs overlap. The revision therefore reports it as a measure of agreement between agents, and draws its conclusions about future prediction from the breakdown above. The trajectory and collision results were measured at low-speed urban intersections, and the revision will state this setting where it reports them.


Revision. The revised PDF will follow this response, with the revised description of the design, the input-matched comparison, and the breakdown by object visibility and motion. We will upload it before the end of the discussion period, after taking your feedback into account.

References

[1] Song, Rui, et al. "Collaborative semantic occupancy prediction with hybrid feature fusion in connected automated vehicles." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Wu, Hanlin, et al. "A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

[3] Yu, Haibao, et al. "End-to-end autonomous driving through v2x cooperation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025.

[4] You, Junwei, et al. "V2X-WAM: A Cooperative World Action Model for End-to-End Autonomous Driving." arXiv preprint arXiv:2609.37098 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.