- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents WM-V2X, a framework for cooperative occupancy forecasting and motion planning using multi-agent sensory observations and satellite imagery. It combines an early- or intermediate-fusion scene tokenizer with a spatial and temporal forecasting module that predicts future latent states autoregressively. The paper extends V2X-Real with occupancy annotations and aligned satellite images, introduces cross-agent consistency metrics, and provides extensive empirical evaluations.
Critical strengths
- This paper proposes and formulates a novel cooperative world model setting.
- This paper extends V2X-Real with occupancy annotations and aligned satellite imagery, providing a useful foundation for further research under this setting.
Critical weaknesses
- V2X-Real with 5772/1253 train/test frames are small in scale, which is reflect in Table 6 where five methods have exact same collision rate for all horizons (three in Table 2 are all 0). 5 semantic categories are also far fewer than 17 in Occ3D-nuScenes.
- WM-V2X performs close to OccProphet (Table 1 & 6) and worse than OccProphet without satellite embedding (Table 4). The performance gain might come from additional satellite input rather than forecasting module.
Questions for authors
- V2X-Real includes two infrastructures and two vehicles, totaling four agents. Could the authors report how performance scales as the total number of participating agents increases from one to four?
- Could the authors provide camera-only WM-V2X performance w/o satellite inputs?
2 comments
[Q1] Performance as the number of agents grows from one to four
[A3]
To avoid mixing scene effects into this curve, we use 157 windows from 5 scenarios in which all three collaborators are in range, and evaluate every subset of collaborators on the same samples. The target is always the fused ground truth of all agents. With 2 agents, the ego is joined by one of the three collaborators, and with 3 agents by two of them. Each case has three possible choices, and we report their mean. The 1-agent row feeds ego-only input to a model trained with collaborators. The table uses the models compared in A2. Avg. is the mean over all six horizons (0.5–3 s).
| Fusion | Agents | 0.5 s | 1.5 s | 3 s | Avg. |
|---|---|---|---|---|---|
| Inter. | 1 (ego only) | 22.27 | 18.02 | 14.59 | 17.82 |
| Inter. | 2 | 32.43 | 26.37 | 21.85 | 26.20 |
| Inter. | 3 | 38.45 | 31.05 | 25.99 | 30.96 |
| Inter. | 4 | 41.95 | 33.28 | 27.51 | 33.22 |
| Early | 1 (ego only) | 28.90 | 22.97 | 18.61 | 22.83 |
| Early | 2 | 36.42 | 29.14 | 24.08 | 29.06 |
| Early | 3 | 40.87 | 32.40 | 26.62 | 32.33 |
| Early | 4 | 43.37 | 34.20 | 27.99 | 34.15 |
Each added agent improves the forecast. On average, the second, third, and fourth agents add 8.4, 4.8, and 2.3 mIoU points under intermediate fusion and 6.2, 3.3, and 1.8 under early fusion.
[Q2] Camera-only performance without satellite input
[A4]
We retrained the camera-only WM-V2X with and without satellite input, both under the protocol described in A2, and the revision will report these results. The model with satellite input receives crops of the observed frames only and keeps the last one during the rollout.
| Method | 0.5 s | 1.5 s | 3 s | Avg. |
|---|---|---|---|---|
| No cooperation | 10.92 | 10.14 | 9.41 | 10.07 |
| WM-V2X, with satellite | 27.38 | 23.92 | 21.10 | 23.74 |
| WM-V2X, without satellite | 27.48 | 23.87 | 20.16 | 23.47 |
Averaged over the six horizons, the camera-only model scores 23.47 without satellite input and 23.74 with it. The difference is 0.27 points, so the camera-only model performs nearly the same without satellite input.
Revision. The revised PDF will follow this response, with the dataset statistics, the input-matched comparison, and the new results on the number of agents and on the camera-only model. We will upload it before the end of the discussion period, after taking your feedback into account.
Response to Reviewer Yd8H
We thank the reviewer for the concise review and for recognizing the value of the cooperative world model setting. Below we address the dataset scale, the collision rates, and the classes (A1), give an input-matched comparison with the baselines (A2), and report the two results you asked for, namely the performance from one to four agents (A3) and the camera-only model with and without satellite input (A4).
[W1] Dataset scale, identical collision rates, and the number of classes
[A1]
The scale is that of V2X-Real itself, and our labels cover all of its training and test frames. Cooperative forecasting needs synchronized recordings from several connected agents, which a larger single-vehicle dataset cannot replace. V2X-Real is one of the few real datasets with two vehicles and two roadside units. We label every agent in its 5,772 training and 1,253 test frames. This gives 21,917 and 5,012 agent frames, each with a dense label of about one million voxels (256 × 256 × 16), together with fused labels in the ego frame. The effects we report are consistent at this scale. For example, every added agent improves the forecast at every horizon and for both fusion types (see the table in A3).
The identical collision rates come from how the collision rate is computed. The collision rate counts the test samples in which the predicted ego box overlaps the box of another object. When the predicted ego positions are within 0.2 m of the ground truth, far below the size of a vehicle, the ego boxes overlap the same objects and the rates coincide. In Table 6, for example, OccWorld and WM-V2X both have a rate of 0.052%, while their position errors are 0.166 m and 0.036 m. The methods with larger position errors have different rates. We therefore compare the methods by position error.
Classes. V2X-Real provides 3D boxes rather than point-level semantic labels. We group the box categories into vehicle, pedestrian, and truck, and split the static scene into ground and a remaining class that we call building. These classes keep the distinction that matters for cooperative forecasting, between static structure and the road users that move and get occluded. Finer static classes would need new annotation. We will discuss this choice and the difference from Occ3D-nuScenes explicitly.
[W2] Close to OccProphet, and worse without satellite
[A2]
With matched inputs and no satellite input for any model, WM-V2X leads OccProphet by +2.58. The comparison in the submission gave satellite input to WM-V2X only. In the revision, we treat the satellite crops as a dataset resource, not as a contribution of the model, and compare all forecasters on the same inputs.
We retrained every model without satellite input, so all forecasters on a tokenizer, ours and the external ones, now share the same latent history and past ego poses. Forecasts use 4 history frames (t−1.5 s to t) and 6 future frames (0.5–3 s), matching the 3 s horizon of OccWorld and DOME on nuScenes. All methods are trained and evaluated this way, and the revision will report these new results.
Values are the mean IoU (mIoU, in %) over the five semantic classes. Avg. is the mean over all six horizons (0.5–3 s). No cooperation feeds ego-only input to components trained with collaborators.
With intermediate fusion, WM-V2X leads OccProphet by +2.58 and OccWorld by +1.60. This lead comes from how WM-V2X uses the shared past poses, by conditioning each rollout step on a constant-velocity extrapolation of the last two observed poses.